[HUDI-1609] How to disable Hive JDBC and enable metastore #1679
Comments
|
hello @selvarajperiyasamy, try option |
|
@lamber-ken Do you mean something like below in data source writer ? |
|
Hi @selvarajperiyasamy, I guess you used |
|
Hi @lamber-ken , is this Spark option available in 0.5.0. I tired , it didn’t work . When I checked in Hudi code base , this string not found anywhere . Attached the image .
|
|
Hi @selvarajperiyasamy, for hudi-0.5.0, use |
|
cc @n3nash as well who made similar changes |
|
I have already used below setting and error is still the same as mentioned in the ticket. option(HIVE_SYNC_ENABLED_OPT_KEY,true). option(HIVE_URL_OPT_KEY,"jdbc:hive2://server1.visa.com:2181,server2.visa.com:2181,server3.visa.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2"). |
|
@selvarajperiyasamy Actually, the stack trace does show it going over thrift to the metastore. This might be an issue with Hive 1.2? (we test with Hive 2.x).. I assume you are running with CDH? @bvaradar who may know about this combo more |
|
@selvarajperiyasamy : This is indeed caused by the version mismatch of Hive. Enabling/Disabling jdbc will not help here. With 0.5.0, Hudi moved to Hive 2.x which was predominantly being used across various deployments. Hive 1.2.x is really old :) and Hive 1.2.x server is not compatible with Hive 2.x clients. Is it possible to upgrade the hive environment to use Hive 2.x (2.3.3 for example) ? |
|
Thanks Balaji . We are using shared cluster and may have impact for other
users if I upgrade to 2.x. Will check with cluster owners and see.
However thanks for all of your support .
Thanks,
Selva
…On Sun, May 31, 2020 at 10:57 AM Balaji Varadarajan < ***@***.***> wrote:
@selvarajperiyasamy <https://github.com/selvarajperiyasamy> : This is
indeed caused by the version mismatch of Hive. Enabling/Disabling jdbc will
not help here. With 0.5.0, Hudi moved to Hive 2.x which was predominantly
being used across various deployments. Hive 1.2.x is really old :) and Hive
1.2.x server is not compatible with Hive 2.x clients. Is it possible to
upgrade the hive environment to use Hive 2.x (2.3.3 for example) ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1679 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AOUU7IBRS4KHFCNYB7CVAATRUKLBJANCNFSM4NNQITBA>
.
|
|
let us know how that goes. @bvaradar raised a JIRA to see what/if we can do something here.. |
|
Sure Vinoth. Thanks !
…On Sun, May 31, 2020 at 11:59 AM vinoth chandar ***@***.***> wrote:
let us know how that goes. @bvaradar <https://github.com/bvaradar> raised
a JIRA to see what/if we can do something here..
But to add my 2c, hadoop/hive vendors are increasingly moving to Hive 3
even.. So ideally upgrading to hive 2 is a good to do things nonetheless.
At least at uber, it improved hive overall from what I remember..
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1679 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AOUU7ICJR2SXSN4RF73YH53RUKSI3ANCNFSM4NNQITBA>
.
|
|
@selvarajperiyasamy : Hope you were able to resolve the issue. Let us know if any help is needed. |
|
@bvaradar I've tested deltastreamer by hudi in master branch, If I set |
|
@cdmikechen : Long time :) Hudi utilities include following hive jars in shaded form Can you attach the whole exception you are seeing. We had a compliance reason for not including org.json classes (due to licensing issues). |
|
@cdmikechen Please let us know the whole exception.. If we can repro, ideally like to fix it before 0.6.0 goes out . |
|
Hi, i'm facing the same issue when trying to sync to hive with hoodie.datasource.hive_sync.use_jdbc=false. This is the complete stacktrace - |
|
I'm also facing the same issue as documented by @ruztbucket |
|
I am also experiencing this error on Hudi 0.6.0, EMR 5.31.0. I've tried referencing the json-1.8.jar found in /usr/lib/hive/lib/json-1.8.jar of my EMR server in my --jars parameter but that does not fix the issue either. |
|
@bvaradar : Can you please follow up on this ticket when you can. |
|
@kimberlyamandalu : Sorry for the delay. This is weird. Can you check if org/json/JSONException is present in /usr/lib/hive/lib/json-1.8.jar ? |
|
I'm also facing the same issue mentioned by @ruztbucket |
nsivabalan
added
the
priority:critical
|
@bvaradar : fyi I have created a sec:critical jira on this https://issues.apache.org/jira/browse/HUDI-1609. Please reduce priority if you feel otherwise. |
nsivabalan
changed the title
|
@bvaradar : I could not reproduce w/ local docker set up. Do you have any pointers on how to go about triaging this. Also, I am running into some other issue locally which I documented in https://issues.apache.org/jira/browse/HUDI-1609. |
|
@kimberlyamandalu : in the mean time, would you mind responding to Balaji's doubts. |
|
Here is my understanding. |
nsivabalan
added
priority:major
|
@nsivabalan I had a different error
|
|
I give another try on First ERROR:
Second ERROR:
|
@bvaradar Sorry for the delayed response. Yes, the JSONException object is present in this jar |
|
this comment might help for the second error #1751 (comment)
|
|
Hey, I'm having the same issues with JSONEXception on EMR as mentioned above. Is there any update around that? Anything I can help with to make it work? Thanks! |
|
I was able to fix the JSONException error on EMR. Just needed to manually add the org.json (https://mvnrepository.com/artifact/org.json/json) package to both executor and driver extraClassPath config when deploying the cluster. |
|
Hudi version: 0.7.0 Hi, when I use:
I have the following exception: Existing tables are updated, but for tables that need to be created I get the exception above. |
|
@diogodilcl Are you able to reproduce this issue consistently ? Could you provide some ways to reproduce it so we can find a resolution. |
n3nash
added
hive
|
Closing this ticket due to inactivity. There is a PR open that will provide ways to disable JDBC. |
Team,
My spark version is 2.3.0
Scala version 2.11.8
Hive version 1.2.2
I see the below comment in Hudi code. How can I start using metastore client for hive registrations? is there a way to disable useJdbc flag?
// Support both JDBC and metastore based implementations for backwards compatiblity. Future users should
// disable jdbc and depend on metastore client for all hive registrations
Below is my log. It makes hive JDBC connection and failing due to method not available error.
Thanks,
Selva
The text was updated successfully, but these errors were encountered: