[HUDI-306] Support Glue catalog and other hive metastore implementations

[umehrot2]

Hudi currently does not work with AWS Glue Catalog or other Hive metastore implementations. The issue/exception it runs into has been reported here as well issue .

As mentioned in the issue, the reason for this is:

• Currently Hudi is interacting with Hive through two different ways:
• Creation of table statement is submitted directly to Hive via JDBC https://github.com/apache/incubator-hudi/blob/master/hudi-hive/src/main/java/org/apache/hudi/hive/HoodieHiveClient.java#L472 . Thus, Hive will internally create the right metastore client (i.e. Glue if hive.metastore.client.factory.class is set to com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory in hive-site)
• Whereas partition listing among other things are being done by directly calling hive metastore APIs using hive metastore client: https://github.com/apache/incubator-hudi/blob/master/hudi-hive/src/main/java/org/apache/hudi/hive/HoodieHiveClient.java#L240
• Now in Hudi code, standard specific implementation of the metastore client (not glue metastore client) is being instantiated: https://github.com/apache/incubator-hudi/blob/master/hudi-hive/src/main/java/org/apache/hudi/hive/HoodieHiveClient.java#L109 .
• Ideally this instantiation of metastore client should be left to Hive through https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L5045 for it to consider other implementations of metastore client that might be configured through hive.metastore.client.factory.class .

That is the reason that table gets created in Glue metastore, but while reading or scanning partitions it is talking to the local hive metastore where it does not find the table created.

Note: We need to removing shading of Hive in hudi-spark-bundle by default, because we would get RuntimeException NoSuchMethod because HiveConf is shaded and relocated to a new namespace. But Hive.java is not shaded and hence Hive.get(conf) results in NoSuchMethod. We cannot shade Hive.java since it is in hive-exec which itself is a huge bundle jar with numerous dependencies. A similar issue already exists in Hudi because of shading of Hive which we have reported here: https://issues.apache.org/jira/browse/HUDI-281 . So this PR will help fix that also.

[vinothchandar]

Changes look fine to me. CI seems to have stalled due to inactivity. Wondering if travis is acting up again.. Rekicked the test

[umehrot2]

@vinothchandar Can you help out with this integration test failure ? I am not sure why it is getting stalled.

Even on my local setup with master branch, the tests seem to be getting stalled. Not sure how to debug this. I did ssh to the docker container but could not find any logs.

[vinothchandar]

@umehrot2 It passes for me all the time locally.. if it fails, then the docker containers may still be running locally.. You can use docker/stop_demo.sh to kill them .. and re-run tests if it helps.. https://travis-ci.org/apache/incubator-hudi/builds used to be spotless. but seeing some travis weirdness off late.

let me take a look at the failure itself

[vinothchandar]

@umehrot2 did not respond here since there was a mailing list thread on the same. Seems like the jetty threads are not exiting cleanly somehow. and its intermittent? do you see that it always stalls. I or others are not seeing that

I did ssh to the docker container but could not find any logs.
stdout is piped back to the test process, that why. you can use jstack and jmap to get dumps and see whats going on.

[umehrot2]

@umehrot2 did not respond here since there was a mailing list thread on the same. Seems like the jetty threads are not exiting cleanly somehow. and its intermittent? do you see that it always stalls. I or others are not seeing that

I did ssh to the docker container but could not find any logs.
stdout is piped back to the test process, that why. you can use jstack and jmap to get dumps and see whats going on.

@vinothchandar sorry for the late response here. Have been busy with oncall. Yeah atleast on ec2 instance with amazon linux I was able to constantly run into integration test hanging case. Is there any progress on debugging that ? Willing to help with this if required.

[vinothchandar]

Latest on that is summarized here
https://issues.apache.org/jira/browse/HUDI-312 We are actively debugging it.. Will try again today.. If you can take a crack at it, sure by all means :)

[umehrot2]

@vinothchandar as the integration tests are now fixed, can we trigger another travis build for this ?

[vinothchandar]

@umehrot2 I was hoping you will rebase and repush :). that would trigger CI

[umehrot2]

@umehrot2 I was hoping you will rebase and repush :). that would trigger CI

Sure. I thought since this merges cleanly, rebase is not required. Will do that.

[umehrot2]

@vinothchandar @bvaradar I would need your input here. Finally the tests got past the being stuck issue, but now the integration test fails with:

 ###### Stderr #######

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.get(Lorg/apache/hudi/org/apache/hadoop_hive/conf/HiveConf;)Lorg/apache/hadoop/hive/ql/metadata/Hive;

	at org.apache.hudi.hive.HoodieHiveClient.<init>(HoodieHiveClient.java:111)

	at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:60)

	at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncHive(DeltaSync.java:443)

	at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:385)

	at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:227)

	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:121)

	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:298)

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.lang.reflect.Method.invoke(Method.java:498)

	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)

	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)

	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)

	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)

	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)

	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

As you can see from the stack trace, and I didn't know this until now that even hudi-utilities-bundle shades Hive. I had removed the shading of Hive from hudi-spark-bundle to unblock this PR, but not from hudi-utilities-bundle and thats why this issue occurs through DeltaStreamer which uses shaded Hive. The reason for this happening is I have added Hive.get in this code, and Hive class exists in hive-exec which has not been shaded in Hudi until now, and ideally should not be because its a fat jar itself. But HiveConf is shaded and relocated.

I think we need to take a larger call here, whether its fine to remove Hive shading from hudi-utilities-bundle or not. And with this change of introducing Hive.get even if Hive is shaded it will still break, because hive-exec is not being shaded while other modules are being shaded. So it will cause this NoSuchMethodError. Need your suggestions on whether we should completely move out of shading Hive.

[vinothchandar]

utilities-bundle is very similar to its spark-bundle.. For now, we could do a similar fix (control via a property) for utilities-bundle as well?

Longer term, I am wondering if both bundles can do away with bundling Hive at all? (shaded or unshaded).. We probably need to design that more carefully?

[vinothchandar]

@umehrot2 we moved the build instructions to README. Do you mind adding a EMR build section to the file?.. Otherwise LGTM. Will merge once you make that change..

[vinothchandar]

remove this line as well?

[umehrot2]

@umehrot2 we moved the build instructions to README. Do you mind adding a EMR build section to the file?.. Otherwise LGTM. Will merge once you make that change..

Hudi will be released as an officially supported application in a day or two. On that note, do we really need EMR specific build instructions any more ? I mean from a customers perspective, Hudi is coming packaged on EMR, they don't have to build it themselves. Thoughts ?

[vinothchandar]

@umehrot2 More than EMR, I was wondering if we should provide some guidance on how/when to use the profiles you added.. I am ok either ways. let me know what you think

[umehrot2]

@umehrot2 More than EMR, I was wondering if we should provide some guidance on how/when to use the profiles you added.. I am ok either ways. let me know what you think

To be honest, the profiles added as part of this PR we should not be enabling them as if we shade Hive now, it will cause failures because of the NoSuchMethod error reported above. I guess they are there just as suggestions from you guys, if someone really wants to easily play around with Hive shaded. Like @n3nash suggested way back that in some cases he requires Hive shaded.