BIGTOP-3803: Fix Hive3.1.3 Metastore service compatible issue with Hadoop3.3.x when kerberos enabled

[timyuer]

Fix Hive3.1.3 Metastore service compatible issue with Hadoop3.3.x when kerberos enabled

Description of PR

How was this patch tested?

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
• Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/

[iwasakims]

We are specifying properties in do-component-build script to adjust versions of products. If just changing the curator.version works, updating the do-component-build should be consistent. @timyuer.

[timyuer]

I will do it. Thanks. @iwasakims

[timyuer]

Update as iwasakims comment.

[iwasakims]

Hmm. I ran unit tests of Hive with -Dcurator.version=4.2.0. Test failures caused by curator version mismatch are observed. Just setting the curator.verion seems not to work.

$ mvn -Dcurator.version=4.2.0 -Dhbase.version=2.4.13 -Dzookeeper.version=3.5.9 -Dhadoop.version=3.3.4 -Dtez.version=0.10.1 -Dspark.version=3.2.1 -Dscala.binary.version=2.12 -Dscala.version=2.12.13 -Dguava.version=27.0-jre -Dmaven.test.failure.ignore test

[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.121 s <<< FAILURE! - in org.apache.hadoop.hive.ql.lockmgr.zookeeper.TestZookeeperLockManager
[ERROR] testDeleteNoChildren(org.apache.hadoop.hive.ql.lockmgr.zookeeper.TestZookeeperLockManager)  Time elapsed: 1.431 s  <<< ERROR!
java.lang.NoSuchMethodError: org.apache.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lorg/apache/curator/framework/api/ProtectACLCreateModePathAndBytesable;
        at org.apache.hadoop.hive.ql.lockmgr.zookeeper.TestZookeeperLockManager.testDeleteNoChildren(TestZookeeperLockManager.java:94)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        ...

[iwasakims]

While I see NoSuchMethodError only in test classes (TestSlotZnode and TestZookeeperLockManager), it should be addressed as part of HIVE-26513. We can see if no defect by running tests again after it.

[timyuer]

While I see NoSuchMethodError only in test classes (TestSlotZnode and TestZookeeperLockManager), it should be addressed as part of HIVE-26513. We can see if no defect by running tests again after it.

The Hive community says it will not be fixed to be compatible with the Hadoop3.3.x, if using the Hadoop3.3.x, they recommend upgrading the Hive to the master branch. @iwasakims

[iwasakims]

Then we should backport HIVE-22205 to Hive 3.1.3 as done for existing patch*.diff. We can not upgrade Hive before Hive 4.0.0 is released. @timyuer.

[iwasakims]

Then we should backport HIVE-22205 to Hive 3.1.3 as done for existing patch*.diff.

I summitted #1003 doing this. @timyuer.

[guyuqi]

Tested this PR with #1003.

./docker-hadoop.sh --create 3 \
--image bigtop/puppet:trunk-centos-7 \
--memory 8g \
--repo file:///bigtop-home/output \
--disable-gpg-check \
--stack hdfs,yarn,mapreduce,hbase,tez,hive \
--smoke-tests hive

..........
.......
...
> Task :bigtop-tests:smoke-tests:hive:test
Finished generating test XML results (0.014 secs) into: /bigtop-home/bigtop-tests/smoke-tests/hive/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.031 secs) into: /bigtop-home/bigtop-tests/smoke-tests/hive/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:hive:test (Thread[Execution worker for ':' Thread 6,5,main]) completed. Took 23.513 secs.

BUILD SUCCESSFUL in 4m 4s
30 actionable tasks: 7 executed, 23 up-to-date
Stopped 1 worker daemon(s).
+ rm -rf buildSrc/build/test-results/binary
+ rm -rf /bigtop-home/.gradle

+1

[guyuqi]

Merged. Thanks, @timyuer @iwasakims