HADOOP-16579 - Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.5

[nkalmar]

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

[nkalmar]

I updated ZooKeeper to 3.5.5 as well, not just curator. This way unit tests pass. I also did an integration test with multiple Hadoop component. There were some test failures if they use the new ZK version from Hadoop classpath, as from 3.5+ you need to explicitly whitelist 4 letter commands in ZooKeeper. For example in Solr there were unit test failures.

But I think this is something other components will wan't to deal anyway, as it is recommended to upgrade to 3.5 ZooKeeper everywhere.

[nkalmar]

org.apache.hadoop.security.TestShellBasedIdMapping failed on a second run, looks unrelated, doing a third run...

[nkalmar]

This failed the second time as well, also org.apache.hadoop.ha.TestZKFailoverController failed.

I will test this further, but I will leave the PR up so the community can see and decide if the ZK upgrade is also feasible on master.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	38	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1337	trunk passed
+1	compile	15	trunk passed
+1	mvnsite	17	trunk passed
+1	shadedclient	2153	branch has no errors when building and testing our client artifacts.
+1	javadoc	18	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	12	the patch passed
+1	compile	10	the patch passed
+1	javac	10	the patch passed
+1	mvnsite	14	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	2	The patch has no ill-formed XML file.
+1	shadedclient	866	patch has no errors when building and testing our client artifacts.
+1	javadoc	14	the patch passed
		_ Other Tests _
+1	unit	13	hadoop-project in the patch passed.
+1	asflicense	28	The patch does not generate ASF License warnings.
		3305

Subsystem	Report/Notes
Docker	Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1629/1/artifact/out/Dockerfile
GITHUB PR	#1629
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux c119d36cc188 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 2d81abc
Default Java	1.8.0_222
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1629/1/testReport/
Max. process+thread count	305 (vs. ulimit of 5500)
modules	C: hadoop-project U: hadoop-project
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1629/1/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[nkalmar]

If we only update Curator to 4.2.0, TestZKSignerSecretProvider will fail because curator is looking for a non-existent field: java.lang.NoSuchFieldError: configFileStr. My understanding is that curator decides runtime if it runs with 3.4 or 3.5 ZK compatibility mode. Error logs show otherwise unfortunately. (I tried both excluding and leaving curator’s ZK dependency version)

If we also update ZooKeeper to 3.5.5, TestZKSignerSecretProvider will run just fine, but TestZKFailoverController will Timeout because ZKFailoverController.doGracefulFailover() no longer works (I think it can’t read back which node became active. Maybe has to do with the need in 3.5 to whitelist 4 letter word commands?)