HADOOP-18587: upgrade to jettison 1.5.3 due to cve

[pjfanning]

Description of PR

• GHSA-x27m-9w8j-5vcw
• https://github.com/jettison-json/jettison/releases

v1.5.2 is flagged as fixing a CVE but a v1.5.3 was quickly released and appears to fix some regressions caused by v1.5.2.
Many hadoop tests fail when jettison 1.5.2 is used.

How was this patch tested?

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[cnauroth]

+1, pending successful CI run. I applied the patch locally and confirmed it by reviewing mvn dependency:tree output.

Thanks, PJ!

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 23s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 44s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 42s		trunk passed
+1 💚	compile	23m 7s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	20m 24s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	mvnsite	19m 6s		trunk passed
-1 ❌	javadoc	1m 22s	/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	7m 34s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	28m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	22m 6s		the patch passed
+1 💚	compile	22m 33s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	22m 33s		the patch passed
+1 💚	compile	20m 31s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	20m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	18m 35s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
-1 ❌	javadoc	1m 11s	/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	root in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	7m 14s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	29m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	859m 28s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 44s		The patch does not generate ASF License warnings.
		1122m 22s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver
	hadoop.hdfs.server.balancer.TestBalancerService
	hadoop.hdfs.TestLeaseRecovery2
	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions
	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfig
	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities
	hadoop.yarn.server.resourcemanager.webapp.fairscheduler.TestRMWebServicesFairSchedulerCustomResourceTypes
	hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2
	hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobConf
	hadoop.mapreduce.v2.app.TestRuntimeEstimators
	hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobConf

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5270/1/artifact/out/Dockerfile
GITHUB PR	#5270
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux 3aecf152636e 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 9cf5d73
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5270/1/testReport/
Max. process+thread count	3490 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5270/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[pjfanning]

@cnauroth there are test failures and some of them seem to relate to jettison changes. I'm not sure how semver applies when they make breaking changes in a patch level release. I'll have a look at the tests.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 41s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 13s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 59s		trunk passed
+1 💚	compile	24m 2s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	20m 58s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	mvnsite	19m 38s		trunk passed
-1 ❌	javadoc	1m 24s	/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	7m 30s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	28m 34s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	23m 14s		the patch passed
+1 💚	compile	23m 49s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	23m 49s		the patch passed
+1 💚	compile	21m 27s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	21m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	19m 41s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
-1 ❌	javadoc	1m 14s	/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	root in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	7m 31s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	shadedclient	29m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	890m 46s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 28s		The patch does not generate ASF License warnings.
		1149m 31s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestLeaseRecovery2
	hadoop.mapreduce.v2.app.TestRuntimeEstimators

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5270/2/artifact/out/Dockerfile
GITHUB PR	#5270
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux 5f73602cd069 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / fc1721a
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5270/2/testReport/
Max. process+thread count	3445 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5270/2/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[pjfanning]

@cnauroth there are still 2 test failures and they seem genuine (not intermittent failures).

I might need someone with more HDFS debugging experience than me to look into this one.

This test just timeouts waiting for a name-node restart. Something behind the scenes is failing and not logging anything.

[INFO] Running org.apache.hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy
[ERROR] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 149.393 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy
[ERROR] testSnapshotsOnErasureCodingDirAfterNNRestart(org.apache.hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy)  Time elapsed: 120.166 s  <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 120000 milliseconds
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:2831)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2320)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2271)
	at org.apache.hadoop.hdfs.TestErasureCodingPolicyWithSnapshot.testSnapshotsOnErasureCodingDirAfterNNRestart(TestErasureCodingPolicyWithSnapshot.java:174)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:750)

Likewise, the mapreduce test failure has no obvious cause.

[ayushtkn]

The HDFS one shouldn't be related. It is being chased at HDFS-16853
Mostly there is an issue with restarts, if you are able to repro any of the HDFS failures locally which is crashing on waitActive(), can try reverting HADOOP-18324 and see if things work post that

[pjfanning]

The mapreduce test failure is tracked as MAPREDUCE-7203

[cnauroth]

+1

I agree that the remaining test failures are unrelated. On Ayush's advice, I tried a local test run of this patch + a revert of HADOOP-18324. The HDFS tests passed.

I'll plan on comitting this to trunk and branch-3.3 tomorrow. (branch-3.2 is still on version 1.1, so if you want it there, please open a separate pull request.)

[cnauroth]

I have committed this to trunk and branch-3.3, after resolving a minor merge conflict in LICENSE-binary.

@pjfanning , thank you for the contribution. @ayushtkn , thank you for advising on the test failures.