YARN-11126. ZKConfigurationStore Java deserialisation vulnerability #4265

tomicooler · 2022-05-05T12:55:01Z

No description provided.

steveloughran

looks good, just need some error text on assertion failures.

every assert needs to have a meaningful error message so that a jenkins/yetus build helps diagnose problems. line numbers/stack traces aren't enough. sorry

steveloughran · 2022-05-05T13:08:45Z

...che/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestZKConfigurationStore.java

+        Thread.sleep(100);
+      }
+
+      Assert.assertFalse(flagFile.exists());


can you add the error text to raise in the assert, including the filename.

Thanks for the review. I added an error message.

I see some test failures with errors Metrics source DelegationTokenSecretManagerMetrics already exists!, that was introduced with this commit (git bisect):

d60262fe0092a9b45ee17830b98750fb00b856b1 is the first bad commit commit d60262fe0092a9b45ee17830b98750fb00b856b1 Author: hchaverri <55413673+hchaverri@users.noreply.github.com> Date: Tue Apr 26 09:20:11 2022 -0700 HADOOP-18167. Add metrics to track delegation token secret manager op… (#4092) * HADOOP-18167. Add metrics to track delegation token secret manager operations .../AbstractDelegationTokenSecretManager.java | 121 +++++++++++++++++- .../token/delegation/TestDelegationToken.java | 136 +++++++++++++++++++++ 2 files changed, 252 insertions(+), 5 deletions(-)

This is fixed with HADOOP-18222.
Please rebase your branch so that UT failures can go away :)

hadoop-yetus · 2022-05-05T16:29:35Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 42s		trunk passed
+1 💚	compile	1m 16s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	1m 7s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 4s		trunk passed
+1 💚	mvnsite	1m 14s		trunk passed
+1 💚	javadoc	1m 5s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 53s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 20s		trunk passed
+1 💚	shadedclient	24m 41s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 56s		the patch passed
+1 💚	compile	1m 4s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	1m 4s		the patch passed
+1 💚	compile	0m 54s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 54s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 45s		the patch passed
+1 💚	mvnsite	0m 59s		the patch passed
+1 💚	javadoc	0m 45s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 41s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 16s		the patch passed
+1 💚	shadedclient	27m 18s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	100m 37s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		213m 15s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
	hadoop.yarn.server.resourcemanager.TestRMHA
	hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher
	hadoop.yarn.webapp.TestRMWithXFSFilter
	hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
	hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
	hadoop.yarn.server.resourcemanager.TestClientRMService

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4265/1/artifact/out/Dockerfile
GITHUB PR	#4265
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 987f7417d408 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / b117ac8226e364844f8fdeb7e11f3095d91c62f5
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4265/1/testReport/
Max. process+thread count	934 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4265/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2022-05-06T09:58:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 42s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 41s		trunk passed
+1 💚	compile	1m 10s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	1m 1s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 53s		trunk passed
+1 💚	mvnsite	1m 6s		trunk passed
+1 💚	javadoc	1m 5s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 45s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 18s		trunk passed
+1 💚	shadedclient	25m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 57s		the patch passed
+1 💚	compile	1m 4s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	1m 4s		the patch passed
+1 💚	compile	0m 56s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 56s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 40s		the patch passed
+1 💚	mvnsite	0m 57s		the patch passed
+1 💚	javadoc	0m 44s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 39s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 9s		the patch passed
+1 💚	shadedclient	22m 46s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	97m 25s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		202m 36s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher
	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
	hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
	hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
	hadoop.yarn.webapp.TestRMWithXFSFilter
	hadoop.yarn.server.resourcemanager.TestRMHA
	hadoop.yarn.server.resourcemanager.TestClientRMService

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4265/2/artifact/out/Dockerfile
GITHUB PR	#4265
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux ce72676ff089 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 7236183cbeb551582886a5bd181486e0ba356bcb
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4265/2/testReport/
Max. process+thread count	1009 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4265/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

szilard-nemeth · 2022-05-11T14:16:06Z

Thanks @tomicooler for working on this.
Patch LGTM, +1
As mentioned in my comment previously, you need to rebase your branch to latest trunk in order to get rid of the unit test failures.
Once we have a green jenkins, I can commit this.
Thanks.

Change-Id: I07162784851480b9b85c5faffa3e01f18b5dc4a7

Change-Id: I7c6f7668ca6be44f5ea38167d275f56b6b7cdfe5

hadoop-yetus · 2022-05-11T21:54:37Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 52s		trunk passed
+1 💚	compile	1m 6s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	compile	0m 58s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 58s		trunk passed
+1 💚	mvnsite	1m 15s		trunk passed
+1 💚	javadoc	0m 56s		trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 43s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 5s		trunk passed
+1 💚	shadedclient	21m 38s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 56s		the patch passed
+1 💚	compile	1m 4s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	1m 4s		the patch passed
+1 💚	compile	0m 53s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 53s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 38s		the patch passed
+1 💚	mvnsite	0m 57s		the patch passed
+1 💚	javadoc	0m 44s		the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javadoc	0m 40s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 2s		the patch passed
+1 💚	shadedclient	21m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	97m 23s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		197m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4265/3/artifact/out/Dockerfile
GITHUB PR	#4265
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 24ff7f0b118f 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ababeb5`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4265/3/testReport/
Max. process+thread count	939 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4265/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

szilard-nemeth · 2022-05-12T11:42:32Z

Thanks @tomicooler ,
Latest patch LGTM, committed to trunk.
Thanks @steveloughran for the review.

steveloughran requested changes May 5, 2022

View reviewed changes

tomicooler added 2 commits May 11, 2022 20:35

YARN-11126. ZKConfigurationStore Java deserialisation vulnerability

f8cc9a7

Change-Id: I07162784851480b9b85c5faffa3e01f18b5dc4a7

add error message to assert failure

ababeb5

Change-Id: I7c6f7668ca6be44f5ea38167d275f56b6b7cdfe5

tomicooler force-pushed the YARN-11126 branch from 7236183 to ababeb5 Compare May 11, 2022 18:35

szilard-nemeth closed this May 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-11126. ZKConfigurationStore Java deserialisation vulnerability #4265

YARN-11126. ZKConfigurationStore Java deserialisation vulnerability #4265

tomicooler commented May 5, 2022

steveloughran left a comment

steveloughran May 5, 2022

tomicooler May 6, 2022

szilard-nemeth May 11, 2022

hadoop-yetus commented May 5, 2022

hadoop-yetus commented May 6, 2022

szilard-nemeth commented May 11, 2022

hadoop-yetus commented May 11, 2022

szilard-nemeth commented May 12, 2022

YARN-11126. ZKConfigurationStore Java deserialisation vulnerability #4265

YARN-11126. ZKConfigurationStore Java deserialisation vulnerability #4265

Conversation

tomicooler commented May 5, 2022

steveloughran left a comment

Choose a reason for hiding this comment

steveloughran May 5, 2022

Choose a reason for hiding this comment

tomicooler May 6, 2022

Choose a reason for hiding this comment

szilard-nemeth May 11, 2022

Choose a reason for hiding this comment

hadoop-yetus commented May 5, 2022

hadoop-yetus commented May 6, 2022

szilard-nemeth commented May 11, 2022

hadoop-yetus commented May 11, 2022

szilard-nemeth commented May 12, 2022