Hadoop-17563 bouncycastle 1.68

[pjfanning]

Description of PR

HADOOP-17563. BouncyCastle to 1.70

CVEs are reported for releases lower than 1.66

How was this patch tested?

CI build

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[jojochuang]

I see no problem in this PR alone. We should make sure the problems in the prior PR (#3405) no longer appears here (namely, build/test failures in downstream applications. Chiefly, Spark and HBase)

[pjfanning]

@steveloughran @aajisaka would either of you be able to look at the build result?

https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3980/1/artifact/out/patch-unit-root.txt seems to complete ok but has been marked as -1 by hadoop-yetus

I'm not an expert in the hadoop build and CI - it would be great if I could someone to tell me what the issue is.

[steveloughran]

@mehakmeet @mukund-thakur ?

[pjfanning]

hadoop-auth builds ok for me locally - not sure why it fails in the CI build

[mukund-thakur]

Change the title with caps HADOOP-17563 jira.

[mukund-thakur]

First of all there are test failure which must be fixed.
Also upgrading this has a history of breaking downstream builds, what testing is done for that ?

Seems like 1.70 is a major upgrade with new features and enhancements. https://www.bouncycastle.org/latest_releases.html can we use 1.68/1.69 to just fix the vulnerabilities ?

[pjfanning]

@mukund-thakur thanks for looking at this. I'm afraid I'm not an expert in the hadoop build. I'm just a random ASF member who has found lots of ASF projects that use insecure dependencies and a lot of them are holding back from upgrading because they support hadoop and it is built with the out of date insecure dependencies.

I tried to build with bouncycastle 1.67 but got similar failures.

• The test failures are timeout exceptions so it is hard to work out why they fail.
• bouncycastle is used for security purposes so it seems even more important to use a version of this jar that has no published security vulnerabilities
• even if other projects will also be forced to upgrade bouncycastle of haddop does, shouldn't this be regarded as a good thing?
• what if bouncycastle 1.60 is discovered to have a new major flaw? - that would likely only be fixed in a bouncycastle 1.7x release

[steveloughran]

i think we just go for the most recent version so as to give us a longer time of being up to date.
the last time we attempted a 1.68 update things downstream of spark failed due to some later java version .class file included.

right now I'm not even getting that far on trunk, something BC related.

[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.3.0:testCompile (scala-test-compile-first) on project spark-sql_2.12: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.3.0:testCompile failed: A required class was missing while executing net.alchim31.maven:scala-maven-plugin:4.3.0:testCompile: org/bouncycastle/jce/provider/BouncyCastleProvider
[ERROR] -----------------------------------------------------
[ERROR] realm =    plugin>net.alchim31.maven:scala-maven-plugin:4.3.0

no idea whatsoever, but as this is with hadoop building with 1.60, it'd be hard for an upgrade to 1.70 to make things worse.

looking at the test failures, they all seem network related, though as some of the mini test clusters use bouncy castle, they may be related.

looking at #2740 i can see we did actually go up to 1.68 there and didn't hit problems. lets try that one again as a pr to see if it now passes/fails.

@pjfanning could you try a version of 1.68 here and see how it goes?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	18m 7s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 7s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 51s		trunk passed
+1 💚	compile	24m 15s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	20m 41s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	mvnsite	26m 39s		trunk passed
+1 💚	javadoc	8m 17s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	8m 9s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	shadedclient	38m 45s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	24m 23s		the patch passed
+1 💚	compile	23m 51s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	23m 51s		the patch passed
+1 💚	compile	20m 41s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 41s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	22m 8s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	8m 48s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	8m 14s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	shadedclient	40m 34s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	982m 27s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 25s		The patch does not generate ASF License warnings.
		1288m 20s

Reason	Tests
Failed junit tests	hadoop.yarn.conf.TestYarnConfigurationFields
	hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3980/8/artifact/out/Dockerfile
GITHUB PR	#3980
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml shellcheck shelldocs
uname	Linux 5bd485e7d16e 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 06c26f3
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3980/8/testReport/
Max. process+thread count	2458 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3980/8/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[steveloughran]

+1

test failures look unrelated here

[INFO] Running org.apache.hadoop.yarn.conf.TestYarnConfigurationFields
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.575 s <<< FAILURE! - in org.apache.hadoop.yarn.conf.TestYarnConfigurationFields
[ERROR] testCompareConfigurationClassAgainstXml  Time elapsed: 0.06 s  <<< FAILURE!
java.lang.AssertionError: class org.apache.hadoop.yarn.conf.YarnConfiguration has 1 variables missing in yarn-default.xml Entries:   yarn.scheduler.app-placement-allocator.class expected:<0> but was:<1>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareConfigurationClassAgainstXml(TestConfigurationFieldsBase.java:493)

distcp tests are all timeouts.

[ERROR] testShutdown(org.apache.hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure)  Time elapsed: 30.073 s  <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 30000 milliseconds

going to consider them unrelated, even though its always possible that test setup could be failing from the change -just unlikely enough I'm not worried

[steveloughran]

merged into trunk; we need to get this into branch-3.3 too