HADOOP-19090. Use protobuf-java 3.23.4. #6593

pjfanning · 2024-02-26T18:03:22Z

Description of PR

DRAFT - See HADOOP-19090

The hadoop-thirdparty change is not merged yet so I am temporarily testing with a snapshot that I published with my groupId.

How was this patch tested?

Local build

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-project/pom.xml

hadoop-yetus · 2024-02-27T11:24:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 14s		Maven dependency ordering for branch
+1 💚	mvninstall	33m 51s		trunk passed
+1 💚	compile	17m 40s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	16m 15s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	mvnsite	20m 56s		trunk passed
+1 💚	javadoc	8m 36s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	7m 40s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	shadedclient	152m 50s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 52s		Maven dependency ordering for patch
+1 💚	mvninstall	30m 57s		the patch passed
+1 💚	compile	16m 55s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	16m 55s		the patch passed
+1 💚	compile	16m 2s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	16m 2s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	16m 13s		the patch passed
+1 💚	javadoc	8m 28s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	7m 51s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	shadedclient	66m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	737m 39s	/patch-unit-root.txt	root in the patch failed.
+1 💚	asflicense	1m 35s		The patch does not generate ASF License warnings.
		1039m 44s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestSecureNameNode

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6593/1/artifact/out/Dockerfile
GITHUB PR	#6593
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux bc53b5a08ee5 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `53109b7`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6593/1/testReport/
Max. process+thread count	4799 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6593/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

pjfanning · 2024-02-27T13:13:02Z

The 1 broken test seems unrelated:

[ERROR] org.apache.hadoop.hdfs.server.namenode.TestSecureNameNode  Time elapsed: 1.417 s  <<< ERROR!
org.apache.kerby.kerberos.kerb.KrbException: Failed to start DefaultKrbServer. Address already in use (Bind failed)

ayushtkn

the change is merged to hadoop-thirdparty & we have a automated build which publishes the snapshot version of that, can we use that, if the build is green we can merge this to get the tests fixed

update fix groupId fix repository url

pjfanning · 2024-03-05T12:12:39Z

Switched PR to use the Apache snapshot

Hexiaoqiao · 2024-03-05T13:54:42Z

Thanks @pjfanning . LGTM. Let's wait what CI will say.
Try link to apache/hadoop-thirdparty#34

hadoop-yetus · 2024-03-06T05:35:54Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 38s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 8s		trunk passed
+1 💚	compile	17m 29s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	15m 56s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	mvnsite	21m 29s		trunk passed
+1 💚	javadoc	8m 36s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	7m 37s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	shadedclient	151m 56s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 53s		Maven dependency ordering for patch
+1 💚	mvninstall	32m 13s		the patch passed
+1 💚	compile	18m 16s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	18m 16s		the patch passed
+1 💚	compile	17m 14s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	17m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	17m 50s		the patch passed
+1 💚	javadoc	8m 52s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	8m 1s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	shadedclient	72m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	731m 59s	/patch-unit-root.txt	root in the patch failed.
+1 💚	asflicense	1m 38s		The patch does not generate ASF License warnings.
		1043m 4s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy
	hadoop.hdfs.tools.TestDFSAdmin

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6593/2/artifact/out/Dockerfile
GITHUB PR	#6593
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux d622216aa576 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9a7e4e6`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6593/2/testReport/
Max. process+thread count	4498 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6593/2/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

pom.xml

ayushtkn

LGTM

hadoop-yetus · 2024-03-06T20:47:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 38s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 16s		Maven dependency ordering for branch
+1 💚	mvninstall	19m 59s		trunk passed
+1 💚	compile	9m 2s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	8m 4s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	mvnsite	13m 7s		trunk passed
+1 💚	javadoc	4m 49s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	4m 58s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	shadedclient	93m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 39s		Maven dependency ordering for patch
+1 💚	mvninstall	18m 30s		the patch passed
+1 💚	compile	8m 41s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	8m 41s		the patch passed
+1 💚	compile	8m 0s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	8m 0s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	8m 20s		the patch passed
+1 💚	javadoc	4m 44s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	5m 7s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	shadedclient	38m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	548m 49s	/patch-unit-root.txt	root in the patch failed.
+0 🆗	asflicense	0m 34s		ASF License check generated no output?
		726m 29s

Reason	Tests
Failed junit tests	hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity
	hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema
	hadoop.hdfs.tools.TestDFSAdmin
	hadoop.hdfs.TestErasureCodingMultipleRacks
	hadoop.tools.TestDistCpWithRawXAttrs

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6593/3/artifact/out/Dockerfile
GITHUB PR	#6593
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 279b9de865a7 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2035ce9`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6593/3/testReport/
Max. process+thread count	4428 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6593/3/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

pjfanning · 2024-03-06T21:31:41Z

Some of the test issues relate to java.lang.OutOfMemoryError: unable to create new native thread

[ERROR] Tests run: 10, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 418.936 s <<< FAILURE! - in org.apache.hadoop.tools.TestDistCpWithRawXAttrs
[ERROR] testUseIterator(org.apache.hadoop.tools.TestDistCpWithRawXAttrs)  Time elapsed: 155.529 s  <<< ERROR!
java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread
	at org.apache.hadoop.hdfs.ExceptionLastSeen.set(ExceptionLastSeen.java:45)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:861)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:719)
	at org.apache.hadoop.hdfs.DataStreamer.initDataStreaming(DataStreamer.java:672)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:752)

Hexiaoqiao

+1 from my side.

steveloughran · 2024-03-08T12:54:39Z

we ready to get this into branch-3.4?

pjfanning · 2024-03-08T13:37:48Z

@steveloughran this PR depends on a snapshot version of hadoop-thirdparty. Can we do a release of hadoop-thirdparty and then backport the uptake of the future hadoop-thirdparty release to the 3.4 branch?

steveloughran · 2024-03-08T21:16:25Z

are we up to date with other dependencies for a new thirdparty release? we should really bump them all up and then rebuild it.
this isn't targeting 3.4.0 so not a blocker there

pjfanning · 2024-03-08T21:23:20Z

are we up to date with other dependencies for a new thirdparty release? we should really bump them all up and then rebuild it. this isn't targeting 3.4.0 so not a blocker there

Guava has a slightly newer release 33.0.0-jre. Is this upgrade needed though?

zhangshuyan0 · 2024-03-11T04:46:55Z

It seems that hadoop-shaded-protobuf_3_23 has not been deployed to the maven central repository yet. I cannot compile trunk with the latest code now. @pjfanning

pjfanning · 2024-03-11T07:36:21Z

@zhangshuyan0 it is in the Apache Snapshot repo. See the version number: 1.3.0-SNAPHOT.

See https://repository.apache.org/content/groups/snapshots/org/apache/hadoop/thirdparty/hadoop-shaded-protobuf_3_23/

See https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6593/3

zhangshuyan0 · 2024-03-11T10:23:13Z

@pjfanning Thanks for your reply. The reason I couldn't compile successfully was due to my local network problem.

… PJ Fanning.

…nning. (#6663) Signed-off-by: Shilun Fan <slfan1989@apache.org>

pjfanning marked this pull request as draft February 26, 2024 18:03

github-actions bot added YARN build trunk Common labels Feb 26, 2024

steveloughran reviewed Feb 27, 2024

View reviewed changes

hadoop-project/pom.xml Show resolved Hide resolved

pjfanning mentioned this pull request Feb 27, 2024

HADOOP-19090. Use protobuf-java 3.23.4 apache/hadoop-thirdparty#34

Merged

ayushtkn reviewed Mar 5, 2024

View reviewed changes

HADOOP-19090. Use protobuf-java 3.23.4.

9a7e4e6

update fix groupId fix repository url

pjfanning force-pushed the HADOOP-19090-protobuf branch from 53109b7 to 9a7e4e6 Compare March 5, 2024 12:11

pjfanning marked this pull request as ready for review March 5, 2024 12:15

ayushtkn reviewed Mar 6, 2024

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

remove extra repository

2035ce9

ayushtkn approved these changes Mar 6, 2024

View reviewed changes

Hexiaoqiao approved these changes Mar 7, 2024

View reviewed changes

ayushtkn merged commit fc166d3 into apache:trunk Mar 7, 2024
1 of 4 checks passed

pjfanning deleted the HADOOP-19090-protobuf branch March 8, 2024 13:37

ZanderXu pushed a commit to ZanderXu/hadoop that referenced this pull request Mar 23, 2024

HADOOP-19090. Use protobuf-java 3.23.4. (apache#6593). Contributed by…

a46c1a7

… PJ Fanning.

slfan1989 pushed a commit that referenced this pull request Mar 24, 2024

HADOOP-19090. Use protobuf-java 3.23.4. (#6593). Contributed by PJ Fa…

d32bc50

…nning. (#6663) Signed-off-by: Shilun Fan <slfan1989@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-19090. Use protobuf-java 3.23.4. #6593

HADOOP-19090. Use protobuf-java 3.23.4. #6593

pjfanning commented Feb 26, 2024

hadoop-yetus commented Feb 27, 2024

pjfanning commented Feb 27, 2024

ayushtkn left a comment

pjfanning commented Mar 5, 2024

Hexiaoqiao commented Mar 5, 2024

hadoop-yetus commented Mar 6, 2024

ayushtkn left a comment

hadoop-yetus commented Mar 6, 2024

pjfanning commented Mar 6, 2024

Hexiaoqiao left a comment

steveloughran commented Mar 8, 2024

pjfanning commented Mar 8, 2024

steveloughran commented Mar 8, 2024

pjfanning commented Mar 8, 2024

zhangshuyan0 commented Mar 11, 2024

pjfanning commented Mar 11, 2024

zhangshuyan0 commented Mar 11, 2024

HADOOP-19090. Use protobuf-java 3.23.4. #6593

HADOOP-19090. Use protobuf-java 3.23.4. #6593

Conversation

pjfanning commented Feb 26, 2024

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Feb 27, 2024

pjfanning commented Feb 27, 2024

ayushtkn left a comment

Choose a reason for hiding this comment

pjfanning commented Mar 5, 2024

Hexiaoqiao commented Mar 5, 2024

hadoop-yetus commented Mar 6, 2024

ayushtkn left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Mar 6, 2024

pjfanning commented Mar 6, 2024

Hexiaoqiao left a comment

Choose a reason for hiding this comment

steveloughran commented Mar 8, 2024

pjfanning commented Mar 8, 2024

steveloughran commented Mar 8, 2024

pjfanning commented Mar 8, 2024

zhangshuyan0 commented Mar 11, 2024

pjfanning commented Mar 11, 2024

zhangshuyan0 commented Mar 11, 2024