HADOOP-19424. [S3A] Upgrade JUnit from 4 to 5 in hadoop-aws. #7752

slfan1989 · 2025-06-19T23:41:23Z

Description of PR

JIRA: HADOOP-19424. [S3A] Upgrade JUnit from 4 to 5 in hadoop-aws.

How was this patch tested?

mvn clean test.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2025-06-20T00:59:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 32 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 3s		trunk passed
+1 💚	compile	0m 43s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 21s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	0m 23s		trunk passed
+1 💚	mvnsite	0m 26s		trunk passed
+1 💚	javadoc	0m 28s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 24s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	0m 48s		trunk passed
+1 💚	shadedclient	20m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 21s		the patch passed
+1 💚	compile	0m 22s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 22s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 14s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 10 new + 65 unchanged - 0 fixed = 75 total (was 65)
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	javadoc	0m 18s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 19s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	0m 40s		the patch passed
+1 💚	shadedclient	20m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	2m 48s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		76m 54s

Reason	Tests
Failed junit tests	hadoop.fs.s3a.TestS3AGetFileStatus
	hadoop.fs.s3a.TestS3AInputPolicies

Subsystem	Report/Notes
Docker	ClientAPI=1.50 ServerAPI=1.50 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7752/1/artifact/out/Dockerfile
GITHUB PR	#7752
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 2b199540d45e 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `09cd4a1`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7752/1/testReport/
Max. process+thread count	562 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7752/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2025-06-21T02:33:50Z

@steveloughran The JUnit 5 upgrade for the hadoop-aws module has been completed and is ready for review. I've also removed the dependency on JUnit 4. Let's take a look at the Yetus build results. If you have time, your review of this PR would be appreciated. Thank you!

hadoop-yetus · 2025-06-21T03:45:33Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 35 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 2s		trunk passed
+1 💚	compile	0m 29s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 24s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	0m 24s		trunk passed
+1 💚	mvnsite	0m 30s		trunk passed
+1 💚	javadoc	0m 29s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 22s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	0m 48s		trunk passed
+1 💚	shadedclient	20m 52s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 22s		the patch passed
+1 💚	compile	0m 22s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 22s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 13s	/results-checkstyle-hadoop-tools_hadoop-aws.txt	hadoop-tools/hadoop-aws: The patch generated 14 new + 68 unchanged - 1 fixed = 82 total (was 69)
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	0m 45s		the patch passed
+1 💚	shadedclient	20m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 47s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		76m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.50 ServerAPI=1.50 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7752/2/artifact/out/Dockerfile
GITHUB PR	#7752
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux 00877fddb6e8 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `001893e`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7752/2/testReport/
Max. process+thread count	559 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7752/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-06-21T06:12:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 35 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	23m 42s		trunk passed
+1 💚	compile	0m 26s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 21s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	0m 23s		trunk passed
+1 💚	mvnsite	0m 31s		trunk passed
+1 💚	javadoc	0m 29s		trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 23s		trunk passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	0m 47s		trunk passed
+1 💚	shadedclient	21m 15s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 21s		the patch passed
+1 💚	compile	0m 23s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 23s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	0m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 14s		hadoop-tools/hadoop-aws: The patch generated 0 new + 68 unchanged - 1 fixed = 68 total (was 69)
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 19s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	0m 45s		the patch passed
+1 💚	shadedclient	20m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 46s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		76m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.50 ServerAPI=1.50 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7752/3/artifact/out/Dockerfile
GITHUB PR	#7752
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux 9cc7f35045e4 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3e30cda`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7752/3/testReport/
Max. process+thread count	555 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7752/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

cnauroth

+1. Before committing this, I would prefer if someone could run the integration tests against real S3. (If necessary, I can fake it with GCS's S3-compatible XML API, but that's not the typical usage pattern.)

slfan1989 · 2025-06-24T00:01:33Z

+1. Before committing this, I would prefer if someone could run the integration tests against real S3. (If necessary, I can fake it with GCS's S3-compatible XML API, but that's not the typical usage pattern.)

@cnauroth Thank you for your help and code review! I completely agree with your point, and we indeed need other members of the community to assist in validating this part of the code.

slfan1989 · 2025-07-05T07:48:28Z

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AInputPolicies.java

+  public void testInputPolicies(S3AInputPolicy pPolicy,
+      long pTargetPos, long pLength, long pContentLength,
+      long pReadahead, long pExpectedLimit) throws Throwable {
+    initTestS3AInputPolicies(pPolicy, pTargetPos, pLength, pContentLength,


@steveloughran Regarding the Parameterized test issue, we have found some solutions, although it will significantly increase the workload. We can call initTestS3AInputPolicies in each unit test and explicitly call the setup method in the init method, which ensures that the tests pass.
cc: @cnauroth

slfan1989 · 2025-07-05T07:55:09Z

@steveloughran I would still prefer to have this PR temporarily approved, as it will help better advance the migration to JUnit 4. If you have a better implementation approach, I would be happy to assist you in further upgrading the code in the future.
cc: @cnauroth

steveloughran · 2025-07-07T16:08:11Z

+1 for the merge and then I can work on top of this

steveloughran

+1 and lets see where we go from here.

FWIW I am really unimpressed how they changed the signatures of asserttrue/assertequals etc, so that those people who put in the effort of writing meaningful assertion messages suffer.

I am so happy I've been using assertj for some time

slfan1989 · 2025-07-09T00:52:50Z

@steveloughran Thanks for supporting the merge of this PR! If any issues arise, feel free to me @slfan1989, and I'll address them as soon as possible. Additionally, once the JUnit 5 upgrade is complete, I plan to prioritize upgrading AssertJ in the Yarn module. I’ve found that AssertJ provides a more user-friendly assertion experience, which is more intuitive and easier to use compared to JUnit 5's native assertions.

steveloughran · 2025-07-09T11:48:56Z

I’ve found that AssertJ provides a more user-friendly assertion experience, which is more intuitive and easier to use compared to JUnit 5's native assertions.

really good for assertions over collections, e.g if an assert about size doesn't match, you get the two collections printed
you can extend them!. Look at IOStatisticAssertions there

steveloughran · 2025-07-09T11:51:51Z

I've generally left junit4 asserts alone in existing test cases, except on code I was actually editing. But given junit5 has broken every single assert with meaningful error messages, well, it's easier to justify, isn't it?

HADOOP-19424. [S3A] Upgrade JUnit from 4 to 5 in hadoop-aws.

09cd4a1

github-actions bot added trunk TOOLS AWS labels Jun 19, 2025

HADOOP-19424. [S3A] Upgrade JUnit from 4 to 5 in hadoop-aws.

001893e

github-actions bot added the build label Jun 21, 2025

slfan1989 marked this pull request as ready for review June 21, 2025 02:29

slfan1989 requested a review from steveloughran June 21, 2025 02:29

HADOOP-19424. Fix CheckStyle.

3e30cda

cnauroth approved these changes Jun 23, 2025

View reviewed changes

slfan1989 commented Jul 5, 2025

View reviewed changes

steveloughran approved these changes Jul 7, 2025

View reviewed changes

steveloughran merged commit f099f08 into apache:trunk Jul 7, 2025
4 checks passed

slfan1989 mentioned this pull request Jul 9, 2025

[DISCUSS] [JDK17] Upgrade JUnit / JavaDoc from 4 to 5. #7337

Draft

74 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19424. [S3A] Upgrade JUnit from 4 to 5 in hadoop-aws. #7752

HADOOP-19424. [S3A] Upgrade JUnit from 4 to 5 in hadoop-aws. #7752

Uh oh!

slfan1989 commented Jun 19, 2025 •

edited

Loading

Uh oh!

hadoop-yetus commented Jun 20, 2025

Uh oh!

slfan1989 commented Jun 21, 2025

Uh oh!

hadoop-yetus commented Jun 21, 2025

Uh oh!

hadoop-yetus commented Jun 21, 2025

Uh oh!

cnauroth left a comment

Uh oh!

slfan1989 commented Jun 24, 2025

Uh oh!

slfan1989 Jul 5, 2025

Uh oh!

slfan1989 commented Jul 5, 2025

Uh oh!

steveloughran commented Jul 7, 2025

Uh oh!

steveloughran left a comment

Uh oh!

Uh oh!

slfan1989 commented Jul 9, 2025 •

edited

Loading

Uh oh!

steveloughran commented Jul 9, 2025

Uh oh!

steveloughran commented Jul 9, 2025

Uh oh!

Uh oh!

HADOOP-19424. [S3A] Upgrade JUnit from 4 to 5 in hadoop-aws. #7752

HADOOP-19424. [S3A] Upgrade JUnit from 4 to 5 in hadoop-aws. #7752

Uh oh!

Conversation

slfan1989 commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Jun 20, 2025

Uh oh!

slfan1989 commented Jun 21, 2025

Uh oh!

hadoop-yetus commented Jun 21, 2025

Uh oh!

hadoop-yetus commented Jun 21, 2025

Uh oh!

cnauroth left a comment

Choose a reason for hiding this comment

Uh oh!

slfan1989 commented Jun 24, 2025

Uh oh!

slfan1989 Jul 5, 2025

Choose a reason for hiding this comment

Uh oh!

slfan1989 commented Jul 5, 2025

Uh oh!

steveloughran commented Jul 7, 2025

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

slfan1989 commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steveloughran commented Jul 9, 2025

Uh oh!

steveloughran commented Jul 9, 2025

Uh oh!

Uh oh!

slfan1989 commented Jun 19, 2025 •

edited

Loading

slfan1989 commented Jul 9, 2025 •

edited

Loading