YARN-11404. Add junit5 dependency to hadoop-mapreduce-client-app to fix few unit test failure

[susheel-gupta]

Description of PR

How was this patch tested?

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 51 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	47m 20s		trunk passed
+1 💚	compile	0m 36s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	0m 32s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	0m 37s		trunk passed
+1 💚	mvnsite	0m 37s		trunk passed
-1 ❌	javadoc	0m 37s	/branch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-app in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	0m 25s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	1m 12s		trunk passed
+1 💚	shadedclient	26m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 42s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 25s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	0m 25s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 23s	/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 27 new + 869 unchanged - 80 fixed = 896 total (was 949)
+1 💚	mvnsite	0m 27s		the patch passed
-1 ❌	javadoc	0m 22s	/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-app in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	0m 21s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	1m 4s		the patch passed
+1 💚	shadedclient	26m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	8m 59s		hadoop-mapreduce-client-app in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		120m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5295/1/artifact/out/Dockerfile
GITHUB PR	#5295
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux d655990733ae 4.15.0-197-generic #208-Ubuntu SMP Tue Nov 1 17:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 43b4f5a
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5295/1/testReport/
Max. process+thread count	1074 (vs. ulimit of 5500)
modules	C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5295/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 51 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	46m 23s		trunk passed
+1 💚	compile	0m 35s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	0m 30s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	0m 37s		trunk passed
+1 💚	mvnsite	0m 35s		trunk passed
-1 ❌	javadoc	0m 37s	/branch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-app in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	0m 26s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	1m 11s		trunk passed
+1 💚	shadedclient	26m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 42s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 26s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	0m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 23s	/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 25 new + 871 unchanged - 78 fixed = 896 total (was 949)
+1 💚	mvnsite	0m 28s		the patch passed
-1 ❌	javadoc	0m 22s	/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-app in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	0m 21s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	1m 3s		the patch passed
+1 💚	shadedclient	27m 31s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	9m 40s		hadoop-mapreduce-client-app in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		121m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5295/2/artifact/out/Dockerfile
GITHUB PR	#5295
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux 0ec12ebfe6c1 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / d3d685a
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5295/2/testReport/
Max. process+thread count	1073 (vs. ulimit of 5500)
modules	C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5295/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 51 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	47m 27s		trunk passed
+1 💚	compile	0m 39s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	0m 32s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	0m 37s		trunk passed
+1 💚	mvnsite	0m 36s		trunk passed
-1 ❌	javadoc	0m 37s	/branch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-app in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	0m 26s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	1m 14s		trunk passed
+1 💚	shadedclient	26m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 43s		the patch passed
+1 💚	compile	0m 32s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	0m 32s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 23s	/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 25 new + 871 unchanged - 78 fixed = 896 total (was 949)
+1 💚	mvnsite	0m 29s		the patch passed
-1 ❌	javadoc	0m 22s	/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-mapreduce-client-app in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	0m 22s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	1m 6s		the patch passed
+1 💚	shadedclient	27m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	9m 2s		hadoop-mapreduce-client-app in the patch passed.
+1 💚	asflicense	0m 33s		The patch does not generate ASF License warnings.
		122m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5295/3/artifact/out/Dockerfile
GITHUB PR	#5295
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux 5b49813e1aae 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 020fb8d
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5295/3/testReport/
Max. process+thread count	1024 (vs. ulimit of 5500)
modules	C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5295/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[szilard-nemeth]

Thanks @susheel-gupta for working on this.
Latest patch LGTM, committed to trunk

[steveloughran]

can we have these versioned imports pulled up into the hadoop-project pom for (a) version maintenance and (b) ease of using an IDE to find where things are used. this is particularly important for mockito as it is so brittle

[szilard-nemeth]

Hi @steveloughran ,
Fair point.

@susheel-gupta Could you please open a follow-up jira for this?

Thanks.

[steveloughran]

couple of belated comments

1. I don't see any need anywhere in the codebase to move to jupiter assertions. It makes backporting harder without offering any tangible benefits.
   If the jupiter team chose to move classes to new packages, well, that's their choice. But if we are going to update test asserts, assertJ is a far better assert framework.
   It's a richer assertion syntax, generates better messages and is already in the 3.3 line -which is why we are using for much of the new tests.
   2, new pom imports should be added to hadoop-project and then referenced, so we can stay on top of the changes.

I'm not going to suggest rolling this back -as it's in, and it was a big piece of work. It's just that in particular the code changes for jupiter assertions wasn't needed and it's potentially counter-productive.

What I would propose is

• followup PR to move the pom declarations up
• no new patches to move to jupiter asserts. stay on org.junit or embrace assertJ

[susheel-gupta]

couple of belated comments

1. I don't see any need anywhere in the codebase to move to jupiter assertions. It makes backporting harder without offering any tangible benefits.
   If the jupiter team chose to move classes to new packages, well, that's their choice. But if we are going to update test asserts, assertJ is a far better assert framework.
   It's a richer assertion syntax, generates better messages and is already in the 3.3 line -which is why we are using for much of the new tests.
   2, new pom imports should be added to hadoop-project and then referenced, so we can stay on top of the changes.

I'm not going to suggest rolling this back -as it's in, and it was a big piece of work. It's just that in particular the code changes for jupiter assertions wasn't needed and it's potentially counter-productive.

What I would propose is

• followup PR to move the pom declarations up
• no new patches to move to jupiter asserts. stay on org.junit or embrace assertJ

Hi @steveloughran,
Thanks for the comments.
I followed this previously merged pr #4771 by @aajisaka .
I will create a followup ticket to move the pom declarations up.

[steveloughran]

thanks for the update, the pom pull would be welcome. and if there are other places we need to do that, don't be afraid to include in the same PR

[ayushtkn]

This seems to have broken the mapreduce tests.
https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1143/testReport/

Reverting this fixed it locally for me, I think there ain't any followup ticket as Steve asked for either. Have reverted this.

In case I have messed up, I will commit this again may be once I figure out the culprit, else this got sacrificed in my attempt to get the build green

Tests passed locally

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.mapreduce.v2.TestSpeculativeExecutionWithMRApp
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.528 s - in org.apache.hadoop.mapreduce.v2.TestSpeculativeExecutionWithMRApp
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.986 s - in org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing
[INFO] Running org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEvents
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.179 s - in org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEvents
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0

PS. The PR was closed not merged, not sure how & why but it was initially confusing to me

[steveloughran]

Caused by: java.lang.ClassNotFoundException: org.junit.jupiter.api.Assertions
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 14 more

problem here is that maven doesn't import transitive dependencies from a -test JAR, so if one module (mapreduce-client) adds a new mandatory dependency (here, junit jupiter stuff, maybe junit-mockito) then everything which imports the suite and uses it will break. We had that when we added assertJ tests to hadoop-common fs contract tests.

Fix here is pom changes everywhere the test jar is referenced.

[ayushtkn]

Update:
This indeed was the culprit, reverting this fixed the tests.
https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1145/#showFailuresLink

as the testcase TestAMWebServicesJobConf, TestAMWebServicesJobs, TestAMWebServices, TestAMWebServicesAttempts, TestAMWebServicesTasks were passing locally but failed at jenkins build in this

From the Jira description : It looks like this PR was supposed to fix some tests but reverting this, didn't break them.
https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1145/testReport/org.apache.hadoop.mapreduce.v2.app.webapp/

[szilard-nemeth]

Hi @ayushtkn ,
What I still don't understand is why the tests were passing before I did the commit.

@susheel-gupta Anyway, the way forward in my opinion is to figure out if we can make the tests pass with your original commit + @steveloughran 's suggested pom fix.

[szilard-nemeth]

PS. The PR was closed not merged, not sure how & why but it was initially confusing to me

@ayushtkn
Yes, because I usually create commits by downloading the diff from github, apply, commit in my CLI, then simply closing the PR.
This way I can preserve author info for commits.

[ayushtkn]

Hi @szilard-nemeth

What I still don't understand is why the tests were passing before I did the commit.

Because those tests weren't executed. The way our Jenkins job is setup, it is like it runs the tests only for the modules which you touch as part of your change. Not even the modules which your change can impact, unfortunate but we can't run the entire test suites for all PRs, it would take around 20 hrs or more for it :)