HADOOP-17786. Parallelize stages in Jenkins #3166

GauthamBanasandra · 2021-07-01T10:46:09Z

We now build Hadoop on multiple platforms
for validation. We need to parallelize them
for faster validation.

hadoop-yetus · 2021-07-01T11:46:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 31s		Maven dependency ordering for branch
+1 💚	shadedclient	15m 26s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 8 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	15m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 31s		The patch does not generate ASF License warnings.
		47m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/2/artifact/out/Dockerfile
GITHUB PR	#3166
Optional Tests	dupname asflicense codespell shellcheck shelldocs
uname	Linux e05c3d46624a 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / d20f84a6956c5e4c3c4c3d9e7edc43585f8133e6
Max. process+thread count	524 (vs. ulimit of 5500)
modules	C: U:
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/2/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-07-27T11:22:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	37m 55s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 59s		trunk passed
+1 💚	compile	5m 43s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	6m 16s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	mvnsite	0m 33s		trunk passed
+1 💚	shadedclient	20m 55s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 31s		the patch passed
+1 💚	compile	5m 42s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	cc	5m 42s		the patch passed
+1 💚	golang	5m 42s		the patch passed
+1 💚	javac	5m 42s		the patch passed
+1 💚	compile	6m 52s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	cc	6m 52s		the patch passed
+1 💚	golang	6m 52s		the patch passed
+1 💚	javac	6m 52s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 34s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	21m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	43m 46s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in the patch failed.
+1 💚	asflicense	1m 24s		The patch does not generate ASF License warnings.
		201m 14s

Reason	Tests
Failed CTEST tests	test_test_libhdfs_ops_hdfs_static
	test_test_libhdfs_threaded_hdfs_static
	test_test_libhdfs_zerocopy_hdfs_static
	test_test_native_mini_dfs
	test_libhdfs_threaded_hdfspp_test_shim_static
	test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static
	libhdfs_mini_stress_valgrind_hdfspp_test_static
	memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static
	test_libhdfs_mini_stress_hdfspp_test_shim_static
	test_hdfs_ext_hdfspp_test_shim_static

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/4/artifact/out/Dockerfile
GITHUB PR	#3166
Optional Tests	dupname asflicense codespell shellcheck shelldocs compile cc mvnsite javac unit golang
uname	Linux aeab8e924c32 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 16ebf88db8bfbc0f8ec60f5c5c1c739dca4baa38
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
CTEST	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/4/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/4/testReport/
Max. process+thread count	520 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/4/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-07-27T11:24:57Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	48m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shellcheck	0m 0s		Shellcheck was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 26s		trunk passed
+1 💚	compile	6m 31s		trunk passed
+1 💚	mvnsite	1m 24s		trunk passed
+1 💚	shadedclient	66m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 28s		the patch passed
+1 💚	compile	4m 48s		the patch passed
+1 💚	cc	4m 48s		the patch passed
+1 💚	golang	4m 48s		the patch passed
+1 💚	javac	4m 48s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 53s		the patch passed
+1 💚	shadedclient	26m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	51m 56s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in the patch failed.
+1 💚	asflicense	1m 45s		The patch does not generate ASF License warnings.
		203m 57s

Reason	Tests
Failed CTEST tests	test_test_libhdfs_threaded_hdfs_static
	test_test_libhdfs_zerocopy_hdfs_static
	test_test_native_mini_dfs
	test_libhdfs_threaded_hdfspp_test_shim_static
	test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static
	libhdfs_mini_stress_valgrind_hdfspp_test_static
	memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static
	test_libhdfs_mini_stress_hdfspp_test_shim_static
	test_hdfs_ext_hdfspp_test_shim_static

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/4/artifact/out/Dockerfile
GITHUB PR	#3166
Optional Tests	dupname asflicense codespell shellcheck shelldocs compile cc mvnsite javac unit golang
uname	Linux d4c936a36643 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 16ebf88db8bfbc0f8ec60f5c5c1c739dca4baa38
Default Java	Red Hat, Inc.-1.8.0_302-b08
CTEST	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/4/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/4/testReport/
Max. process+thread count	615 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/4/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-07-27T12:36:32Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	34m 38s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	50m 52s		trunk passed
+1 💚	compile	6m 24s		trunk passed
+1 💚	mvnsite	0m 45s		trunk passed
+1 💚	shadedclient	28m 31s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 25s		the patch passed
+1 💚	compile	5m 54s		the patch passed
+1 💚	cc	5m 54s		the patch passed
+1 💚	golang	5m 54s		the patch passed
+1 💚	javac	5m 54s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	26m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	113m 5s		hadoop-hdfs-native-client in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		275m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/4/artifact/out/Dockerfile
GITHUB PR	#3166
Optional Tests	dupname asflicense codespell shellcheck shelldocs compile cc mvnsite javac unit golang
uname	Linux b43ef5bce072 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 16ebf88db8bfbc0f8ec60f5c5c1c739dca4baa38
Default Java	Debian-11.0.11+9-post-Debian-1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/4/testReport/
Max. process+thread count	532 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/4/console
versions	git=2.30.2 maven=3.6.3 shellcheck=0.7.1
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-07-27T14:02:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	16m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
-1 ❌	mvninstall	3m 21s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	4m 44s		trunk passed
+1 💚	mvnsite	0m 22s		trunk passed
+1 💚	shadedclient	20m 26s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	2m 27s		the patch passed
+1 💚	cc	2m 27s		the patch passed
+1 💚	golang	2m 27s		the patch passed
+1 💚	javac	2m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 16s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	17m 33s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	39m 54s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in the patch failed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		109m 25s

Reason	Tests
Failed CTEST tests	test_libhdfs_threaded_hdfspp_test_shim_static

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/5/artifact/out/Dockerfile
GITHUB PR	#3166
Optional Tests	dupname asflicense codespell shellcheck shelldocs compile cc mvnsite javac unit golang
uname	Linux 7b209ad31bfe 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 3e46d9e6f788fa3578092e6ad2e903ff9cbbb1e6
Default Java	Debian-11.0.11+9-post-Debian-1
CTEST	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/5/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/5/testReport/
Max. process+thread count	563 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/5/console
versions	git=2.30.2 maven=3.6.3 shellcheck=0.7.1
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

* We now build Hadoop on multiple platforms for validation. We need to parallelize them for faster validation.

* Added an option to the pipeline that fails as soon as any of the parallel stages fails.

* Parallellization of the stages must happen after the code has been checked out for each of the environments.

* Added an option parallelsAlwaysFailFast to fail the full CI immediately when any stage fails.

hadoop-yetus · 2021-08-03T09:39:52Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	46m 41s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shellcheck	0m 0s		Shellcheck was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 51s		trunk passed
+1 💚	compile	5m 18s		trunk passed
+1 💚	mvnsite	0m 48s		trunk passed
+1 💚	shadedclient	66m 55s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 22s		the patch passed
+1 💚	compile	3m 2s		the patch passed
+1 💚	cc	3m 2s		the patch passed
+1 💚	golang	3m 2s		the patch passed
+1 💚	javac	3m 2s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 23s		the patch passed
+1 💚	shadedclient	18m 3s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	84m 48s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt	hadoop-hdfs-native-client in the patch failed.
+1 💚	asflicense	1m 33s		The patch does not generate ASF License warnings.
		224m 50s

Reason	Tests
Failed CTEST tests	test_test_libhdfs_ops_hdfs_static
	test_test_libhdfs_threaded_hdfs_static
	test_test_libhdfs_zerocopy_hdfs_static
	test_test_native_mini_dfs

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/6/artifact/out/Dockerfile
GITHUB PR	#3166
Optional Tests	dupname asflicense codespell shellcheck shelldocs compile cc mvnsite javac unit golang
uname	Linux c9fc6c9774f1 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `26529d0`
Default Java	Red Hat, Inc.-1.8.0_302-b08
CTEST	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/6/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/6/testReport/
Max. process+thread count	601 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3166/6/console
versions	git=2.27.0 maven=3.6.3
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

GauthamBanasandra force-pushed the llize-stages branch from 9fb432c to d20f84a Compare July 1, 2021 10:57

GauthamBanasandra marked this pull request as draft July 1, 2021 11:35

GauthamBanasandra force-pushed the llize-stages branch from 9cadc30 to 16ebf88 Compare July 27, 2021 07:59

GauthamBanasandra added 7 commits August 3, 2021 11:21

HADOOP-17786. Parallelize stages in Jenkins

ac10e39

* We now build Hadoop on multiple platforms for validation. We need to parallelize them for faster validation.

Fail fast when any parallel stage fails

7b52a2b

* Added an option to the pipeline that fails as soon as any of the parallel stages fails.

Have multiple checkouts

f31bac9

Trigger CI

8ef6fde

Parallelize after code checkout

84cb1b3

* Parallellization of the stages must happen after the code has been checked out for each of the environments.

Fail fast when any parallel stage fails

437eccf

* Added an option parallelsAlwaysFailFast to fail the full CI immediately when any stage fails.

Rebase with trunk

26529d0

GauthamBanasandra force-pushed the llize-stages branch from 3e46d9e to 26529d0 Compare August 3, 2021 05:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-17786. Parallelize stages in Jenkins #3166

HADOOP-17786. Parallelize stages in Jenkins #3166

GauthamBanasandra commented Jul 1, 2021

hadoop-yetus commented Jul 1, 2021

hadoop-yetus commented Jul 27, 2021

hadoop-yetus commented Jul 27, 2021

hadoop-yetus commented Jul 27, 2021

hadoop-yetus commented Jul 27, 2021

hadoop-yetus commented Aug 3, 2021

HADOOP-17786. Parallelize stages in Jenkins #3166

Are you sure you want to change the base?

HADOOP-17786. Parallelize stages in Jenkins #3166

Conversation

GauthamBanasandra commented Jul 1, 2021

hadoop-yetus commented Jul 1, 2021

hadoop-yetus commented Jul 27, 2021

hadoop-yetus commented Jul 27, 2021

hadoop-yetus commented Jul 27, 2021

hadoop-yetus commented Jul 27, 2021

hadoop-yetus commented Aug 3, 2021