HADOOP-17224. Install Intel ISA-L library in Dockerfile. #2537

tasanuma · 2020-12-10T04:29:00Z

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

hadoop-yetus · 2020-12-10T04:31:26Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/1/console in case of problems.

hadoop-yetus · 2020-12-10T05:18:15Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	3m 29s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+1 💚	shadedclient	25m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	hadolint	0m 2s		There were no new hadolint issues.
+1 💚	shellcheck	0m 0s		There were no new shellcheck issues.
+1 💚	shelldocs	0m 13s		There were no new shelldocs issues.
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	17m 1s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		48m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/1/artifact/out/Dockerfile
GITHUB PR	#2537
Optional Tests	dupname asflicense shellcheck shelldocs hadolint
uname	Linux ec85cb8065e4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c2cecfc`
Max. process+thread count	629 (vs. ulimit of 5500)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/1/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6 hadolint=1.11.1-0-g0e692dd
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-12-10T05:40:34Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/2/console in case of problems.

hadoop-yetus · 2020-12-10T22:23:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 7s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 40s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 8s		trunk passed
+1 💚	compile	27m 6s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	22m 21s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	3m 20s		trunk passed
+1 💚	mvnsite	30m 34s		trunk passed
+1 💚	shadedclient	19m 8s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	8m 47s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	8m 59s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	44m 33s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	47m 7s		trunk passed
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	27m 49s		the patch passed
+1 💚	compile	24m 57s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	24m 57s		the patch passed
+1 💚	compile	18m 20s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	18m 20s		the patch passed
-0 ⚠️	checkstyle	2m 47s	/diff-checkstyle-root.txt	root: The patch generated 1 new + 9 unchanged - 1 fixed = 10 total (was 10)
+1 💚	hadolint	0m 3s		There were no new hadolint issues.
+1 💚	mvnsite	20m 58s		the patch passed
+1 💚	shellcheck	0m 1s		There were no new shellcheck issues.
+1 💚	shelldocs	0m 14s		There were no new shelldocs issues.
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	17m 2s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 23s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	8m 12s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	39m 36s		the patch passed
			_ Other Tests _
-1 ❌	unit	622m 22s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 33s		The patch does not generate ASF License warnings.
		1003m 43s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestBPOfferService
	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.TestPersistBlocks
	hadoop.tools.dynamometer.TestDynamometerInfra
	hadoop.yarn.applications.distributedshell.TestDistributedShell
	hadoop.yarn.client.api.impl.TestAMRMClient
	hadoop.yarn.server.router.webapp.TestRouterWebServicesREST
	hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor
	hadoop.yarn.server.nodemanager.TestNodeStatusUpdater
	hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocation
	hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
	hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/2/artifact/out/Dockerfile
GITHUB PR	#2537
Optional Tests	dupname asflicense shellcheck shelldocs hadolint compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 52a6192b363a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c2cecfc`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/2/testReport/
Max. process+thread count	3239 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/2/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 hadolint=1.11.1-0-g0e692dd
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-12-11T12:44:23Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/3/console in case of problems.

hadoop-yetus · 2020-12-13T05:13:14Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/4/console in case of problems.

hadoop-yetus · 2020-12-13T20:08:35Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	3m 10s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 1s		Maven dependency ordering for branch
+1 💚	mvninstall	21m 3s		trunk passed
+1 💚	compile	21m 2s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	17m 57s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	2m 52s		trunk passed
+1 💚	mvnsite	25m 4s		trunk passed
+1 💚	shadedclient	15m 37s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 15s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	7m 27s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	32m 3s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	34m 2s		trunk passed
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for patch
+1 💚	mvninstall	21m 10s		the patch passed
+1 💚	compile	21m 9s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	21m 9s		the patch passed
+1 💚	compile	17m 36s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	17m 36s		the patch passed
-0 ⚠️	checkstyle	2m 46s	/diff-checkstyle-root.txt	root: The patch generated 1 new + 9 unchanged - 1 fixed = 10 total (was 10)
+1 💚	hadolint	0m 2s		There were no new hadolint issues.
+1 💚	mvnsite	20m 9s		the patch passed
+1 💚	shellcheck	0m 1s		There were no new shellcheck issues.
+1 💚	shelldocs	0m 19s		There were no new shelldocs issues.
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 25s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 10s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	7m 28s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	34m 21s		the patch passed
			_ Other Tests _
-1 ❌	unit	571m 41s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 51s		The patch does not generate ASF License warnings.
		896m 32s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
	hadoop.yarn.server.router.webapp.TestRouterWebServicesREST
	hadoop.yarn.applications.distributedshell.TestDistributedShell
	hadoop.tools.dynamometer.TestDynamometerInfra
	hadoop.yarn.sls.appmaster.TestAMSimulator

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/4/artifact/out/Dockerfile
GITHUB PR	#2537
Optional Tests	dupname asflicense shellcheck shelldocs hadolint compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 98a24f8c13ed 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3234e5e`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/4/testReport/
Max. process+thread count	4214 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/4/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 hadolint=1.11.1-0-g0e692dd
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-12-13T23:54:54Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/5/console in case of problems.

hadoop-yetus · 2020-12-14T06:27:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 1s		Maven dependency ordering for branch
+1 💚	mvninstall	21m 14s		trunk passed
+1 💚	compile	20m 17s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	17m 30s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	2m 49s		trunk passed
+1 💚	mvnsite	25m 8s		trunk passed
+1 💚	shadedclient	15m 15s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 12s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	7m 34s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	32m 6s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	34m 8s		trunk passed
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	21m 15s		the patch passed
+1 💚	compile	19m 30s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	19m 30s		the patch passed
+1 💚	compile	17m 33s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	17m 33s		the patch passed
-0 ⚠️	checkstyle	2m 49s	/diff-checkstyle-root.txt	root: The patch generated 1 new + 9 unchanged - 1 fixed = 10 total (was 10)
+1 💚	hadolint	0m 3s		There were no new hadolint issues.
+1 💚	mvnsite	20m 13s		the patch passed
+1 💚	shellcheck	0m 0s		There were no new shellcheck issues.
+1 💚	shelldocs	0m 18s		There were no new shelldocs issues.
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 23s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 42s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	8m 21s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	38m 17s		the patch passed
			_ Other Tests _
-1 ❌	unit	70m 31s	/patch-unit-root.txt	root in the patch failed.
+0 🆗	asflicense	0m 38s		ASF License check generated no output?
		393m 44s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestEncryptedTransfer
	hadoop.hdfs.TestFileChecksum
	hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
	hadoop.hdfs.TestMaintenanceState
	hadoop.hdfs.TestDecommissionWithStriped
	hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy
	hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
	hadoop.hdfs.TestDFSStartupVersions
	hadoop.hdfs.tools.TestECAdmin
	hadoop.hdfs.TestReconstructStripedFile
	hadoop.hdfs.TestDFSRollback
	hadoop.hdfs.TestDFSStripedInputStream
	hadoop.hdfs.TestAclsEndToEnd
	hadoop.hdfs.TestDecommissionWithBackoffMonitor
	hadoop.hdfs.TestViewDistributedFileSystem
	hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy
	hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy
	hadoop.hdfs.TestDecommission
	hadoop.hdfs.TestDFSStripedOutputStream
	hadoop.hdfs.TestErasureCodingExerciseAPIs
	hadoop.hdfs.TestWriteReadStripedFile
	hadoop.hdfs.TestReservedRawPaths
	hadoop.hdfs.TestSetrepIncreasing
	hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
	hadoop.metrics2.source.TestJvmMetrics

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/5/artifact/out/Dockerfile
GITHUB PR	#2537
Optional Tests	dupname asflicense shellcheck shelldocs hadolint compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 683aabfa135a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3234e5e`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/5/testReport/
Max. process+thread count	2638 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/5/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 hadolint=1.11.1-0-g0e692dd
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2020-12-15T04:53:13Z

@tasanuma In the last build, indeed some EC tests failed with OOM
May be it is because of ISA-L only?

iwasakims · 2020-12-15T13:14:25Z

All OOMs are "unable to create new native thread" indicating ulimit or resource shortage to create LWP. The first OOM is in TestJvmMetrics in hadoop-common. If ISA-L is related, the cause should be in the code path of ErasureCodeNative#loadLibrary. I don't have clear insight yet. I think we have been familiar with test failures by "unable to create new native thread" for a long time..

tasanuma · 2020-12-15T13:28:38Z

This PR ran Jenkins 4 times. The best is 25 failures and the worst is 274 failures.

According to some of the latest result of QBT, the best is 17 failures(#349) and the worst is 224 failures(#348). The average is about 50 failures.

I can't say for sure, but I don't think ISA-L has much to do with OOM.

tasanuma · 2020-12-15T13:42:27Z

@amahussein What do you think about it?

amahussein · 2020-12-16T21:40:26Z

All OOMs are "unable to create new native thread" indicating ulimit or resource shortage to create LWP. The first OOM is in TestJvmMetrics in hadoop-common. If ISA-L is related, the cause should be in the code path of ErasureCodeNative#loadLibrary. I don't have clear insight yet. I think we have been familiar with test failures by "unable to create new native thread" for a long time..

@iwasakims , I cannot fully confident that ErasureCodeNative#loadLibrary is a strong indication that ISLA-L does not contribute to the OOM.
ISA-L is a native library; therefore loading this library means different memory allocations and possibly some background threads.

For sure, we do not want to blame those pre-existing failures to ISA-L. However, adding ISA-L could increase failures because of the hadoop code, or the native code.

I think there are two approaches:

Profile the memory. Then compare the two profiles with and without ISA-L. If there is no Yetus hookup to do that, then it will have to be done on a local machine for a sample of unit tests.
Add another commit that ignores the failures frequently reported in QBT report. In addition I suggest adding "ignore" to TestDistributredShell#testDistributedShellWithResourcesWithLargeContainers and TestDistributredShell#testDistributedShellWithResources. Those two tests leave two ApplicationMaster processes running in the background. After ignoring the "every-day" failures, we can look at the remaining failures as possible consequences of loading ISA-L.

amahussein · 2020-12-17T15:56:49Z

@tasanuma, @iwasakims
did anyone consider increasing the resources allocated to the docker image? For example, increase the memory and see whether or not OOM disappears?

Based on Yetus documentations :

maybe we can try increasing --dockermemlimit=20g to 22 or 24g. Well, cautious needs to be given to avoid bringing the entire server down. So, increase it "responsibly" :)
Apache Yetus also sets the --oom-score-adj to 500 in order to offer itself as the first processes to be killed if memory is low.

tasanuma · 2020-12-18T05:21:57Z

@amahussein I didn't consider it. Thanks for trying it on HADOOP-17438. Let's see the result.

I'm also paying attention to #2556 that Akira is trying to reduce threadCount for unit tests. The result seems very good for now.

amahussein · 2021-01-12T13:55:34Z

@amahussein I didn't consider it. Thanks for trying it on HADOOP-17438. Let's see the result.

I'm also paying attention to #2556 that Akira is trying to reduce threadCount for unit tests. The result seems very good for now.

Hey @tasanuma and @iwasakims .
HADOOP-17438 has been merged to "3.3.1, 3.4.0, 3.1.5, 2.10.2, 3.2.3".
You might try to trigger the Yetus build one more time, hopefully without getting OOM.

hadoop-yetus · 2021-01-13T01:00:56Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/6/console in case of problems.

tasanuma · 2021-01-13T01:10:20Z

Thanks for letting me know, @amahussein. I rebased and pushed it again.

hadoop-yetus · 2021-01-13T16:35:05Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	3m 48s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	2m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	23m 32s		trunk passed
+1 💚	compile	21m 32s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	18m 21s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	2m 51s		trunk passed
+1 💚	mvnsite	21m 15s		trunk passed
+1 💚	shadedclient	17m 4s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 19s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	7m 32s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	34m 35s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	36m 39s		trunk passed
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	23m 50s		the patch passed
+1 💚	compile	20m 57s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	20m 57s		the patch passed
+1 💚	compile	18m 12s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	18m 12s		the patch passed
-0 ⚠️	checkstyle	2m 52s	/diff-checkstyle-root.txt	root: The patch generated 1 new + 9 unchanged - 1 fixed = 10 total (was 10)
+1 💚	hadolint	0m 3s		There were no new hadolint issues.
+1 💚	mvnsite	20m 57s		the patch passed
+1 💚	shellcheck	0m 0s		There were no new shellcheck issues.
+1 💚	shelldocs	0m 14s		There were no new shelldocs issues.
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	17m 9s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 21s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	7m 37s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	37m 17s		the patch passed
			_ Other Tests _
-1 ❌	unit	610m 16s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 49s		The patch does not generate ASF License warnings.
		934m 49s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.ha.TestStandbyInProgressTail
	hadoop.hdfs.TestReconstructStripedFile
	hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized
	hadoop.tools.dynamometer.TestDynamometerInfra
	hadoop.yarn.sls.TestReservationSystemInvariants
	hadoop.yarn.sls.appmaster.TestAMSimulator
	hadoop.yarn.server.nodemanager.TestNodeStatusUpdater
	hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/6/artifact/out/Dockerfile
GITHUB PR	#2537
Optional Tests	dupname asflicense shellcheck shelldocs hadolint compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 904cdcb03ab7 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `724edb0`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/6/testReport/
Max. process+thread count	3113 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/6/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 hadolint=1.11.1-0-g0e692dd
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-01-14T03:36:20Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/7/console in case of problems.

hadoop-yetus · 2021-01-14T19:24:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	29m 10s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 19s		Maven dependency ordering for branch
+1 💚	mvninstall	21m 8s		trunk passed
+1 💚	compile	20m 16s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	17m 38s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	2m 53s		trunk passed
+1 💚	mvnsite	25m 14s		trunk passed
+1 💚	shadedclient	15m 43s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 17s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	7m 35s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	37m 30s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	39m 29s		trunk passed
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	25m 52s		the patch passed
+1 💚	compile	25m 4s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	25m 4s		the patch passed
-1 ❌	compile	11m 14s	/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt	root in the patch failed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.
-1 ❌	javac	11m 14s	/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt	root in the patch failed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.
-0 ⚠️	checkstyle	3m 24s	/diff-checkstyle-root.txt	root: The patch generated 1 new + 9 unchanged - 1 fixed = 10 total (was 10)
+1 💚	hadolint	0m 3s		There were no new hadolint issues.
-1 ❌	mvnsite	17m 29s	/patch-mvnsite-root.txt	root in the patch failed.
+1 💚	shellcheck	0m 0s		There were no new shellcheck issues.
-1 ❌	shelldocs	0m 30s	/diff-patch-shelldocs.txt	The patch generated 334 new + 236 unchanged - 0 fixed = 570 total (was 236)
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	18m 52s		patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	1m 17s	/patch-javadoc-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt	root in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.
+1 💚	javadoc	8m 43s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	34m 50s		the patch passed
			_ Other Tests _
-1 ❌	unit	592m 29s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 53s		The patch does not generate ASF License warnings.
		949m 15s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
	hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
	hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor
	hadoop.tools.dynamometer.TestDynamometerInfra

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/7/artifact/out/Dockerfile
GITHUB PR	#2537
Optional Tests	dupname asflicense shellcheck shelldocs hadolint compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 1e70facff171 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `768e2f4`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/7/testReport/
Max. process+thread count	4309 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/7/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 hadolint=1.11.1-0-g0e692dd
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

tasanuma · 2021-01-15T07:36:31Z

I triggered the Yetus build to run all unit tests twice, and OOM does not cause the failed tests. I think we can commit this again.

hadoop-yetus · 2021-01-15T07:40:20Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/8/console in case of problems.

hadoop-yetus · 2021-01-15T22:44:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 5s		Maven dependency ordering for branch
+1 💚	mvninstall	21m 50s		trunk passed
+1 💚	compile	22m 29s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	18m 51s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	2m 50s		trunk passed
+1 💚	mvnsite	25m 30s		trunk passed
+1 💚	shadedclient	15m 41s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 22s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	7m 43s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	32m 24s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	34m 26s		trunk passed
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for patch
+1 💚	mvninstall	21m 22s		the patch passed
+1 💚	compile	19m 44s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	19m 44s		the patch passed
+1 💚	compile	17m 46s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	17m 46s		the patch passed
+1 💚	checkstyle	2m 51s		the patch passed
+1 💚	hadolint	0m 3s		There were no new hadolint issues.
+1 💚	mvnsite	20m 24s		the patch passed
+1 💚	shellcheck	0m 0s		There were no new shellcheck issues.
+1 💚	shelldocs	0m 18s		There were no new shelldocs issues.
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 43s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	7m 16s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	7m 33s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	34m 34s		the patch passed
			_ Other Tests _
-1 ❌	unit	578m 30s	/patch-unit-root.txt	root in the patch passed.
+1 💚	asflicense	1m 54s		The patch does not generate ASF License warnings.
		904m 43s

Reason	Tests
Failed junit tests	hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor
	hadoop.tools.dynamometer.TestDynamometerInfra

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/8/artifact/out/Dockerfile
GITHUB PR	#2537
Optional Tests	dupname asflicense shellcheck shelldocs hadolint compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 8e432e8c61e1 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `630f8dd`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/8/testReport/
Max. process+thread count	5011 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2537/8/console
versions	git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 hadolint=1.11.1-0-g0e692dd
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

amahussein · 2021-01-20T20:22:47Z

I triggered the Yetus build to run all unit tests twice, and OOM does not cause the failed tests. I think we can commit this again.

@tasanuma Have you verified that all the EC unit tests are passing after adding the library (i.e., TestBlockTokenWithDFSStriped.testEnd2End)?

tasanuma · 2021-01-21T02:57:22Z

Yes, the failed tests succeeded with ISA-L in my local computer, except for TestDynamometerInfra(#2471, #2622).

amahussein · 2021-01-21T14:35:41Z

+1 (non-binding)

iwasakims

+1 to commit this again.

tasanuma · 2021-01-22T01:28:08Z

Merged it. Thanks for your comments and reviews, @amahussein, @iwasakims, @ayushtkn!

(cherry picked from commit d09e3c9)

tasanuma added 6 commits January 13, 2021 09:58

HADOOP-17224. Install Intel ISA-L library in Dockerfile.

6b09a0f

increment timeout in Jenkinsfile

c7b500a

edit a java file to run all unit tests

2ef05f0

edit a java file to run all unit tests 2

4424d98

edit a java file to run all unit tests 3

b6afebe

edit a java file to run all unit tests 4

4107c9b

tasanuma force-pushed the HADOOP-17224 branch from 65dbd42 to 4107c9b Compare January 13, 2021 00:59

edit a java file to run all unit tests 5

2e0b3db

revert Jenkinsfile

d7a583a

iwasakims approved these changes Jan 22, 2021

View reviewed changes

tasanuma merged commit d09e3c9 into apache:trunk Jan 22, 2021

tasanuma deleted the HADOOP-17224 branch January 22, 2021 01:27

steveloughran mentioned this pull request Mar 9, 2021

HADOOP-17472. Upgrade hadoop default version in hadoop.tools.dynamometer.TestDynamometerInfra #2622

Closed

asfgit pushed a commit that referenced this pull request May 19, 2021

HADOOP-17224. Install Intel ISA-L library in Dockerfile. (#2537)

932a168

(cherry picked from commit d09e3c9)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-17224. Install Intel ISA-L library in Dockerfile. #2537

HADOOP-17224. Install Intel ISA-L library in Dockerfile. #2537

tasanuma commented Dec 10, 2020

hadoop-yetus commented Dec 10, 2020

hadoop-yetus commented Dec 10, 2020

hadoop-yetus commented Dec 10, 2020

hadoop-yetus commented Dec 10, 2020

hadoop-yetus commented Dec 11, 2020

hadoop-yetus commented Dec 13, 2020

hadoop-yetus commented Dec 13, 2020

hadoop-yetus commented Dec 13, 2020

hadoop-yetus commented Dec 14, 2020

ayushtkn commented Dec 15, 2020

iwasakims commented Dec 15, 2020

tasanuma commented Dec 15, 2020

tasanuma commented Dec 15, 2020

amahussein commented Dec 16, 2020

amahussein commented Dec 17, 2020

tasanuma commented Dec 18, 2020

amahussein commented Jan 12, 2021

hadoop-yetus commented Jan 13, 2021

tasanuma commented Jan 13, 2021

hadoop-yetus commented Jan 13, 2021

hadoop-yetus commented Jan 14, 2021

hadoop-yetus commented Jan 14, 2021

tasanuma commented Jan 15, 2021 •

edited

hadoop-yetus commented Jan 15, 2021

hadoop-yetus commented Jan 15, 2021

amahussein commented Jan 20, 2021

tasanuma commented Jan 21, 2021

amahussein commented Jan 21, 2021

iwasakims left a comment

tasanuma commented Jan 22, 2021

HADOOP-17224. Install Intel ISA-L library in Dockerfile. #2537

HADOOP-17224. Install Intel ISA-L library in Dockerfile. #2537

Conversation

tasanuma commented Dec 10, 2020

NOTICE

hadoop-yetus commented Dec 10, 2020

hadoop-yetus commented Dec 10, 2020

hadoop-yetus commented Dec 10, 2020

hadoop-yetus commented Dec 10, 2020

hadoop-yetus commented Dec 11, 2020

hadoop-yetus commented Dec 13, 2020

hadoop-yetus commented Dec 13, 2020

hadoop-yetus commented Dec 13, 2020

hadoop-yetus commented Dec 14, 2020

ayushtkn commented Dec 15, 2020

iwasakims commented Dec 15, 2020

tasanuma commented Dec 15, 2020

tasanuma commented Dec 15, 2020

amahussein commented Dec 16, 2020

amahussein commented Dec 17, 2020

tasanuma commented Dec 18, 2020

amahussein commented Jan 12, 2021

hadoop-yetus commented Jan 13, 2021

tasanuma commented Jan 13, 2021

hadoop-yetus commented Jan 13, 2021

hadoop-yetus commented Jan 14, 2021

hadoop-yetus commented Jan 14, 2021

tasanuma commented Jan 15, 2021 • edited

hadoop-yetus commented Jan 15, 2021

hadoop-yetus commented Jan 15, 2021

amahussein commented Jan 20, 2021

tasanuma commented Jan 21, 2021

amahussein commented Jan 21, 2021

iwasakims left a comment

Choose a reason for hiding this comment

tasanuma commented Jan 22, 2021

tasanuma commented Jan 15, 2021 •

edited