HDFS-15413. add dfs.client.read.striped.datanode.max.attempts to fix read ecfile timeout #5829

Neilxzn · 2023-07-11T12:42:24Z

Description of PR

https://issues.apache.org/jira/browse/HDFS-15413

Offer a available patch to fix HDFS-15413. This patch add dfs.client.read.striped.datanode.max.attempts config to allow users to adjust the number of dn retries to solve the problem of Datanode timeout when reading EC files.

How was this patch tested?

no add test. just test in our cluster

For code changes:

add dfs.client.read.striped.datanode.max.attempts config

hadoop-yetus · 2023-07-11T20:15:05Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 55s		Maven dependency ordering for branch
+1 💚	mvninstall	37m 31s		trunk passed
+1 💚	compile	6m 12s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	5m 55s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	1m 30s		trunk passed
+1 💚	mvnsite	2m 24s		trunk passed
+1 💚	javadoc	1m 53s		trunk passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	2m 17s		trunk passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	6m 2s		trunk passed
+1 💚	shadedclient	44m 7s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 7s		the patch passed
+1 💚	compile	6m 25s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	6m 25s		the patch passed
+1 💚	compile	6m 6s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	6m 6s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	1m 27s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 8 new + 45 unchanged - 0 fixed = 53 total (was 45)
+1 💚	mvnsite	2m 6s		the patch passed
+1 💚	javadoc	1m 33s		the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	2m 5s		the patch passed with JDK Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	spotbugs	6m 8s		the patch passed
+1 💚	shadedclient	40m 45s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 22s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	253m 30s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 52s		The patch does not generate ASF License warnings.
		451m 11s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.ha.TestObserverNode

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/1/artifact/out/Dockerfile
GITHUB PR	#5829
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 2fa23e2f4762 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `5366ce9`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/1/testReport/
Max. process+thread count	2328 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao · 2023-11-03T03:20:52Z

cc @zhangshuyan0 Would you mind to take a review?

ayushtkn

Should add tests around this, which can reproduce these issues, maybe by setting a lower value for socket timeout.

Should cover scenarios, where

Connection to DN containing DataBlock is established.
Connection to DN containing ParityBlock is established.
When there are missing/lost nodes in the pipeline

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java

Hexiaoqiao · 2023-11-06T04:49:55Z

Please also check the checkstyle and blannks reported by Yetus. Thanks. @Neilxzn

Neilxzn · 2023-11-06T14:01:26Z

Please also check the checkstyle and blannks reported by Yetus. Thanks. @Neilxzn

Fix these checkstyle and add unit test. Please review it again. Thanks

hadoop-yetus · 2023-11-06T19:16:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	9m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 3s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 34s		trunk passed
+1 💚	compile	3m 7s		trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	compile	3m 4s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	checkstyle	0m 52s		trunk passed
+1 💚	mvnsite	1m 31s		trunk passed
+1 💚	javadoc	1m 22s		trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 40s		trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	3m 23s		trunk passed
+1 💚	shadedclient	21m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	2m 59s		the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javac	2m 59s		the patch passed
+1 💚	compile	2m 51s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	javac	2m 51s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 41s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 1 new + 45 unchanged - 0 fixed = 46 total (was 45)
+1 💚	mvnsite	1m 17s		the patch passed
+1 💚	javadoc	1m 5s		the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 30s		the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚	spotbugs	3m 25s		the patch passed
+1 💚	shadedclient	21m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 56s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	192m 14s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		315m 36s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSUtil
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/2/artifact/out/Dockerfile
GITHUB PR	#5829
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux a7218a7ce8bd 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ff4b113`
Default Java	Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/2/testReport/
Max. process+thread count	3329 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

bbeaudreault · 2023-11-08T23:53:58Z

@ayushtkn @zhangshuyan0 looks like the remaining failing checks are unrelated, and the feedback was addressed. Any chance for another look?

ayushtkn

I am not sure if this test is reproducing the issue for me, I reverted the changes in StripeReader and ran the test & it still passed.

If that gets sorted, We should add a test, where one DN is dead, like same test, but kill a DN

...t/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStreamWithTimeout.java

Hexiaoqiao · 2023-11-17T03:47:01Z

Hi @Neilxzn Any progress here? Thanks.

bbeaudreault · 2023-12-03T16:50:18Z

Hi @Neilxzn , any chance you have time to finish this up?

Neilxzn · 2023-12-04T11:08:06Z

Hi @Neilxzn , any chance you have time to finish this up?

Sorry for the late reply. I have been busy with other things recently. I will try to submit a new unit test tomorrow.

hadoop-yetus · 2023-12-05T15:04:46Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 2s		Maven dependency ordering for branch
+1 💚	mvninstall	35m 6s		trunk passed
+1 💚	compile	6m 4s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	5m 51s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	1m 27s		trunk passed
+1 💚	mvnsite	2m 22s		trunk passed
+1 💚	javadoc	1m 52s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 21s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	5m 56s		trunk passed
+1 💚	shadedclient	40m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 59s		the patch passed
+1 💚	compile	5m 54s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	5m 54s		the patch passed
+1 💚	compile	5m 42s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	5m 42s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	1m 17s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 2 new + 45 unchanged - 0 fixed = 47 total (was 45)
+1 💚	mvnsite	2m 6s		the patch passed
+1 💚	javadoc	1m 32s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	2m 6s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	5m 59s		the patch passed
+1 💚	shadedclient	40m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 22s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	251m 52s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		439m 20s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSStripedInputStreamWithTimeout
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/3/artifact/out/Dockerfile
GITHUB PR	#5829
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 15e76d99d238 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `db52c34`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/3/testReport/
Max. process+thread count	2531 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-12-06T11:46:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 24s		Maven dependency ordering for branch
+1 💚	mvninstall	19m 14s		trunk passed
+1 💚	compile	2m 51s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	2m 47s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 44s		trunk passed
+1 💚	mvnsite	1m 16s		trunk passed
+1 💚	javadoc	1m 3s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 29s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 3s		trunk passed
+1 💚	shadedclient	20m 9s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 2s		the patch passed
+1 💚	compile	2m 47s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	2m 47s		the patch passed
+1 💚	compile	2m 42s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	2m 42s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 35s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 1 new + 45 unchanged - 0 fixed = 46 total (was 45)
+1 💚	mvnsite	1m 6s		the patch passed
+1 💚	javadoc	0m 50s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 20s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 5s		the patch passed
+1 💚	shadedclient	20m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 49s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	189m 58s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 27s		The patch does not generate ASF License warnings.
		293m 19s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSStripedInputStreamWithTimeout

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/4/artifact/out/Dockerfile
GITHUB PR	#5829
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 807603bf2dcf 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `bb46dbd`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/4/testReport/
Max. process+thread count	3787 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Neilxzn · 2023-12-07T04:31:56Z

I can pass the unit test hadoop.hdfs.TestDFSStripedInputStreamWithTimeout in my local development environment, but it fails on GitHub Jenkins.

Check if the test log of the development environment is consistent with the assumption. When the client reads the file for the first time and stops for 10 seconds, the connection between the client and the datanode server will be automatically disconnected, resulting in a failed subsequent read by the client. @ayushtkn Any other suggestions?

ayushtkn · 2023-12-07T10:49:59Z

@Neilxzn I tried & it fails locally

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithTimeout
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.64 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithTimeout
[ERROR] testPreadTimeout(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithTimeout)  Time elapsed: 17.509 s  <<< FAILURE!
java.lang.AssertionError: It Should fail to read striped time out with 1 attempt . 
        at org.junit.Assert.fail(Assert.java:89)
        at org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithTimeout.testPreadTimeout(TestDFSStripedInputStreamWithTimeout.java:145)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:750)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   TestDFSStripedInputStreamWithTimeout.testPreadTimeout:145 It Should fail to read striped time out with 1 attempt . 
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO] 
[ERROR] There are test failures.

To reproduce:
in the hadoop root directory there is file named start-build-env.sh, run that bash start-build-env.sh, it will give you a docker env, run the test inside that & it will fail

Neilxzn · 2023-12-11T09:44:49Z

@Neilxzn I tried & it fails locally

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithTimeout
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.64 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithTimeout
[ERROR] testPreadTimeout(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithTimeout)  Time elapsed: 17.509 s  <<< FAILURE!
java.lang.AssertionError: It Should fail to read striped time out with 1 attempt . 
        at org.junit.Assert.fail(Assert.java:89)
        at org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithTimeout.testPreadTimeout(TestDFSStripedInputStreamWithTimeout.java:145)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
        at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.lang.Thread.run(Thread.java:750)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   TestDFSStripedInputStreamWithTimeout.testPreadTimeout:145 It Should fail to read striped time out with 1 attempt . 
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO] 
[ERROR] There are test failures.

To reproduce: in the hadoop root directory there is file named start-build-env.sh, run that bash start-build-env.sh, it will give you a docker env, run the test inside that & it will fail

Thank you. I will check it again soon.

Neilxzn · 2024-03-11T02:49:09Z

Hi @Neilxzn Any progress here? Thanks.

this PR is still necessary, there are some similar problems in our environment~

@haiyang1987 Our online environment (70 PB EC Data cluster， spark + hive olap) has already applied this patch. So far, everything is running normally.

Neilxzn · 2024-03-11T02:51:18Z

Rebase it to branch trunk. And The new test org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithTimeout passed on my lolcal env.

haiyang1987 · 2024-03-11T02:54:03Z

Hi @Neilxzn Any progress here? Thanks.
this PR is still necessary, there are some similar problems in our environment~

@haiyang1987 Our online environment (70 PB EC Data cluster， spark + hive olap) has already applied this patch. So far, everything is running normally.

Noted, thanks for you work ~

hadoop-yetus · 2024-03-11T08:03:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 55s		Maven dependency ordering for branch
+1 💚	mvninstall	20m 51s		trunk passed
+1 💚	compile	3m 1s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	2m 53s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 46s		trunk passed
+1 💚	mvnsite	1m 21s		trunk passed
+1 💚	javadoc	1m 10s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 33s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
-1 ❌	spotbugs	1m 30s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html	hadoop-hdfs-project/hadoop-hdfs-client in trunk has 1 extant spotbugs warnings.
+1 💚	shadedclient	22m 5s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	22m 18s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 10s		the patch passed
+1 💚	compile	3m 5s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	3m 5s		the patch passed
+1 💚	compile	3m 1s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	3m 1s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 40s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 1 new + 45 unchanged - 0 fixed = 46 total (was 45)
+1 💚	mvnsite	1m 11s		the patch passed
+1 💚	javadoc	0m 58s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 26s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 4s		the patch passed
+1 💚	shadedclient	22m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 49s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	217m 9s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		330m 42s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
	hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
	hadoop.hdfs.TestDFSStripedInputStreamWithTimeout
	hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor
	hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
	hadoop.hdfs.TestFileChecksum
	hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
	hadoop.hdfs.tools.TestDFSAdmin

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/6/artifact/out/Dockerfile
GITHUB PR	#5829
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux e826827ee6d4 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `7e4992b`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/6/testReport/
Max. process+thread count	4313 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-03-11T08:53:04Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 40s		Maven dependency ordering for branch
+1 💚	mvninstall	21m 13s		trunk passed
+1 💚	compile	3m 2s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	3m 5s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	checkstyle	0m 48s		trunk passed
+1 💚	mvnsite	1m 19s		trunk passed
+1 💚	javadoc	1m 8s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 48s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
-1 ❌	spotbugs	1m 39s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html	hadoop-hdfs-project/hadoop-hdfs-client in trunk has 1 extant spotbugs warnings.
+1 💚	shadedclient	22m 50s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	23m 3s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 1s		the patch passed
+1 💚	compile	2m 55s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	2m 55s		the patch passed
+1 💚	compile	2m 56s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	2m 56s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 44s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 1 new + 45 unchanged - 0 fixed = 46 total (was 45)
+1 💚	mvnsite	1m 13s		the patch passed
+1 💚	javadoc	0m 53s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 34s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	spotbugs	3m 23s		the patch passed
+1 💚	shadedclient	23m 7s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 54s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	235m 57s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		351m 39s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestEncryptionZonesWithKMS
	hadoop.hdfs.TestDFSStripedInputStreamWithTimeout
	hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
	hadoop.hdfs.server.namenode.TestFSEditLogLoader
	hadoop.hdfs.server.namenode.TestAuditLogs
	hadoop.hdfs.tools.TestDFSAdmin
	hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/7/artifact/out/Dockerfile
GITHUB PR	#5829
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux f03f54ac86d1 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `7e4992b`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/7/testReport/
Max. process+thread count	4049 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/7/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java

zhangshuyan0

As @haiyang1987 said, we should make sure pread work well after retrying.

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java

zhangshuyan0 · 2024-04-15T10:46:19Z

@Neilxzn Hi, this patch is very useful, would you mind further fixing this PR?

Neilxzn · 2024-04-17T10:29:53Z

@Neilxzn Hi, this patch is very useful, would you mind further fixing this PR?

Sorry for my late reply. I have updated the patch based on the suggestions above. Please review it again. @haiyang1987 @zhangshuyan0

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java

hadoop-yetus · 2024-04-17T15:41:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 56s		Maven dependency ordering for branch
+1 💚	mvninstall	20m 11s		trunk passed
+1 💚	compile	2m 56s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	2m 53s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	0m 47s		trunk passed
+1 💚	mvnsite	1m 24s		trunk passed
+1 💚	javadoc	1m 9s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 31s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	3m 17s		trunk passed
+1 💚	shadedclient	21m 19s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	21m 32s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 3s		the patch passed
+1 💚	compile	2m 51s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	2m 51s		the patch passed
+1 💚	compile	2m 44s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	2m 44s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 34s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 1 new + 45 unchanged - 0 fixed = 46 total (was 45)
+1 💚	mvnsite	1m 8s		the patch passed
+1 💚	javadoc	0m 54s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 28s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	3m 10s		the patch passed
+1 💚	shadedclient	21m 13s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 54s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	209m 14s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		318m 25s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSStripedInputStreamWithTimeout
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/8/artifact/out/Dockerfile
GITHUB PR	#5829
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 574d2feade6c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `5d597a1`
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/8/testReport/
Max. process+thread count	3853 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/8/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

haiyang1987 · 2024-04-18T01:45:49Z

The UT hadoop.hdfs.TestDFSStripedInputStreamWithTimeout run failed.

hadoop-yetus · 2024-04-19T12:59:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 0s		Maven dependency ordering for branch
+1 💚	mvninstall	19m 50s		trunk passed
+1 💚	compile	2m 58s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	2m 51s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	0m 48s		trunk passed
+1 💚	mvnsite	1m 21s		trunk passed
+1 💚	javadoc	1m 12s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 34s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	3m 10s		trunk passed
+1 💚	shadedclient	21m 16s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	21m 30s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 7s		the patch passed
+1 💚	compile	2m 52s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	2m 52s		the patch passed
+1 💚	compile	2m 49s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	2m 49s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 37s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 1 new + 45 unchanged - 0 fixed = 46 total (was 45)
+1 💚	mvnsite	1m 6s		the patch passed
+1 💚	javadoc	0m 54s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 26s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	3m 12s		the patch passed
+1 💚	shadedclient	21m 22s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 51s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	211m 45s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 31s		The patch does not generate ASF License warnings.
		321m 9s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDFSStripedInputStreamWithTimeout

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/9/artifact/out/Dockerfile
GITHUB PR	#5829
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 911c52bf0959 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `381f5ca`
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/9/testReport/
Max. process+thread count	4391 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/9/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Neilxzn · 2024-04-19T13:08:01Z

The UT hadoop.hdfs.TestDFSStripedInputStreamWithTimeout run failed.

add Datanode$closeDataXceiverServer method to close connnect for testing.

hadoop-yetus · 2024-04-19T18:19:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 14s		Maven dependency ordering for branch
+1 💚	mvninstall	20m 8s		trunk passed
+1 💚	compile	2m 58s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	2m 49s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	0m 46s		trunk passed
+1 💚	mvnsite	1m 22s		trunk passed
+1 💚	javadoc	1m 8s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 35s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	3m 10s		trunk passed
+1 💚	shadedclient	21m 22s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	21m 36s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 7s		the patch passed
+1 💚	compile	2m 52s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	2m 52s		the patch passed
+1 💚	compile	2m 43s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	2m 43s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 38s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 2 new + 113 unchanged - 0 fixed = 115 total (was 113)
+1 💚	mvnsite	1m 9s		the patch passed
+1 💚	javadoc	0m 55s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 27s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	3m 8s		the patch passed
+1 💚	shadedclient	23m 50s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 51s		hadoop-hdfs-client in the patch passed.
-1 ❌	unit	212m 5s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		325m 11s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/10/artifact/out/Dockerfile
GITHUB PR	#5829
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 91e3e1ef2a9e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c5afc65`
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/10/testReport/
Max. process+thread count	4589 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/10/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

haiyang1987 · 2024-04-20T04:46:45Z

please fix checkstyle, thanks~

Neilxzn · 2024-04-22T01:39:31Z

./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java:171:  public DfsClientConf(Configuration conf) {:3: Method length is 156 lines (max allowed is 150). [MethodLength]

Should we suppress this checkstyle warning? Or are there any better suggestions?

hadoop-yetus · 2024-04-22T07:06:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 4s		Maven dependency ordering for branch
+1 💚	mvninstall	19m 54s		trunk passed
+1 💚	compile	3m 0s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	2m 48s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	0m 47s		trunk passed
+1 💚	mvnsite	1m 18s		trunk passed
+1 💚	javadoc	1m 8s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 37s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	3m 2s		trunk passed
+1 💚	shadedclient	20m 53s		branch has no errors when building and testing our client artifacts.
-0 ⚠️	patch	21m 7s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 4s		the patch passed
+1 💚	compile	2m 50s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	2m 50s		the patch passed
+1 💚	compile	2m 51s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	2m 51s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 38s	/results-checkstyle-hadoop-hdfs-project.txt	hadoop-hdfs-project: The patch generated 1 new + 113 unchanged - 0 fixed = 114 total (was 113)
+1 💚	mvnsite	1m 8s		the patch passed
+1 💚	javadoc	0m 55s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	1m 27s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	3m 8s		the patch passed
+1 💚	shadedclient	21m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 49s		hadoop-hdfs-client in the patch passed.
+1 💚	unit	214m 41s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 26s		The patch does not generate ASF License warnings.
		329m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/11/artifact/out/Dockerfile
GITHUB PR	#5829
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux f0536dd0a688 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `d698a94`
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/11/testReport/
Max. process+thread count	4213 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5829/11/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

add dfs.client.read.striped.datanode.max.attempts to fix read ec issue

5366ce9

Hexiaoqiao requested a review from zhangshuyan0 November 3, 2023 03:20

ayushtkn reviewed Nov 3, 2023

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java Outdated Show resolved Hide resolved

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java Outdated Show resolved Hide resolved

zhangshuyan0 requested changes Nov 5, 2023

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java Outdated Show resolved Hide resolved

xiezhineng added 2 commits November 6, 2023 21:36

fix checkstyle

e471564

add unit test

ff4b113

github-actions bot added HDFS trunk labels Nov 6, 2023

ayushtkn reviewed Nov 10, 2023

View reviewed changes

...t/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStreamWithTimeout.java Show resolved Hide resolved

...t/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedInputStreamWithTimeout.java Outdated Show resolved Hide resolved

add testReadFileWithAttempt for unit test

db52c34

fix chekstyle.

bb46dbd

xiezhineng added 5 commits December 7, 2023 12:21

add dfs.client.read.striped.datanode.max.attempts to fix read ec issue

abac7c8

fix checkstyle

3646123

add unit test

0a0b5e3

add testReadFileWithAttempt for unit test

bfc97a8

fix chekstyle.

ff3de47

github-actions bot removed TOOLS AWS ABFS labels Mar 11, 2024

haiyang1987 reviewed Mar 20, 2024

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java Outdated Show resolved Hide resolved

haiyang1987 reviewed Mar 20, 2024

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java Outdated Show resolved Hide resolved

haiyang1987 reviewed Mar 20, 2024

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java Outdated Show resolved Hide resolved

zhangshuyan0 requested changes Apr 8, 2024

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java Outdated Show resolved Hide resolved

add offsetInBlock to avoid reading duplicate data from datanode.

5d597a1

haiyang1987 reviewed Apr 17, 2024

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java Outdated Show resolved Hide resolved

haiyang1987 reviewed Apr 17, 2024

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java Outdated Show resolved Hide resolved

haiyang1987 reviewed Apr 17, 2024

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java Outdated Show resolved Hide resolved

haiyang1987 reviewed Apr 17, 2024

View reviewed changes

hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java Outdated Show resolved Hide resolved

xiezhineng added 2 commits April 19, 2024 15:37

fix check style && log.

381f5ca

fix tests

c5afc65

fix checkstyle

d698a94

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-15413. add dfs.client.read.striped.datanode.max.attempts to fix read ecfile timeout #5829

HDFS-15413. add dfs.client.read.striped.datanode.max.attempts to fix read ecfile timeout #5829

Neilxzn commented Jul 11, 2023

hadoop-yetus commented Jul 11, 2023

Hexiaoqiao commented Nov 3, 2023

ayushtkn left a comment •

edited

Hexiaoqiao commented Nov 6, 2023

Neilxzn commented Nov 6, 2023

hadoop-yetus commented Nov 6, 2023

bbeaudreault commented Nov 8, 2023

ayushtkn left a comment

Hexiaoqiao commented Nov 17, 2023

bbeaudreault commented Dec 3, 2023

Neilxzn commented Dec 4, 2023

hadoop-yetus commented Dec 5, 2023

hadoop-yetus commented Dec 6, 2023

Neilxzn commented Dec 7, 2023

ayushtkn commented Dec 7, 2023

Neilxzn commented Dec 11, 2023

Neilxzn commented Mar 11, 2024

Neilxzn commented Mar 11, 2024

haiyang1987 commented Mar 11, 2024

hadoop-yetus commented Mar 11, 2024

hadoop-yetus commented Mar 11, 2024

zhangshuyan0 left a comment •

edited

zhangshuyan0 commented Apr 15, 2024

Neilxzn commented Apr 17, 2024

hadoop-yetus commented Apr 17, 2024

haiyang1987 commented Apr 18, 2024

hadoop-yetus commented Apr 19, 2024

Neilxzn commented Apr 19, 2024

hadoop-yetus commented Apr 19, 2024

haiyang1987 commented Apr 20, 2024

Neilxzn commented Apr 22, 2024

hadoop-yetus commented Apr 22, 2024

HDFS-15413. add dfs.client.read.striped.datanode.max.attempts to fix read ecfile timeout #5829

Are you sure you want to change the base?

HDFS-15413. add dfs.client.read.striped.datanode.max.attempts to fix read ecfile timeout #5829

Conversation

Neilxzn commented Jul 11, 2023

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Jul 11, 2023

Hexiaoqiao commented Nov 3, 2023

ayushtkn left a comment • edited

Choose a reason for hiding this comment

Hexiaoqiao commented Nov 6, 2023

Neilxzn commented Nov 6, 2023

hadoop-yetus commented Nov 6, 2023

bbeaudreault commented Nov 8, 2023

ayushtkn left a comment

Choose a reason for hiding this comment

Hexiaoqiao commented Nov 17, 2023

bbeaudreault commented Dec 3, 2023

Neilxzn commented Dec 4, 2023

hadoop-yetus commented Dec 5, 2023

hadoop-yetus commented Dec 6, 2023

Neilxzn commented Dec 7, 2023

ayushtkn commented Dec 7, 2023

Neilxzn commented Dec 11, 2023

Neilxzn commented Mar 11, 2024

Neilxzn commented Mar 11, 2024

haiyang1987 commented Mar 11, 2024

hadoop-yetus commented Mar 11, 2024

hadoop-yetus commented Mar 11, 2024

zhangshuyan0 left a comment • edited

Choose a reason for hiding this comment

zhangshuyan0 commented Apr 15, 2024

Neilxzn commented Apr 17, 2024

hadoop-yetus commented Apr 17, 2024

haiyang1987 commented Apr 18, 2024

hadoop-yetus commented Apr 19, 2024

Neilxzn commented Apr 19, 2024

hadoop-yetus commented Apr 19, 2024

haiyang1987 commented Apr 20, 2024

Neilxzn commented Apr 22, 2024

hadoop-yetus commented Apr 22, 2024

ayushtkn left a comment •

edited

zhangshuyan0 left a comment •

edited