HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. #3544

jianghuazhu · 2021-10-12T09:35:32Z

Description of PR

When using NNThroughputBenchmark to test blockReport, some exceptions will be prompted.
E.g:

The purpose of this issue is to solve this problem.

How was this patch tested?

When using the same command, the same exception will not occur.

hadoop-yetus · 2021-10-12T16:43:28Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 56s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 12s		trunk passed
+1 💚	compile	1m 22s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 13s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	1m 24s		trunk passed
+1 💚	javadoc	0m 58s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 23s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 19s		trunk passed
+1 💚	shadedclient	23m 55s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 11s		the patch passed
+1 💚	compile	1m 16s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 16s		the patch passed
+1 💚	compile	1m 9s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 9s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 55s		the patch passed
+1 💚	mvnsite	1m 13s		the patch passed
+1 💚	javadoc	0m 47s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 21s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 20s		the patch passed
+1 💚	shadedclient	24m 1s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	323m 11s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 40s		The patch does not generate ASF License warnings.
		426m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/1/artifact/out/Dockerfile
GITHUB PR	#3544
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux dbace46031da 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / e66171dd599c8ccf6dfb79d06edfed1af0e9da1d
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/1/testReport/
Max. process+thread count	2154 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

jianghuazhu · 2021-10-13T01:52:20Z

@ayushtkn @virajjasani, are you willing to spend some time reviewing this pr.
Thank you very much.

jojochuang

+1

jianghuazhu · 2021-10-18T03:51:44Z

Thank you @jojochuang for your comments and reviews.

jianghuazhu · 2021-10-18T04:01:37Z

Hi @aajisaka @ferhui @Hexiaoqiao , are you willing to spend some time to help review this pr.
Thank you very much.

jojochuang

question: will it guarantee each block is added to one TinyDataNode? It seems so when i ran the test but not sure how NN generates xferport.

jianghuazhu · 2021-10-18T14:23:09Z

Thanks @jojochuang for the comment.
Thank you for your reminder. Sorry, I overlooked this point.
I also found this phenomenon when I tested the online cluster.
Before the test, there are 3 DN nodes in the online cluster, for example:
'
hadoop1.xxxx.org:50010
hadoop2..xxxxorg:50010
hadoop3..xxxxorg:50010
'
The Xferport of these 3 DN nodes are the same.
When I try to connect to the online namenode, 3 new DataNode services will be created locally, for example:
’
xxxx.xxxx.xxxx.xxxx:1
xxxx.xxxx.xxxx.xxxx:2
xxxx.xxxx.xxxx.xxxx:3
‘
When creating a file, you need to get the location of the datanode from the remote namenode through addBlock(). At this time, you may get hadoop1.xxxx.org:50010, hadoop2..xxxxorg:50010, hadoop3..xxxxorg:50010.
For the safety of the entire link, I think these DNs should be filtered out when executing addBlock().

hadoop-yetus · 2021-10-19T15:45:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 2s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 7s		trunk passed
+1 💚	compile	1m 22s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 16s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	1m 25s		trunk passed
+1 💚	javadoc	0m 56s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 25s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 17s		trunk passed
+1 💚	shadedclient	24m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 16s		the patch passed
+1 💚	compile	1m 17s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 17s		the patch passed
+1 💚	compile	1m 9s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 9s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 53s		the patch passed
+1 💚	mvnsite	1m 14s		the patch passed
+1 💚	javadoc	0m 48s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 19s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 19s		the patch passed
+1 💚	shadedclient	23m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	321m 43s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		440m 46s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/2/artifact/out/Dockerfile
GITHUB PR	#3544
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 6540fb6efbaa 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 288b4c58a17ccc48200b2ce80910d04324d121f2
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/2/testReport/
Max. process+thread count	1905 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

jianghuazhu · 2021-10-20T03:37:57Z

Some changes have been made here.
When executing addBlock(), some nodes will be excluded. These nodes include:

Non-locally activated DN node.
If xferport is greater than the length of datanodes.

Below are some verified examples.
step 1:
There are some existing DN nodes, [hostname=hadoop1.jdhadoop.xxx.org, ip=xxxx.xxxx.xxxx.xxxx, xferport=50010]
dn[hostname=hadoop2.jdhadoop.xxx.org, ip=xxxx.xxxx.xxxx.xxxx, xferport=50010]
dn[hostname=hadoop3.jdhadoop.xxx.org, ip=xxxx.xxxx.xxxx.xxxx, xferport=50010]
Step 2:
Excuting an order.
./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs hdfs://xxxx -op blockReport -datanodes 3

Step 3:
We will print some additional information.
All surviving dn nodes: [hostname=hadoop1.jdhadoop.xxxx.org, ip=xxxx.xxxx.xxxx.xxxx xferport=1]
dn[hostname=hadoop1.jdhadoop.xxxx.org, ip=xxxx, xferport=2]
dn[hostname=hadoop1.jdhadoop.xxxx.org, ip=xxxx.xxxx.xxxx.xxxx.xxxx, xferport=3]
dn[hostname=hadoop1.jdhadoop.xxxx.org, ip=xxx.xxx.xxx.xxx, xferport=50010]
dn[hostname=hadoop2.jdhadoop.xxxx.org, ip=xxxx.xxx.xxx.xxx, xferport=50010]
dn[hostname=hadoop3.jdhadoop.xxxx.org, ip=xxx.xxx.xxx.xxx.xxx, xferport=50010]
The dn node that needs to be filtered: [hostname=hadoop1.jdhadoop.xxx.org, ip=xxx.xxx.xxx.xxx, xferport=50010]
dn[hostname=hadoop2.jdhadoop.xxxx.org, ip=xxx.xxx.xxx.xxx, xferport=50010]
dn[hostname=hadoop3.jdhadoop.xxx.org, ip=xxx.xxx.xxx.xxx, xferport=50010]
Returned dn node: [hostname=hadoop1.jdhadoop.xxx.org, ip=xxxx.xxx.xxx.xxx, xferport=1],dn[hostname=hadoop1.jdhadoop.xxxx.org, ip=xxx.xxx.xxx .xxx, xferport=3]
dn[hostname=hadoop1.jdhadoop.xxxx.org, ip=xxxx.xxx.xxx.xxx, xferport=2]

@jojochuang, would you like to take some time to help review again.
Thank you very much.

aajisaka

The fix mostly looks good to me.

...oop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java

jojochuang

nice. thanks for catching the bug.

hadoop-yetus · 2021-10-28T16:02:50Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 55s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 37s		trunk passed
+1 💚	compile	1m 22s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 15s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 59s		trunk passed
+1 💚	mvnsite	1m 21s		trunk passed
+1 💚	javadoc	0m 58s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 25s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 17s		trunk passed
+1 💚	shadedclient	25m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 15s		the patch passed
+1 💚	compile	1m 16s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 16s		the patch passed
+1 💚	compile	1m 8s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 8s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 54s		the patch passed
+1 💚	mvnsite	1m 15s		the patch passed
+1 💚	javadoc	0m 47s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 16s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 20s		the patch passed
+1 💚	shadedclient	24m 53s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	320m 19s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		425m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/3/artifact/out/Dockerfile
GITHUB PR	#3544
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 5bc5577d7cca 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 75c4e6dd4eb801d5918361c95320fd66d632213b
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/3/testReport/
Max. process+thread count	1877 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

jianghuazhu · 2021-10-29T02:01:31Z

Thanks for your comments and reviews, @aajisaka @jojochuang .

jianghuazhu · 2021-10-29T02:05:35Z

@virajjasani @ferhui, are you willing to spend some time to help review this pr.
Thank you very much.

hadoop-yetus · 2021-10-29T13:28:48Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	33m 17s		trunk passed
+1 💚	compile	1m 22s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 13s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	1m 23s		trunk passed
+1 💚	javadoc	0m 56s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 24s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 20s		trunk passed
+1 💚	shadedclient	24m 52s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	1m 18s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 18s		the patch passed
+1 💚	compile	1m 8s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 8s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 52s		the patch passed
+1 💚	mvnsite	1m 14s		the patch passed
+1 💚	javadoc	0m 47s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 19s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 22s		the patch passed
+1 💚	shadedclient	25m 18s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	319m 0s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		422m 57s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/4/artifact/out/Dockerfile
GITHUB PR	#3544
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 9d661a554853 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 6979d3bce5df2e8504afae2def9212ae83fe529f
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/4/testReport/
Max. process+thread count	1909 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-10-30T20:41:54Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 58s		trunk passed
+1 💚	compile	1m 22s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 14s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 0s		trunk passed
+1 💚	mvnsite	1m 27s		trunk passed
+1 💚	javadoc	0m 57s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 26s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 17s		trunk passed
+1 💚	shadedclient	24m 41s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	1m 19s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 19s		the patch passed
+1 💚	compile	1m 7s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 7s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 53s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 129 unchanged - 0 fixed = 130 total (was 129)
+1 💚	mvnsite	1m 14s		the patch passed
+1 💚	javadoc	0m 47s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 19s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 20s		the patch passed
+1 💚	shadedclient	25m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	318m 17s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		423m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/5/artifact/out/Dockerfile
GITHUB PR	#3544
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux f8720398b3d2 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / d88938632f65843c57e392d66c35461a3c06517b
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/5/testReport/
Max. process+thread count	1871 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

jianghuazhu · 2021-10-31T01:40:40Z

I have made some changes, can you review this pr again, @aajisaka .
Thank you very much.

...oop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java

hadoop-yetus · 2021-11-01T13:22:08Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 3s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	37m 15s		trunk passed
+1 💚	compile	1m 33s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 25s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 4s		trunk passed
+1 💚	mvnsite	1m 31s		trunk passed
+1 💚	javadoc	1m 5s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 33s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 35s		trunk passed
+1 💚	shadedclient	25m 35s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 19s		the patch passed
+1 💚	compile	1m 28s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 28s		the patch passed
+1 💚	compile	1m 16s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 16s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 56s		the patch passed
+1 💚	mvnsite	1m 24s		the patch passed
+1 💚	javadoc	0m 55s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 25s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 39s		the patch passed
+1 💚	shadedclient	25m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	376m 41s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 45s		The patch does not generate ASF License warnings.
		487m 51s

Reason	Tests
Failed junit tests	hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/6/artifact/out/Dockerfile
GITHUB PR	#3544
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 55004c7946a0 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `bfedca8`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/6/testReport/
Max. process+thread count	2103 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

aajisaka · 2021-11-01T15:56:29Z

Merged. Thank you @jianghuazhu @ferhui @jojochuang

#3544) Reviewed-by: Fei Hui <feihui.ustc@gmail.com> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org> (cherry picked from commit 618fea2) Conflicts: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java

apache#3544) Reviewed-by: Fei Hui <feihui.ustc@gmail.com> Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

jojochuang approved these changes Oct 18, 2021

View reviewed changes

jojochuang reviewed Oct 18, 2021

View reviewed changes

jianghuazhu force-pushed the HDFS-16269 branch from e66171d to 288b4c5 Compare October 19, 2021 08:23

aajisaka reviewed Oct 28, 2021

View reviewed changes

...oop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java Outdated Show resolved Hide resolved

jianghuazhu force-pushed the HDFS-16269 branch from 288b4c5 to 75c4e6d Compare October 28, 2021 08:55

jojochuang approved these changes Oct 28, 2021

View reviewed changes

ferhui approved these changes Oct 29, 2021

View reviewed changes

jianghuazhu force-pushed the HDFS-16269 branch from 75c4e6d to 6979d3b Compare October 29, 2021 06:24

jianghuazhu force-pushed the HDFS-16269 branch from 6979d3b to d889386 Compare October 30, 2021 13:36

aajisaka reviewed Oct 31, 2021

View reviewed changes

...oop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNThroughputBenchmark.java Outdated Show resolved Hide resolved

HDFS-16269.[Fix] Improve NNThroughputBenchmark#blockReport operation.

bfedca8

jianghuazhu force-pushed the HDFS-16269 branch from d889386 to bfedca8 Compare November 1, 2021 05:12

aajisaka approved these changes Nov 1, 2021

View reviewed changes

aajisaka merged commit 618fea2 into apache:trunk Nov 1, 2021

aajisaka changed the title ~~HDFS-16269.[Fix] Improve NNThroughputBenchmark#blockReport operation.~~ HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. Nov 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. #3544

HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. #3544

jianghuazhu commented Oct 12, 2021

hadoop-yetus commented Oct 12, 2021

jianghuazhu commented Oct 13, 2021

jojochuang left a comment

jianghuazhu commented Oct 18, 2021

jianghuazhu commented Oct 18, 2021

jojochuang left a comment

jianghuazhu commented Oct 18, 2021

hadoop-yetus commented Oct 19, 2021

jianghuazhu commented Oct 20, 2021

aajisaka left a comment

jojochuang left a comment

hadoop-yetus commented Oct 28, 2021

jianghuazhu commented Oct 29, 2021

jianghuazhu commented Oct 29, 2021

hadoop-yetus commented Oct 29, 2021

hadoop-yetus commented Oct 30, 2021

jianghuazhu commented Oct 31, 2021 •

edited

hadoop-yetus commented Nov 1, 2021

aajisaka commented Nov 1, 2021

Navigation Menu

HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. #3544

HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. #3544

Conversation

jianghuazhu commented Oct 12, 2021

Description of PR

How was this patch tested?

hadoop-yetus commented Oct 12, 2021

jianghuazhu commented Oct 13, 2021

jojochuang left a comment

Choose a reason for hiding this comment

jianghuazhu commented Oct 18, 2021

jianghuazhu commented Oct 18, 2021

jojochuang left a comment

Choose a reason for hiding this comment

jianghuazhu commented Oct 18, 2021

hadoop-yetus commented Oct 19, 2021

jianghuazhu commented Oct 20, 2021

aajisaka left a comment

Choose a reason for hiding this comment

jojochuang left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Oct 28, 2021

jianghuazhu commented Oct 29, 2021

jianghuazhu commented Oct 29, 2021

hadoop-yetus commented Oct 29, 2021

hadoop-yetus commented Oct 30, 2021

jianghuazhu commented Oct 31, 2021 • edited

hadoop-yetus commented Nov 1, 2021

aajisaka commented Nov 1, 2021

jianghuazhu commented Oct 31, 2021 •

edited