HDFS-17761. [ARR] RouterNetworkTopologyServlet adapts to async router rpc. #7533

hfutatzhanghb · 2025-03-25T07:10:48Z

Description of PR

RouterNetworkTopologyServlet adapts to async router rpc.
Currently, we will get NPE when request RouterNetworkTopologyServlet using async router rpc. Logs as below:

2025-03-25 13:53:26,300 [qtp1085041526-1172] WARN  server.HttpChannel (HttpChannel.java:handleException(776)) - /topology
java.lang.NullPointerException
	at java.util.Objects.requireNonNull(Objects.java:203)
	at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
	at java.util.Arrays.asList(Arrays.java:3800)
	at org.apache.hadoop.hdfs.server.federation.router.RouterNetworkTopologyServlet.doGet(RouterNetworkTopologyServlet.java:55)

How was this patch tested?

Add an unit test.

hfutatzhanghb · 2025-03-25T07:53:06Z

@KeeProMise Could you please help review this pr when have free time?
@slfan1989 Could you please help check the unit test style of this PR?
Thank you all.

hadoop-yetus · 2025-03-25T10:08:13Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	36m 11s		trunk passed
+1 💚	compile	1m 5s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 42s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
-0 ⚠️	checkstyle	0m 25s	/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	The patch fails to run checkstyle in hadoop-hdfs-rbf
+1 💚	mvnsite	0m 46s		trunk passed
+1 💚	javadoc	0m 44s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 34s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 33s		trunk passed
+1 💚	shadedclient	34m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 36s		the patch passed
+1 💚	compile	0m 58s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 58s		the patch passed
+1 💚	compile	0m 34s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	0m 34s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 14s	/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	The patch fails to run checkstyle in hadoop-hdfs-rbf
+1 💚	mvnsite	0m 36s		the patch passed
+1 💚	javadoc	0m 30s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 26s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 32s		the patch passed
+1 💚	shadedclient	36m 3s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	52m 18s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 40s		The patch does not generate ASF License warnings.
		173m 49s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7533/2/artifact/out/Dockerfile
GITHUB PR	#7533
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 170033f94e50 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `af2df38`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7533/2/testReport/
Max. process+thread count	3129 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7533/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hfutatzhanghb · 2025-03-25T10:10:35Z

The checkstyle problem was caused by #7535 , which has been reverted.

hadoop-yetus · 2025-03-25T10:33:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	20m 46s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 20s		trunk passed
+1 💚	compile	1m 15s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 40s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	0m 32s		trunk passed
+1 💚	mvnsite	0m 47s		trunk passed
+1 💚	javadoc	0m 44s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 32s		trunk passed
+1 💚	shadedclient	39m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 35s		the patch passed
+1 💚	compile	1m 4s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 4s		the patch passed
+1 💚	compile	0m 34s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	0m 34s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 20s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚	mvnsite	0m 38s		the patch passed
+1 💚	javadoc	0m 31s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 29s		the patch passed
+1 💚	shadedclient	39m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	47m 33s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		201m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7533/1/artifact/out/Dockerfile
GITHUB PR	#7533
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux c3022903cbef 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `af2df38`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7533/1/testReport/
Max. process+thread count	2744 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7533/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

KeeProMise

Hi @hfutatzhanghb Thanks for your contribution. Leave some comments.

.../main/java/org/apache/hadoop/hdfs/server/federation/router/RouterNetworkTopologyServlet.java

...t/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterNetworkTopologyServlet.java

hadoop-yetus · 2025-03-26T06:57:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	20m 29s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 21s		trunk passed
+1 💚	compile	1m 13s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 41s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	0m 33s		trunk passed
+1 💚	mvnsite	0m 47s		trunk passed
+1 💚	javadoc	0m 43s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 33s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 32s		trunk passed
+1 💚	shadedclient	39m 30s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 35s		the patch passed
+1 💚	compile	1m 4s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 4s		the patch passed
+1 💚	compile	0m 34s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	0m 34s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 21s		the patch passed
+1 💚	mvnsite	0m 37s		the patch passed
+1 💚	javadoc	0m 31s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 30s		the patch passed
+1 💚	shadedclient	38m 55s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	47m 24s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		202m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7533/5/artifact/out/Dockerfile
GITHUB PR	#7533
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 4141f94bdb32 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `820b870`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7533/5/testReport/
Max. process+thread count	2773 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7533/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

KeeProMise

LGTM! hi @slfan1989 If you have time, please help review the unit test.

slfan1989 · 2025-03-26T15:34:38Z

LGTM! hi @slfan1989 If you have time, please help review the unit test.

@KeeProMise Thank you for inviting me to review this PR! I will check the related unit tests as soon as possible. We are in the process of upgrading the HDFS Router to JUnit5, and one PR has already been merged, while another PR (#7531) has been approved. Once #7531 is merged, I plan to monitor it for 1-2 days to ensure everything is working correctly, and then we will merge this PR. Therefore, the merge of this PR may be delayed until this weekend or early next week.

cc: @hfutatzhanghb

hfutatzhanghb · 2025-03-27T01:36:34Z

No problems! Thanks @slfan1989 @KeeProMise .
I find there are some conflicts after #7531 merged into trunk, will fix soonly.

… rpc.

hadoop-yetus · 2025-03-27T04:45:32Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 11s		trunk passed
+1 💚	compile	1m 12s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 41s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	0m 33s		trunk passed
+1 💚	mvnsite	0m 46s		trunk passed
+1 💚	javadoc	0m 43s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 33s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 34s		trunk passed
+1 💚	shadedclient	39m 12s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 35s		the patch passed
+1 💚	compile	1m 4s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 4s		the patch passed
+1 💚	compile	0m 33s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 20s		the patch passed
+1 💚	mvnsite	0m 39s		the patch passed
+1 💚	javadoc	0m 31s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 30s		the patch passed
+1 💚	shadedclient	40m 6s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	47m 29s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		182m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7533/6/artifact/out/Dockerfile
GITHUB PR	#7533
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 3501b66207f6 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `85f28de`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7533/6/testReport/
Max. process+thread count	2782 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7533/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

github-actions bot added HDFS trunk RBF labels Mar 25, 2025

hfutatzhanghb force-pushed the HDFS-17761 branch from e538f20 to af2df38 Compare March 25, 2025 07:13

KeeProMise reviewed Mar 25, 2025

View reviewed changes

.../main/java/org/apache/hadoop/hdfs/server/federation/router/RouterNetworkTopologyServlet.java Outdated Show resolved Hide resolved

...t/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterNetworkTopologyServlet.java Outdated Show resolved Hide resolved

KeeProMise approved these changes Mar 26, 2025

View reviewed changes

hfutatzhanghb added 4 commits March 27, 2025 09:41

HDFS-17761. [ARR] RouterNetworkTopologyServlet adapts to async router…

6fbb190

… rpc.

optimize throw IOE

4e77225

fix checkstyle.

131d3b0

trigger yetus.

85f28de

hfutatzhanghb force-pushed the HDFS-17761 branch from 820b870 to 85f28de Compare March 27, 2025 01:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-17761. [ARR] RouterNetworkTopologyServlet adapts to async router rpc. #7533

HDFS-17761. [ARR] RouterNetworkTopologyServlet adapts to async router rpc. #7533

hfutatzhanghb commented Mar 25, 2025

hfutatzhanghb commented Mar 25, 2025

hadoop-yetus commented Mar 25, 2025

hfutatzhanghb commented Mar 25, 2025

hadoop-yetus commented Mar 25, 2025

KeeProMise left a comment

hadoop-yetus commented Mar 26, 2025

KeeProMise left a comment •

edited

Loading

slfan1989 commented Mar 26, 2025 •

edited

Loading

hfutatzhanghb commented Mar 27, 2025

hadoop-yetus commented Mar 27, 2025

HDFS-17761. [ARR] RouterNetworkTopologyServlet adapts to async router rpc. #7533

Are you sure you want to change the base?

HDFS-17761. [ARR] RouterNetworkTopologyServlet adapts to async router rpc. #7533

Conversation

hfutatzhanghb commented Mar 25, 2025

Description of PR

How was this patch tested?

hfutatzhanghb commented Mar 25, 2025

hadoop-yetus commented Mar 25, 2025

hfutatzhanghb commented Mar 25, 2025

hadoop-yetus commented Mar 25, 2025

KeeProMise left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Mar 26, 2025

KeeProMise left a comment • edited Loading

Choose a reason for hiding this comment

slfan1989 commented Mar 26, 2025 • edited Loading

hfutatzhanghb commented Mar 27, 2025

hadoop-yetus commented Mar 27, 2025

KeeProMise left a comment •

edited

Loading

slfan1989 commented Mar 26, 2025 •

edited

Loading