HDFS-16310. RBF: Add client port to CallerContext for Router #3635

tomscut · 2021-11-09T11:20:07Z

We mentioned in #3538 that adding the client port to the CallerContext of the Router.

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallerContext.java

hadoop-yetus · 2021-11-09T21:31:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 3s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	11m 19s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 18s		trunk passed
+1 💚	compile	23m 31s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	19m 47s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	3m 51s		trunk passed
+1 💚	mvnsite	3m 49s		trunk passed
+1 💚	javadoc	2m 50s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	4m 8s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	7m 10s		trunk passed
+1 💚	shadedclient	23m 18s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 40s		the patch passed
+1 💚	compile	22m 43s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	22m 43s		the patch passed
+1 💚	compile	19m 55s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	19m 55s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 43s		the patch passed
+1 💚	mvnsite	3m 49s		the patch passed
+1 💚	javadoc	2m 54s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	4m 35s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	9m 4s		the patch passed
+1 💚	shadedclient	25m 0s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 23s		hadoop-common in the patch passed.
-1 ❌	unit	331m 12s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
-1 ❌	unit	37m 38s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		609m 52s

Reason	Tests
Failed junit tests	hadoop.hdfs.tools.TestDFSAdminWithHA
	hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination
	hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure
	hadoop.hdfs.server.federation.router.TestRouterRpc

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3635/1/artifact/out/Dockerfile
GITHUB PR	#3635
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux a54cef6a1f5f 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / c95077422057f600eb89d98f43d2b73fea1fb0db
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3635/1/testReport/
Max. process+thread count	2913 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3635/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-11-11T12:38:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 56s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 15s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 38s		trunk passed
+1 💚	compile	26m 42s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	19m 56s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	3m 49s		trunk passed
+1 💚	mvnsite	3m 48s		trunk passed
+1 💚	javadoc	2m 52s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	4m 12s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	7m 19s		trunk passed
+1 💚	shadedclient	23m 12s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 40s		the patch passed
+1 💚	compile	22m 36s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	22m 36s		the patch passed
+1 💚	compile	20m 1s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	20m 1s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 45s		the patch passed
+1 💚	mvnsite	3m 45s		the patch passed
+1 💚	javadoc	2m 49s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	4m 6s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	7m 52s		the patch passed
+1 💚	shadedclient	23m 49s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 24s		hadoop-common in the patch passed.
+1 💚	unit	328m 57s		hadoop-hdfs in the patch passed.
-1 ❌	unit	37m 17s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	1m 9s		The patch does not generate ASF License warnings.
		609m 55s

Reason	Tests
Failed junit tests	hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3635/2/artifact/out/Dockerfile
GITHUB PR	#3635
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 6e591de47bbb 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / f1f402f949d45da4a2afab76885ba374b69d0fd0
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3635/2/testReport/
Max. process+thread count	2375 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3635/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-11-16T15:39:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 1s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 38s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 21s		trunk passed
+1 💚	compile	5m 54s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	5m 22s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 13s		trunk passed
+1 💚	mvnsite	2m 4s		trunk passed
+1 💚	javadoc	1m 40s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 20s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	4m 36s		trunk passed
+1 💚	shadedclient	23m 6s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 23s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 49s		the patch passed
+1 💚	compile	5m 52s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	5m 52s		the patch passed
+1 💚	compile	5m 31s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	5m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 6s		the patch passed
+1 💚	mvnsite	1m 57s		the patch passed
+1 💚	javadoc	1m 28s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	2m 12s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	4m 47s		the patch passed
+1 💚	shadedclient	23m 15s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	356m 42s		hadoop-hdfs in the patch passed.
-1 ❌	unit	33m 3s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 57s		The patch does not generate ASF License warnings.
		525m 11s

Reason	Tests
Failed junit tests	hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3635/3/artifact/out/Dockerfile
GITHUB PR	#3635
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 2c5b168f9454 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0bad062`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3635/3/testReport/
Max. process+thread count	2227 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3635/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

goiri · 2021-11-16T19:47:37Z

There's a failure with TestRouterDistCpProcedure but I don't see how it can be related.
Can you confirm?

tomscut · 2021-11-17T01:21:01Z

There's a failure with TestRouterDistCpProcedure but I don't see how it can be related. Can you confirm?

Thank @goiri for your review. This failed unit test is unrelated to the change. It failed because timeout and work fine locally.

tomscut · 2021-11-17T01:22:40Z

Hi @tasanuma @jojochuang @ferhui @ayushtkn , could you please take a look at this. Thank you very much.

tasanuma · 2021-11-17T09:18:15Z

@tomscut With the latest commit, it seems Router adds a client port even if dfs.namenode.audit.log.with.remote.port=false. We may want to add a client port only when dfs.namenode.audit.log.with.remote.port=true. What do you think of it?

tasanuma · 2021-11-17T09:23:27Z

...-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java

+    CallerContext.setCurrent(
+        new CallerContext.Builder(origContext, contextFieldSeparator)
+            .appendIfAbsent(CLIENT_IP_STR, Server.getRemoteAddress())
+            .appendIfAbsent(CLIENT_PORT_STR,


Just a note, we allow the line length up to 100 after HADOOP-17813.

Just a note, we allow the line length up to 100 after HADOOP-17813.

Thank you for reminding me. This is good.

tomscut · 2021-11-17T12:13:15Z

@tomscut With the latest commit, it seems Router adds a client port even if dfs.namenode.audit.log.with.remote.port=false. We may want to add a client port only when dfs.namenode.audit.log.with.remote.port=true. What do you think of it?

Thanks @tasanuma for your careful consideration and comments. This problem does exist at present. But dfs.namenode.audit.log.with.remote.port should only work within the Namenode. We can treat the Router as the client of Namenode. From the perspective of the Namenode, the behavior of the client is not controllable because the real client may also set the clientPort. What do you think of this?

tasanuma · 2021-11-18T02:55:05Z

@tomscut Thank you for your explanation. It makes sense to me. Now I'm +1.

tasanuma · 2021-11-18T04:48:44Z

Merged it. Thanks for all your contributions to this work, @tomscut. Thanks for your review, @goiri.

tomscut · 2021-11-18T05:26:18Z

Thanks @tasanuma for your comment and the merge.

Cherry-picked from 5b05068 by Owen O'Malley

…3635)

goiri reviewed Nov 9, 2021

View reviewed changes

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallerContext.java Outdated Show resolved Hide resolved

tomscut mentioned this pull request Nov 11, 2021

HADOOP-18003. Add a method appendIfAbsent for CallerContext #3644

Merged

HDFS-16310. RBF: Add client port to CallerContext for Router

0bad062

tomscut force-pushed the HDFS-16310 branch from f1f402f to 0bad062 Compare November 16, 2021 06:52

tomscut requested a review from goiri November 16, 2021 15:47

goiri approved these changes Nov 16, 2021

View reviewed changes

tasanuma self-requested a review November 17, 2021 08:24

tasanuma reviewed Nov 17, 2021

View reviewed changes

tasanuma approved these changes Nov 18, 2021

View reviewed changes

tasanuma merged commit 5b05068 into apache:trunk Nov 18, 2021

omalley pushed a commit that referenced this pull request Mar 14, 2022

HDFS-16310. RBF: Add client port to CallerContext for Router (#3635)

496657c

Cherry-picked from 5b05068 by Owen O'Malley

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-16310. RBF: Add client port to CallerContext for Router (apache#…

f9c3791

…3635)

LiuGuH pushed a commit to LiuGuH/hadoop that referenced this pull request Mar 26, 2024

HDFS-16310. RBF: Add client port to CallerContext for Router (apache#…

0d8af49

…3635)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16310. RBF: Add client port to CallerContext for Router #3635

HDFS-16310. RBF: Add client port to CallerContext for Router #3635

tomscut commented Nov 9, 2021

hadoop-yetus commented Nov 9, 2021

hadoop-yetus commented Nov 11, 2021

hadoop-yetus commented Nov 16, 2021

goiri commented Nov 16, 2021

tomscut commented Nov 17, 2021

tomscut commented Nov 17, 2021

tasanuma commented Nov 17, 2021

tasanuma Nov 17, 2021

tomscut Nov 17, 2021

tomscut commented Nov 17, 2021

tasanuma commented Nov 18, 2021

tasanuma commented Nov 18, 2021

tomscut commented Nov 18, 2021

HDFS-16310. RBF: Add client port to CallerContext for Router #3635

HDFS-16310. RBF: Add client port to CallerContext for Router #3635

Conversation

tomscut commented Nov 9, 2021

hadoop-yetus commented Nov 9, 2021

hadoop-yetus commented Nov 11, 2021

hadoop-yetus commented Nov 16, 2021

goiri commented Nov 16, 2021

tomscut commented Nov 17, 2021

tomscut commented Nov 17, 2021

tasanuma commented Nov 17, 2021

tasanuma Nov 17, 2021

Choose a reason for hiding this comment

tomscut Nov 17, 2021

Choose a reason for hiding this comment

tomscut commented Nov 17, 2021

tasanuma commented Nov 18, 2021

tasanuma commented Nov 18, 2021

tomscut commented Nov 18, 2021