HBASE-27195 Clean up netty worker/thread pool configuration by apurtell · Pull Request #4619 · apache/hbase

apurtell · 2022-07-13T21:34:16Z

The configuration settings "hbase.netty.worker.count" and "hbase.netty.eventloop.rpcserver.thread.count" appear to duplicate each other.

Also, formalizes another setting found in NettyEventLoopGroupConfig, "hbase.netty.nativetransport".

Also, native epoll is not limited to amd64. aarch64 supports it too.

Apache-HBase · 2022-07-13T22:26:08Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 12s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 24s	master passed
+1 💚	compile	2m 48s	master passed
+1 💚	checkstyle	0m 45s	master passed
+1 💚	spotless	0m 43s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 46s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 12s	the patch passed
+1 💚	compile	2m 44s	the patch passed
+1 💚	javac	2m 44s	the patch passed
+1 💚	checkstyle	0m 44s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	hadoopcheck	11m 42s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotless	0m 42s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 58s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 19s	The patch does not generate ASF License warnings.
		35m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4619
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile xml
uname	Linux 23ea1b7f4d21 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `01b45e2`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2022-07-14T00:49:49Z

hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java

-      channelClass = NioServerSocketChannel.class;
+    // Get the event loop group configuration from the server class if available.
+    NettyEventLoopGroupConfig config = null;
+    if (server != null) {


Still better to check whether server is an instance of HBaseServerBase? Or at least add a comment to say for now, if we pass a non-null server then it must be HBaseServerBase(or HRegionServer for branch-2.x). Otherwise other developers may be confused on this direct cast.

I will add the comment, sounds good.

Apache-HBase · 2022-07-14T01:33:29Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 22s	master passed
+1 💚	compile	0m 48s	master passed
+1 💚	shadedjars	4m 1s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 7s	the patch passed
+1 💚	compile	0m 49s	the patch passed
+1 💚	javac	0m 49s	the patch passed
+1 💚	shadedjars	4m 3s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 34s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 24s	hbase-common in the patch passed.
-1 ❌	unit	203m 53s	hbase-server in the patch failed.
		223m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4619
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux bb5ebe05968d 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `01b45e2`
Default Java	AdoptOpenJDK-1.8.0_282-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/1/testReport/
Max. process+thread count	2555 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-07-14T01:37:51Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 6s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 48s	master passed
+1 💚	compile	1m 4s	master passed
+1 💚	shadedjars	3m 44s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 43s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 14s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 44s	the patch passed
+1 💚	compile	1m 4s	the patch passed
+1 💚	javac	1m 4s	the patch passed
+1 💚	shadedjars	3m 44s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 42s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 45s	hbase-common in the patch passed.
-1 ❌	unit	205m 56s	hbase-server in the patch failed.
		227m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4619
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 8fced36602b3 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `01b45e2`
Default Java	AdoptOpenJDK-11.0.10+9
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/1/testReport/
Max. process+thread count	2462 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

apurtell · 2022-07-14T17:59:20Z

The test failure is related, let me check it.

[org.apache.hadoop.hbase.security.token.TestTokenAuthentication.testTokenCreation[1: rpcServerImpl=org.apache.hadoop.hbase.ipc.NettyRpcServer]](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/1/testReport/org.apache.hadoop.hbase.security.token/TestTokenAuthentication

The configuration settings "hbase.netty.worker.count" and "hbase.netty.eventloop.rpcserver.thread.count" appear to duplicate each other. Also, formalizes another setting found in NettyEventLoopGroupConfig, "hbase.netty.nativetransport". Also, native epoll is not limited to amd64. aarch64 supports it too.

apurtell · 2022-07-15T23:50:11Z

Rebased.
The test failure pointed out that a null check is not enough, we need an instanceof check.

Apache-HBase · 2022-07-16T00:30:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 40s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 21s	master passed
+1 💚	compile	2m 56s	master passed
+1 💚	checkstyle	0m 39s	master passed
+1 💚	spotless	0m 41s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 38s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 26s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 8s	the patch passed
+1 💚	compile	2m 47s	the patch passed
+1 💚	javac	2m 47s	the patch passed
+1 💚	checkstyle	0m 37s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	hadoopcheck	11m 43s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotless	0m 39s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 49s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		35m 1s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4619
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile xml
uname	Linux adb5a64a61f4 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `02f2636`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-07-16T03:39:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 40s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 20s	master passed
+1 💚	compile	0m 49s	master passed
+1 💚	shadedjars	4m 2s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 41s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 20s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 9s	the patch passed
+1 💚	compile	0m 48s	the patch passed
+1 💚	javac	0m 48s	the patch passed
+1 💚	shadedjars	3m 59s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 33s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 23s	hbase-common in the patch passed.
+1 💚	unit	204m 31s	hbase-server in the patch passed.
		223m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4619
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 546ded10df2c 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `02f2636`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/2/testReport/
Max. process+thread count	2670 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-07-16T03:44:28Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 10s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 46s	master passed
+1 💚	compile	1m 6s	master passed
+1 💚	shadedjars	3m 46s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 43s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 16s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 45s	the patch passed
+1 💚	compile	1m 4s	the patch passed
+1 💚	javac	1m 4s	the patch passed
+1 💚	shadedjars	3m 46s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 43s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 2s	hbase-common in the patch passed.
+1 💚	unit	207m 3s	hbase-server in the patch passed.
		229m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4619
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux f88fb0a06098 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `02f2636`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/2/testReport/
Max. process+thread count	2472 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4619/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

The configuration settings "hbase.netty.worker.count" and "hbase.netty.eventloop.rpcserver.thread.count" appear to duplicate each other. Also, formalizes another setting found in NettyEventLoopGroupConfig, "hbase.netty.nativetransport". Also, native epoll is not limited to amd64. aarch64 supports it too. Signed-off-by: Duo Zhang <zhangduo@apache.org> Conflicts: hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java

apurtell requested review from Apache9 and virajjasani July 13, 2022 21:34

apurtell mentioned this pull request Jul 13, 2022

HBASE-27195 Clean up netty worker/thread pool configuration #4618

Closed

Apache9 approved these changes Jul 14, 2022

View reviewed changes

apurtell added 2 commits July 15, 2022 16:45

Fix for TestTokenAuthentication failure

74dc0f1

apurtell force-pushed the HBASE-27195 branch from a4a1a26 to 74dc0f1 Compare July 15, 2022 23:49

Apache9 approved these changes Jul 16, 2022

View reviewed changes

apurtell merged commit 745142a into apache:master Jul 18, 2022

apurtell deleted the HBASE-27195 branch July 18, 2022 22:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-27195 Clean up netty worker/thread pool configuration#4619

HBASE-27195 Clean up netty worker/thread pool configuration#4619
apurtell merged 2 commits intoapache:masterfrom
apurtell:HBASE-27195

apurtell commented Jul 13, 2022

Uh oh!

Apache-HBase commented Jul 13, 2022

Uh oh!

Apache9 Jul 14, 2022

Uh oh!

apurtell Jul 14, 2022

Uh oh!

Apache-HBase commented Jul 14, 2022

Uh oh!

Apache-HBase commented Jul 14, 2022

Uh oh!

apurtell commented Jul 14, 2022

Uh oh!

apurtell commented Jul 15, 2022

Uh oh!

Apache-HBase commented Jul 16, 2022

Uh oh!

Apache-HBase commented Jul 16, 2022

Uh oh!

Apache-HBase commented Jul 16, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

apurtell commented Jul 13, 2022

Uh oh!

Apache-HBase commented Jul 13, 2022

Uh oh!

Apache9 Jul 14, 2022

Choose a reason for hiding this comment

Uh oh!

apurtell Jul 14, 2022

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Jul 14, 2022

Uh oh!

Apache-HBase commented Jul 14, 2022

Uh oh!

apurtell commented Jul 14, 2022

Uh oh!

apurtell commented Jul 15, 2022

Uh oh!

Apache-HBase commented Jul 16, 2022

Uh oh!

Apache-HBase commented Jul 16, 2022

Uh oh!

Apache-HBase commented Jul 16, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants