HBASE-25528: Dedicated merge dispatch thread pool on master #2904

bharathv · 2021-01-25T21:06:34Z

Adds "hbase.master.executor.merge.dispatch.threads" and defaults to 2.

Also adds additional logging that includes the number of split plans
and merge plans computed for each normalizer run.

Apache-HBase · 2021-01-25T21:58:48Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 23s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 41s	master passed
+1 💚	compile	4m 0s	master passed
+1 💚	checkstyle	1m 31s	master passed
+1 💚	spotbugs	2m 49s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 14s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 31s	the patch passed
+1 💚	compile	4m 1s	the patch passed
+1 💚	javac	4m 1s	the patch passed
-0 ⚠️	checkstyle	1m 8s	hbase-server: The patch generated 2 new + 146 unchanged - 1 fixed = 148 total (was 147)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	18m 12s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	3m 9s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 22s	The patch does not generate ASF License warnings.
		51m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2904
JIRA Issue	HBASE-25528
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 23c06ab8c4c4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0b0cdca`
Default Java	AdoptOpenJDK-1.8.0_232-b09
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	96 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-01-26T01:03:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 3s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 21s	Maven dependency ordering for branch
+1 💚	mvninstall	4m 3s	master passed
+1 💚	compile	1m 20s	master passed
+1 💚	shadedjars	7m 10s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 58s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 14s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 49s	the patch passed
+1 💚	compile	1m 22s	the patch passed
+1 💚	javac	1m 22s	the patch passed
+1 💚	shadedjars	7m 11s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 57s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 56s	hbase-common in the patch passed.
+1 💚	unit	203m 20s	hbase-server in the patch passed.
		235m 49s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2904
JIRA Issue	HBASE-25528
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux adeeaa60bb98 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0b0cdca`
Default Java	AdoptOpenJDK-1.8.0_232-b09
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/1/testReport/
Max. process+thread count	2830 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-01-26T01:05:02Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 18s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 21s	Maven dependency ordering for branch
+1 💚	mvninstall	4m 43s	master passed
+1 💚	compile	1m 37s	master passed
+1 💚	shadedjars	7m 24s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 5s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for patch
+1 💚	mvninstall	4m 37s	the patch passed
+1 💚	compile	1m 39s	the patch passed
+1 💚	javac	1m 39s	the patch passed
+1 💚	shadedjars	7m 29s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 4s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 8s	hbase-common in the patch passed.
+1 💚	unit	201m 52s	hbase-server in the patch passed.
		237m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2904
JIRA Issue	HBASE-25528
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux ed83a5a12fee 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0b0cdca`
Default Java	AdoptOpenJDK-11.0.6+10
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/1/testReport/
Max. process+thread count	2747 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

virajjasani · 2021-01-26T13:54:12Z

...e-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java

+    LOG.debug("Computed {} normalization plans for table {}. Split plans: {}, merge plans: {}",
+        plans.size(), table, splitPlansCount, mergePlansCount);


Nice log. One small nit: Computed total {} normalization plans....

Adds "hbase.master.executor.merge.dispatch.threads" and defaults to 2. Also adds additional logging that includes the number of split plans and merge plans computed for each normalizer run.

Apache-HBase · 2021-01-28T00:54:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 24s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 45s	master passed
+1 💚	compile	3m 58s	master passed
+1 💚	checkstyle	1m 32s	master passed
+1 💚	spotbugs	2m 43s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 29s	the patch passed
+1 💚	compile	3m 57s	the patch passed
+1 💚	javac	3m 57s	the patch passed
-0 ⚠️	checkstyle	1m 7s	hbase-server: The patch generated 2 new + 146 unchanged - 1 fixed = 148 total (was 147)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	17m 30s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	3m 16s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 24s	The patch does not generate ASF License warnings.
		51m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2904
JIRA Issue	HBASE-25528
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux e95b23834424 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b07549f`
Default Java	AdoptOpenJDK-1.8.0_232-b09
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	96 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-01-28T02:53:35Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 25s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 46s	master passed
+1 💚	compile	1m 23s	master passed
+1 💚	shadedjars	6m 30s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 0s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 16s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 33s	the patch passed
+1 💚	compile	1m 22s	the patch passed
+1 💚	javac	1m 22s	the patch passed
+1 💚	shadedjars	6m 38s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 0s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 34s	hbase-common in the patch passed.
+1 💚	unit	139m 54s	hbase-server in the patch passed.
		170m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2904
JIRA Issue	HBASE-25528
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c481751e423d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b07549f`
Default Java	AdoptOpenJDK-1.8.0_232-b09
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/2/testReport/
Max. process+thread count	3680 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-01-28T03:54:46Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 18s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 22s	Maven dependency ordering for branch
+1 💚	mvninstall	4m 46s	master passed
+1 💚	compile	1m 38s	master passed
+1 💚	shadedjars	7m 23s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 6s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for patch
+1 💚	mvninstall	4m 38s	the patch passed
+1 💚	compile	1m 38s	the patch passed
+1 💚	javac	1m 38s	the patch passed
+1 💚	shadedjars	7m 22s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 4s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 11s	hbase-common in the patch passed.
+1 💚	unit	195m 40s	hbase-server in the patch passed.
		231m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2904
JIRA Issue	HBASE-25528
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 73d07db2ee94 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b07549f`
Default Java	AdoptOpenJDK-11.0.6+10
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/2/testReport/
Max. process+thread count	2691 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2904/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

wchevreuil approved these changes Jan 26, 2021

View reviewed changes

virajjasani approved these changes Jan 26, 2021

View reviewed changes

HBASE-25528: Dedicated merge dispatch threadpool on master

36b4698

Adds "hbase.master.executor.merge.dispatch.threads" and defaults to 2. Also adds additional logging that includes the number of split plans and merge plans computed for each normalizer run.

bharathv force-pushed the HBASE-25528 branch from 0847346 to 36b4698 Compare January 28, 2021 00:02

bharathv merged commit de51a40 into apache:master Jan 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-25528: Dedicated merge dispatch thread pool on master #2904

HBASE-25528: Dedicated merge dispatch thread pool on master #2904

bharathv commented Jan 25, 2021

Apache-HBase commented Jan 25, 2021

Apache-HBase commented Jan 26, 2021

Apache-HBase commented Jan 26, 2021

virajjasani Jan 26, 2021

bharathv Jan 28, 2021

Apache-HBase commented Jan 28, 2021

Apache-HBase commented Jan 28, 2021

Apache-HBase commented Jan 28, 2021

		LOG.debug("Computed {} normalization plans for table {}. Split plans: {}, merge plans: {}",
		plans.size(), table, splitPlansCount, mergePlansCount);

HBASE-25528: Dedicated merge dispatch thread pool on master #2904

HBASE-25528: Dedicated merge dispatch thread pool on master #2904

Conversation

bharathv commented Jan 25, 2021

Apache-HBase commented Jan 25, 2021

Apache-HBase commented Jan 26, 2021

Apache-HBase commented Jan 26, 2021

virajjasani Jan 26, 2021

Choose a reason for hiding this comment

bharathv Jan 28, 2021

Choose a reason for hiding this comment

Apache-HBase commented Jan 28, 2021

Apache-HBase commented Jan 28, 2021

Apache-HBase commented Jan 28, 2021