HBASE-22349 slop in StochasticLoadBalancer by d-c-manning · Pull Request #4371 · apache/hbase

d-c-manning · 2022-04-26T07:31:09Z

Slop is a concept used within SimpleLoadBalancer but has always been referenced in StochasticLoadBalancer without any effect. This change proposes that we start using the concept of slop, as another trigger for a balancer run. This is similar to HBASE-24139, where a server with 0 regions is considered a trigger to run the balancer.

In this way we can guarantee balancer attempts to run in other cases that a balancer run is desirable, such as only 1 region on a server, or 2x or 3x the average region count on a server.

I remove the normalizeSlop method for two reasons:

A negative value will indicate the slop value should not be used at all
Values greater than 1 can still be useful (setting slop to 2 would mean to trigger a balancer run if a server had >3x the average region count for the cluster.)

As far as I can tell, the default slop of 0.001f never had any meaning for the StochasticLoadBalancer before this change. I choose to reuse the default slop of 0.2f as used in the SimpleLoadBalancer.

Apache-HBase · 2022-04-26T08:19:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 41s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 44s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 4s	master passed
+1 💚	compile	2m 59s	master passed
+1 💚	checkstyle	0m 52s	master passed
+0 🆗	refguide	2m 3s	branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	spotbugs	1m 55s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 10s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 13s	the patch passed
+1 💚	compile	3m 11s	the patch passed
+1 💚	javac	3m 11s	the patch passed
+1 💚	checkstyle	0m 12s	The patch passed checkstyle in hbase-common
+1 💚	checkstyle	0m 7s	hbase-balancer: The patch generated 0 new + 15 unchanged - 64 fixed = 15 total (was 79)
+1 💚	checkstyle	0m 33s	The patch passed checkstyle in hbase-server
-0 ⚠️	whitespace	0m 0s	The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+0 🆗	refguide	1m 38s	patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	hadoopcheck	14m 30s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	3m 20s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 26s	The patch does not generate ASF License warnings.
		44m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4371/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4371
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile refguide xml
uname	Linux a6ce397cd769 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `af13c6d`
Default Java	AdoptOpenJDK-1.8.0_282-b08
refguide	https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4371/1/yetus-general-check/output/branch-site/book.html
whitespace	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4371/1/artifact/yetus-general-check/output/whitespace-eol.txt
refguide	https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-4371/1/yetus-general-check/output/patch-site/book.html
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-common hbase-balancer hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4371/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-04-26T11:41:22Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 44s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 17s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 23s	master passed
+1 💚	compile	1m 0s	master passed
+1 💚	shadedjars	3m 59s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 44s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 7s	the patch passed
+1 💚	compile	1m 0s	the patch passed
+1 💚	javac	1m 0s	the patch passed
+1 💚	shadedjars	4m 1s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 48s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 26s	hbase-common in the patch passed.
+1 💚	unit	11m 11s	hbase-balancer in the patch passed.
+1 💚	unit	213m 7s	hbase-server in the patch passed.
		246m 1s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4371/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4371
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 83283906398a 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `af13c6d`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4371/1/testReport/
Max. process+thread count	2718 (vs. ulimit of 30000)
modules	C: hbase-common hbase-balancer hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4371/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-04-26T11:52:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 6s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 45s	master passed
+1 💚	compile	1m 20s	master passed
+1 💚	shadedjars	3m 42s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 55s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 33s	the patch passed
+1 💚	compile	1m 19s	the patch passed
+1 💚	javac	1m 19s	the patch passed
+1 💚	shadedjars	4m 1s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 56s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 38s	hbase-common in the patch passed.
+1 💚	unit	11m 36s	hbase-balancer in the patch passed.
-1 ❌	unit	221m 58s	hbase-server in the patch failed.
		257m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4371/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4371
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux ab83f86587a7 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `af13c6d`
Default Java	AdoptOpenJDK-11.0.10+9
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4371/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4371/1/testReport/
Max. process+thread count	2672 (vs. ulimit of 30000)
modules	C: hbase-common hbase-balancer hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4371/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

d-c-manning · 2022-04-28T19:15:36Z

test failures don't appear to be related.

apurtell

Approved for commit to branch-2.5, branch-2, and master. Please push a fix for the reported whitespace issue and I will then merge this. If backport to branch-2 is nontrivial please consider providing a branch-2 PR as well, although I will try to cherry pick this change otherwise.

d-c-manning · 2022-04-28T19:25:50Z

Oops, missed the whitespace comment. Will fix.

Signed-off-by: Andrew Purtell <apurtell@apache.org> Conflicts: hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java

HBASE-22349 slop in StochasticLoadBalancer

f4e8256

apurtell approved these changes Apr 28, 2022

View reviewed changes

remove unnecessary whitespace

b4c72cd

apurtell merged commit f848c61 into apache:master Apr 28, 2022

d-c-manning deleted the HBASE-22349 branch April 28, 2022 19:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HBASE-22349 slop in StochasticLoadBalancer#4371

HBASE-22349 slop in StochasticLoadBalancer#4371
apurtell merged 2 commits into
apache:masterfrom
d-c-manning:HBASE-22349

d-c-manning commented Apr 26, 2022

Uh oh!

Apache-HBase commented Apr 26, 2022

Uh oh!

Apache-HBase commented Apr 26, 2022

Uh oh!

Apache-HBase commented Apr 26, 2022

Uh oh!

d-c-manning commented Apr 28, 2022

Uh oh!

apurtell left a comment •

edited

Loading

Uh oh!

d-c-manning commented Apr 28, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

d-c-manning commented Apr 26, 2022

Uh oh!

Apache-HBase commented Apr 26, 2022

Uh oh!

Apache-HBase commented Apr 26, 2022

Uh oh!

Apache-HBase commented Apr 26, 2022

Uh oh!

d-c-manning commented Apr 28, 2022

Uh oh!

apurtell left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d-c-manning commented Apr 28, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

apurtell left a comment •

edited

Loading