HBASE-26872 Load rate calculator for cost functions should be more precise #4253

sunhelly · 2022-03-21T04:32:54Z

No description provided.

…ecise

Apache-HBase · 2022-03-21T05:15:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 47s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 19s	master passed
+1 💚	compile	0m 33s	master passed
+1 💚	checkstyle	0m 13s	master passed
+1 💚	spotbugs	0m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 49s	the patch passed
+1 💚	compile	0m 26s	the patch passed
+1 💚	javac	0m 26s	the patch passed
+1 💚	checkstyle	0m 10s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	14m 14s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	0m 35s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		30m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4253
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 23996c647168 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0d7638d`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	60 (vs. ulimit of 30000)
modules	C: hbase-balancer U: hbase-balancer
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-03-21T05:19:36Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 40s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 42s	master passed
+1 💚	compile	0m 17s	master passed
+1 💚	shadedjars	4m 42s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 17s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 18s	the patch passed
+1 💚	compile	0m 17s	the patch passed
+1 💚	javac	0m 17s	the patch passed
+1 💚	shadedjars	4m 41s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 14s	the patch passed
		_ Other Tests _
+1 💚	unit	10m 10s	hbase-balancer in the patch passed.
		30m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4253
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux ffe4e57827db 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0d7638d`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/testReport/
Max. process+thread count	190 (vs. ulimit of 30000)
modules	C: hbase-balancer U: hbase-balancer
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-03-21T05:21:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 55s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 36s	master passed
+1 💚	compile	0m 24s	master passed
+1 💚	shadedjars	5m 40s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 18s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 40s	the patch passed
+1 💚	compile	0m 23s	the patch passed
+1 💚	javac	0m 23s	the patch passed
+1 💚	shadedjars	6m 8s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 17s	the patch passed
		_ Other Tests _
+1 💚	unit	12m 26s	hbase-balancer in the patch passed.
		35m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4253
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 1bc43c365842 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0d7638d`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/testReport/
Max. process+thread count	161 (vs. ulimit of 30000)
modules	C: hbase-balancer U: hbase-balancer
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

sunhelly · 2022-03-24T11:57:57Z

Hi, @bbeaudreault , would you want to review this PR in your convenience? Thanks.

bbeaudreault

This change makes sense! The code change itself looks good. I had 1 minor request for the test, if it makes sense

bbeaudreault · 2022-03-24T14:56:18Z

...lancer/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java

@@ -519,6 +517,31 @@ public void testRegionLoadCost() {
    assertEquals(2.5, result, 0.01);
  }

+  @Test
+  public void testRegionLoadCostWhenDecrease() {


correct me if im wrong, but i believe this is testing the case where region loads go 1, 2, 0, 4. So it uses a 0 value in the 3rd slot. This works to verify your change, but I wonder if it would be more comprehensive to test 1, 2, 1, 4. This was just be more indicative of a real reset where the value starts at 0 but then gets incremented again.

Yeah, good idea, @bbeaudreault. It's reasonable to test the [1, 2, 1, 4], since it's rare to see 0 in the real scenarios, and I used 0 to describe the issue more clearly. Thanks.

Apache-HBase · 2022-03-25T07:17:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 46s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 24s	master passed
+1 💚	compile	0m 20s	master passed
+1 💚	checkstyle	0m 8s	master passed
+1 💚	spotbugs	0m 22s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 3s	the patch passed
+1 💚	compile	0m 17s	the patch passed
+1 💚	javac	0m 17s	the patch passed
+1 💚	checkstyle	0m 7s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	11m 2s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	0m 23s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 7s	The patch does not generate ASF License warnings.
		22m 36s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4253
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 84bd4ab8d986 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4ac600b`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	60 (vs. ulimit of 30000)
modules	C: hbase-balancer U: hbase-balancer
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-03-25T07:24:28Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 8s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 50s	master passed
+1 💚	compile	0m 16s	master passed
+1 💚	shadedjars	3m 44s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 15s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 42s	the patch passed
+1 💚	compile	0m 14s	the patch passed
+1 💚	javac	0m 14s	the patch passed
+1 💚	shadedjars	3m 44s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 13s	the patch passed
		_ Other Tests _
+1 💚	unit	13m 19s	hbase-balancer in the patch passed.
		29m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4253
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c32b3b25f9fc 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4ac600b`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/testReport/
Max. process+thread count	195 (vs. ulimit of 30000)
modules	C: hbase-balancer U: hbase-balancer
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-03-25T07:28:13Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 53s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 38s	master passed
+1 💚	compile	0m 23s	master passed
+1 💚	shadedjars	5m 39s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 16s	the patch passed
+1 💚	compile	0m 17s	the patch passed
+1 💚	javac	0m 17s	the patch passed
+1 💚	shadedjars	5m 38s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 20s	the patch passed
		_ Other Tests _
+1 💚	unit	10m 50s	hbase-balancer in the patch passed.
		32m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4253
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux dbba970960ef 5.4.0-1043-aws #45~18.04.1-Ubuntu SMP Fri Apr 9 23:32:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4ac600b`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/testReport/
Max. process+thread count	162 (vs. ulimit of 30000)
modules	C: hbase-balancer U: hbase-balancer
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4253/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…ecise (#4253) Signed-off-by: Bryan Beaudreault <bbeaudreault@hubspot.com> Signed-off-by: Viraj Jasani<virajjasani@apache.org>

… more precise (#4253)" This reverts commit 7ae6be7.

… more precise (#4253)" This reverts commit 568b335.

…ecise (#4253) Signed-off-by: Bryan Beaudreault <bbeaudreault@hubspot.com> Signed-off-by: Viraj Jasani<virajjasani@apache.org>

HBASE-26872 Load rate calculator for cost functions should be more pr…

a2d0071

…ecise

sunhelly changed the title ~~HBASE-26872 Load rate calculator for cost functions should be more pr…~~ HBASE-26872 Load rate calculator for cost functions should be more precise Mar 21, 2022

sunhelly requested a review from Apache9 March 21, 2022 12:34

bbeaudreault approved these changes Mar 24, 2022

View reviewed changes

update the testcase

b090103

virajjasani approved these changes Mar 26, 2022

View reviewed changes

sunhelly merged commit ad70ac2 into apache:master Mar 27, 2022

asfgit pushed a commit that referenced this pull request Mar 27, 2022

HBASE-26872 Load rate calculator for cost functions should be more pr…

7ae6be7

…ecise (#4253) Signed-off-by: Bryan Beaudreault <bbeaudreault@hubspot.com> Signed-off-by: Viraj Jasani<virajjasani@apache.org>

asfgit pushed a commit that referenced this pull request Mar 27, 2022

HBASE-26872 Load rate calculator for cost functions should be more pr…

568b335

…ecise (#4253) Signed-off-by: Bryan Beaudreault <bbeaudreault@hubspot.com> Signed-off-by: Viraj Jasani<virajjasani@apache.org>

asfgit pushed a commit that referenced this pull request Mar 28, 2022

Revert "HBASE-26872 Load rate calculator for cost functions should be…

7623900

… more precise (#4253)" This reverts commit 7ae6be7.

asfgit pushed a commit that referenced this pull request Mar 28, 2022

Revert "HBASE-26872 Load rate calculator for cost functions should be…

2452290

… more precise (#4253)" This reverts commit 568b335.

asfgit pushed a commit that referenced this pull request Mar 28, 2022

HBASE-26872 Load rate calculator for cost functions should be more pr…

f0088df

…ecise (#4253) Signed-off-by: Bryan Beaudreault <bbeaudreault@hubspot.com> Signed-off-by: Viraj Jasani<virajjasani@apache.org>

asfgit pushed a commit that referenced this pull request Mar 28, 2022

HBASE-26872 Load rate calculator for cost functions should be more pr…

170570b

…ecise (#4253) Signed-off-by: Bryan Beaudreault <bbeaudreault@hubspot.com> Signed-off-by: Viraj Jasani<virajjasani@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-26872 Load rate calculator for cost functions should be more precise #4253

HBASE-26872 Load rate calculator for cost functions should be more precise #4253

sunhelly commented Mar 21, 2022 •

edited

Loading

Apache-HBase commented Mar 21, 2022

Apache-HBase commented Mar 21, 2022

Apache-HBase commented Mar 21, 2022

sunhelly commented Mar 24, 2022

bbeaudreault left a comment

bbeaudreault Mar 24, 2022

sunhelly Mar 25, 2022

Apache-HBase commented Mar 25, 2022

Apache-HBase commented Mar 25, 2022

Apache-HBase commented Mar 25, 2022

HBASE-26872 Load rate calculator for cost functions should be more precise #4253

HBASE-26872 Load rate calculator for cost functions should be more precise #4253

Conversation

sunhelly commented Mar 21, 2022 • edited Loading

Apache-HBase commented Mar 21, 2022

Apache-HBase commented Mar 21, 2022

Apache-HBase commented Mar 21, 2022

sunhelly commented Mar 24, 2022

bbeaudreault left a comment

Choose a reason for hiding this comment

bbeaudreault Mar 24, 2022

Choose a reason for hiding this comment

sunhelly Mar 25, 2022

Choose a reason for hiding this comment

Apache-HBase commented Mar 25, 2022

Apache-HBase commented Mar 25, 2022

Apache-HBase commented Mar 25, 2022

sunhelly commented Mar 21, 2022 •

edited

Loading