HBASE-26984 Chaos Monkey thread dies in ITBLL Chaos GracefulRollingRe… by huaxiangsun · Pull Request #4383 · apache/hbase

huaxiangsun · 2022-04-29T00:36:10Z

…startRsAction

There are two cases here:

Chaos Monkey thread died and there is no chaos after that.
Sometimes, regions are being moved back too quick that region server has not finished its initliazation yet.
wait sometime to make sure that region server finishes its initialization.

Apache-HBase · 2022-04-29T02:16:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	7m 39s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 8s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 47s	master passed
+1 💚	compile	7m 4s	master passed
+1 💚	checkstyle	1m 26s	master passed
+1 💚	spotbugs	9m 26s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 10s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 49s	the patch passed
+1 💚	compile	7m 49s	the patch passed
+1 💚	javac	7m 49s	the patch passed
+1 💚	checkstyle	1m 24s	the patch passed
-0 ⚠️	whitespace	0m 0s	The patch has 19 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚	hadoopcheck	14m 12s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	9m 42s	the patch passed
		_ Other Tests _
-1 ❌	asflicense	0m 20s	The patch generated 4 ASF License warnings.
		70m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4383
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux cb29604e9bb6 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `f4eb907`
Default Java	AdoptOpenJDK-1.8.0_282-b08
whitespace	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/1/artifact/yetus-general-check/output/whitespace-eol.txt
asflicense	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/1/artifact/yetus-general-check/output/patch-asflicense-problems.txt
Max. process+thread count	138 (vs. ulimit of 30000)
modules	C: hbase-it . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-04-29T05:28:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 40s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 16s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 52s	master passed
+1 💚	compile	1m 48s	master passed
+1 💚	shadedjars	3m 49s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 6s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 10s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 35s	the patch passed
+1 💚	compile	1m 47s	the patch passed
+1 💚	javac	1m 47s	the patch passed
-1 ❌	shadedjars	0m 7s	patch has 7 errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 3s	the patch passed
		_ Other Tests _
+1 💚	unit	240m 33s	root in the patch passed.
		262m 1s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4383
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 475ed9c0fe2d 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `f4eb907`
Default Java	AdoptOpenJDK-11.0.10+9
shadedjars	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/1/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/1/testReport/
Max. process+thread count	5446 (vs. ulimit of 30000)
modules	C: hbase-it . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-04-29T07:02:48Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 18s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 25s	master passed
+1 💚	compile	1m 38s	master passed
+1 💚	shadedjars	3m 40s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 41s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 13s	the patch passed
+1 💚	compile	1m 34s	the patch passed
+1 💚	javac	1m 34s	the patch passed
-1 ❌	shadedjars	0m 7s	patch has 7 errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 40s	the patch passed
		_ Other Tests _
+1 💚	unit	337m 4s	root in the patch passed.
		357m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4383
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux eb0c9ca4cd13 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `f4eb907`
Default Java	AdoptOpenJDK-1.8.0_282-b08
shadedjars	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/1/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/1/testReport/
Max. process+thread count	5081 (vs. ulimit of 30000)
modules	C: hbase-it . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

wchevreuil · 2022-04-29T13:15:08Z

+     getLogger().info("Starting region server: {}", server);
+     startRs(server);
+    // Sleep 2 seconds to make sure RS is online.
+    sleep(2000);


Make it configurable?

Just figured that sleepTime is less than 1 second, so I think that is the perfect sleep time. Adding one more configuration is probably too much, what do you think? Thanks.

Yeah, and I just realised we already have similar hardcoded sleep calls elsewhere in this code, so I guess this extra one is inline with that approach.

…startRsAction There are two cases here: 1. Chaos Monkey thread died and there is no chaos after that. 2. Sometimes, regions are being moved back too quick that region server has not finished its initliazation yet. wait sometime to make sure that region server finishes its initialization.

Apache-HBase · 2022-05-02T23:51:40Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 21s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 34s	master passed
+1 💚	compile	0m 18s	master passed
+1 💚	shadedjars	3m 41s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 14s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 15s	the patch passed
+1 💚	compile	0m 18s	the patch passed
+1 💚	javac	0m 18s	the patch passed
+1 💚	shadedjars	3m 40s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 11s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 40s	hbase-it in the patch passed.
		15m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4383
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux ee1edbf16d6e 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `da55154`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/2/testReport/
Max. process+thread count	560 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-05-02T23:51:52Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 35s	master passed
+1 💚	compile	0m 16s	master passed
+1 💚	shadedjars	3m 41s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 12s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 20s	the patch passed
+1 💚	compile	0m 17s	the patch passed
+1 💚	javac	0m 17s	the patch passed
+1 💚	shadedjars	3m 41s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 10s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 34s	hbase-it in the patch passed.
		15m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4383
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 3c4861661b09 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `da55154`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/2/testReport/
Max. process+thread count	589 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-05-03T00:02:48Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 40s	master passed
+1 💚	compile	0m 25s	master passed
+1 💚	checkstyle	0m 9s	master passed
+1 💚	spotbugs	0m 32s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 6s	the patch passed
+1 💚	compile	0m 24s	the patch passed
+1 💚	javac	0m 24s	the patch passed
+1 💚	checkstyle	0m 7s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	11m 54s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotbugs	0m 31s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		25m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4383
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 769ea506d29c 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `da55154`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-it U: hbase-it
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4383/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

huaxiangsun · 2022-05-03T15:07:25Z

Hi @wchevreuil, does the new patch look good to you?

…startRsAction (apache#4383) There are two cases here: 1. Chaos Monkey thread died and there is no chaos after that. 2. Sometimes, regions are being moved back too quick that region server has not finished its initliazation yet. wait sometime to make sure that region server finishes its initialization. Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>

…startRsAction (#4383) (#4408) There are two cases here: 1. Chaos Monkey thread died and there is no chaos after that. 2. Sometimes, regions are being moved back too quick that region server has not finished its initliazation yet. wait sometime to make sure that region server finishes its initialization. Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>

…startRsAction (#4383) (#4409) There are two cases here: 1. Chaos Monkey thread died and there is no chaos after that. 2. Sometimes, regions are being moved back too quick that region server has not finished its initliazation yet. wait sometime to make sure that region server finishes its initialization. Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>

wchevreuil requested changes Apr 29, 2022

View reviewed changes

huaxiangsun force-pushed the master.HBASE-26984 branch from 28ceac3 to af7f84f Compare May 2, 2022 23:17

wchevreuil approved these changes May 5, 2022

View reviewed changes

huaxiangsun merged commit 597098e into apache:master May 5, 2022

This was referenced May 5, 2022

HBASE-26984 Chaos Monkey thread dies in ITBLL Chaos GracefulRollingRe… #4408

Merged

HBASE-26984 Chaos Monkey thread dies in ITBLL Chaos GracefulRollingRe… #4409

Merged

Navigation Menu

Uh oh!

HBASE-26984 Chaos Monkey thread dies in ITBLL Chaos GracefulRollingRe… - #4383

HBASE-26984 Chaos Monkey thread dies in ITBLL Chaos GracefulRollingRe…#4383
huaxiangsun merged 1 commit into
apache:masterfrom
huaxiangsun:master.HBASE-26984

huaxiangsun commented Apr 29, 2022

Uh oh!

Apache-HBase commented Apr 29, 2022

Uh oh!

Apache-HBase commented Apr 29, 2022

Uh oh!

Apache-HBase commented Apr 29, 2022

Uh oh!

wchevreuil Apr 29, 2022

Uh oh!

huaxiangsun May 2, 2022

Uh oh!

wchevreuil May 5, 2022

Uh oh!

Uh oh!

Uh oh!

Apache-HBase commented May 2, 2022

Uh oh!

Apache-HBase commented May 2, 2022

Uh oh!

Apache-HBase commented May 3, 2022

Uh oh!

huaxiangsun commented May 3, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

huaxiangsun commented Apr 29, 2022

Uh oh!

Apache-HBase commented Apr 29, 2022

Uh oh!

Apache-HBase commented Apr 29, 2022

Uh oh!

Apache-HBase commented Apr 29, 2022

Uh oh!

wchevreuil Apr 29, 2022

Choose a reason for hiding this comment

Uh oh!

huaxiangsun May 2, 2022

Choose a reason for hiding this comment

Uh oh!

wchevreuil May 5, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Apache-HBase commented May 2, 2022

Uh oh!

Apache-HBase commented May 2, 2022

Uh oh!

Apache-HBase commented May 3, 2022

Uh oh!

huaxiangsun commented May 3, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants