HBASE-25334 TestRSGroupsFallback.testFallback is flaky #2728

sunhelly · 2020-12-01T09:58:02Z

No description provided.

sunhelly · 2020-12-01T10:09:09Z

@Apache9 I'm sorry, could you test this patch?
I think the problem is at TEST_UTIL.waitUntilAllRegionsAssigned(table), which just waiting for table regions be OPEN and checking the RS for table regions are online. While before AM executing the TRSP procedures, this assertion will be true.

Apache-HBase · 2020-12-01T10:49:36Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 46s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 25s	master passed
+1 💚	checkstyle	1m 23s	master passed
+1 💚	spotbugs	2m 48s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 36s	the patch passed
+1 💚	checkstyle	1m 29s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	22m 43s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 59s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 16s	The patch does not generate ASF License warnings.
		50m 44s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux 10fa8c8d5249 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `86bb037`
Max. process+thread count	95 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-12-01T12:56:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 27s	master passed
+1 💚	compile	1m 9s	master passed
+1 💚	shadedjars	6m 51s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 43s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 10s	the patch passed
+1 💚	compile	1m 8s	the patch passed
+1 💚	javac	1m 8s	the patch passed
+1 💚	shadedjars	7m 9s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 51s	the patch passed
		_ Other Tests _
+1 💚	unit	148m 10s	hbase-server in the patch passed.
		177m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 817e25fa2876 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `86bb037`
Default Java	AdoptOpenJDK-11.0.6+10
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/1/testReport/
Max. process+thread count	4317 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-12-01T13:03:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 53s	master passed
+1 💚	compile	0m 55s	master passed
+1 💚	shadedjars	6m 30s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 35s	the patch passed
+1 💚	compile	0m 56s	the patch passed
+1 💚	javac	0m 56s	the patch passed
+1 💚	shadedjars	6m 30s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	the patch passed
		_ Other Tests _
+1 💚	unit	158m 14s	hbase-server in the patch passed.
		184m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux a247883c01b3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `86bb037`
Default Java	AdoptOpenJDK-1.8.0_232-b09
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/1/testReport/
Max. process+thread count	3652 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

yuqi1129 · 2020-12-01T15:14:32Z

@sunhelly Yes, that is exactly the cause, in HBASE-25282, i use the ServerCrashProcedure to judge whether the Region Server is in processing, Sorry to introduce such a problem.

sunhelly · 2020-12-01T15:18:28Z

@sunhelly Yes, that is exactly the cause, in HBASE-25282, i use the ServerCrashProcedure to judge whether the Region Server is in processing, Sorry to introduce such a problem.

@yuqi1129 Please help to check if this patch can fix the problem, thanks.

yuqi1129 · 2020-12-03T12:18:58Z

@sunhelly Sorry for late reply, your PR seems OK to me，LGTM

Apache9 · 2020-12-04T03:00:30Z

Tried locally, TestRSGroupsFallback still failed. Could you two please confirm it again? And mind explaining a bit why HBASE-25282 could make this test fail? What's the root cause?

Thanks.

sunhelly · 2020-12-04T06:25:04Z

Tried locally, TestRSGroupsFallback still failed. Could you two please confirm it again? And mind explaining a bit why HBASE-25282 could make this test fail? What's the root cause?

@Apache9 @yuqi1129 I looked through the patch of HBASE-25282 and found a little clue. I have commented at the PR, PR of HBASE-25282 .
Does it fails at assertRegionsInGroup after crashRsInGroup(groupName) ? @Apache9 (I can't reproduce the failure of this UT, while few days ago it failed after MASTER.balance() as I have attached the links in the description of HBASE-25334).
I think to fix this failure, two places need to be corrected. One is modify ServerManager.expireServer, move the calling of moveFromOnlineToDeadServers(serverName); after submit SCP.
The other is change AssignmentTestingUtil.waitForRsToBeDead(), it should check both server is in the dead list and no SCP procedures exists.

sunhelly · 2020-12-04T06:47:53Z

Oh, the method balance() used
if (serverManager.**areDeadServersInProgress**()) { LOG.debug("Not running balancer because processing dead regionserver(s): {}", serverManager.getDeadServers()); return false; }
So in the UT, if the SCP has not been finished, the balancer will break.Though it wait for the same host port RS to be online by TEST_UTIL.getMiniHBaseCluster().startRegionServerAndWait(60000), master deals them as two servers...
As a result, assertRegionsInGroup can fail after MASTER.balance(), as we saw in the test results...
@Apache9 @yuqi1129
This is directly relevant to the patch of HBASE-25282.

sunhelly · 2020-12-06T09:06:55Z

I'm so sorry, didn't merge the master branch, so the UT is always passing. @Apache9
I think the root cause is that, HBASE-25282 makes ServerManager.areDeadServersInProgress wrong, shouldn't uses getProcedures, and should not make assertion from false to true in testCrashProcedureReplay.
I have updated the PR, all UTs pass now.

Apache-HBase · 2020-12-06T09:38:51Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 25s	master passed
+1 💚	checkstyle	1m 4s	master passed
+1 💚	spotbugs	1m 57s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 28s	the patch passed
+1 💚	checkstyle	1m 4s	the patch passed
+1 💚	whitespace	0m 1s	The patch has no whitespace issues.
+1 💚	hadoopcheck	16m 54s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 8s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		37m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux 4aa7f93bc650 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7d0a687`
Max. process+thread count	94 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-12-06T11:41:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 12s	master passed
+1 💚	compile	1m 6s	master passed
+1 💚	shadedjars	6m 39s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 44s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 1s	the patch passed
+1 💚	compile	1m 5s	the patch passed
+1 💚	javac	1m 5s	the patch passed
+1 💚	shadedjars	6m 42s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 40s	the patch passed
		_ Other Tests _
+1 💚	unit	133m 12s	hbase-server in the patch passed.
		161m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 036495c84a7f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7d0a687`
Default Java	AdoptOpenJDK-11.0.6+10
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/2/testReport/
Max. process+thread count	4513 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-12-06T11:47:14Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 29s	master passed
+1 💚	compile	0m 56s	master passed
+1 💚	shadedjars	6m 29s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 27s	the patch passed
+1 💚	compile	0m 54s	the patch passed
+1 💚	javac	0m 54s	the patch passed
+1 💚	shadedjars	6m 29s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	the patch passed
		_ Other Tests _
+1 💚	unit	140m 38s	hbase-server in the patch passed.
		166m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux ac7a6512e0b0 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7d0a687`
Default Java	AdoptOpenJDK-1.8.0_232-b09
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/2/testReport/
Max. process+thread count	4885 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-12-06T12:28:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 45s	master passed
+1 💚	checkstyle	1m 6s	master passed
+1 💚	spotbugs	2m 3s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 29s	the patch passed
-0 ⚠️	checkstyle	1m 3s	hbase-server: The patch generated 2 new + 14 unchanged - 0 fixed = 16 total (was 14)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	17m 4s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 9s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		38m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux 48acdd6b9ceb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7d0a687`
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	94 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/3/console
versions	git=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-12-06T14:30:15Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 58s	master passed
+1 💚	compile	1m 7s	master passed
+1 💚	shadedjars	6m 46s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 39s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 1s	the patch passed
+1 💚	compile	1m 5s	the patch passed
+1 💚	javac	1m 5s	the patch passed
+1 💚	shadedjars	6m 37s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 41s	the patch passed
		_ Other Tests _
+1 💚	unit	133m 2s	hbase-server in the patch passed.
		160m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 52df72c1d3d6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7d0a687`
Default Java	AdoptOpenJDK-11.0.6+10
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/3/testReport/
Max. process+thread count	4773 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/3/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-12-06T14:35:50Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 30s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 28s	master passed
+1 💚	compile	0m 57s	master passed
+1 💚	shadedjars	6m 32s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 28s	the patch passed
+1 💚	compile	0m 56s	the patch passed
+1 💚	javac	0m 56s	the patch passed
+1 💚	shadedjars	6m 36s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	140m 21s	hbase-server in the patch passed.
		166m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux f4dc8e25f6b4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7d0a687`
Default Java	AdoptOpenJDK-1.8.0_232-b09
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/3/testReport/
Max. process+thread count	4967 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/3/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9

So this is the root cause? I suggest that we still use getProcedures but first filter out the finished procedures before checking whehter it is SCP.

Apache9 · 2020-12-06T15:03:12Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java

   */
  public boolean areDeadServersInProgress() throws IOException {
-    return master.getProcedures().stream().anyMatch(p -> p instanceof ServerCrashProcedure);
+    return master.getMasterProcedureExecutor().getActiveProceduresNoCopy().stream()


The getActiveProceduresNoCopy is only expected to be called when initialization, where we can make sure there is no race.

Apache-HBase · 2020-12-07T02:54:32Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 29s	master passed
+1 💚	checkstyle	1m 5s	master passed
+1 💚	spotbugs	1m 56s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 26s	the patch passed
-0 ⚠️	checkstyle	1m 1s	hbase-server: The patch generated 2 new + 14 unchanged - 0 fixed = 16 total (was 14)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	16m 56s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 9s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		37m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux 3fad14a8ee02 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7d0a687`
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	94 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/4/console
versions	git=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-12-07T05:02:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 25s	master passed
+1 💚	compile	0m 56s	master passed
+1 💚	shadedjars	6m 35s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 28s	the patch passed
+1 💚	compile	0m 55s	the patch passed
+1 💚	javac	0m 55s	the patch passed
+1 💚	shadedjars	6m 28s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	140m 41s	hbase-server in the patch passed.
		166m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 984ea2707f06 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7d0a687`
Default Java	AdoptOpenJDK-1.8.0_232-b09
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/4/testReport/
Max. process+thread count	5138 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/4/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-12-07T05:07:06Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 29s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 3s	master passed
+1 💚	compile	1m 7s	master passed
+1 💚	shadedjars	6m 43s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 40s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 5s	the patch passed
+1 💚	compile	1m 5s	the patch passed
+1 💚	javac	1m 6s	the patch passed
+1 💚	shadedjars	6m 39s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 40s	the patch passed
		_ Other Tests _
-1 ❌	unit	142m 58s	hbase-server in the patch failed.
		170m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 9c1dfec31ec0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `7d0a687`
Default Java	AdoptOpenJDK-11.0.6+10
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/4/testReport/
Max. process+thread count	4432 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/4/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-12-08T07:55:09Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 40s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 45s	master passed
+1 💚	checkstyle	1m 22s	master passed
+1 💚	spotbugs	2m 43s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 44s	the patch passed
-0 ⚠️	checkstyle	1m 21s	hbase-server: The patch generated 2 new + 14 unchanged - 0 fixed = 16 total (was 14)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	22m 36s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 55s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		50m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux 43221084bfd3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `979ad0f`
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/5/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	95 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/5/console
versions	git=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-12-08T09:50:48Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 26s	master passed
+1 💚	compile	0m 58s	master passed
+1 💚	shadedjars	6m 33s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 30s	the patch passed
+1 💚	compile	0m 56s	the patch passed
+1 💚	javac	0m 56s	the patch passed
+1 💚	shadedjars	6m 31s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	the patch passed
		_ Other Tests _
+1 💚	unit	140m 24s	hbase-server in the patch passed.
		166m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 9f0c7a0d0c51 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `979ad0f`
Default Java	AdoptOpenJDK-1.8.0_232-b09
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/5/testReport/
Max. process+thread count	4546 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/5/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2020-12-08T10:55:11Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 20s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 49s	master passed
+1 💚	compile	1m 14s	master passed
+1 💚	shadedjars	7m 27s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 43s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 34s	the patch passed
+1 💚	compile	1m 14s	the patch passed
+1 💚	javac	1m 14s	the patch passed
+1 💚	shadedjars	7m 23s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 40s	the patch passed
		_ Other Tests _
+1 💚	unit	199m 19s	hbase-server in the patch passed.
		230m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2728
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 4e1b3e2e90c3 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `979ad0f`
Default Java	AdoptOpenJDK-11.0.6+10
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/5/testReport/
Max. process+thread count	3250 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2728/5/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

sunhelly · 2020-12-09T03:02:36Z

Ping @Apache9 @yuqi1129 for review，thanks.

Signed-off-by: Duo Zhang <zhangduo@apache.org> Change-Id: I2f78f6108cd4c3049b3df797d9f6aba4d81b5da8

…allback.testFallback is flaky" into branch-2 and branch-2.6 (#7472) #7476 (#7476) The change in HBASE-25334 (introduced by commit 32c4432 in branch-2 and branch-2.6) has diverged between the master (#2728) and branch-2 branches. This is a following up changes after HBASE-29720 and have the complete functional changes of HBASE-25334 that uses SCP as the source of dead servers in progress. Signed-off-by: Istvan Toth <stoty@apache.org> Co-authored-by: Kevin Geiszler <kevin.j.geiszler@gmail.com>

HBASE-25334 TestRSGroupsFallback.testFallback is flaky

8475af7

Merge branch 'master' into HBASE-25334

0cfeb4f

sunhelly force-pushed the HBASE-25334 branch from 3c9b685 to 92cf513 Compare December 6, 2020 09:14

sunhelly force-pushed the HBASE-25334 branch from 92cf513 to fe491f0 Compare December 6, 2020 10:19

Apache9 requested changes Dec 6, 2020

View reviewed changes

sunhelly force-pushed the HBASE-25334 branch from fe491f0 to e190110 Compare December 7, 2020 02:15

fix testCrashProcedureReplay and testFallback

d4d5a50

sunhelly force-pushed the HBASE-25334 branch from e190110 to d4d5a50 Compare December 8, 2020 07:03

Apache9 approved these changes Dec 10, 2020

View reviewed changes

Apache9 merged commit a9b8c10 into apache:master Dec 10, 2020

kgeisz mentioned this pull request Nov 20, 2025

HBASE-29724: Backport "HBASE-25334: TestRSGroupsFallback.testFallback is flaky" into branch-2 and branch-2.6 #7472

Merged

kgeisz pushed a commit to kgeisz/hbase that referenced this pull request Nov 20, 2025

HBASE-25334 TestRSGroupsFallback.testFallback is flaky (apache#2728)

01797bb

Signed-off-by: Duo Zhang <zhangduo@apache.org> Change-Id: I2f78f6108cd4c3049b3df797d9f6aba4d81b5da8

taklwu mentioned this pull request Nov 24, 2025

HBASE-29724: Backport missing changes of "HBASE-25334: TestRSGroupsFallback.testFallback is flaky" into branch-2 and branch-2.6 (#7472) #7476

Merged

HBASE-25334 TestRSGroupsFallback.testFallback is flaky #2728

HBASE-25334 TestRSGroupsFallback.testFallback is flaky #2728

Uh oh!

Conversation

sunhelly commented Dec 1, 2020

Uh oh!

sunhelly commented Dec 1, 2020

Uh oh!

Apache-HBase commented Dec 1, 2020

Uh oh!

Apache-HBase commented Dec 1, 2020

Uh oh!

Apache-HBase commented Dec 1, 2020

Uh oh!

yuqi1129 commented Dec 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sunhelly commented Dec 1, 2020

Uh oh!

yuqi1129 commented Dec 3, 2020

Uh oh!

Apache9 commented Dec 4, 2020

Uh oh!

sunhelly commented Dec 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sunhelly commented Dec 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sunhelly commented Dec 6, 2020

Uh oh!

Apache-HBase commented Dec 6, 2020

Uh oh!

Apache-HBase commented Dec 6, 2020

Uh oh!

Apache-HBase commented Dec 6, 2020

Uh oh!

Apache-HBase commented Dec 6, 2020

Uh oh!

Apache-HBase commented Dec 6, 2020

Uh oh!

Apache-HBase commented Dec 6, 2020

Uh oh!

Apache9 left a comment

Choose a reason for hiding this comment

Uh oh!

Apache9 Dec 6, 2020

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Dec 7, 2020

Uh oh!

Apache-HBase commented Dec 7, 2020

Uh oh!

Apache-HBase commented Dec 7, 2020

Uh oh!

Apache-HBase commented Dec 8, 2020

Uh oh!

Apache-HBase commented Dec 8, 2020

Uh oh!

Apache-HBase commented Dec 8, 2020

Uh oh!

sunhelly commented Dec 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yuqi1129 commented Dec 1, 2020 •

edited

Loading

sunhelly commented Dec 4, 2020 •

edited

Loading

sunhelly commented Dec 4, 2020 •

edited

Loading