HBASE-24545 Add backoff to SCP check on WAL split completion

[saintstack]

Add simple backoff where we'll check every 100ms if < 10 WALs to complete else wait up to a minute between checks.

[Apache9]

Have you tried to set 'hbase.split.wal.zk.coordinated' to false to enable procedure based wal splitting? It will schedule sub procedures and remote procedures to do wal splitting, which will not block the SCP thread.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 0s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ branch-2.3 Compile Tests _
+1 💚	mvninstall	4m 43s	branch-2.3 passed
+1 💚	checkstyle	1m 22s	branch-2.3 passed
+1 💚	spotbugs	2m 17s	branch-2.3 passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 7s	the patch passed
+1 💚	checkstyle	1m 26s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	18m 12s	Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1.
+1 💚	spotbugs	2m 20s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		43m 23s

Subsystem	Report/Notes
Docker	Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1891/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1891
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux fd758a99d58b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2.3 / 66abc07
Max. process+thread count	94 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1891/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

[virajjasani]

Have you tried to set 'hbase.split.wal.zk.coordinated' to false to enable procedure based wal splitting? It will schedule sub procedures and remote procedures to do wal splitting, which will not block the SCP thread.

@Apache9 This option is for Procedures only setup right? For ZK based approach, we still need this patch right? Just trying to understand.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	7m 58s	Docker mode activated.
-0 ⚠️	yetus	0m 7s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ branch-2.3 Compile Tests _
+1 💚	mvninstall	5m 21s	branch-2.3 passed
+1 💚	compile	1m 16s	branch-2.3 passed
+1 💚	shadedjars	6m 34s	branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 46s	hbase-server in branch-2.3 failed.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 36s	the patch passed
+1 💚	compile	1m 11s	the patch passed
+1 💚	javac	1m 11s	the patch passed
+1 💚	shadedjars	6m 57s	patch has no errors when building our shaded downstream artifacts.
-0 ⚠️	javadoc	0m 53s	hbase-server in the patch failed.
		_ Other Tests _
-1 ❌	unit	212m 40s	hbase-server in the patch failed.
		250m 21s

Subsystem	Report/Notes
Docker	Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1891/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#1891
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 5f43c1dbae5a 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2.3 / 66abc07
Default Java	2020-01-14
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1891/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt
javadoc	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1891/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt
unit	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1891/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1891/1/testReport/
Max. process+thread count	2425 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1891/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	7m 57s	Docker mode activated.
-0 ⚠️	yetus	0m 5s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ branch-2.3 Compile Tests _
+1 💚	mvninstall	4m 34s	branch-2.3 passed
+1 💚	compile	1m 5s	branch-2.3 passed
+1 💚	shadedjars	5m 45s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	branch-2.3 passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 46s	the patch passed
+1 💚	compile	1m 1s	the patch passed
+1 💚	javac	1m 1s	the patch passed
+1 💚	shadedjars	5m 33s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	223m 10s	hbase-server in the patch passed.
		256m 28s

Subsystem	Report/Notes
Docker	Client=19.03.11 Server=19.03.11 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1891/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR	#1891
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 452b699a3e78 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2.3 / 66abc07
Default Java	1.8.0_232
Test Results	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1891/1/testReport/
Max. process+thread count	2326 (vs. ulimit of 12500)
modules	C: hbase-server U: hbase-server
Console output	https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1891/1/console
versions	git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)
Powered by	Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

[Apache9]

Have you tried to set 'hbase.split.wal.zk.coordinated' to false to enable procedure based wal splitting? It will schedule sub procedures and remote procedures to do wal splitting, which will not block the SCP thread.

@Apache9 This option is for Procedures only setup right? For ZK based approach, we still need this patch right? Just trying to understand.

Yes, exactly. We still need this for zk based split.

[saintstack]

Have you tried to set 'hbase.split.wal.zk.coordinated' to false to enable procedure based wal splitting?

No. Forgot it exists. Let me try it (on non-production system). Thanks.