HBASE-25459 WAL can't be cleaned with phoenix5

[leyangyueshan]

No description provided.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 29s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 22s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 39s	master passed
+1 💚	checkstyle	1m 30s	master passed
+1 💚	spotbugs	2m 41s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 28s	the patch passed
-0 ⚠️	checkstyle	0m 24s	hbase-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-0 ⚠️	checkstyle	1m 4s	hbase-server: The patch generated 4 new + 11 unchanged - 0 fixed = 15 total (was 11)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	17m 26s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
-1 ❌	spotbugs	2m 8s	hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
+1 💚	asflicense	0m 26s	The patch does not generate ASF License warnings.
		44m 20s

Reason	Tests
FindBugs	module:hbase-server
	org.apache.hadoop.hbase.regionserver.wal.SequenceIdAccounting.getLowestSequenceIdWithPhoenix(Map) makes inefficient use of keySet iterator instead of entrySet iterator At SequenceIdAccounting.java:keySet iterator instead of entrySet iterator At SequenceIdAccounting.java:[line 274]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2842
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux b815ca370a43 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 8ecfced
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-common.txt
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
spotbugs	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html
Max. process+thread count	94 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[wchevreuil]

I don't think we should add logic dealing with the specifics of a 3rd party project (phoenix) into hbase wal handling.

What about changing HBase WALFactory to allow pluggable custom wal providers implementations? Then this phoenix specific logic could be implemented on a Phoenix wal provider, keeping hbase agnostic to phoenix. AbstractFSWALProvider/WALProvider interfaces are all marked as IA Private, though, so it would be challenging maintain compatibility of custom wal providers.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 35s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 27s	Maven dependency ordering for branch
+1 💚	mvninstall	4m 54s	master passed
+1 💚	compile	1m 37s	master passed
+1 💚	shadedjars	7m 23s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 3s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 16s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 36s	the patch passed
+1 💚	compile	1m 23s	the patch passed
+1 💚	javac	1m 23s	the patch passed
+1 💚	shadedjars	6m 38s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 58s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 33s	hbase-common in the patch passed.
-1 ❌	unit	17m 1s	hbase-server in the patch failed.
		50m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2842
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 206187575ab4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 8ecfced
Default Java	AdoptOpenJDK-1.8.0_232-b09
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/1/testReport/
Max. process+thread count	794 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 5s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 28s	Maven dependency ordering for branch
+1 💚	mvninstall	6m 28s	master passed
+1 💚	compile	2m 8s	master passed
+1 💚	shadedjars	9m 11s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 39s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 18s	Maven dependency ordering for patch
+1 💚	mvninstall	6m 25s	the patch passed
+1 💚	compile	2m 12s	the patch passed
+1 💚	javac	2m 12s	the patch passed
+1 💚	shadedjars	9m 38s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 29s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 23s	hbase-common in the patch passed.
-1 ❌	unit	22m 23s	hbase-server in the patch failed.
		69m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2842
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c5af760d1714 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 8ecfced
Default Java	AdoptOpenJDK-11.0.6+10
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/1/testReport/
Max. process+thread count	425 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 22s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 41s	master passed
+1 💚	checkstyle	1m 28s	master passed
+1 💚	spotbugs	2m 44s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 25s	the patch passed
-0 ⚠️	checkstyle	0m 24s	hbase-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-0 ⚠️	checkstyle	1m 2s	hbase-server: The patch generated 4 new + 11 unchanged - 0 fixed = 15 total (was 11)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	17m 11s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
-1 ❌	spotbugs	2m 9s	hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
+1 💚	asflicense	0m 26s	The patch does not generate ASF License warnings.
		41m 52s

Reason	Tests
FindBugs	module:hbase-server
	org.apache.hadoop.hbase.regionserver.wal.SequenceIdAccounting.getLowestSequenceIdWithPhoenix(Map) makes inefficient use of keySet iterator instead of entrySet iterator At SequenceIdAccounting.java:keySet iterator instead of entrySet iterator At SequenceIdAccounting.java:[line 274]

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2842
Optional Tests	dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle
uname	Linux a7715e0796d6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 8ecfced
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-common.txt
checkstyle	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
spotbugs	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/2/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html
Max. process+thread count	94 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 14s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 35s	master passed
+1 💚	compile	1m 20s	master passed
+1 💚	shadedjars	6m 50s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 56s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 16s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 39s	the patch passed
+1 💚	compile	1m 19s	the patch passed
+1 💚	javac	1m 19s	the patch passed
+1 💚	shadedjars	6m 47s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 59s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 40s	hbase-common in the patch passed.
-1 ❌	unit	17m 28s	hbase-server in the patch failed.
		47m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2842
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 0e80dfa78e69 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 8ecfced
Default Java	AdoptOpenJDK-1.8.0_232-b09
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/2/testReport/
Max. process+thread count	788 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 2s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 17s	Maven dependency ordering for branch
+1 💚	mvninstall	6m 38s	master passed
+1 💚	compile	2m 52s	master passed
+1 💚	shadedjars	9m 24s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 31s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 20s	Maven dependency ordering for patch
+1 💚	mvninstall	6m 24s	the patch passed
+1 💚	compile	2m 18s	the patch passed
+1 💚	javac	2m 18s	the patch passed
+1 💚	shadedjars	9m 25s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 18s	hbase-common in the patch passed.
-1 ❌	unit	22m 6s	hbase-server in the patch failed.
		69m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2842
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux e5a742c7fe80 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 8ecfced
Default Java	AdoptOpenJDK-11.0.6+10
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/2/testReport/
Max. process+thread count	424 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2842/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[leyangyueshan]

I don't think we should add logic dealing with the specifics of a 3rd party project (phoenix) into hbase wal handling.

What about changing HBase WALFactory to allow pluggable custom wal providers implementations? Then this phoenix specific logic could be implemented on a Phoenix wal provider, keeping hbase agnostic to phoenix. AbstractFSWALProvider/WALProvider interfaces are all marked as IA Private, though, so it would be challenging maintain compatibility of custom wal providers.

I think you are right! I will close this pr!

[leyangyueshan]

• [ ]