HBASE-25680 Non-idempotent test in TestReplicationHFileCleaner #2984

lzx404243 · 2021-02-25T06:56:25Z

The test org.apache.hadoop.hbase.master.cleaner.TestReplicationHFileCleaner.testIsFileDeletable is not idempotent and fail if run twice in the same JVM, because it pollutes some states shared among tests. It may be good to clean this state pollution so that some other tests do not fail in the future due to the shared state polluted by this test.

Detail

Running TestReplicationHFileCleaner.testIsFileDeletable twice would result in the second run failing due to the following assertion error:

java.lang.AssertionError: Cleaner should allow to delete this file as there is no hfile reference node for it in the queue.

The root cause is that the a hfile reference is added during the first test run, which doesn't get removed upon test exits. Therefore, in the second test run , cleaner.isFileDeletable(fs.getFileStatus(file))) would return false, resulting in the assertion error.

The suggested fix is to remove the hfile reference created during the test when the test exits.

With the proposed fix, the test does not pollute the shared state (and passes when run twice in the same JVM).

Jira link: https://issues.apache.org/jira/browse/HBASE-25680

…able

Apache-HBase · 2021-02-25T07:42:54Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 40s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 57s	master passed
+1 💚	compile	3m 14s	master passed
+1 💚	checkstyle	1m 5s	master passed
+1 💚	spotbugs	2m 2s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 26s	the patch passed
+1 💚	compile	3m 8s	the patch passed
+1 💚	javac	3m 8s	the patch passed
+1 💚	checkstyle	1m 2s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	17m 11s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 9s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		45m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2984
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 2507c611c2ac 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `ed2693f`
Default Java	AdoptOpenJDK-1.8.0_232-b09
Max. process+thread count	95 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-02-25T09:48:48Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 30s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 54s	master passed
+1 💚	compile	0m 58s	master passed
+1 💚	shadedjars	6m 33s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 39s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 30s	the patch passed
+1 💚	compile	0m 58s	the patch passed
+1 💚	javac	0m 58s	the patch passed
+1 💚	shadedjars	6m 41s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	the patch passed
		_ Other Tests _
+1 💚	unit	145m 4s	hbase-server in the patch passed.
		171m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2984
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 453dc5b61060 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `ed2693f`
Default Java	AdoptOpenJDK-1.8.0_232-b09
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/1/testReport/
Max. process+thread count	4852 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-02-25T10:54:39Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 18s	Docker mode activated.
-0 ⚠️	yetus	0m 9s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 56s	master passed
+1 💚	compile	1m 13s	master passed
+1 💚	shadedjars	7m 27s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 44s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 39s	the patch passed
+1 💚	compile	1m 13s	the patch passed
+1 💚	javac	1m 13s	the patch passed
+1 💚	shadedjars	7m 21s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 43s	the patch passed
		_ Other Tests _
+1 💚	unit	204m 43s	hbase-server in the patch passed.
		237m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2984
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 25a4b692817f 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `ed2693f`
Default Java	AdoptOpenJDK-11.0.6+10
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/1/testReport/
Max. process+thread count	3350 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

...server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java

saintstack · 2021-03-15T22:41:43Z

Is there a JIRA that goes w/ this PR? Thanks.

saintstack · 2021-03-18T19:30:20Z

@lzx404243 Please see above comments. Thanks.

lzx404243 · 2021-03-19T03:25:08Z

@saintstack I've created a jira issue for this: https://issues.apache.org/jira/browse/HBASE-25680
Thanks!

saintstack · 2021-03-19T03:28:14Z

Thanks @lzx404243 I fixed the subject so it points to your new JIRA. Did you see @Apache9 's comment above? Thanks

lzx404243 · 2021-07-07T02:31:11Z

Thanks @saintstack for updating the subject.

@Apache9 Please let me know what you think of my reply to your comment (See above). Thanks!

Apache9 · 2021-07-07T06:24:00Z

Thanks @saintstack for updating the subject.

@Apache9 Please let me know what you think of my reply to your comment (See above). Thanks!

Sorry, missed your previous reply. It will be good to remove all HFileRefs in tearDown.

Apache-HBase · 2021-07-08T04:05:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	4m 0s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 59s	master passed
+1 💚	compile	3m 6s	master passed
+1 💚	checkstyle	1m 3s	master passed
+1 💚	spotbugs	2m 4s	master passed
-0 ⚠️	patch	2m 13s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 39s	the patch passed
+1 💚	compile	3m 6s	the patch passed
+1 💚	javac	3m 6s	the patch passed
+1 💚	checkstyle	1m 1s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	18m 8s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 13s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		50m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#2984
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 29b9120f46c3 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e65fc92`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	96 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-07-08T06:11:43Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 57s	master passed
+1 💚	compile	1m 1s	master passed
+1 💚	shadedjars	8m 3s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 41s	master passed
-0 ⚠️	patch	8m 55s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 43s	the patch passed
+1 💚	compile	1m 2s	the patch passed
+1 💚	javac	1m 2s	the patch passed
+1 💚	shadedjars	8m 4s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	145m 59s	hbase-server in the patch passed.
		176m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#2984
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 438c1052b087 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e65fc92`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/2/testReport/
Max. process+thread count	5039 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-07-08T07:11:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 7s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	5m 6s	master passed
+1 💚	compile	1m 21s	master passed
+1 💚	shadedjars	9m 6s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 45s	master passed
-0 ⚠️	patch	10m 1s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 47s	the patch passed
+1 💚	compile	1m 17s	the patch passed
+1 💚	javac	1m 17s	the patch passed
+1 💚	shadedjars	9m 18s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 41s	the patch passed
		_ Other Tests _
+1 💚	unit	201m 48s	hbase-server in the patch passed.
		237m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#2984
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 184f8e841a84 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e65fc92`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/2/testReport/
Max. process+thread count	3161 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2984/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

lzx404243 · 2021-08-14T01:54:55Z

Thanks @Apache9 for the approval! Is there anything I can do before this can be merged? Thanks!

Apache9 · 2021-08-14T02:15:40Z

Oh, sorry... Will merge soon.

Thanks for the reminding~

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Remove HFile reference in TestReplicationHFileCleaner.testIsFileDelet…

d2bbd5b

…able

Apache9 reviewed Mar 3, 2021

View reviewed changes

...server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java Outdated Show resolved Hide resolved

saintstack changed the title ~~Remove HFile reference in TestReplicationHFileCleaner.testIsFileDeletable~~ HBASE-25680 Non-idempotent test in TestReplicationHFileCleaner Mar 19, 2021

Remove HFileRefs after each test

4afb81a

Apache9 approved these changes Jul 8, 2021

View reviewed changes

Apache9 merged commit 44d5624 into apache:master Aug 14, 2021

Apache9 pushed a commit that referenced this pull request Aug 14, 2021

HBASE-25680 Non-idempotent test in TestReplicationHFileCleaner (#2984)

b34d0cd

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Aug 14, 2021

HBASE-25680 Non-idempotent test in TestReplicationHFileCleaner (#2984)

627e0d6

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Aug 14, 2021

HBASE-25680 Non-idempotent test in TestReplicationHFileCleaner (#2984)

4aec407

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-25680 Non-idempotent test in TestReplicationHFileCleaner #2984

HBASE-25680 Non-idempotent test in TestReplicationHFileCleaner #2984

lzx404243 commented Feb 25, 2021 •

edited

Loading

Apache-HBase commented Feb 25, 2021

Apache-HBase commented Feb 25, 2021

Apache-HBase commented Feb 25, 2021

saintstack commented Mar 15, 2021

saintstack commented Mar 18, 2021

lzx404243 commented Mar 19, 2021

saintstack commented Mar 19, 2021

lzx404243 commented Jul 7, 2021

Apache9 commented Jul 7, 2021

Apache-HBase commented Jul 8, 2021

Apache-HBase commented Jul 8, 2021

Apache-HBase commented Jul 8, 2021

lzx404243 commented Aug 14, 2021

Apache9 commented Aug 14, 2021

HBASE-25680 Non-idempotent test in TestReplicationHFileCleaner #2984

HBASE-25680 Non-idempotent test in TestReplicationHFileCleaner #2984

Conversation

lzx404243 commented Feb 25, 2021 • edited Loading

Detail

Apache-HBase commented Feb 25, 2021

Apache-HBase commented Feb 25, 2021

Apache-HBase commented Feb 25, 2021

saintstack commented Mar 15, 2021

saintstack commented Mar 18, 2021

lzx404243 commented Mar 19, 2021

saintstack commented Mar 19, 2021

lzx404243 commented Jul 7, 2021

Apache9 commented Jul 7, 2021

Apache-HBase commented Jul 8, 2021

Apache-HBase commented Jul 8, 2021

Apache-HBase commented Jul 8, 2021

lzx404243 commented Aug 14, 2021

Apache9 commented Aug 14, 2021

lzx404243 commented Feb 25, 2021 •

edited

Loading