HBASE-27750: Update the list of prefetched Hfiles upon block eviction #5140

Kota-SH · 2023-03-24T15:36:34Z

Currently, we maintain a list of Hfiles on disk for which prefetch is complete and avoid prefetching those files after a restart. But we don't handle cases where blocks are evicted from the cache. This ticket is for updating the list of prefetched files upon simple block eviction.

Jira: https://issues.apache.org/jira/browse/HBASE-27750

wchevreuil · 2023-03-24T17:22:29Z

...e-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCachePersister.java

@@ -119,21 +120,36 @@ public void testPrefetchPersistenceCrash() throws Exception {

  @Test
  public void testPrefetchPersistenceCrashNegative() throws Exception {
-    long bucketCachePersistInterval = 3000;
+    long bucketCachePersistInterval = 5000;


Actually, we should make BucketCachePersister a daemon thread. Then we could set this interval to Long.MAX_VALUE.

Apache-HBase · 2023-03-24T18:06:39Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 19s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 32s	master passed
+1 💚	compile	2m 25s	master passed
+1 💚	checkstyle	0m 37s	master passed
+1 💚	spotless	0m 43s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 29s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 18s	the patch passed
+1 💚	compile	2m 25s	the patch passed
+1 💚	javac	2m 25s	the patch passed
+1 💚	checkstyle	0m 35s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 22s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 34s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 11s	The patch does not generate ASF License warnings.
		37m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5140
JIRA Issue	HBASE-27750
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 806cb8aea9a2 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `735fb43`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-03-24T21:18:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 25s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 7s	master passed
+1 💚	compile	0m 35s	master passed
+1 💚	shadedjars	4m 39s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 45s	the patch passed
+1 💚	compile	0m 34s	the patch passed
+1 💚	javac	0m 34s	the patch passed
+1 💚	shadedjars	4m 35s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	the patch passed
		_ Other Tests _
+1 💚	unit	207m 33s	hbase-server in the patch passed.
		229m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5140
JIRA Issue	HBASE-27750
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 9095a5a8991c 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `735fb43`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/1/testReport/
Max. process+thread count	2677 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-03-24T21:26:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 53s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 31s	master passed
+1 💚	compile	0m 46s	master passed
+1 💚	shadedjars	4m 25s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 21s	the patch passed
+1 💚	compile	0m 46s	the patch passed
+1 💚	javac	0m 46s	the patch passed
+1 💚	shadedjars	4m 19s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	214m 3s	hbase-server in the patch failed.
		237m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5140
JIRA Issue	HBASE-27750
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 00377f6ba339 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `735fb43`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/1/testReport/
Max. process+thread count	2868 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-03-24T22:06:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 59s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 27s	master passed
+1 💚	compile	2m 23s	master passed
+1 💚	checkstyle	0m 36s	master passed
+1 💚	spotless	0m 43s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 29s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 18s	the patch passed
+1 💚	compile	2m 25s	the patch passed
+1 💚	javac	2m 25s	the patch passed
+1 💚	checkstyle	0m 36s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 19s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	0m 53s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 43s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		39m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5140
JIRA Issue	HBASE-27750
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux cf5f4d0ae60a 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `735fb43`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

wchevreuil · 2023-03-24T23:02:35Z

Latest UT failure seems unrelated to me, but have re-triggered the pre-commit job.

Apache-HBase · 2023-03-25T01:17:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 23s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 44s	master passed
+1 💚	compile	0m 35s	master passed
+1 💚	shadedjars	4m 36s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 44s	the patch passed
+1 💚	compile	0m 34s	the patch passed
+1 💚	javac	0m 34s	the patch passed
+1 💚	shadedjars	4m 36s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 20s	the patch passed
		_ Other Tests _
+1 💚	unit	208m 35s	hbase-server in the patch passed.
		229m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5140
JIRA Issue	HBASE-27750
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c5a658545b9c 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `735fb43`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/2/testReport/
Max. process+thread count	2654 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-03-25T01:23:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 51s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 13s	master passed
+1 💚	compile	0m 47s	master passed
+1 💚	shadedjars	4m 21s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 16s	the patch passed
+1 💚	compile	0m 47s	the patch passed
+1 💚	javac	0m 47s	the patch passed
+1 💚	shadedjars	4m 20s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	the patch passed
		_ Other Tests _
+1 💚	unit	213m 41s	hbase-server in the patch passed.
		236m 18s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5140
JIRA Issue	HBASE-27750
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 333fecaba57f 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `735fb43`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/2/testReport/
Max. process+thread count	2473 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5140/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…#5140) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>

…les upon block eviction (apache#5140) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org> Change-Id: Ifdfa755a800c1c926403b90ec991d7d2e9f7da38

Kota-SH force-pushed the hbase-27743 branch from 5a37b66 to ae4941e Compare March 24, 2023 16:34

wchevreuil reviewed Mar 24, 2023

View reviewed changes

HBASE-27750: Update the list of prefetched Hfiles upon block eviction

0e30312

Kota-SH force-pushed the hbase-27743 branch from ae4941e to 0e30312 Compare March 24, 2023 18:13

Kota-SH requested a review from wchevreuil March 24, 2023 18:16

wchevreuil approved these changes Mar 24, 2023

View reviewed changes

wchevreuil merged commit a6dad70 into apache:master Mar 27, 2023

wchevreuil pushed a commit that referenced this pull request Mar 27, 2023

HBASE-27750: Update the list of prefetched Hfiles upon block eviction (…

61e18a3

…#5140) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-27750: Update the list of prefetched Hfiles upon block eviction #5140

HBASE-27750: Update the list of prefetched Hfiles upon block eviction #5140

Kota-SH commented Mar 24, 2023

wchevreuil Mar 24, 2023

Apache-HBase commented Mar 24, 2023

Apache-HBase commented Mar 24, 2023

Apache-HBase commented Mar 24, 2023

Apache-HBase commented Mar 24, 2023

wchevreuil commented Mar 24, 2023

Apache-HBase commented Mar 25, 2023

Apache-HBase commented Mar 25, 2023

HBASE-27750: Update the list of prefetched Hfiles upon block eviction #5140

HBASE-27750: Update the list of prefetched Hfiles upon block eviction #5140

Conversation

Kota-SH commented Mar 24, 2023

wchevreuil Mar 24, 2023

Choose a reason for hiding this comment

Apache-HBase commented Mar 24, 2023

Apache-HBase commented Mar 24, 2023

Apache-HBase commented Mar 24, 2023

Apache-HBase commented Mar 24, 2023

wchevreuil commented Mar 24, 2023

Apache-HBase commented Mar 25, 2023

Apache-HBase commented Mar 25, 2023