HBASE-28204 : Region Canary can take lot more time If any region (except the first region) starts with delete markers #5675

mihir6692 · 2024-02-10T04:11:56Z

… first region) starts with delete markers

virajjasani

Left minor nits, +1 otherwise. Really nice improvement!

virajjasani · 2024-02-10T04:24:20Z

hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java

+        // regions in sequence until it can find first available row.
+        //
+        // This could result in multiple millions of scans based on the size of table and number of
+        // empty regions in sequence. In test environment, A table no data and 1000 empty regions,


nit: A table with no data

virajjasani · 2024-02-10T04:26:07Z

hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java

      } else {
        scan = new Scan();
+        // In case of first region of the HBase Table, we do not have start-key for the region.
+        // For Region Canary, we only need scan a single row/cell in the region to make sure that


nit: we only need to scan

Apache-HBase · 2024-02-10T04:48:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 1s	master passed
+1 💚	compile	2m 33s	master passed
+1 💚	checkstyle	0m 38s	master passed
+1 💚	spotless	0m 44s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 33s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 44s	the patch passed
+1 💚	compile	2m 27s	the patch passed
+1 💚	javac	2m 27s	the patch passed
+1 💚	checkstyle	0m 36s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	4m 49s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 38s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		28m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5675
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux c623837cefdc 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b3ffc4a`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	82 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

d-c-manning

At least this change does not change any behavior of the Canary, other than avoiding unnecessary scans for empty tables/regions, and avoiding a Get of a very large deleted row, if it exists. Seems to be a strict improvement in that sense.

If anyone was relying on the RPCs being Gets, by using separate RPC handlers for Gets vs. Scans, this will change that behavior. But that seems like weird behavior to rely on.

+1

d-c-manning · 2024-02-10T05:42:33Z

hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java

+        // do canary run for the table.
+        //
+        // Since First region of the table doesn't have any start key, We should set End Key as
+        // stop row and set inclusive=false to limit scan to single region only.


We could even add a comment that we think this would be the better approach for all regions (not only the first region of the table), but who knows what performance people may expect from the canary, and whether they are counting on most Get requests (now Scans) not returning any data, and therefore performing extra fast (?)

Added some more comments as well as TODO for future.

Apache-HBase · 2024-02-10T10:06:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 33s	master passed
+1 💚	compile	1m 6s	master passed
+1 💚	shadedjars	6m 42s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 54s	the patch passed
+1 💚	compile	0m 59s	the patch passed
+1 💚	javac	0m 59s	the patch passed
+1 💚	shadedjars	6m 36s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	315m 48s	hbase-server in the patch passed.
		346m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5675
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 5f9debf74642 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b3ffc4a`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/testReport/
Max. process+thread count	4623 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-02-10T10:18:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 57s	master passed
+1 💚	compile	0m 50s	master passed
+1 💚	shadedjars	6m 32s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 20s	the patch passed
+1 💚	compile	0m 59s	the patch passed
+1 💚	javac	0m 59s	the patch passed
+1 💚	shadedjars	6m 17s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	the patch passed
		_ Other Tests _
-1 ❌	unit	330m 19s	hbase-server in the patch failed.
		358m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5675
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 8262b6cf7abc 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b3ffc4a`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/testReport/
Max. process+thread count	4416 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

mihir6692 · 2024-02-10T18:15:25Z

At least this change does not change any behavior of the Canary, other than avoiding unnecessary scans for empty tables/regions, and avoiding a Get of a very large deleted row, if it exists. Seems to be a strict improvement in that sense.

If anyone was relying on the RPCs being Gets, by using separate RPC handlers for Gets vs. Scans, this will change that behavior. But that seems like weird behavior to rely on.

@d-c-manning 's suggestion is valid. Should we include this as JIRA release note? So it's documented for every branch that has this JIRA? @virajjasani

virajjasani · 2024-02-10T18:48:17Z

At least this change does not change any behavior of the Canary, other than avoiding unnecessary scans for empty tables/regions, and avoiding a Get of a very large deleted row, if it exists. Seems to be a strict improvement in that sense.
If anyone was relying on the RPCs being Gets, by using separate RPC handlers for Gets vs. Scans, this will change that behavior. But that seems like weird behavior to rely on.

@d-c-manning 's suggestion is valid. Should we include this as JIRA release note? So it's documented for every branch that has this JIRA? @virajjasani

Sounds good, feel free to update release note on the Jira.

Apache-HBase · 2024-02-10T18:48:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 24s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 5s	master passed
+1 💚	compile	2m 27s	master passed
+1 💚	checkstyle	0m 32s	master passed
+1 💚	spotless	0m 41s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 42s	the patch passed
+1 💚	compile	2m 23s	the patch passed
+1 💚	javac	2m 23s	the patch passed
+1 💚	checkstyle	0m 34s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	5m 3s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 38s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 32s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 9s	The patch does not generate ASF License warnings.
		27m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5675
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 85380ad2a503 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b3ffc4a`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	77 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

mihir6692 · 2024-02-10T20:10:42Z

Updated Release note on the JIRA @virajjasani

Apache-HBase · 2024-02-10T22:28:54Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 29s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 46s	master passed
+1 💚	compile	0m 38s	master passed
+1 💚	shadedjars	5m 33s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 25s	the patch passed
+1 💚	compile	0m 38s	the patch passed
+1 💚	javac	0m 38s	the patch passed
+1 💚	shadedjars	5m 30s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	the patch passed
		_ Other Tests _
+1 💚	unit	225m 51s	hbase-server in the patch passed.
		248m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5675
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 3354719fc5f1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b3ffc4a`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/testReport/
Max. process+thread count	5597 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-02-10T22:34:02Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 2s	master passed
+1 💚	compile	0m 48s	master passed
+1 💚	shadedjars	5m 9s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 51s	the patch passed
+1 💚	compile	0m 48s	the patch passed
+1 💚	javac	0m 48s	the patch passed
+1 💚	shadedjars	5m 8s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	229m 38s	hbase-server in the patch passed.
		253m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5675
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 4e4140ac47df 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `b3ffc4a`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/testReport/
Max. process+thread count	5134 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5675/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…ept the first region) starts with delete markers (#5675) Signed-off-by: David Manning <david.manning@salesforce.com> Signed-off-by: Viraj Jasani <vjasani@apache.org>

HBASE-28204 : Canary can take lot more time If any region (except the…

58dcee0

… first region) starts with delete markers

virajjasani approved these changes Feb 10, 2024

View reviewed changes

d-c-manning approved these changes Feb 10, 2024

View reviewed changes

Added more comments

222b34f

mihir6692 changed the title ~~HBASE-28204 : Canary can take lot more time If any region (except the first region) starts with delete markers~~ HBASE-28204 : Region Canary can take lot more time If any region (except the first region) starts with delete markers Feb 10, 2024

virajjasani merged commit 3e281bd into apache:master Feb 12, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-28204 : Region Canary can take lot more time If any region (except the first region) starts with delete markers #5675

HBASE-28204 : Region Canary can take lot more time If any region (except the first region) starts with delete markers #5675

mihir6692 commented Feb 10, 2024 •

edited by virajjasani

Loading

virajjasani left a comment

virajjasani Feb 10, 2024

mihir6692 Feb 10, 2024

virajjasani Feb 10, 2024

mihir6692 Feb 10, 2024

Apache-HBase commented Feb 10, 2024

d-c-manning left a comment •

edited

Loading

d-c-manning Feb 10, 2024

mihir6692 Feb 10, 2024

Apache-HBase commented Feb 10, 2024

Apache-HBase commented Feb 10, 2024

mihir6692 commented Feb 10, 2024 •

edited

Loading

virajjasani commented Feb 10, 2024

Apache-HBase commented Feb 10, 2024

mihir6692 commented Feb 10, 2024

Apache-HBase commented Feb 10, 2024

Apache-HBase commented Feb 10, 2024

HBASE-28204 : Region Canary can take lot more time If any region (except the first region) starts with delete markers #5675

HBASE-28204 : Region Canary can take lot more time If any region (except the first region) starts with delete markers #5675

Conversation

mihir6692 commented Feb 10, 2024 • edited by virajjasani Loading

virajjasani left a comment

Choose a reason for hiding this comment

virajjasani Feb 10, 2024

Choose a reason for hiding this comment

mihir6692 Feb 10, 2024

Choose a reason for hiding this comment

virajjasani Feb 10, 2024

Choose a reason for hiding this comment

mihir6692 Feb 10, 2024

Choose a reason for hiding this comment

Apache-HBase commented Feb 10, 2024

d-c-manning left a comment • edited Loading

Choose a reason for hiding this comment

d-c-manning Feb 10, 2024

Choose a reason for hiding this comment

mihir6692 Feb 10, 2024

Choose a reason for hiding this comment

Apache-HBase commented Feb 10, 2024

Apache-HBase commented Feb 10, 2024

mihir6692 commented Feb 10, 2024 • edited Loading

virajjasani commented Feb 10, 2024

Apache-HBase commented Feb 10, 2024

mihir6692 commented Feb 10, 2024

Apache-HBase commented Feb 10, 2024

Apache-HBase commented Feb 10, 2024

mihir6692 commented Feb 10, 2024 •

edited by virajjasani

Loading

d-c-manning left a comment •

edited

Loading

mihir6692 commented Feb 10, 2024 •

edited

Loading