HBASE-25736 Scan should be limited by read capacity unit quota if read size quota is not set #3127

mymeiyi · 2021-04-06T09:01:45Z

No description provided.

Apache-HBase · 2021-04-06T09:49:01Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 56s	master passed
+1 💚	compile	3m 7s	master passed
+1 💚	checkstyle	1m 5s	master passed
+1 💚	spotbugs	2m 6s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 37s	the patch passed
+1 💚	compile	3m 12s	the patch passed
+1 💚	javac	3m 12s	the patch passed
+1 💚	checkstyle	1m 2s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	17m 43s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 13s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		46m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3127
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux 3af459748d5a 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `048ca4e`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	96 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-04-06T11:51:42Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 27s	master passed
+1 💚	compile	1m 11s	master passed
+1 💚	shadedjars	7m 44s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 43s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 19s	the patch passed
+1 💚	compile	1m 14s	the patch passed
+1 💚	javac	1m 14s	the patch passed
+1 💚	shadedjars	7m 38s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 41s	the patch passed
		_ Other Tests _
+1 💚	unit	138m 19s	hbase-server in the patch passed.
		169m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3127
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux a7d788b1fc25 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `048ca4e`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/1/testReport/
Max. process+thread count	3622 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-04-06T13:07:37Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 12s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 20s	master passed
+1 💚	compile	1m 6s	master passed
+1 💚	shadedjars	8m 23s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 58s	the patch passed
+1 💚	compile	1m 3s	the patch passed
+1 💚	javac	1m 3s	the patch passed
+1 💚	shadedjars	8m 24s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	213m 25s	hbase-server in the patch passed.
		245m 1s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3127
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c974f381b638 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `048ca4e`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/1/testReport/
Max. process+thread count	3684 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…d size quota is not set

Apache-HBase · 2021-04-08T05:37:08Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 2s	master passed
+1 💚	compile	3m 19s	master passed
+1 💚	checkstyle	1m 2s	master passed
+1 💚	spotbugs	2m 6s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 35s	the patch passed
+1 💚	compile	3m 16s	the patch passed
+1 💚	javac	3m 16s	the patch passed
+1 💚	checkstyle	1m 2s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	21m 4s	Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚	spotbugs	2m 42s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		52m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#3127
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
uname	Linux d073e9dc323e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d9f4f41`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	96 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-04-08T07:37:35Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 42s	master passed
+1 💚	compile	1m 17s	master passed
+1 💚	shadedjars	7m 53s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 43s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 23s	the patch passed
+1 💚	compile	1m 17s	the patch passed
+1 💚	javac	1m 17s	the patch passed
+1 💚	shadedjars	7m 43s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 41s	the patch passed
		_ Other Tests _
-1 ❌	unit	141m 21s	hbase-server in the patch failed.
		172m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#3127
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 7a6687f44338 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d9f4f41`
Default Java	AdoptOpenJDK-11.0.10+9
unit	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/2/testReport/
Max. process+thread count	3921 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2021-04-08T08:51:34Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 10s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 2s	master passed
+1 💚	compile	1m 3s	master passed
+1 💚	shadedjars	8m 22s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 59s	the patch passed
+1 💚	compile	1m 2s	the patch passed
+1 💚	javac	1m 2s	the patch passed
+1 💚	shadedjars	8m 22s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	215m 29s	hbase-server in the patch passed.
		246m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#3127
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 70c080758241 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d9f4f41`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/2/testReport/
Max. process+thread count	3358 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3127/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2021-04-15T07:03:36Z

hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/DefaultOperationQuota.java

@@ -122,6 +126,9 @@ public void close() {

  @Override
  public long getReadAvailable() {
+    if (readAvailable == Long.MAX_VALUE && readCapacityUnitAvailable != Long.MAX_VALUE) {


Mind explaing a bit more here? It will also effect the quota for operations other than scan?

Scan use this value to limit the result size:
long maxQuotaResultSize = Math.min(maxScannerResultSize, quota.getReadAvailable()); long maxResultSize; if (scanner.getMaxResultSize() > 0) { maxResultSize = Math.min(scanner.getMaxResultSize(), maxQuotaResultSize); } else { maxResultSize = maxQuotaResultSize; }

Now it only considers the 'read size' type quota, and if it is not set, the scan can not be limited. The 'read capacity unit' is also a type of size quota, we can use it to calculate the available read size.

Now only scan use it.

HBASE-25736 Scan should be limited by read capacity unit quota if rea…

59a6513

…d size quota is not set

mymeiyi force-pushed the quota2 branch from e0b22f5 to 59a6513 Compare April 8, 2021 04:44

Apache9 requested changes Apr 15, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-25736 Scan should be limited by read capacity unit quota if read size quota is not set #3127

HBASE-25736 Scan should be limited by read capacity unit quota if read size quota is not set #3127

mymeiyi commented Apr 6, 2021

Apache-HBase commented Apr 6, 2021

Apache-HBase commented Apr 6, 2021

Apache-HBase commented Apr 6, 2021

Apache-HBase commented Apr 8, 2021

Apache-HBase commented Apr 8, 2021

Apache-HBase commented Apr 8, 2021

Apache9 Apr 15, 2021

mymeiyi Apr 15, 2021

mymeiyi Apr 15, 2021

HBASE-25736 Scan should be limited by read capacity unit quota if read size quota is not set #3127

Are you sure you want to change the base?

HBASE-25736 Scan should be limited by read capacity unit quota if read size quota is not set #3127

Conversation

mymeiyi commented Apr 6, 2021

Apache-HBase commented Apr 6, 2021

Apache-HBase commented Apr 6, 2021

Apache-HBase commented Apr 6, 2021

Apache-HBase commented Apr 8, 2021

Apache-HBase commented Apr 8, 2021

Apache-HBase commented Apr 8, 2021

Apache9 Apr 15, 2021

Choose a reason for hiding this comment

mymeiyi Apr 15, 2021

Choose a reason for hiding this comment

mymeiyi Apr 15, 2021

Choose a reason for hiding this comment