HBASE-27166 WAL value compression minor improvements #4584

apurtell · 2022-06-27T22:36:04Z

A larger IO buffer for absorbing WALCodec writes can improve the compression ratio of larger values, because the compressor will be given a larger internal buffer over which there will be more match opportunities. Does not impact the ability to read existing written files.

Also, reset the BAOS internal buffer on the way out of compress() so potential large-ish buffers do not linger on the heap longer than necessary.

A larger IO buffer for absorbing WALCodec writes can improve the compression ratio of larger values, because the compressor will be given a larger internal buffer over which there will be more match opportunities. Does not impact the ability to read existing written files. Also, reset the BAOS internal buffer on the way out of compress() so potential large-ish buffers do not linger on the heap longer than necessary.

Apache-HBase · 2022-06-27T23:23:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 7s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 32s	master passed
+1 💚	compile	2m 14s	master passed
+1 💚	checkstyle	0m 33s	master passed
+1 💚	spotless	0m 44s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 18s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 13s	the patch passed
+1 💚	compile	2m 13s	the patch passed
+1 💚	javac	2m 13s	the patch passed
+1 💚	checkstyle	0m 31s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	11m 37s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotless	0m 40s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 24s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 11s	The patch does not generate ASF License warnings.
		32m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4584/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4584
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux cb58127ba3d3 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d7f6861`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4584/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-06-28T02:22:50Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 53s	master passed
+1 💚	compile	0m 41s	master passed
+1 💚	shadedjars	3m 49s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 27s	the patch passed
+1 💚	compile	0m 41s	the patch passed
+1 💚	javac	0m 41s	the patch passed
+1 💚	shadedjars	3m 46s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	193m 44s	hbase-server in the patch passed.
		211m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4584/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4584
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux a3698bfc2a25 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d7f6861`
Default Java	AdoptOpenJDK-11.0.10+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4584/1/testReport/
Max. process+thread count	2437 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4584/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-06-28T03:43:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 22s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 43s	master passed
+1 💚	compile	0m 43s	master passed
+1 💚	shadedjars	5m 18s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 20s	the patch passed
+1 💚	compile	0m 55s	the patch passed
+1 💚	javac	0m 55s	the patch passed
+1 💚	shadedjars	5m 32s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 32s	the patch passed
		_ Other Tests _
-1 ❌	unit	267m 10s	hbase-server in the patch failed.
		292m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4584/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4584
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 8594e154f38b 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d7f6861`
Default Java	AdoptOpenJDK-1.8.0_282-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4584/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4584/1/testReport/
Max. process+thread count	2442 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4584/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

A larger IO buffer for absorbing WALCodec writes can improve the compression ratio of larger values, because the compressor will be given a larger internal buffer over which there will be more match opportunities. Does not impact the ability to read existing written files. Also, reset the BAOS internal buffer on the way out of compress() so potential large-ish buffers do not linger on the heap longer than necessary. Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 approved these changes Jun 28, 2022

View reviewed changes

apurtell merged commit ca34bdc into apache:master Jun 28, 2022

apurtell deleted the HBASE-27166 branch June 28, 2022 19:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-27166 WAL value compression minor improvements #4584

HBASE-27166 WAL value compression minor improvements #4584

apurtell commented Jun 27, 2022

Apache-HBase commented Jun 27, 2022

Apache-HBase commented Jun 28, 2022

Apache-HBase commented Jun 28, 2022

HBASE-27166 WAL value compression minor improvements #4584

HBASE-27166 WAL value compression minor improvements #4584

Conversation

apurtell commented Jun 27, 2022

Apache-HBase commented Jun 27, 2022

Apache-HBase commented Jun 28, 2022

Apache-HBase commented Jun 28, 2022