use lz4 instead of gzip for StringEncoding compress/decompress #1196

zhoney · 2020-09-28T12:22:02Z

No description provided.

Change-Id: Ie4458ba481ac3625139458cc92e9dbc0631898da

Change-Id: I39cbfe2f643b46ca736e5076ef73ec6203b0be3a

codecov · 2020-09-28T13:56:24Z

Codecov Report

Merging #1196 into master will increase coverage by 0.01%.
The diff coverage is 71.42%.

@@             Coverage Diff              @@
##             master    #1196      +/-   ##
============================================
+ Coverage     65.75%   65.76%   +0.01%     
+ Complexity     5754     5753       -1     
============================================
  Files           361      360       -1     
  Lines         29548    29593      +45     
  Branches       4172     4180       +8     
============================================
+ Hits          19428    19463      +35     
- Misses         8147     8152       +5     
- Partials       1973     1978       +5

Impacted Files	Coverage Δ	Complexity Δ
...aidu/hugegraph/backend/serializer/BytesBuffer.java	`87.75% <60.00%> (-0.18%)`	`175.00 <3.00> (ø)`
...rc/main/java/com/baidu/hugegraph/util/LZ4Util.java	`70.00% <66.66%> (+70.00%)`	`3.00 <3.00> (+3.00)`
.../java/com/baidu/hugegraph/util/StringEncoding.java	`85.07% <77.77%> (+0.65%)`	`29.00 <5.00> (+2.00)`
...c/main/java/com/baidu/hugegraph/task/HugeTask.java	`69.84% <100.00%> (ø)`	`82.00 <0.00> (ø)`
...idu/hugegraph/backend/store/raft/StoreCommand.java	`56.52% <0.00%> (-31.72%)`	`6.00% <0.00%> (ø%)`
.../hugegraph/backend/store/raft/StoreSerializer.java	`58.73% <0.00%> (-1.93%)`	`6.00% <0.00%> (ø%)`
...ph/backend/store/AbstractBackendStoreProvider.java	`92.59% <0.00%> (-1.16%)`	`23.00% <0.00%> (ø%)`
...m/baidu/hugegraph/backend/store/raft/RaftNode.java	`0.00% <0.00%> (ø)`	`0.00% <0.00%> (ø%)`
...hugegraph/backend/store/raft/RaftBackendStore.java	`0.00% <0.00%> (ø)`	`0.00% <0.00%> (ø%)`
...ugegraph/backend/store/raft/RaftSharedContext.java	`0.00% <0.00%> (ø)`	`0.00% <0.00%> (ø%)`
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db8bb17...3751061. Read the comment docs.

javeme

explain the effect in commit message

javeme · 2020-09-29T05:44:35Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/StringEncoding.java

                buf.write(bytes, 0, len);
            }
-            return decode(buf.bytes());
+            return decode(LZ4Util.decompress(buf.bytes(), BLOCK_SIZE).array());


to avoid copy bytes, add method decode(bytes, offset, length), and let BytesBuffer implements OutputStream that BytesBuffer instance could be pass into LZ4BlockOutputStream.

address it:
use BytesBuffer instead of ByteArrayOutputStream in compress and decompress method, and init BytesBuffer estimate size

javeme · 2020-09-29T05:54:54Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/StringEncoding.java

            byte[] bytes = new byte[value.length];
            int len;
-            while ((len = in.read(bytes)) > 0) {
+            while ((len = bis.read(bytes)) > 0) {


no need to write a buf, just LZ4Util.decompress(value)

javeme · 2020-09-29T05:56:17Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/StringEncoding.java

-            out.write(bytes);
-            out.finish();
+            bytes = LZ4Util.compress(bytes, BLOCK_SIZE).array();
+            bos.write(bytes);


just return LZ4Util.compress(bytes, BLOCK_SIZE).bytes()

Change-Id: I408bc15c3be531239c92b3da1d75af192a532347

javeme · 2020-09-29T09:44:04Z

hugegraph-test/src/main/resources/hugegraph.properties

-backend=${backend}
-serializer=${serializer}
+backend=rocksdb
+serializer=binary


keep origin

javeme · 2020-09-29T09:48:57Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/StringEncoding.java

                buf.write(bytes, 0, len);
            }
-            return decode(buf.bytes());
+            return decode(LZ4Util.decompress(buf.bytes(), BLOCK_SIZE).array());


address it:
use BytesBuffer instead of ByteArrayOutputStream in compress and decompress method, and init BytesBuffer estimate size

Change-Id: I812c0ba5a2abda0d11cb2e7af5ebc00da0c26ef6

javeme · 2020-09-29T10:47:20Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/StringEncoding.java

@@ -135,11 +144,13 @@ public static String encodeBase64(byte[] bytes) {
    }

    public static byte[] compress(String value) {
-        return LZ4Util.compress(encode(value), BLOCK_SIZE).array();
+        BytesBuffer bf = LZ4Util.compress(encode(value), BLOCK_SIZE);


rename to buf

javeme · 2020-09-29T10:47:28Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/StringEncoding.java

    }

    public static String decompress(byte[] value) {
-        return decode(LZ4Util.decompress(value, BLOCK_SIZE).array());
+        BytesBuffer bf = LZ4Util.decompress(value, BLOCK_SIZE);


javeme · 2020-09-29T10:47:55Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/LZ4Util.java

        ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
-        ByteArrayOutputStream baos = new ByteArrayOutputStream(blockSize);
+        BytesBuffer bf = new BytesBuffer(bytes.length * 8);


rename to buf

javeme · 2020-09-29T10:47:59Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/LZ4Util.java

@@ -37,35 +36,35 @@
    public static BytesBuffer compress(byte[] bytes, int blockSize) {
        LZ4Factory factory = LZ4Factory.fastestInstance();
        LZ4Compressor compressor = factory.fastCompressor();
-        ByteArrayOutputStream baos = new ByteArrayOutputStream();
+        BytesBuffer bf = new BytesBuffer(bytes.length / 8);


rename to buf

javeme · 2020-09-29T10:51:44Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/serializer/BytesBuffer.java

        require(BYTE_LEN * val.length);
        this.buffer.put(val);
        return this;
    }

-    public BytesBuffer write(byte[] val, int offset, int length) {
+    public BytesBuffer writeByteArray(byte[] val, int offset, int length) {


prefer keep these methods returning void

Change-Id: I3bc4db83bdf3d6eed14567a771f25c744d486d44

javeme · 2020-09-29T12:05:31Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/serializer/BytesBuffer.java

    }

-    public BytesBuffer write(byte[] val, int offset, int length) {
+    public void write(byte[] val, int offset, int length) {


add "@OverRide" for 3 write() methods

Change-Id: Ia3b3b4cff5cb4967815deeb27d0bcd5093d3c75e

javeme · 2020-09-29T13:44:44Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/LZ4Util.java

@@ -36,7 +36,7 @@
    public static BytesBuffer compress(byte[] bytes, int blockSize) {
        LZ4Factory factory = LZ4Factory.fastestInstance();
        LZ4Compressor compressor = factory.fastCompressor();
-        BytesBuffer buf = new BytesBuffer(bytes.length / 8);
+        BytesBuffer buf = new BytesBuffer(bytes.length);


add a const var buf_ratio, let init_size=bytes.length/buf_ratio, also update decompress to bytes.length*buf_ratio

Change-Id: Ie15e6bf9c6eed48a448c4ed84d35e4f27197b67b

Change-Id: Ib68648def8946433666211b25782775151fb4c52

Change-Id: I0919c57e990f50a0df18a4d942d2de67947f928e

javeme

ci failed

javeme · 2020-10-23T12:51:20Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/StringEncoding.java

+        return compress(value, 0.0F);
+    }
+
+    public static byte[] compress(String value, float ratio) {


prefer to not add this method

javeme · 2020-10-23T12:51:26Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/StringEncoding.java

+        return decompress(value, 0.0F);
+    }
+
+    public static String decompress(byte[] value, float ratio) {


rename to bufferRatio

zhoney · 2020-10-26T02:49:29Z

cassandra

[ERROR] Tests run: 683, Failures: 1, Errors: 0, Skipped: 21, Time elapsed: 482.552 s <<< FAILURE! - in com.baidu.hugegraph.core.CoreTestSuite
[ERROR] testAddEdgeWithTtlAndTtlStartTime(com.baidu.hugegraph.core.EdgeCoreTest)  Time elapsed: 1.796 s  <<< FAILURE!
java.lang.AssertionError
	at com.baidu.hugegraph.core.EdgeCoreTest.testAddEdgeWithTtlAndTtlStartTime(EdgeCoreTest.java:754)

hbase

[ERROR] Tests run: 38, Failures: 3, Errors: 5, Skipped: 0, Time elapsed: 13.07 s <<< FAILURE! - in com.baidu.hugegraph.api.ApiTestSuite
[ERROR] testList(com.baidu.hugegraph.api.TaskApiTest)  Time elapsed: 0.103 s  <<< FAILURE!
java.lang.AssertionError: Response with status 400 and content {"exception":"class com.baidu.hugegraph.exception.ExistedException","message":"The index label 'personByCity' has existed","cause":""} expected:<202> but was:<400>
	at com.baidu.hugegraph.api.TaskApiTest.prepareSchema(TaskApiTest.java:42)
[ERROR] testList(com.baidu.hugegraph.api.TaskApiTest)  Time elapsed: 0.103 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to list indexlabels
[ERROR] testCancel(com.baidu.hugegraph.api.TaskApiTest)  Time elapsed: 0.058 s  <<< FAILURE!
java.lang.AssertionError: Response with status 400 and content {"exception":"class com.baidu.hugegraph.exception.ExistedException","message":"The index label 'personByCity' has existed","cause":""} expected:<202> but was:<400>
	at com.baidu.hugegraph.api.TaskApiTest.prepareSchema(TaskApiTest.java:42)
[ERROR] testCancel(com.baidu.hugegraph.api.TaskApiTest)  Time elapsed: 0.058 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to list indexlabels
[ERROR] testDelete(com.baidu.hugegraph.api.TaskApiTest)  Time elapsed: 0.033 s  <<< FAILURE!
java.lang.AssertionError: Response with status 400 and content {"exception":"class com.baidu.hugegraph.exception.ExistedException","message":"The index label 'personByCity' has existed","cause":""} expected:<202> but was:<400>
	at com.baidu.hugegraph.api.TaskApiTest.prepareSchema(TaskApiTest.java:42)
[ERROR] testDelete(com.baidu.hugegraph.api.TaskApiTest)  Time elapsed: 0.034 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to list indexlabels
[ERROR] com.baidu.hugegraph.api.GremlinApiTest  Time elapsed: 0.122 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to list indexlabels
[ERROR] com.baidu.hugegraph.api.MetricsApiTest  Time elapsed: 0.191 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to list indexlabels
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   GremlinApiTest>BaseApiTest.init:71->BaseApiTest.clearData:459->BaseApiTest.clearSchema:417->BaseApiTest.lambda$clearSchema$5:396 » Huge
[ERROR]   MetricsApiTest>BaseApiTest.init:71->BaseApiTest.clearData:459->BaseApiTest.clearSchema:417->BaseApiTest.lambda$clearSchema$5:396 » Huge
[ERROR] com.baidu.hugegraph.api.TaskApiTest.testCancel(com.baidu.hugegraph.api.TaskApiTest)
[ERROR]   Run 1: TaskApiTest.prepareSchema:42->BaseApiTest.initIndexLabel:279->BaseApiTest.assertResponseStatus:476 Response with status 400 and content {"exception":"class com.baidu.hugegraph.exception.ExistedException","message":"The index label 'personByCity' has existed","cause":""} expected:<202> but was:<400>
[ERROR]   Run 2: TaskApiTest>BaseApiTest.teardown:81->BaseApiTest.clearData:459->BaseApiTest.clearSchema:417->BaseApiTest.lambda$clearSchema$5:396 » Huge
[INFO] 
[ERROR] com.baidu.hugegraph.api.TaskApiTest.testDelete(com.baidu.hugegraph.api.TaskApiTest)
[ERROR]   Run 1: TaskApiTest.prepareSchema:42->BaseApiTest.initIndexLabel:279->BaseApiTest.assertResponseStatus:476 Response with status 400 and content {"exception":"class com.baidu.hugegraph.exception.ExistedException","message":"The index label 'personByCity' has existed","cause":""} expected:<202> but was:<400>
[ERROR]   Run 2: TaskApiTest>BaseApiTest.teardown:81->BaseApiTest.clearData:459->BaseApiTest.clearSchema:417->BaseApiTest.lambda$clearSchema$5:396 » Huge
[INFO] 
[ERROR] com.baidu.hugegraph.api.TaskApiTest.testList(com.baidu.hugegraph.api.TaskApiTest)
[ERROR]   Run 1: TaskApiTest.prepareSchema:42->BaseApiTest.initIndexLabel:279->BaseApiTest.assertResponseStatus:476 Response with status 400 and content {"exception":"class com.baidu.hugegraph.exception.ExistedException","message":"The index label 'personByCity' has existed","cause":""} expected:<202> but was:<400>
[ERROR]   Run 2: TaskApiTest>BaseApiTest.teardown:81->BaseApiTest.clearData:459->BaseApiTest.clearSchema:417->BaseApiTest.lambda$clearSchema$5:396 » Huge

Change-Id: If332999b94a2ef918b2b9d125a99de6ae15b59fe

javeme · 2020-10-26T08:27:06Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/StringEncoding.java

-        } catch (IOException e) {
-            throw new BackendException("Failed to compress: %s", e, value);
-        }
+        return compress(value, 0.0F);


define const

javeme · 2020-10-26T08:27:20Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/StringEncoding.java

+        return decompress(value, 0.0F);
+    }
+
+    public static String decompress(byte[] value, float ratio) {


rename to bufferRatio

Change-Id: I1970aab20cea373a6e5e1c11dea364aa38b7d3d4

use lz4 instead of gzip for StringEncoding compress/decompress

f00f547

Change-Id: Ie4458ba481ac3625139458cc92e9dbc0631898da

zhoney force-pushed the task-result-lz4 branch from f4de848 to f00f547 Compare September 28, 2020 12:24

improve

0910c1c

Change-Id: I39cbfe2f643b46ca736e5076ef73ec6203b0be3a

javeme requested changes Sep 29, 2020

View reviewed changes

improve

5b2b44c

Change-Id: I408bc15c3be531239c92b3da1d75af192a532347

javeme reviewed Sep 29, 2020

View reviewed changes

use BytesBuffer improve LZ4Util

be7e3ee

Change-Id: I812c0ba5a2abda0d11cb2e7af5ebc00da0c26ef6

javeme reviewed Sep 29, 2020

View reviewed changes

improve

0019088

Change-Id: I3bc4db83bdf3d6eed14567a771f25c744d486d44

javeme reviewed Sep 29, 2020

View reviewed changes

improve

99f44ff

Change-Id: Ia3b3b4cff5cb4967815deeb27d0bcd5093d3c75e

javeme requested changes Sep 29, 2020

View reviewed changes

zhoney added 3 commits September 30, 2020 10:17

improve

866a933

Change-Id: Ie15e6bf9c6eed48a448c4ed84d35e4f27197b67b

improve

e593b53

Change-Id: Ib68648def8946433666211b25782775151fb4c52

improve

c703c6d

Change-Id: I0919c57e990f50a0df18a4d942d2de67947f928e

javeme reviewed Oct 23, 2020

View reviewed changes

improve

e457ec0

Change-Id: If332999b94a2ef918b2b9d125a99de6ae15b59fe

javeme reviewed Oct 26, 2020

View reviewed changes

improve

3751061

Change-Id: I1970aab20cea373a6e5e1c11dea364aa38b7d3d4

javeme approved these changes Oct 26, 2020

View reviewed changes

Linary approved these changes Oct 26, 2020

View reviewed changes

zhoney merged commit 0e4eaa0 into master Oct 26, 2020

zhoney deleted the task-result-lz4 branch October 26, 2020 10:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use lz4 instead of gzip for StringEncoding compress/decompress #1196

use lz4 instead of gzip for StringEncoding compress/decompress #1196

zhoney commented Sep 28, 2020

codecov bot commented Sep 28, 2020 •

edited

javeme left a comment

javeme Sep 29, 2020

javeme Sep 29, 2020

javeme Sep 29, 2020

javeme Sep 29, 2020

javeme Sep 29, 2020

javeme Sep 29, 2020

javeme Sep 29, 2020

javeme Sep 29, 2020

javeme Sep 29, 2020

javeme Sep 29, 2020

javeme Sep 29, 2020

javeme Sep 29, 2020

javeme Sep 29, 2020

javeme left a comment

javeme Oct 23, 2020

javeme Oct 23, 2020

javeme Oct 26, 2020

zhoney commented Oct 26, 2020

javeme Oct 26, 2020

javeme Oct 26, 2020

use lz4 instead of gzip for StringEncoding compress/decompress #1196

use lz4 instead of gzip for StringEncoding compress/decompress #1196

Conversation

zhoney commented Sep 28, 2020

codecov bot commented Sep 28, 2020 • edited

Codecov Report

javeme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhoney commented Oct 26, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 28, 2020 •

edited