Move lz4 decompress to backend executor #1237

Linary · 2020-11-02T10:44:04Z

Change-Id: Idf2e796739fda1dc5ce5ffcfa16bc61b6e1a6281

javeme

update the title to "Move lz4 decompress to backend executor"

javeme · 2020-11-02T12:41:41Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java

@@ -230,7 +230,8 @@ private void waitIfBusy() {
            if (this.busyCounter.get() > 0) {
                synchronized (this) {
                    if (this.busyCounter.get() > 0) {
-                        this.busyCounter.decrementAndGet();
+                        LOG.info("Decrease busy counter: [{}]",
+                                 this.busyCounter.decrementAndGet());


prefer to define a var

javeme · 2020-11-02T12:42:41Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java

+        /**
+         * Maybe useful in the future
+         */
+        private boolean isRPCTimeout(Status status) {


prefer isRpcTimeout

Change-Id: I4b4c470228caa4ba7e44df9618a7aa1f35da06ef

zhoney · 2020-11-03T02:53:29Z

hugegraph-core/src/main/java/com/baidu/hugegraph/config/CoreOptions.java

+    public static final ConfigOption<Integer> RAFT_RPC_BUF_HIGH_WATER_MARK =
+            new ConfigOption<>(
+                    "raft.rpc_buf_high_water_mark",
+                    "",


zhoney · 2020-11-03T02:53:48Z

hugegraph-core/src/main/java/com/baidu/hugegraph/config/CoreOptions.java

+    public static final ConfigOption<Integer> RAFT_RPC_BUF_LOW_WATER_MARK =
+            new ConfigOption<>(
+                    "raft.rpc_buf_low_water_mark",
+                    "",


prefer write some description

javeme · 2020-11-03T06:13:33Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftSharedContext.java

@@ -266,6 +267,9 @@ public void notifyCache(HugeType type, Id id) {
            eventHub = this.params.graphEventHub();
        } else if (type.isSchema()) {
            eventHub = this.params.schemaEventHub();
+            if (id.number() && id.asLong() < 0) {
+                return;


add some comment

javeme · 2020-11-03T06:33:07Z

hugegraph-core/src/main/java/com/baidu/hugegraph/util/LZ4Util.java

@@ -54,6 +54,10 @@ public static BytesBuffer compress(byte[] bytes, int blockSize,
        } catch (IOException e) {
            throw new BackendException("Failed to compress", e);
        }
+        /*
+         * If need perform reading outside the method,


need to perform

Change-Id: I70c2fcfd92ed923d801394cd6f97b31afa8475d4

javeme · 2020-11-03T10:19:36Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/cache/CachedSchemaTransaction.java

@@ -107,7 +107,9 @@ private void listenChanges() {
            if ("invalid".equals(args[0])) {
                HugeType type = (HugeType) args[1];
                Id id = (Id) args[2];
-                this.arrayCaches.remove(type, id);
+                if (id.number() && id.asLong() > 0) {
+                    this.arrayCaches.remove(type, id);


move to remove() method

Change-Id: I8008c86b8e7c92d72dac205eca8894826a39edb8

javeme

ci failure

javeme · 2020-11-04T07:41:42Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/cache/CachedSchemaTransaction.java

                this.set(schema.type(), id, schema);
            }
        }

        public V get(HugeType type, Id id) {
-            assert id.number() && id.asLong() > 0 : id;
+            assert id.number();
+            if (id.asLong() <= 0L) {


define var longId=id.asLong()

javeme · 2020-11-04T07:41:59Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/cache/CachedSchemaTransaction.java

@@ -348,7 +349,10 @@ public V get(HugeType type, Id id) {
        }

        public void set(HugeType type, Id id, V value) {
-            assert id.number() && id.asLong() > 0 : id;
+            assert id.number();
+            if (id.asLong() <= 0L) {


javeme · 2020-11-04T07:42:06Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/cache/CachedSchemaTransaction.java

@@ -373,7 +377,10 @@ public void set(HugeType type, Id id, V value) {
        }

        public void remove(HugeType type, Id id) {
-            assert id.number() && id.asLong() > 0 : id;
+            assert id.number();
+            if (id.asLong() <= 0L) {


Change-Id: Ia15672c9e0564cb9673a0b8ab983272c3943c276

javeme · 2020-11-04T08:36:20Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/cache/CachedSchemaTransaction.java

-            assert id.number() && id.asLong() > 0 : id;
+            assert id.number();
+            if (id.asLong() <= 0L) {
+                return;


add assert false : id;

Change-Id: I4d7a90c5e40117e00c1e017917ac5d83824df3ce

Change-Id: Ic7770f0b8cb9a81dbb714ea9ce8ef99d5b710807

codecov · 2020-11-04T10:12:07Z

Codecov Report

Merging #1237 into master will increase coverage by 0.20%.
The diff coverage is 23.07%.

@@             Coverage Diff              @@
##             master    #1237      +/-   ##
============================================
+ Coverage     62.21%   62.42%   +0.20%     
- Complexity     5736     5758      +22     
============================================
  Files           371      374       +3     
  Lines         31153    31284     +131     
  Branches       4381     4393      +12     
============================================
+ Hits          19382    19529     +147     
+ Misses         9795     9772      -23     
- Partials       1976     1983       +7

Impacted Files	Coverage Δ	Complexity Δ
...hugegraph/backend/store/raft/RaftBackendStore.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...aidu/hugegraph/backend/store/raft/RaftClosure.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...m/baidu/hugegraph/backend/store/raft/RaftNode.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...ugegraph/backend/store/raft/RaftSharedContext.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...ugegraph/backend/store/raft/StoreStateMachine.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...rc/main/java/com/baidu/hugegraph/util/LZ4Util.java	`70.00% <ø> (ø)`	`3.00 <0.00> (ø)`
...gegraph/backend/cache/CachedSchemaTransaction.java	`89.80% <31.57%> (-3.53%)`	`33.00 <5.00> (ø)`
...n/java/com/baidu/hugegraph/config/CoreOptions.java	`99.44% <100.00%> (+0.01%)`	`2.00 <0.00> (ø)`
...com/baidu/hugegraph/job/computer/ComputerPool.java	`0.00% <0.00%> (ø)`	`0.00% <0.00%> (ø%)`
...baidu/hugegraph/job/computer/AbstractComputer.java	`0.00% <0.00%> (ø)`	`0.00% <0.00%> (ø%)`
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 76d0cb3...21178e0. Read the comment docs.

* use whitebox to set LOW_WATER_MARK and HIGH_WATER_MARK Change-Id: Ic7770f0b8cb9a81dbb714ea9ce8ef99d5b710807

Let lz4 decompress in thread pool

445a2bf

Change-Id: Idf2e796739fda1dc5ce5ffcfa16bc61b6e1a6281

javeme reviewed Nov 2, 2020

View reviewed changes

tiny improve

f3ac544

Change-Id: I4b4c470228caa4ba7e44df9618a7aa1f35da06ef

zhoney reviewed Nov 3, 2020

View reviewed changes

javeme reviewed Nov 3, 2020

View reviewed changes

Linary changed the title ~~Let lz4 decompress in thread pool~~ Move lz4 decompress to backend executor Nov 3, 2020

some improve

408689e

Change-Id: I70c2fcfd92ed923d801394cd6f97b31afa8475d4

zhoney previously approved these changes Nov 3, 2020

View reviewed changes

javeme reviewed Nov 3, 2020

View reviewed changes

Linary dismissed zhoney’s stale review via da6a0bd November 3, 2020 13:04

tiny improve

d1189c7

Change-Id: I8008c86b8e7c92d72dac205eca8894826a39edb8

Linary force-pushed the improve-raft-perf branch from da6a0bd to d1189c7 Compare November 4, 2020 02:09

javeme previously approved these changes Nov 4, 2020

View reviewed changes

javeme reviewed Nov 4, 2020

View reviewed changes

use whitebox to set LOW_WATER_MARK and HIGH_WATER_MARK

25f7e50

Change-Id: Ia15672c9e0564cb9673a0b8ab983272c3943c276

Linary dismissed javeme’s stale review via 25f7e50 November 4, 2020 08:34

javeme reviewed Nov 4, 2020

View reviewed changes

Linary added 2 commits November 4, 2020 16:42

tiny improve

ec8cb4d

Change-Id: I4d7a90c5e40117e00c1e017917ac5d83824df3ce

tiny improve

21178e0

Change-Id: Ic7770f0b8cb9a81dbb714ea9ce8ef99d5b710807

javeme approved these changes Nov 4, 2020

View reviewed changes

zhoney approved these changes Nov 4, 2020

View reviewed changes

Linary merged commit 636bcbb into master Nov 5, 2020

Linary deleted the improve-raft-perf branch November 5, 2020 02:48

tmljob pushed a commit to tmljob/hugegraph that referenced this pull request Dec 10, 2020

Move lz4 decompress to backend executor (apache#1237)

f41ae0d

* use whitebox to set LOW_WATER_MARK and HIGH_WATER_MARK Change-Id: Ic7770f0b8cb9a81dbb714ea9ce8ef99d5b710807

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move lz4 decompress to backend executor #1237

Move lz4 decompress to backend executor #1237

Linary commented Nov 2, 2020

javeme left a comment

javeme Nov 2, 2020

javeme Nov 2, 2020

zhoney Nov 3, 2020

zhoney Nov 3, 2020

javeme Nov 3, 2020

javeme Nov 3, 2020

javeme Nov 3, 2020

javeme left a comment

javeme Nov 4, 2020

javeme Nov 4, 2020

javeme Nov 4, 2020

javeme Nov 4, 2020

codecov bot commented Nov 4, 2020 •

edited

Move lz4 decompress to backend executor #1237

Move lz4 decompress to backend executor #1237

Conversation

Linary commented Nov 2, 2020

javeme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Nov 4, 2020 • edited

Codecov Report

codecov bot commented Nov 4, 2020 •

edited