Fix queryByRaft readIndex failed #1121

Linary · 2020-07-29T14:28:10Z

ReadIndex request rejected because leader has not committed any log entry at its term

Change-Id: I6f5c6fd3770da732caf0c6acd41f0441aa4acb55

ReadIndex request rejected because leader has not committed any log entry at its term Change-Id: I6f5c6fd3770da732caf0c6acd41f0441aa4acb55

codecov · 2020-07-29T14:49:33Z

Codecov Report

Merging #1121 into master will increase coverage by 7.66%.
The diff coverage is 16.21%.

@@             Coverage Diff              @@
##             master    #1121      +/-   ##
============================================
+ Coverage     58.20%   65.86%   +7.66%     
- Complexity     4801     5652     +851     
============================================
  Files           354      354              
  Lines         28748    28836      +88     
  Branches       4056     4062       +6     
============================================
+ Hits          16732    18994    +2262     
+ Misses        10190     7964    -2226     
- Partials       1826     1878      +52

Impacted Files	Coverage Δ	Complexity Δ
...a/com/baidu/hugegraph/auth/HugeGraphAuthProxy.java	`4.18% <0.00%> (-0.04%)`	`0.00 <0.00> (ø)`
...e/src/main/java/com/baidu/hugegraph/HugeGraph.java	`65.30% <ø> (ø)`	`9.00 <0.00> (ø)`
...hugegraph/backend/store/raft/RaftBackendStore.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...h/backend/store/raft/RaftBackendStoreProvider.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...m/baidu/hugegraph/backend/store/raft/RaftNode.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...ugegraph/backend/store/raft/RaftSharedContext.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...ugegraph/backend/store/raft/StoreStateMachine.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...in/java/com/baidu/hugegraph/StandardHugeGraph.java	`80.54% <88.88%> (+1.41%)`	`110.00 <2.00> (+8.00)`
...in/java/com/baidu/hugegraph/core/GraphManager.java	`57.14% <100.00%> (+2.66%)`	`0.00 <0.00> (ø)`
...ph/backend/store/AbstractBackendStoreProvider.java	`93.75% <100.00%> (+0.07%)`	`23.00 <1.00> (+1.00)`
... and 134 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2df4f68...338ca00. Read the comment docs.

javeme · 2020-07-30T04:00:47Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftBackendStore.java

+            this.node().node().readIndex(BytesUtil.EMPTY_BYTES,
+                                         readIndexClosure);
+            try {
+                return closure.waitFinished();


rename StoreClosure closure to future

other place used closure

javeme · 2020-07-30T04:04:43Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftBackendStore.java

+                             query);
+                    closure.failure(status, new RaftException(
+                            "Failed to execute query '%s' because of leader " +
+                            "has not committed any log entry at its term",


move to "status.getRaftError() == RaftError.EAGAIN" block to line 247, just throw exception in else block

javeme · 2020-07-30T04:06:51Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftBackendStore.java

        }
+        throw new BackendException("Failed to query in the case of %s " +
+                                   "retries times");


collect last error message like "retries times with error: %s"

Change-Id: Ic39d81623efa89b3f35b794f7d183201348ab871

Change-Id: Ied4194a6e20c3365dede4df922d4eade9b4f7477

javeme · 2020-07-30T09:11:20Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftBackendStore.java

        try {
            return closure.waitFinished();
        } catch (Throwable t) {
-            throw new BackendException("Failed to query", t);
+            throw new BackendException("Failed to execute query", t);


move LOG.warn at line 231 here

javeme · 2020-07-30T09:15:00Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java

+        @Override
+        public void onHeartbeated(PeerId peer) {
+            LOG.info("The node {} replicator has heartbeated", peer);
+            started = true;


RaftNode.this.started

javeme · 2020-07-30T09:15:06Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java

@@ -266,6 +300,7 @@ private boolean isWriteBufferOverflow(Status status) {
        @Override
        public void onDestroyed(PeerId peer) {
            LOG.warn("The node {} prepare to offline", peer);
+            started = false;


javeme · 2020-07-30T09:16:37Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java

@@ -165,6 +175,24 @@ protected void waitLeaderElected() {
                  this.group(), WAIT_LEADER_TIMEOUT);
    }

+    protected boolean waitHeartbeated() {


rename waitStarted

javeme · 2020-07-30T09:18:31Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java

+            }
+            try {
+                Thread.sleep(heartbeatInterval);
+            } catch (InterruptedException ex) {


rename ex to e

javeme · 2020-07-30T09:18:33Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java

+            try {
+                Thread.sleep(heartbeatInterval);
+            } catch (InterruptedException ex) {
+                throw new BackendException("Waiting heartbeated is interrupted");


add cause param

javeme · 2020-07-30T09:19:45Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftBackendStore.java

        StoreCommand command = new StoreCommand(StoreAction.QUERY);
        StoreClosure closure = new StoreClosure(command);
-        RaftNode raftNode = this.node();
-        raftNode.node().readIndex(BytesUtil.EMPTY_BYTES, new ReadIndexClosure() {
+        ReadIndexClosure readIndexClosure = new ReadIndexClosure() {


rename StoreClosure closure to future

javeme · 2020-07-30T09:36:54Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftBackendStore.java

+        RaftNode node = this.node();
+        node.waitLeaderElected();
+        if (node.node().isLeader()) {
+            if (!node.waitHeartbeated()) {


add a common waitFor(future, timeout) method

zhoney · 2020-07-30T09:20:47Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftBackendStore.java

+        node.waitLeaderElected();
+        if (node.node().isLeader()) {
+            if (!node.waitHeartbeated()) {
+                throw new BackendException("Wait raft node heartbeat failed");


Failed to wait raft node heartbeat

Change-Id: I689149d598ad902cc1ff5175d76557fd7b1491a0

zhoney · 2020-07-31T03:33:01Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/StoreStateMachine.java

+
+    @Override
+    public void onStopFollowing(LeaderChangeContext ctx) {
+        LOG.info("The node {} become to follower", this.node.nodeId());


abdicated from follower

zhoney · 2020-07-31T03:35:08Z

hugegraph-core/src/main/java/com/baidu/hugegraph/task/ServerInfoManager.java

@@ -337,7 +335,7 @@ private HugeServerInfo serverInfo(Id server) {
    }

    private HugeServerInfo removeSelfServerInfo() {
-        if (this.initialized()) {
+        if (this.graph.initialized()) {


should use this.call() if graph.initlalized() will open tx (ensure graph tx closed when graph close)

javeme · 2020-07-31T03:23:43Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftBackendStore.java

-            if (!node.waitHeartbeated()) {
-                throw new BackendException("Wait raft node heartbeat failed");
-            }
+            node.waitHeartbeated(-1);


also define NO_TIMEOUT

javeme · 2020-07-31T03:24:02Z

...raph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftBackendStoreProvider.java

@@ -105,6 +105,13 @@ public void open(String name) {
        this.provider.open(name);
    }

+    @Override
+    public void waitStoreStarted() {


check not null

Change-Id: I1218ec628755434be909f520fe9655ff2b01af85

Change-Id: I2a2e0cc6f0a2e65e23a1b9b2192dd50d8b28224c

javeme · 2020-07-31T10:33:46Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java

-                return;
-            } else {
+        long beginTime = System.currentTimeMillis();
+        int internalTimeout = 3000;


define const interval

javeme · 2020-07-31T10:34:56Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java

+            }
+        };
+        long beginTime = System.currentTimeMillis();
+        int internalTimeout = 3000;


javeme · 2020-07-31T10:35:07Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/raft/RaftNode.java

+            }
+            try {
+                Thread.sleep(heartbeatInterval);
+            } catch (InterruptedException ex) {


Change-Id: I1f191a261979a9ffe232827d0c639642547ba5b7

Fix queryByRaft readIndex failed

1760059

ReadIndex request rejected because leader has not committed any log entry at its term Change-Id: I6f5c6fd3770da732caf0c6acd41f0441aa4acb55

javeme reviewed Jul 30, 2020

View reviewed changes

Linary added 2 commits July 30, 2020 15:17

use heartbeat interval as retry interval

87e6ab6

Change-Id: Ic39d81623efa89b3f35b794f7d183201348ab871

add listener on heartbeated

c859c0a

Change-Id: Ied4194a6e20c3365dede4df922d4eade9b4f7477

javeme reviewed Jul 30, 2020

View reviewed changes

zhoney requested changes Jul 30, 2020

View reviewed changes

Linary force-pushed the fix-read-index branch 2 times, most recently from e55434b to b42e21e Compare July 31, 2020 03:08

Add wait started logic

e81fadc

Change-Id: I689149d598ad902cc1ff5175d76557fd7b1491a0

Linary force-pushed the fix-read-index branch from b42e21e to e81fadc Compare July 31, 2020 03:25

zhoney requested changes Jul 31, 2020

View reviewed changes

javeme reviewed Jul 31, 2020

View reviewed changes

use polling to wait heartbeat

3a3a5a5

Change-Id: I1218ec628755434be909f520fe9655ff2b01af85

Linary force-pushed the fix-read-index branch from 8f53adb to 3a3a5a5 Compare July 31, 2020 07:29

use closed field as serverInfoManager flag

fccc553

Change-Id: I2a2e0cc6f0a2e65e23a1b9b2192dd50d8b28224c

javeme reviewed Jul 31, 2020

View reviewed changes

tiny improve

338ca00

Change-Id: I1f191a261979a9ffe232827d0c639642547ba5b7

javeme approved these changes Jul 31, 2020

View reviewed changes

zhoney approved these changes Jul 31, 2020

View reviewed changes

Linary merged commit 52d2eac into master Jul 31, 2020

Linary deleted the fix-read-index branch July 31, 2020 11:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix queryByRaft readIndex failed #1121

Fix queryByRaft readIndex failed #1121

Linary commented Jul 29, 2020

codecov bot commented Jul 29, 2020 •

edited

javeme Jul 30, 2020

Linary Jul 30, 2020

javeme Jul 30, 2020

javeme Jul 30, 2020

javeme Jul 30, 2020

javeme Jul 30, 2020

javeme Jul 30, 2020

javeme Jul 30, 2020

javeme Jul 30, 2020

javeme Jul 31, 2020

javeme Jul 30, 2020

javeme Jul 30, 2020

javeme Jul 30, 2020

zhoney Jul 30, 2020

zhoney Jul 31, 2020

zhoney Jul 31, 2020

javeme Jul 31, 2020

javeme Jul 31, 2020

javeme Jul 31, 2020

javeme Jul 31, 2020

javeme Jul 31, 2020

Fix queryByRaft readIndex failed #1121

Fix queryByRaft readIndex failed #1121

Conversation

Linary commented Jul 29, 2020

codecov bot commented Jul 29, 2020 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 29, 2020 •

edited