Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix raft applied index out of range #880

Merged
merged 6 commits into from
Dec 2, 2019

Conversation

zzy96
Copy link
Contributor

@zzy96 zzy96 commented Nov 19, 2019

Fix #845

In raft mode, hard state & entries are written into WAL before raft entries are processed, but applied index is written into the node's local level db only after completion of processing each raft entry. If a node is stopped/ restarted during a large list of entries under processing (particularly when calling applyRaftSnapshot which takes time to sync blocks from peers). hard state & entries are moving forward but applied index is not updated.

This fix can successfully recover the node from applied out of range panic state by resetting last applied index to last persisted applied index from WAL on raft restart.

@zzy96 zzy96 changed the title fix raft applied index out of range Fix raft applied index out of range Nov 19, 2019
jbhurat
jbhurat previously approved these changes Nov 26, 2019
Copy link
Contributor

@jbhurat jbhurat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

trung
trung previously approved these changes Nov 27, 2019
Copy link
Contributor

@trung trung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zzy96 zzy96 force-pushed the fix-raft-applied-index-out-of-range branch from 8bb47eb to 9036022 Compare November 28, 2019 05:53
@jpmsam jpmsam merged commit ef99f6d into master Dec 2, 2019
@zzy96 zzy96 deleted the fix-raft-applied-index-out-of-range branch December 2, 2019 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

'raft applied is out of range' causing panic
4 participants