-
I'm observing the following behavior after some restarts of nodes in an rqlite cluster. A node eventually wins the election and becomes the leader. However, the node quickly reverts to the follower state. There are several warnings about "appendEntries rejected" and "sending older logs," which indicates that there is an inconsistency in the Raft log between the nodes. The node continuously tries to send older logs to synchronize with other nodes. I was seeing this after making some changes to a Nomad rqlite job-spec, which caused staggered restarts. How do I recover from this?
|
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 2 replies
-
What kind of environment are you running in? OS? Containers? What does restart exactly mean? What do you change? Can I also see the output for /status and /nodes from each node before the restart, and after the restart. |
Beta Was this translation helpful? Give feedback.
-
Also /debug/vars. |
Beta Was this translation helpful? Give feedback.
-
I suggest you try to simplify your setup as much as possible -- see if you can reproduce this without TLS enabled, etc. |
Beta Was this translation helpful? Give feedback.
I suggest you try to simplify your setup as much as possible -- see if you can reproduce this without TLS enabled, etc.