-
Notifications
You must be signed in to change notification settings - Fork 763
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Leader election and dqlite errors when recovering nodes in HA cluster #2819
Comments
I am having the same issue on a single node cluster on Debian 11 and Microk8s v1.22.4 rev 2695. After leader election is lost, apparently the process is terminated and restarted by systemd, which throws all pods into turmoil. |
@bc185174 your observation is about right. In a raft cluster (either etcd or dqlite), there must be a majority. With regards to that error on your node |
Thank you for clarifying. In the dqlite repo there is some documentation on the |
@bc185174, the error |
It's not planned immediately but was already discussed, and imo is useful to add. Will try to do it within a reasonable timeframe. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Hello,
We have a HA cluster setup with three nodes, each using version
1.21.7
:We came across an issue when two nodes
masterA
andmasterB
were removed ungracefully and shutdown. The elected leader node wasmasterA
. The following errors occurred around the same time on the remaining nodemasterC
:At this point, the elected leader node remained the same (
masterA
) which is still powered-off; however, when we powered-onmasterB
, it failed to start the kubelite service:Could the segment be corrupt or would this suggest that it cannot sync the dqlite files to the elected leader
masterA
that is still unavailable? If so is there a way we can validate the checksum? In order to recovermasterB
, I had to delete the mentioned dqlite file and restart the kubelite service. OncemasterB
andmasterC
were available, a new leader node was elected (masterB
) and we was able to recover the cluster.Checking the HA documentation, with only a single node available, would this render the cluster inoperable? Essentially, would we need more than one node available at anytime?
There was a few other suggestions such as increasing the following arguments in the kube-scheduler and kube-controller-manager (source):
A number of comments mentioned that they had the same issue using microk8s v1.21. The last potential issue was a "resource crunch or network issue" mentioned here. We have not yet been able to replicate the issue but would appreciate if anyone could shed some light on this.
The text was updated successfully, but these errors were encountered: