Sleep between cluster role reshuffling when sending tokens #26
Conversation
…in first round Signed-off-by: Max Asnaashari <max.asnaashari@canonical.com>
…bers join To avoid issues when a new join request is being handled in the middle of a role reshuffle, Sleep for a few seconds after creating a cluster of size 3, before proceeding with the rest of the cluster members. Signed-off-by: Max Asnaashari <max.asnaashari@canonical.com>
Signed-off-by: Max Asnaashari <max.asnaashari@canonical.com>
What issues was this causing? I ask because its my observation (from working on similar issues with the LXD test suite) that role re-balancing doesn't always occur within a set time limit, and thus adding a pause for a fixed time period may not fully solve the issue. |
In this case, the error was a "dqlite leader not found" error on exactly one peer. The join request seems to be completely dropped because the join token gets used up too. You're probably right that the sleep here is just a bandaid, but the error was pretty consistently reproducible with MicroCloud because every peer attempts to join at the almost exact same time. I'm also not sure if it's exclusive to LXD's dqlite implementation, because I'm pretty sure the cluster got stuck joining MicroCeph too at one point. |
|
I see thanks. |
On
init, MicroCloud starts an mdns broadcast with all the join tokens for each service and peer as the text body. What this means is that all peers will attempt to join through that first initial member.In some cases this can cause issues as a cluster rebalances roles as it grows. So this PR splits the broadcast in two. First we send out a broadcast with just 2 members, and wait for a few seconds before continuing with the rest.