mon: paxos add the timeout function when peon recovery #10359
Conversation
Now the peon only have 10s from init to active.But when the leader from init to active he may start twice or more recovery collect when meet a higher pn than himself.It means the leader from init to active may spend more than 10S.So it is not reasonable for the peon only have one 10s timeout event. Signed-off-by: song baisen <song.baisen@zte.com.cn>
|
I don't think this is correct -- peons can't unilaterally grant themselves more time before their leases are stale and might be outdated. We also need to timeout the leader if it gets stuck in some kind of recovery loop, so that we do another election and pick one that's better-behaved. Did you track this down as a result of some specific behavior, or was it by inspection? |
|
@gregsfortytwo HI! Thank you for review. I think the here what I change is not conflict with what you said.The main reason I change here is because as what i said above. |
|
I've seen this behavior being triggered in the past and I agree it's something we should fix. For instance, if the leader takes some time before sending the collect messages, we may easily incur in a scenario in which the peons only handle the messages after their lease times out, or way before they get their proper lease. There may be way too much time in-between, triggering unnecessary elections. After all, we're still in the recovery state by this time, and there are no reads or writes going through paxos, we can be a bit lenient the lease timeout. I think this patch solves the issue and will not affect the leases. After all, the patch itself is only resetting the lease timeout, not setting the lease (Paxos::reset_lease_timeout() does not set the lease, that happens only on Paxos::handle_lease()). Also, there's no chance that if get stuck in a loop, unless we have a misbehaved leader that keeps on sending collects for all eternity. If the leader is somehow bad, we still have a timeout set that will trigger a new election after 10 seconds. The only difference now is that we will push the timeout window 10 seconds into the future by resetting the timeout each time we get a collect message (which makes sense, given not only the leader is resetting its timeout but we're going to do IO during this period, and IO may take time). This LGTM but some testing would be nice. |
|
Also, forgot to mention that this patch will still maintain the same behavior of timing out after 10 seconds if the leader does not send the collect messages. If the leader is broken somehow, the peons will timeout after 10 seconds have passed between Paxos::peon_init() and waiting for the collect messages. The only thing that's different is that the peons will reset this timeout on each collect they get. And while on one hand I would love to use 'collect_timeout_event' instead for this, I gather that using the lease timeout is the simpler, albeit not as intuitive, approach. After all, as it has been since time immemorial, lease timeouts only exist in a peon's context, while collect timeouts only exist in the leader's context, so not changing this to collect_timeout_event is not the worst thing ever. |
mon: paxos add the timeout function when peon recovery
Now the peon only have 10s from init to active.But when the leader
from init to active he may start twice or more recovery collect when
meet a higher pn than himself.It means the leader from init to active
may spend more than 10S. So it is not reasonable for the peon only have
one 10s timeout event.
Signed-off-by: song baisen song.baisen@zte.com.cn