Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
kv: retry liveness heartbeat on race with insufficient expiration
Fixes #124693. Fixes #125287. This commit adds logic to retry the synchronous liveness heartbeat which ensures that the liveness record has a later expiration than the prior lease by the time the lease promotion goes into effect, which was added in #123442. This heartbeat may need to be retried because it may have raced with another liveness heartbeat which did not extend the liveness expiration far enough. We opt to allow this race and retry across it instead of detecting it and returning an error from `NodeLiveness.Heartbeat` because: 1. returning an error would have a larger blast radius and could cause other issues. 2. returning an error wouldn't actually fix the tests that are failing, because they would still get an error, just a different one. Before this commit, `TestLeaseQueueProactiveEnqueueOnPreferences` would hit this case fail under deadlock and stress every ~150 iterations. After this commit, the test passes continuously under deadlock stress for over 2000 runs. This makes #123442 even uglier. The nicer solution is #125235, but that is not backportable. Still, we're planning to address that issue as part of the leader leases work, so this is a temporary fix. This also removes a TODO added in 1dc18df. As mentioned above, we don't address it, but instead document the behavior. Release note (bug fix): resolves a concerning log message that says "expiration of liveness record ... is not greater than expiration of the previous lease ... after liveness heartbeat". This message is no longer possible.
- Loading branch information