kvserver: use shorter lease durations

[erikgrinaker]

Currently, range leases are fairly long-lived at 9 seconds:

multiplier * electionTimeoutTicks * tickInterval = 3 * 15 * 200ms = 9s

cockroach/pkg/base/config.go

Lines 442 to 444 in 8817c28

	func (cfg RaftConfig) RangeLeaseDurations() (rangeLeaseActive, rangeLeaseRenewal time.Duration) {
	rangeLeaseActive = time.Duration(cfg.RangeLeaseRaftElectionTimeoutMultiplier *
	float64(cfg.RaftElectionTimeout()))

This is also true for epoch-based leases, since the node liveness timeout simply calls through to the range lease interval:

cockroach/pkg/base/config.go

Lines 471 to 472 in 8817c28

	func (cfg RaftConfig) NodeLivenessDurations() (livenessActive, livenessRenewal time.Duration) {
	livenessActive = cfg.RangeLeaseActiveDuration()

This can result in a fairly long delay when the leaseholder is lost. We should try to reduce this down e.g. 5 seconds. However, many other components also rely on node liveness, so we must balance this against the impact of node flapping (or consider using separate intervals for node liveness and epoch-based lease expiration).

Jira issue: CRDB-14889

[bdarnell]

We should consider the raft heartbeat and election constants at the same time. All of these constants were set very early and never seriously reevaluated. Network timeouts too.

The entire recovery flow is complicated and it's more than just the 9s lease timeout, although that's the single biggest piece. Cutting the lease timeout in half is probably the best thing we can do to speed up recovery, but beyond that I'm not sure whether further work on the lease timeout would be as effective as improving other areas.

Raft election interval is currently set to 3s, but within raft it's actually transformed into a random number between X and 2X. So we could theoretically be spending 6s waiting on raft elections regardless of the lease duration. (it's quantized by the 200ms tick interval, and each replica picks independently. So each node has a 1/5 chance of picking a 6s timeout, but the faster node wins, so with a replication factor of 3 (and one down node) there's a 1/25 chance that we have to wait the full 6s. A replication factor of 5 reduces that to 1/625).

The raft election interval is currently set to 3x the heartbeat interval, which is common in the raft literature but I'm not certain it makes sense. It's useful for unreliable transports, so that a single dropped packet doesn't trigger an election, but since we're using TCP for everything we could probably run with the election and heartbeat intervals closer together. Something like electionTimeout = heartbeatInterval + 2*RTT.

Network RTT is a factor in a lot of these considerations. It would be great if we could give ranges different configurations based on whether they're contained within a single region or not.

Because we use the raft prevote feature, it's less harmful to set the raft election timeout too short than it is to set the lease duration too short. In theory, the range can keep operating as normal if one node incorrectly thinks the leader has failed because network congestion prevented the timely delivery of a heartbeat.

Setting the raft heartbeat interval too low has a cost in both network traffic and cpu on a per-range basis. We "coalesce" heartbeats to reduce this cost but it's still there and it adds up; this has implications for our maximum data density per node.

Tweaking the tick interval probably doesn't make too much difference either way, but reducing it might help by reducing the chance of contested elections (due to two replicas picking the same effective election timeout).

Similarly, setting the node liveness heartbeat interval too low increases pressure on this critical range, and setting the lease duration too low increases the chance that a lease will appear to be expired while the node is in fact still up. This is disruptive because when a liveness epoch expires, other replicas can attempt to invalidate it so they can steal the lease.

From memory, the overall recovery flow (for the worst case of a node that vanishes just after sending all its heartbeats) is something like this:

• Some SQL gateway node tries to talk to a range on the dead node, and waits for the 3s network timeout. (This can be cut short if something in the network is able to send a TCP RST packet)
• That gateway tries to talk to other replicas of the same range. One of those replicas will check the lease, so this is where we wait for it to expire (with the gateway bouncing between replicas). We've already waited 3s on the network timeout, so we wait 6 more seconds.
• If the range is not quiesced, one of the surviving replicas might try to start a raft election during this time (does this state of affairs automatically unquiesce? I can't remember).
• Once the liveness epoch has expired, one of the living replicas sends a CPut to the node liveness range to mark the node as dead and waits for it to commit.
• If the range is still quiesced, this is where the raft election starts. The timeout has already elapsed, so we just go through the 3 RTT election process (prevote, vote, append).
• One (or both) of the living replicas attempt to acquire the lease through a raft append operation.
• If the lease and leaseholder are not colocated, we go through another raft election/leader transfer (because this is a cooperative leadership transfer, no election timeout and just the 3 RTTs. This might even be a fast path and do 2 RTTs instead).
• Now the new lease is valid and usable

So that's a 9s wait plus 2 global RTTs (writing to node liveness) plus about 6 RTTs for the range's quorum. If we reduce the 9s lease duration to 5s, the network and raft election timeouts start to become relevant.

[rail]

Manually synced with Jira

[tbg]

From a discussion yesterday: maybe the single most impactful thing we can do here is to make sure that the expiration-based lease duration on the liveness range is such that a non-cooperative lease transfer happens quickly enough to avoid invalidating all epoch-based leases in the cluster.

[erikgrinaker]

Wrote up a separate issue to track this: #88443

[erikgrinaker]

Initial PR in #91947, reducing the lease interval to 5.0s and Raft election timeout to 2.0s.