kvserver: add MaxInflightMsgBytes to raft

[tbg]

Is your feature request related to a problem? Please describe.

raft.RawNode has two options MaxSizePerMsg, MaxInflightMsgs:

cockroach/pkg/kv/kvserver/store.go

Lines 284 to 285 in 0c9a5da

	MaxSizePerMsg: storeCfg.RaftMaxSizePerMsg,
	MaxInflightMsgs: storeCfg.RaftMaxInflightMsgs,

We set them to 32kb ¹ and 128², respectively.

Consider a slow follower that the leader is replicating a stream of 16mb AddSST entries to. The follower will, for this one range, have to hold up to 128*16m = 2G worth of entries in memory. If this happens on even a handful of ranges, an OOM is likely to follow; we see this happen in practice³.

Note that the quota pool (I think defaulting to 8mb right now) can sort of inhibit this undesirable behavior in the steady state. But once a node falls out of the quota pool, for example since it was recently restarted or something like that (I assume slow nodes frequently fall out of the quota pool, since we haven't really seen range write stalls caused by them, though that doesn't mean they don't happen - see #79756) there's no stopping the raft leader from catching up the follower in this unintended overly aggressive way.

Describe the solution you'd like

RawNode should have options that allow us to specify the max inflight bytes. We could set this to, say, 64MiB to say that no more than 64MiB should be outstanding at any point in time. (This is an etcd/raft change).

In an ideal world, we'd also be able to budget how much can be outstanding across raft instances, but this is a much harder problem. For example, there may be 1000 ranges for which the leader is sending us 16mb AddSSTs. We would need to have these leaders coordinate (unlikely) or the follower would need to reject messages (this being essentially ³).

Describe alternatives you've considered

Additional context

Jira issue: CRDB-20690

Footnotes

1. https://github.com/cockroachdb/cockroach/blob/2c3916be699e01165193d148e6caef7cb79ad110/pkg/base/config.go#L136-L137 ↩

2. https://github.com/cockroachdb/cockroach/blob/2c3916be699e01165193d148e6caef7cb79ad110/pkg/base/config.go#L145-L148 ↩

3. https://github.com/cockroachdb/cockroach/issues/79752 ↩ ↩²

[blathers-crl]

cc @cockroachdb/replication

[pav-kv]

The current prototype is in etcd-io/etcd#14624.
It requires etcd-io/etcd#14633 to be merged first.

[pav-kv]

The etcd/raft side of work is done. Next steps:

• Import to CRDB: *: upgrade etcd/raft to 8651478c #93066
• Set the MaxInflightBytes parameter of raft.Config: kvserver: use MaxInflightBytes raft config option #94692
• Figure out the optimal default for `MaxInflightBytes: base: reduce RaftMaxInflightBytes to 32 MB #101437.

[pav-kv]

The option is implemented and plumbed into CRDB. The default is a generous 256MB which should not disrupt existing workflows. We may revisit this setting in the future.

UPD: it is 32 MiB now [#101437].