Commits on Apr 13, 2023

1. base: reduce RaftMaxInflightBytes to 32 MB …

   This patch reduces the default `RaftMaxInflightBytes` from 256 MB to 32
   MB, to reduce the out-of-memory incidence during bulk operations like
   `RESTORE` on clusters with overloaded disks.
   
   `RaftMaxInflightBytes` specifies the maximum aggregate byte size of Raft
   log entries that a leader will send to a follower without hearing
   responses. As such, it also bounds the amount of replication data
   buffered in memory on the receiver. Individual messages can still exceed
   this limit (consider the default command size limit at 64 MB).
   
   Normally, `RaftMaxInflightMsgs` * `RaftMaxSizePerMsg` will bound this at
   4 MB (128 messages at 32 KB each). However, individual messages are
   allowed to exceed the 32 KB limit, typically large AddSSTable commands
   that can be around 10 MB each. To prevent followers running out of
   memory, we place an additional total byte limit of 32 MB, which is 8
   times more than normal.
   
   A survey of CC clusters over the past 30 days showed that, excluding a
   single outlier cluster, the total outstanding `raft.rcvd.queued_bytes`
   of any individual node never exceeded 500 MB, and was roughly 0 across
   all clusters for the majority of time.
   
   Epic: none
   Release note (ops change): the amount of replication traffic in flight
   from a single Raft leader to a follower has been reduced from 256 MB to
   32 MB, in order to reduce the chance of running out of memory during
   bulk write operations. This can be controlled via the environment
   variable `COCKROACH_RAFT_MAX_INFLIGHT_BYTES`.

   

   erikgrinaker committed Apr 13, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for 8136e8e
   • Browse repository at this point

   Copy the full SHA

   8136e8e View commit details

   Browse the repository at this point in the history