kvserver: cpu spike during import due to split stats computation

[kvoli]

Describe the problem

High CPU utilization during an import, which ranges are being split and scattered. The CPU is attributed to AdminSplitRequest computing the user MVCC stats of the (to be) LHS range:

cockroach/pkg/kv/kvserver/replica_command.go

Line 537 in 610688b

userOnlyLeftStats, err = rditer.ComputeStatsForRangeUserOnly(

To Reproduce

See (internal) thread.

Expected behavior

CPU doesn't spike beyond a nominal 10-15%.

Additional data / screenshots

There are profiles and cluster information linked in the above (internal) thread.

Environment:

• CockroachDB version V24.1.0-ALPHA.5-DEV-C43F54CDDE5B7578F4A0CA61DE41463F0D690993

Jira issue: CRDB-37797

[miraradeva]

@kvoli and I looked at this together and it does seem likely that the new split behavior, introduced in #119894, is causing instability in two ways (see below). The precondition here is that the import is issuing tens (we see 60 in the logs for n11) of manual splits on the same non-empty range. In the new split behavior, each split first re-computes the MVCC stats for the range (to prevent stats from drifting across successive splits), and then computes the stats for the user data on the left hand side (to pass to the split trigger).

1. Issuing multiple RecomputeStats requests concurrently for the same range causes latch contention on the range descriptor. This results in many failed splits because many of the RecomputeStats request wait for a while and the split times out.

2. Computing the LHS user-data stats tens of times concurrently for the same range can easily account for the CPU utilization we see. If the range is half-full, at 256MB, 60 such scans would amount to scanning over 15GB of data. The reason this was not an issue with the old accurate-stats splits is because the stats computation would happen in the split trigger while holding latches; acquiring latches would naturally sequence these splits, and not cause a CPU spike.

To confirm the above, we can disable estimated-stats splits (kv.split.estimated_mvcc_stats.enabled) and ensure there are no CPU spikes. To address 1, we can try disabling stats re-computation (kv.split.mvcc_stats_recomputation.enabled). For 2, we would need to change the current behavior. One option is to revert back to accurate-stats computation for manual AdminSplit requests. That should be ok because the main use cases for the new estimate-stats splits were mostly around size-based and load-based splits.