kv: chronic intent-cleanup wedge on hot-key UPSERT workload; scan-storm saturates leaseholder, lease migration and splits don't fix it

[cpj2195]

Summary

On a long-running internal DRT chaos cluster, a high-concurrency UPSERT workload (gitload) produces a chronic, repeatable wedge on a single hot range (r138893) that recurs every ~20 minutes for >2 hours, even though:

• The cluster has no node failures, panics, OOMs, or disk stalls.
• Lease migrations (n3→n5→n4→n1) don't fix it — the wedge follows the lease.
• The splitter does fire (+25 ranges observed), but the post-split right-hand side still co-locates the workload's hot tables and inherits the contention.
• A goroutine dump during a brief recovery window shows the leaseholder is healthy at the goroutine layer (no lock-table waiters, no PushTxn pile-up, 17 runnable goroutines, healthy state distribution).
• Cleanup backlog at the SQL gateway node (78 intentResolver.gcTxnRecord goroutines all blocked in requestbatcher.Send) is consistent with a downstream slow range that the cleanup pipeline can't drain.

We don't have a confident root-cause hypothesis from the data we've gathered so far. Filing this as an investigation to:

1. Track the reproducer.
2. Ask for guidance on what additional profile (block / mutex / Pebble-level) would surface the actual bottleneck.

Environment

• CockroachDB master, 2026-06-25
• 6-node AWS cluster, EBS storage
• Concurrent workloads on the same cluster:
  • tpcc (warehouses=12000)
  • kv (concurrency=128)
  • gitload (UPSERT-heavy custom workload; 32 agents; INSERT … ON CONFLICT DO UPDATE WHERE last_topo_order < excluded.last_topo_order against ~500 hot PRIMARY KEY values on a newly-created file_latest table)
• Time window observed: 2026-06-25T13:14:00Z onwards (~2.5 h continuously)

The reproducer

• 32 agents UPSERTing concurrently into a small PK hotspot (~500 keys on file_latest) + ~hundreds of related INSERTs into adjacent tables (commits, commit_diffs, commit_parents).
• All three tables landed on the existing trailing range r138893 when the gitload DB was first created, so r138893 spans /{Table/4092/12/‹11763›/‹3›/‹2766›-Max} covering both TPCC table 4092 and gitload tables 4102/4103/4106.
• Within ~10 min of workload start, r138893 begins exhibiting the wedge pattern.

The wedge pattern (4 windows of observation)

Each ~20 min cycle:

1. intentcount climbs from ~100 to 2400–2700.
2. Hundreds of slow range RPC warnings fire on r138893: ConditionalPut /Table/4106/1/"dir_N/file_M.txt"/0 taking 500–1193 s; pusher … waiting … for pushee … PushTxn calls timing out at 60–1190 s.
3. closed_timestamp.max_behind_nanos climbs monotonically (2.3s → 10.84s over 2 h, never recovers).
4. Lease eventually transfers (e.g., (n3,s9):8 → (n6,s24):6); the pile-up clears within seconds; cluster has a ~7–8 min "clean window."
5. Re-saturates within minutes; next cycle starts.

The pushee txn IDs rotate every cycle (5 distinct sets observed across 4 windows) but the contended keys are stable — gitload's hot UPSERT keys on Table/4102 and Table/4103, plus a transient locker hot-spot on Table/4106/1/"dir_2/file_2.txt" in one window.

Why "lock-table starvation" is NOT supported (corrected from earlier hypothesis)

Goroutine dumps from all three replicas of r138893 (n1 leaseholder, n4, n6) during a brief recovery window between wedge cycles show:

• 0 goroutines in lockTableWaiterImpl.WaitOn
• 0 goroutines in PushTxn paths
• 0 goroutines in concurrency.Manager.SequenceReq or related lock-table code

State distribution on n1 is healthy: 1416 select / 200 sync.Cond.Wait / 102 IO wait / 17 runnable (out of 1837 total). No goroutine death spiral; no piled-up backend goroutines.

So the wedge is not visible at the goroutine layer on the leaseholder — at least not in the dump we captured.

What IS visible at the goroutine layer (post-wedge backlog)

On node n4 (likely a heavy gateway node based on connection count), 78 distinct goroutines are stuck in:

intentResolver.cleanupFinishedTxnIntents.func2
  → intentResolver.gcTxnRecord  (intent_resolver.go:888)
    → requestbatcher.(*RequestBatcher).Send   ← blocked here
      [RunWithTimeout 20s]

Each is a post-commit txn-record GC request that can't land on its destination range. n1 (leaseholder) has 20 of these; n6 has 3. The asymmetry plus the consistent backlog pattern suggests these are accumulated backlog from a prior wedge cycle that's draining (consistent with our hypothesis that the dump was taken in a brief recovery window).

Concrete observations from 4 successive 30-min Datadog windows

Window	r138893 wedged	intentcount oscillating	Closed-ts lag	Lease moves	Splitter
13:14–13:36	yes (22 min)	n/a	2.3s → 5.3s	(n3,s9):8 stuck, eventually moves to (n6,s24):6	no split during window
14:00–14:25	yes	n/a	5.3s → ~7s	replica churn every few min	no split
14:34–15:04	yes	2 oscillation cycles, brief recovery 14:50–14:57	~7s → 9.71s	(n3,s9):8 → (n1,s1):19	+25 ranges (split at ~14:54)
14:53–15:23	yes	2 oscillation cycles	9.06s → 10.84s	(n1,s1):19 (stable) → followers churn	brief +43 then -28 (split + merge)

What's notable:

• Every window has the wedge active or recovering from it; not a single 30-min window has been clean.
• The split at 14:54 produced +25 ranges, but the post-split RHS still owns all gitload tables and re-wedged within minutes.
• The lease has migrated 3 times in 2 h. The wedge migrates with it.
• closed-ts lag is monotonic and decoupled from intent state: during the 8-min clean window at 15:06–15:16 when intentcount dropped to ~100, lag continued climbing at ~60ms/sec.

Mechanism (high confidence; captured during an active wedge)

A subsequent active-wedge capture showed the bottleneck concretely:

Range stats on the currently-hot gitload range:

• intent_count: 1,116 active intents
• key_count: 360,111 total MVCC keys
• live_count: 86,283 live rows
• key-to-live-row ratio: 4.2× (each row has ~4 MVCC versions stacked on top)

Leaseholder CPU profile (30s sample window during wedge):

• ~7.7 cores actively burning in Pebble iteration:
  • pebble.(*Iterator).internalNextPrefix cumulative 13.2%
  • pebble.(*mergingIter).nextEntry cumulative 10.6%
  • cockroachkvs.Compare cumulative 2.8%
• Not goroutine-blocked; actively scanning MVCC data.

Stuck goroutines on leaseholder:

• 256 goroutines in intentResolver.cleanupFinishedTxnIntents →
  intentResolver.gcTxnRecord (intent_resolver.go:888) →
  requestbatcher.(*RequestBatcher).Send, each holding a 20s RunWithTimeout.

Mutex profile:

• 59.9% of mutex delay cumulative through kv.(*DB).sendUsingSender
• 51.4% through kv.(*Txn).Commit
• Confirms the request batcher's lock serialization is the queue point.

Reconstructed feedback loop:

1. Concurrent UPSERTs on hot keys deposit write intents.
2. After commit, cleanupFinishedTxnIntents issues an MVCC scan to find each txn's intents to resolve.
3. Scans are expensive because hot keys carry many MVCC versions (4.2× ratio observed).
4. Scans saturate leaseholder CPU → cleanups don't drain → intent count climbs.
5. New UPSERTs scan past more intents → take longer.
6. Lease transfer doesn't help — wherever the lease goes, the same per-key scan cost reproduces.
7. Splits don't help — splitter fired naturally (+25 ranges observed in one 30-min window) but the post-split RHS inherited the hot keys.

The mechanism is per-key, not per-range. Manual pre-splitting was not tested.

Asks

1. Is this interaction (per-txn cleanupFinishedTxnIntents issuing MVCC scans that saturate leaseholder CPU on a hot keyspace) a known issue or already tracked? Closest hit in our search was concurrency: committed transaction's unresolved intents can cause spurious lock timeout errors #165097 (parallel-commit + async intent resolution → spurious PushTxn timeouts), but the mechanism we observed is one layer deeper — the cleanup scans themselves, not the PushTxn waits.
2. Is there a cluster setting or admin hatch to:
   • Throttle / batch / coalesce intent-cleanup scans on hot keyspaces
   • Make gcTxnRecord async rather than blocking the cleanup goroutine for 20s
   • Trigger aggressive MVCC GC on a specific hot range to drain accumulated versions
3. Is MVCC GC expected to keep the version-to-live ratio low under sustained hot-key UPSERT load? The 4.2× ratio we observed (key_count=360k vs live_count=86k) is what makes the scans expensive; if GC could keep this ratio at 1.5× or 2×, the scan cost would presumably stay manageable.
4. Manual pre-splitting was not tested. Would you expect aggressive pre-splits of the hot tables (e.g., ~50 splits across file_latest's 500-key PK space) to mitigate, or does the per-key scan cost make sharding irrelevant?

Related issues (for context — not asserting any of these match)

• concurrency: committed transaction's unresolved intents can cause spurious lock timeout errors #165097 (parallel-commit + async intent resolution → spurious PushTxn timeouts) — mechanism is plausible but we don't see goroutine-level evidence
• admission: bypassing QueryTxn/PushTxn work can starve out non-bypassing AC work #99253 (PushTxn bypasses admission control → starves other work) — likely amplifier if the underlying mechanism is PushTxn-related
• storage: a node with a faulty (but not failed) disk causes a cluster availability outage #129579 (silently-slow leaseholder) — ruled out for this incident; leaseholder metrics are healthy
• kv: operations on a range can stall during lease changes when under heavy load #32367 (closed v19.1; same external symptom) — possible regression worth checking
• kv: closed timestamp can starve read-write txns that take longer than 600ms #51294 (closed; closed-ts climbing under contention) — related to symptom fix golint reports #3

Reproduction notes

Observed continuously over 2.5 h on a long-running internal test cluster under the workload mix described above. We have not deliberately reproduced this in isolation. If a focused repro is useful, the rough recipe is:

• 6-node cluster with sparse range layout (recently-created tables on a single trailing range)
• TPCC + KV running steady-state
• Add an UPSERT-heavy workload doing 32-concurrent INSERT … ON CONFLICT DO UPDATE WHERE … against ~500 distinct PK values on a fresh table whose data initially lands in the existing trailing range
• Watch for slow range RPC warnings on that table's keyspace within ~10–20 min of workload start

Happy to provide a deterministic repro script if helpful — let me know.

Epic: none

Jira issue: CRDB-65153

[nicktrav]

In order to debug this further, we're going to need some artifacts (debug.zip, link to cluster, etc.). @cpj2195 - can you help us with this? Sending it back your way.

[cpj2195]

@nicktrav — reproduced this on the same cluster and captured the artifacts you asked for. The wedge is currently live and continuous under the workload (it oscillates every ~10–20 min butnever fully clears — floor ~1–1.5K intents/node — rather than the ~20-min clean windows seen originally), so it's a standing repro. Happy to give the KV team direct access while it's up.

Same signature as filed: hot-key UPSERT convoy on the trailing range co-locating the gitload tables (file_latest, commits, commit_diffs, commit_parents; table IDs 4220–4224). Live
symptoms:

• intentcount oscillating ~1.5K–22K per node, never returning to baseline
• slow replica RPC on Get/PushTxn (PUSH_ABORT/PUSH_TIMESTAMP, ABORT_REASON_PUSHER_ABORTED); a single head-of-line blocker txn with ~13 txns queued behind it on one file_latest PK (e.g.
  one blocker holding /Table/4220/26/"…" with a dozen Get(Exclusive)/PushTxn waiters)
• WriteTooOldError storms on the adjacent tables

New quantified finding — the MVCC version pileup is far worse than first measured (SHOW RANGES … WITH TABLES, DETAILS):

• hottest file_latest range: 1,332,984 keys / 3,884 live → 343× versions per live row
• commits / commit_diffs / commit_parents ranges: 40–92×. vs the ~4.2× in the original report — roughly 80× worse.

Artifacts (uploaded to gs://drt-artifacts):

• debug.zip — cluster-wide, captured during an active wedge
• CPU + goroutine + mutex profiles from the hot-range leaseholder and from a node holding the frozen closed-ts, captured during an active crest
• cluster_locks convoy snapshot (head-of-line blocker + waiters)
• SHOW RANGES … WITH TABLES, DETAILS version-ratio table
• per-range status JSON for the worst-ratio ranges + problemranges

Clustername: drt-chaos-aws
workload node: workload-chaos-aws
Cluster URL: https://3.148.104.236:26258/. (will share the creds internally to whosoever picks this up)