kvserver: ambiguous result under high latencies when running TPCC

[aayushshah15]

Describe the problem

When running TPCC 15K on a 20.2 cluster with 15 c5.4xlarge nodes (and in general, when running large TPCC workloads on clusters that are perhaps too weak to handle the load), the workload generator receives a result is ambiguous (intent missing and record aborted) error after p95 latencies degrade into the dozens of seconds.

To Reproduce

Reproduction steps are identical to the ones in cockroachdb/pebble#934, except that running with 15000 active warehouses will reliably reproduce the error.

roachprod run $CLUSTER:16 './workload run tpcc --warehouses=15000 --partitions=5 --ramp=1m --duration=50m {pgurl:1-15}'

Additional context

Related to #53156. After this PR, the only case where we should be getting these errors is when the transaction record of the errant transaction has been garbage collected. By default, this should only happen after 10 minutes. However, this seems not to be the case (see attached workload logs).
pebble15.txt

Jira issue: CRDB-3705

[blathers-crl]

Hi @aayushshah15, I've guessed the C-ategory of your issue and suitably labeled it. Please re-label if inaccurate.

While you're here, please consider adding an A- label to help keep our repository tidy.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[nvanbenschoten]

This is unexpected. As you mentioned, #53156 was supposed to place some harder guarantees around this ambiguous error case. I bet we could figure out what's going wrong with a few well-placed log statements around EndTxn, QueryTxn, PushTxn, and RecoverTxn requests.

[aayushshah15]

My repro steps mention a large cluster since that's what I was playing with at the time, but I think we should be able to repro this on a TPCC 3K run on 3 of those c5d.4xlarge nodes, for example.