release-21.1: kvserver: improve intent cleanup for disconnected clients

[erikgrinaker]

Backport 2/2 commits from #64869.
Backport 1/1 commits from #65385.
Backport 1/1 commits from #65461.

TestReliableIntentCleanup turned out to be flaky, and the flake uncovered additional bugs. Therefore, the test has to be skipped here (to avoid introducing flake), and will be enabled again when #65592 is backported. The test has been verified to pass most of the time on release-21.1.

/cc @cockroachdb/release

---

kvserver: improve intent cleanup for disconnected clients

Whenever a client disconnects during an open transaction or in-flight
DML statement, the server-side session will asynchronously clean up the
transaction record and intents by rolling back the transaction.
However, this had a very short timeout of 3 seconds, and even though the
actual IntentResolver.CleanupTxnIntentsAsync() cleanup task runs
asynchronously and independently of the client's context in the typical
case, the short timeout could cause transaction cleanup to be cancelled
if the EndTxn request took more than 3 seconds to get all the way
through Raft execution or if the async task limit was exceeded such that
the cleanup task kept using the client's context. This in turn could
lead to intents building up over time.

This patch increases the timeout when rolling back transactions for
disconnected clients to 1 minute, and also tries to perform transaction
cleanup when a client disconnects while waiting for an EndTxn command
to go through Raft.

Resolves #64770, touches #60585.

Release note (bug fix): improved transaction cleanup for disconnected
clients, to reduce intent buildup.

kvserver: fix race condition during synchronous txn record cleanup

Transaction records and intents are normally cleaned up asynchronously
via IntentResolver.CleanupTxnIntentsAsync(), separately from the
client's context. When the async task limit is exceeded, cleanup instead
becomes synchronous and attached to the client context. However, the
final gcTxnRecord() call to remove the transaction record is
asynchronous even when intent cleanup is synchronous, to avoid holding
onto the intent resolver task slot. This call will typically return to
the caller before gcTxnRecord() completes, which may cause the caller to
cancel the context (either via defer cancel() or a client
disconnection) and in turn cancel the gcTxnRecord() call.

This patch gives the async gcTxnRecord() call a separate background
context that's independent of the client's context even in the
synchronous case, with a 20 second timeout to avoid goroutine leaks.

Resolves #64868, touches #60585.

Release note (bug fix): Fixed a race condition during transaction
cleanup that could leave old transaction records behind until MVCC
garbage collection.

/cc @cockroachdb/kv

[cockroach-teamcity]

This change is 

[nvanbenschoten]

Reviewed 1 of 1 files at r1, 4 of 4 files at r2, 1 of 1 files at r3, 1 of 1 files at r4.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @aliher1911)