kv,kvcoord: assign high priority for admission control if the txn has…

[sumeerbhola]

… locks

This is a crude way to limit priority inversion where a txn holding locks
could be waiting in an admission queue while admitted requests are waiting
in the lock table queues for this txn to make progress and release locks.

It can also fare better than no admission control, since work from txns
holding locks will get prioritized, versus no prioritization in goroutine
scheduler.

A tpcc run with 3000 warehouses shows 2x reduction in lock waiters and
10+% improvement in txn throughput with this change (both before and
after experiments running with admission control enabled). When compared
with admission control disabled, we see even higher improvements in lock
waiters and txn throughput.

Some before/after graphs from running tpcc with 3000 warehouses, and
comparison with no admission control.

before (lock waiters):

after (lock waiters):

no admission control (lock waiters):

before (txn throughput and latency):

after (txn throughput and latency):

no admission control (txn throughput and latency):

Informs #65955

Release note: None

Release justification: Low-risk update to new functionality.

[cockroach-teamcity]

This change is 

[RaduBerinde]

What a dramatic difference in tail latencies!

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @ajwerner)

[sumeerbhola]

TFTR!
I've also done a run with no admission control, since lack of prioritization of lock holders could make that worse. The throughput and latency is worse with no admission control. See attached screenshots. In this experiment both cpu and storage became a bottleneck in one node, whose runnable goroutines exceeded 100 and L0 sub-levels exceeded 30.

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @ajwerner)

[RaduBerinde]

Very cool results!

[sumeerbhola]

bors r+

[craig]

Build succeeded:

• GitHub CI (Cockroach)