kv: wait on latches on each key in reverse acquisition order #109349
Conversation
|
It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR? 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
|
This change is |
|
The following test demonstrates the expected improvement from this PR. I ran the following test, which spins up 512 concurrent workload threads to UPSERT the same key. roachprod create local -n1
roachprod put local cockroach
roachprod start local
roachprod run local -- ./cockroach workload run kv --init --cycle-length=1 --concurrency=512 --duration=5mFor the first 5 minutes, I ran with 
|
This commit allocates latch IDs from the top of the uint64 space and in reverse order. This is done to order latches in the tree on a same key in reverse order of acquisition. Doing so ensures that when we iterate over the tree and see a key with many conflicting latches, we visit the latches on that key in the reverse order that they will be released. In doing so, we minimize the number of open channels that we wait on (calls to `waitForSignal`) and minimize the number of goroutine scheduling points. This is important to avoid spikes in runnable goroutine after each request completes, which can negatively affect node health. Epic: None Release note (performance improvement): The impact of high concurrency blind writes to the same key on goroutine scheduling latency was reduced.
nvanbenschoten
force-pushed
the
nvanbenschoten/latchWaitOrder
branch
from
43a7df2
to
c506290
Compare
nvanbenschoten
changed the title
nvanbenschoten
added
the
backport-23.1.x
|
Adding the |
There was a problem hiding this comment.
Reviewed 1 of 1 files at r1, all commit messages.
Reviewable status:
|
TFTR! bors r=arulajmani,kvoli |
|
Build succeeded: |
shralex
added
O-support
This commit allocates latch IDs from the top of the uint64 space and in reverse order. This is done to order latches in the tree on the same key in reverse order of acquisition. Doing so ensures that when we iterate over the tree and see a key with many conflicting latches, we visit the latches on that key in the reverse order that they will be released. In doing so, we minimize the number of open channels that we wait on (calls to
waitForSignal) and minimize the number of goroutine scheduling points. This is important to avoid spikes in runnable goroutine after each request completes, which can negatively affect node health.See experiments below.
Epic: None
Release note (performance improvement): The impact of high concurrency blind writes to the same key on goroutine scheduling latency was reduced.