New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kvserver: fix Replica.tenantLimiter race #110806
Conversation
It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR? 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
Writing a test for it. |
4616453
to
406552e
Compare
Stressed the new test a bunch, works. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, thanks!
Release note: none Epic: none
If a Replica is destroyed while or shortly before Send waits for the tenant rate limiter, the rate limiter returns a quotapool.ErrClosed which propagates all the way up to the SQL client. This commit fixes maybeRateLimitBatch to return the Replica destruction status error instead, so that the Send stack retries the request. Release note (bug fix): fixed a race condition in Replica lifecycle which could result in a failed SQL request in cases where it could be successfully retried. Epic: none
406552e
to
c3f532f
Compare
bors r=erikgrinaker |
Build succeeded: |
Encountered an error creating backports. Some common things that can go wrong:
You might need to create your backport manually using the backport tool. error creating merge commit from a1202b9 to blathers/backport-release-22.2-110806: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict [] you may need to manually resolve merge conflicts with the backport tool. Backport to branch 22.2.x failed. See errors above. error creating merge commit from a1202b9 to blathers/backport-release-23.1-110806: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict [] you may need to manually resolve merge conflicts with the backport tool. Backport to branch 23.1.x failed. See errors above. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
If a
Replica
is destroyed while or shortly beforeSend
waits for the tenant rate limiter, the rate limiter returns aquotapool.ErrClosed
which propagates all the way up to the SQL client. This commit fixesmaybeRateLimitBatch
to return theReplica
destruction status error instead, so that theSend
stack retries the request.Release note (bug fix): fixed a race condition in
Replica
lifecycle which could result in a failed SQL request in cases where it could be successfully retried.Fixes #109729
Epic: none