release-20.1: kvserver: synchronize replica removal with read-only requests #64377
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
Makes progress on cockroachdb#8299 This commit introduces a new type of ManualClock a HybridManualClock. and wires it into the TestCluster. This clock follows the physical wall time of a regular clock, but allows the developer to move it forward. This is needed to be able to test functionality around lease expiration and other time based mechanisms. To verify that the clock is usefull in tests, a single test in client_merge_tests.go is converted to use TestCluster with a HybridManualClock. To make the test possible we also need a simple way to create a range with an expiration based lease. This is done through the new TestCluster.ScratchRangeWithExpirationLease function. Release note: None
This backports the `HybridManualClock` changes from cockroachdb#59333, but not the client replica test changes which have additional backport dependencies. --- We introduce a new function on hlc.HybridManualClock to pause the clock. This allows a test to conduct specific time based measurements, which are otherwise tricky to do when the time is moving. Release note: None
Pause() reads the clock and saved the reading to be returned from now on. The bgu was that the reading and the saving were not atomic - it was possible to read, then the still-not-paused clock to hand out higher timestamps to others, then to write. This created a backwards clock jump, which is unexpected. In particular, such a jump broke the logic in TestLeaseholdersRejectClockUpdateWithJump. Fixes cockroachdb#60914 Release note: None
The clock pretended to support different underlying time sources, but it didn't really. This patch removes indirection when calling UnixNano(). Release note: None
erikgrinaker
force-pushed
the
backport20.1-64324
branch
2 times, most recently
from
9ff3fcb
to
288cdd4
Compare
This backports `TestCluster.MoveRangeLeaseNonCooperatively` and related test infra from cockroachdb#59086, avoiding backporting the code changes. Release note: None
Replica removal did not synchronize with in-flight read-only requests, which could cause them to be evaluated on a removed (empty) replica, returning an empty result. This patch fixes the problem by locking `Replica.readOnlyCmdMu` during replica removal, thus either waiting for read-only requests to complete or not evaluating them. Release note (bug fix): Fixed a race condition where read-only requests during replica removal (e.g. during range merges or rebalancing) could be evaluated on the removed replica, returning an empty result.
erikgrinaker
force-pushed
the
backport20.1-64324
branch
from
288cdd4
to
47b5050
Compare
tbg
approved these changes
Apr 30, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport 1/1 commits from #64324. Also backport necessary test infrastructure from #59059, #59086 (partial), #59333 (partial), and #64242.
/cc @cockroachdb/release @cockroachdb/kv
Replica removal did not synchronize with in-flight read-only requests,
which could cause them to be evaluated on a removed (empty) replica,
returning an empty result.
This patch fixes the problem by locking
Replica.readOnlyCmdMuduringreplica removal, thus either waiting for read-only requests to complete
or not evaluating them.
Resolves #64325.
Release note (bug fix): Fixed a race condition where read-only requests
during replica removal (e.g. during range merges or rebalancing) could
be evaluated on the removed replica, returning an empty result.