Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spanconfig: correctly order updates when applying in the KVSubscriber #118001

Merged
merged 1 commit into from Feb 5, 2024

Conversation

arulajmani
Copy link
Collaborator

Previously, there was no ordering guarantee between KVSubscriber events at the same timestamp. As a result, if a batch of events included updates to overlapping spans at the same timestamp, we could apply additions before deletions -- this would cause the additions to get clobbered, which was not the intention. This could lead to missing span configurations, resulting in bugs such as the linked issue.

This patch fixes the issue by sorting deletions before additions if two span configuration events have the same timestamp.

Closes #110908

Release note (bug fix): Previously, altering from a Regional By Row table to a Regional By Table table could cause leaseholders to never move to the databse's primary region. This is now fixed.

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@arulajmani arulajmani added backport-22.2.x Flags PRs that need to be backported to 22.2. backport-23.1.x Flags PRs that need to be backported to 23.1 backport-23.2.x Flags PRs that need to be backported to 23.2. labels Jan 19, 2024
Copy link
Collaborator

@andrewbaptist andrewbaptist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Just need to resolve some of the build / lint failures.

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @arulajmani)


pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber.go line 455 at r1 (raw file):

		if events[i].Timestamp().Less(events[j].Timestamp()) {
			return true
		} else if events[j].Timestamp().Less(events[i].Timestamp()) {

nit: remove the unnecessary else here.

Also it might be slightly cleaner to call Compare() once and switch rather than calling Less() twice.

Copy link
Collaborator Author

@arulajmani arulajmani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TFTR! The linter should be happy now.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @andrewbaptist)


pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber.go line 455 at r1 (raw file):

Previously, andrewbaptist (Andrew Baptist) wrote…

nit: remove the unnecessary else here.

Also it might be slightly cleaner to call Compare() once and switch rather than calling Less() twice.

Done.

Copy link
Collaborator

@andrewbaptist andrewbaptist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Thanks for the updates.

@arulajmani
Copy link
Collaborator Author

TFTR!

bors r=andrewbaptist

@arulajmani
Copy link
Collaborator Author

Looks like the test I introduced is failing under stress -- leaseholders aren't moving quick enough. I'll deflake before merging.

bors r-

@craig
Copy link
Contributor

craig bot commented Feb 2, 2024

Canceled.

Previously, there was no ordering guarantee between KVSubscriber
events at the same timestamp. As a result, if a batch of events
included updates to overlapping spans at the same timestamp, we could
apply additions before deletions -- this would cause the additions to
get clobbered, which was not the intention. This could lead to missing
span configurations, resulting in bugs such as the linked issue.

This patch fixes the issue by sorting deletions before additions if
two span configuration events have the same timestamp.

Closes cockroachdb#110908

Release note (bug fix): Previously, altering from a Regional By Row
table to a Regional By Table table could cause leaseholders to never
move to the databse's primary region. This is now fixed.
@arulajmani
Copy link
Collaborator Author

Deflaked and merging. I just bumped the scan interval for this test, to ensure leases end up in the right place quickly.

bors r+

@craig
Copy link
Contributor

craig bot commented Feb 5, 2024

Build succeeded:

@craig craig bot merged commit f68f0fe into cockroachdb:master Feb 5, 2024
8 of 9 checks passed
Copy link

blathers-crl bot commented Feb 5, 2024

Encountered an error creating backports. Some common things that can go wrong:

  1. The backport branch might have already existed.
  2. There was a merge conflict.
  3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.


error creating merge commit from 73cb0da to blathers/backport-release-22.2-118001: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.2.x failed. See errors above.


error creating merge commit from 73cb0da to blathers/backport-release-23.1-118001: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 23.1.x failed. See errors above.


error setting reviewers, but backport branch blathers/backport-release-23.2-118001 is ready: POST https://api.github.com/repos/cockroachdb/cockroach/pulls/118794/requested_reviewers: 422 Reviews may only be requested from collaborators. One or more of the teams you specified is not a collaborator of the cockroachdb/cockroach repository. []

Backport to branch 23.2.x failed. See errors above.


🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-22.2.x Flags PRs that need to be backported to 22.2. backport-23.1.x Flags PRs that need to be backported to 23.1 backport-23.2.x Flags PRs that need to be backported to 23.2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

spanconfig: incorrect span config after switching to from regional by row to regional by table
3 participants