backupccl: pause scheduled backup if resumed on different cluster #111578
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
msbutler
changed the title
msbutler
force-pushed
the
butler-c2c-cutover-schedule
branch
3 times, most recently
from
c273cba
to
e215128
Compare
msbutler
requested review from
dt,
jayshrivastava,
srosenberg,
smg260 and
adityamaru
and removed request for
a team,
jayshrivastava,
srosenberg and
smg260
|
@dt @adityamaru This is ready for a look. The first two commits are getting reviewed in other prs: Test server change is here, and the clusterID/Version I have one high level design q: should I check for the clusterID in the |
dt
reviewed
Oct 2, 2023
There was a problem hiding this comment.
2¢: firing up a whole c2c end to end test with two tenants and cutover and schedules all involved feels like a kinda expensive way to cover this one behavior of schedules to me, and far away from the code in question. I might vote to just test this by manually creating a schedule with a different (and empty!) creator in create_scheduled_backup_test.go, removing all streamingccl/ diffs?
msbutler
force-pushed
the
butler-c2c-cutover-schedule
branch
2 times, most recently
from
b2c50ae
to
5840e76
Compare
|
@dt ready for another look! Added a better test in create_scheduled_backup_test.go |
dt
approved these changes
Oct 3, 2023
msbutler
changed the title
msbutler
force-pushed
the
butler-c2c-cutover-schedule
branch
2 times, most recently
from
050bc49
to
800de37
Compare
Previously, after cluster restore or c2c cutover, a backed up/ replicated backup schedule would begin executing on the new cluster. If the backup / source cluster was still available and executing the schedule, then the two schedules would compete to run, as outlined in cockroachdb#108028. This patch prevents this poor UX by pausing the backup schedule if the schedule realizes it is running on a new cluster. It is then up to the user to resume backups on the new cluster and prevent the backup collision problem. Fixes cockroachdb#108028 Release note (sql change): if a scheduled backup resumes on a new cluster (e.g. after C2C cutover or a tenant restore), the backup schedule will pause. The user may resume the schedule without changing it, but should take special care to ensure no other schedule is backing up to the same collection. The user may also want to cancel the paused schedule and start a new one.
msbutler
force-pushed
the
butler-c2c-cutover-schedule
branch
from
800de37
to
ace81b6
Compare
msbutler
added a commit
to msbutler/cockroach
that referenced
this pull request
Oct 5, 2023
This patch adds an e2e c2c test to verify that replicated backup schedules pause on the destination tenant after cutover. This functionality was added in cockroachdb#111578. Informs cockroachdb#108028 Release note: none
|
bors r=dt |
|
Build succeeded: |
msbutler
added a commit
to msbutler/cockroach
that referenced
this pull request
Oct 5, 2023
This patch adds an e2e c2c test to verify that replicated backup schedules pause on the destination tenant after cutover. This functionality was added in cockroachdb#111578. Informs cockroachdb#108028 Release note: none
craig bot
pushed a commit
that referenced
this pull request
Oct 5, 2023
111804: streamingccl: verify backup schedule pauses after cutover r=stevendanna a=msbutler This patch adds an e2e c2c test to verify that replicated backup schedules pause on the destination tenant after cutover. This functionality was added in #111578. Informs #108028 Release note: none Co-authored-by: Michael Butler <butler@cockroachlabs.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, after cluster restore or c2c cutover, a backed up/ replicated
backup schedule would begin executing on the new cluster. If the backup /
source cluster was still available and executing the schedule, then the two
schedules would compete to run, as outlined in #108028.
This patch prevents this poor UX by pausing the backup schedule if the schedule
realizes it is running on a new cluster. It is then up to the user to resume
backups on the new cluster and prevent the backup collision problem.
Fixes #108028
Release note (sql change): if a scheduled backup resumes on a new cluster (e.g.
after C2C cutover or a tenant restore), the backup schedule will pause. The
user may resume the schedule without changing it, but should take special care
to ensure no other schedule is backing up to the same collection. The user may
also want to cancel the paused schedule and start a new one.