Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

There should be only one stream of preparing disk snapshot backup #16382

Closed
YuJuncen opened this issue Jan 15, 2024 · 4 comments · Fixed by #16388
Closed

There should be only one stream of preparing disk snapshot backup #16382

YuJuncen opened this issue Jan 15, 2024 · 4 comments · Fixed by #16388
Labels
affects-6.5 affects-7.1 affects-7.5 component/backup-restore Component: backup, import, external_storage severity/major type/enhancement Type: Issue - Enhancement

Comments

@YuJuncen
Copy link
Contributor

YuJuncen commented Jan 15, 2024 •
edited

Development Task

For now, we allow multi streams of preparing disk snapshot backup and just write some logs if there are many.
But it isn't good enough for some cases that a user mistakenly run two backups: the former backup will probably finish prior than the latter one, and the former one's its Finish call will in fact break the latter one's stream with an inconsistency error.

Also it is possible to bring some ABA problems, that is:

  1. Client A starts prepare stream.
  2. Client B starts prepare stream, and soon exits. (So, TiKV isn't in the prepared state and it is possible to execute commands that may ruin the backup)
  3. Client C starts a prepare stream again.
  4. Client A checks the lease, everything is alright. But the invariance of backup has been broken.
@YuJuncen YuJuncen added the type/enhancement Type: Issue - Enhancement label Jan 15, 2024
@YuJuncen YuJuncen mentioned this issue Jan 16, 2024
Merged
6 tasks
@YuJuncen
Copy link
Contributor Author

(Strictly say this should also be picked to release-7.1 and release-7.5, but most of our efforts of enhancing disk snapshot backup focus on release-6.5, let's bring it back only to there firstly.)

ti-chi-bot bot pushed a commit that referenced this issue Jan 25, 2024
@YuJuncen
snap_backup: abort last connection of preparing while there are many (#…
5cf15aa 
…16388)

close #16382

Now, a newly established prepare disk snapshot backup stream will abort the former one.

Signed-off-by: Yu Juncen <yu745514916@live.com>
@BornChanger
Copy link
Contributor

/severity major

@BornChanger
Copy link
Contributor

/label affects-7.1

@BornChanger
Copy link
Contributor

/label affects-7.5

@ti-chi-bot ti-chi-bot mentioned this issue Jan 25, 2024
Merged
6 tasks
ti-chi-bot bot added a commit that referenced this issue Jan 25, 2024
@ti-chi-bot @YuJuncen @ti-chi-bot
snap_backup: abort last connection of preparing while there are many (#…
cde0928 
…16388) (#16448)

close #16382

Now, a newly established prepare disk snapshot backup stream will abort the former one.

Signed-off-by: hillium <yujuncen@pingcap.com>
Signed-off-by: Yu Juncen <yu745514916@live.com>

Co-authored-by: hillium <yujuncen@pingcap.com>
Co-authored-by: Yu Juncen <yu745514916@live.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@jebter jebter added the component/backup-restore Component: backup, import, external_storage label Feb 20, 2024
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Feb 20, 2024
@YuJuncen @ti-chi-bot
This is an automated cherry-pick of tikv#16388
62cdee1 
close tikv#16382

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Feb 20, 2024
Closed
6 tasks
YuJuncen added a commit to ti-chi-bot/tikv that referenced this issue Feb 20, 2024
@YuJuncen
snap_backup: abort last connection of preparing while there are many (t…
c062464 
…ikv#16388)

close tikv#16382

Now, a newly established prepare disk snapshot backup stream will abort the former one.

Signed-off-by: Yu Juncen <yu745514916@live.com>
YuJuncen added a commit to ti-chi-bot/tikv that referenced this issue Feb 20, 2024
@YuJuncen
snap_backup: abort last connection of preparing while there are many (t…
433aa74 
…ikv#16388)

close tikv#16382

Now, a newly established prepare disk snapshot backup stream will abort the former one.

Signed-off-by: Yu Juncen <yu745514916@live.com>
dbsid pushed a commit to dbsid/tikv that referenced this issue Mar 24, 2024
@YuJuncen @dbsid
snap_backup: abort last connection of preparing while there are many (t…
b940097 
…ikv#16388)

close tikv#16382

Now, a newly established prepare disk snapshot backup stream will abort the former one.

Signed-off-by: Yu Juncen <yu745514916@live.com>
Signed-off-by: dbsid <chenhuansheng@pingcap.com>
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Apr 11, 2024
@YuJuncen @ti-chi-bot
This is an automated cherry-pick of tikv#16388
8146985 
close tikv#16382

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Apr 11, 2024
Open
6 tasks
mittalrishabh pushed a commit to mittalrishabh/tikv that referenced this issue May 8, 2024
@YuJuncen
snap_backup: abort last connection of preparing while there are many (t…
b359e35 
…ikv#16388) (tikv#14)

close tikv#16382

Now, a newly established prepare disk snapshot backup stream will abort the former one.

Signed-off-by: Yu Juncen <yu745514916@live.com>
Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.5 affects-7.1 affects-7.5 component/backup-restore Component: backup, import, external_storage severity/major type/enhancement Type: Issue - Enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

snap_backup: abort last connection of preparing while there are many YuJuncen/tikv
3 participants
@jebter @YuJuncen @BornChanger