Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

br: Adjust the backup organization structure #12958

Merged
merged 10 commits into from Jul 20, 2022
Merged

br: Adjust the backup organization structure #12958

merged 10 commits into from Jul 20, 2022

Conversation

MoCuishle28
Copy link
Contributor

@MoCuishle28 MoCuishle28 commented Jul 5, 2022 •
edited

Signed-off-by: Gaoming zhanggaoming028@gmail.com

What is changed and how it works?

Issue Number: Close #13063

What's Changed:

Adjust the backup organization structure and add a store_id related prefix under the backup path.

Related changes

None

Check List

Tests

  • Manual test (add detailed scripts or steps below)

Manual test

  • run tiup br backup db to backup the db to S3.
  • drop the backup db.
  • Run tiup br restore db to restore the backup on S3.
  • Compatibility Test Case 1: Use the pre-modified br for backup, and the after-modified br for recovery.
  • Compatibility Test Case 2: Use the after-modified br for backup, and the pre-modified br for recovery.

Test output

  • the directory
    Screenshot from 2022-07-05 15-55-44
  • run tiup br restore db successfully.
  • Compatibility Test Case 1 passed.
  • Compatibility Test Case 2 passed.

Release note 

In the new backup organization structure, we will see:

./br backup --pd "127.0.0.1:2379" -s "s3://backup/20220621" 
  - After br command finished, we will have the structure below.
➜  backup tree .
.
└── 20220621
    ├── backupmeta
    ├── store1
    │   └── backup-xxx.sst
    ├── store100
    │   └── backup-yyy.sst
    ├── store2
    │   └── backup-zzz.sst
    ├── store3
    ├── store4
    └── store5

@MoCuishle28
Adjust the backup organization structure (pingcap/tidb#30087)
562d17f 
Signed-off-by: Gaoming <zhanggaoming028@gmail.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jul 5, 2022 •
edited

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • 3pointer
  • overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot
Copy link
Member

Welcome @MoCuishle28!

It looks like this is your first PR to tikv/tikv 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to tikv/tikv. 😃

@MoCuishle28
Copy link
Contributor Author

/run-integration-br-tests

@MoCuishle28
Adjust the backup organization structure (pingcap/tidb#30087)
e3b4276 
Signed-off-by: Gaoming <zhanggaoming028@gmail.com>
@MoCuishle28
Copy link
Contributor Author

/run-integration-br-tests

@MoCuishle28
Copy link
Contributor Author

/run-integration-br-tests

@MoCuishle28
Copy link
Contributor Author

/rebuild

@MoCuishle28
Copy link
Contributor Author

/run-integration-br-tests

1 similar comment
@3pointer
Copy link
Contributor

/run-integration-br-tests

@MoCuishle28
Copy link
Contributor Author

/run-build

@MoCuishle28
Copy link
Contributor Author

/run-integration-br-tests

@MoCuishle28
Merge branch 'master' into issue-30087
1b3bed6 
Signed-off-by: MoCuishle28 <32541204+MoCuishle28@users.noreply.github.com>
overvenus
overvenus reviewed Jul 14, 2022
View reviewed changes
components/backup/src/endpoint.rs Outdated
region.get_region_epoch().get_version()
),
Some(k) => {
if storage_name == S3Storage::name() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add some comments and unit tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments and unit tests have been added. PTLA

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

match (key, storage_name) {
Some(k), S3Storage::name()|LocalStorage::name() =>
Some(k), _ =>
None, S3Storage::name()|LocalStorage::name() =>
None, _ =>
}

@MoCuishle28
Adjust the backup organization structure
fe2bef2 
Signed-off-by: zhanggaoming <gaoming.zhang@pingcap.com>
@ti-chi-bot ti-chi-bot added this to the Pool milestone Jul 20, 2022
ti-srebot pushed a commit to ti-srebot/tikv that referenced this pull request Jul 20, 2022
@MoCuishle28 @ti-srebot
cherry pick tikv#12958 to release-5.4
8596ad0 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Jul 20, 2022
Merged
@ti-srebot
Copy link
Contributor

cherry pick to release-5.4 in PR #13064

ti-srebot pushed a commit to ti-srebot/tikv that referenced this pull request Jul 20, 2022
@MoCuishle28 @ti-srebot
cherry pick tikv#12958 to release-6.0
bd042dc 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Jul 20, 2022
Closed
@ti-srebot
Copy link
Contributor

cherry pick to release-6.0 in PR #13065

ti-srebot pushed a commit to ti-srebot/tikv that referenced this pull request Jul 20, 2022
@MoCuishle28 @ti-srebot
cherry pick tikv#12958 to release-6.1
3e04c66 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Jul 20, 2022
Merged
@ti-srebot
Copy link
Contributor

cherry pick to release-6.1 in PR #13066

This was referenced Jul 20, 2022
Closed
Merged
LintianShi pushed a commit to LintianShi/tikv that referenced this pull request Jul 27, 2022
@MoCuishle28 @ti-chi-bot
br: Adjust the backup organization structure (tikv#12958)
0094101 
close tikv#13063

Adjust the backup organization structure and add a store_id related prefix under the backup path.

Signed-off-by: Gaoming <zhanggaoming028@gmail.com>
Signed-off-by: MoCuishle28 <32541204+MoCuishle28@users.noreply.github.com>
Signed-off-by: zhanggaoming <gaoming.zhang@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
MoCuishle28 added a commit to ti-srebot/tikv that referenced this pull request Aug 4, 2022
@MoCuishle28
cherry pick tikv#12958 to release-6.1
d57ee06 
Signed-off-by: MoCuishle28 <zhanggaoming028@gmail.com>
MoCuishle28 added a commit to ti-srebot/tikv that referenced this pull request Aug 4, 2022
@MoCuishle28
cherry pick tikv#12958 to release-6.0
b6ed3fb 
Signed-off-by: MoCuishle28 <zhanggaoming028@gmail.com>
MoCuishle28 added a commit to ti-srebot/tikv that referenced this pull request Aug 4, 2022
@MoCuishle28
cherry pick tikv#12958 to release-5.4
044532f 
Signed-off-by: MoCuishle28 <zhanggaoming028@gmail.com>
MoCuishle28 added a commit to ti-srebot/tikv that referenced this pull request Aug 8, 2022
@MoCuishle28
cherry pick tikv#12958 to release-6.0
2a4c73f 
Signed-off-by: zhanggaoming <gaoming.zhang@pingcap.com>
MoCuishle28 added a commit to ti-srebot/tikv that referenced this pull request Aug 8, 2022
@MoCuishle28
cherry pick tikv#12958 to release-5.4
06511e3 
Signed-off-by: zhanggaoming <gaoming.zhang@pingcap.com>
ti-chi-bot added a commit that referenced this pull request Aug 22, 2022
@ti-srebot @MoCuishle28 @ti-chi-bot
br: Adjust the backup organization structure (#12958) (#13066)
ee8d3ca 
ref #12958, close #13063

Adjust the backup organization structure and add a store_id related prefix under the backup path.

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
Signed-off-by: MoCuishle28 <zhanggaoming028@gmail.com>

Co-authored-by: MoCuishle28 <32541204+MoCuishle28@users.noreply.github.com>
Co-authored-by: MoCuishle28 <zhanggaoming028@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
This was referenced Aug 24, 2022
Merged
Closed
3pointer pushed a commit to 3pointer/tikv that referenced this pull request Aug 24, 2022
@MoCuishle28 @3pointer
cherry pick tikv#12958 to release-5.4
dcca164 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
3pointer pushed a commit to 3pointer/tikv that referenced this pull request Aug 24, 2022
@MoCuishle28 @3pointer
cherry pick tikv#12958 to release-5.4
ecbb5b0 
Signed-off-by: MoCuishle28 <zhanggaoming028@gmail.com>
3pointer pushed a commit to 3pointer/tikv that referenced this pull request Aug 24, 2022
@MoCuishle28 @3pointer
cherry pick tikv#12958 to release-5.4
d03a233 
Signed-off-by: zhanggaoming <gaoming.zhang@pingcap.com>
@3pointer 3pointer mentioned this pull request Aug 24, 2022
Merged
3pointer pushed a commit to 3pointer/tikv that referenced this pull request Aug 24, 2022
@MoCuishle28 @3pointer
cherry pick tikv#12958 to release-5.4
85c0abb 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
Signed-off-by: 3pointer <luancheng@pingcap.com>
3pointer pushed a commit to 3pointer/tikv that referenced this pull request Aug 24, 2022
@MoCuishle28 @3pointer
cherry pick tikv#12958 to release-5.4
304989f 
Signed-off-by: MoCuishle28 <zhanggaoming028@gmail.com>
Signed-off-by: 3pointer <luancheng@pingcap.com>
3pointer pushed a commit to 3pointer/tikv that referenced this pull request Aug 24, 2022
@MoCuishle28 @3pointer
cherry pick tikv#12958 to release-5.4
ac4e13d 
Signed-off-by: zhanggaoming <gaoming.zhang@pingcap.com>
Signed-off-by: 3pointer <luancheng@pingcap.com>
ti-chi-bot pushed a commit that referenced this pull request Aug 24, 2022
@3pointer @MoCuishle28
br: Adjust the backup organization structure (#13332)
90a5cf7 
ref #12958, close #13063

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: MoCuishle28 <zhanggaoming028@gmail.com>
Signed-off-by: zhanggaoming <gaoming.zhang@pingcap.com>

Co-authored-by: MoCuishle28 <32541204+MoCuishle28@users.noreply.github.com>
Co-authored-by: MoCuishle28 <zhanggaoming028@gmail.com>
Co-authored-by: zhanggaoming <gaoming.zhang@pingcap.com>
ti-chi-bot pushed a commit that referenced this pull request Aug 30, 2022
@ti-srebot @MoCuishle28
br: Adjust the backup organization structure (#12958) (#13064)
80093e3 
ref #12958, close #13063

Adjust the backup organization structure and add a store_id related prefix under the backup path.

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
Signed-off-by: MoCuishle28 <zhanggaoming028@gmail.com>
Signed-off-by: zhanggaoming <gaoming.zhang@pingcap.com>

Co-authored-by: MoCuishle28 <32541204+MoCuishle28@users.noreply.github.com>
Co-authored-by: MoCuishle28 <zhanggaoming028@gmail.com>
Co-authored-by: zhanggaoming <gaoming.zhang@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@overvenus overvenus overvenus approved these changes

@3pointer 3pointer 3pointer approved these changes

Assignees
No one assigned
Labels
contribution Type: PR - From contributors first-time-contributor needs-cherry-pick-release-5.4 Type: Need cherry pick to release-5.4 needs-cherry-pick-release-6.0 needs-cherry-pick-release-6.1 release-note size/L status/can-merge Status: Can merge to base branch status/LGT2 Status: PR - There are already 2 approvals
Projects
None yet
Milestone
Pool
Development

Successfully merging this pull request may close these issues.

br: failed to backup to s3 due to the limit of request rate.
5 participants
@MoCuishle28 @ti-chi-bot @3pointer @overvenus @ti-srebot