Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

log-backup: support calculating dynamic shift-start-ts when restoring kv-files belonging to default-cf #36089

Merged
merged 13 commits into from
Jul 12, 2022
Merged

log-backup: support calculating dynamic shift-start-ts when restoring kv-files belonging to default-cf #36089

merged 13 commits into from
Jul 12, 2022

Conversation

joccau
Copy link
Member

@joccau joccau commented Jul 11, 2022 •
edited

What problem does this PR solve?

Issue Number: close #36138

Problem Summary:

  • Calculate the min-begin-ts(shift-ts) according to the start-ts in kv-pair at write-cf
  • get the kv-count and total-size when restore log

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Manual test

test1
  • backup
    • create a log backup task with command br log start
    • begin transaction and insert 5 records
    • wait 30 minutes
    • create a full backup(ts=434516875680940033)
    • insert 5 record at the uncommitted transaction
    • commit the transaction
  • restore
    • PiTR from snapshot-ts to target restore-ts with the comand br restore point
  • the result
    • restore successfully and the down-stream cluster has 10 record
    • the log like blow, it has calculate the shift-start-ts
      [2022/07/11 23:13:45.818 +08:00] [INFO] [client.go:301] ["set restore range ts"] [shift-start-ts=434516138702667777] [start-ts=434516875680940033] [restored-ts=434517233926930433]
      • shift-start-ts=2022-07-11 21:52:06.179 +0800 CST
      • start-ts=2022-07-11 22:38:57.528 +0800 CST
      • restore-ts=2022-07-11 23:01:44.128 +0800 CST
test2
  • truncate log
    • truncate log with the command br log truncate --until=434516875680940033
    • the summary below
      Reading Metadata... DONE; take = 450.577875ms
      We are going to remove 51 files, until 2022-07-11 22:38:57.5280.
      Sure? (y/N) y
      Removing metadata... DONE; take = 495.779542ms
      Clearing data files done. kv-count = 174, totalSize = 18722DONE; take = 127.192625ms
    • the low below about calculating shift-ts
      [2022/07/12 12:54:53.754 +08:00] [WARN] [stream_metas.go:70] ["calculate shift-ts"] [start-ts=434516875680940033] [shift-ts=434516138702667777]
      • 434516138702667777=2022-07-11 21:52:06.179 +0800 CST
      • 434516875680940033=2022-07-11 22:38:57.528 +0800 CST
  • restore point
    • PiTR from snapshot-ts to target restore-ts with the comand br restore point
    • the log like blow, it prove log truncate
the restore summary:

[2022/07/11 23:13:48.274 +08:00] [INFO] [collector.go:69] ["restore log success summary"] [total-take=4.90154525s] [restore-from=434516875680940033] [restore-to=434517233926930433] [total-kv-count=10] [total-size=820]

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@joccau
print the size and kv count of truncated log
d248508 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
calculate shift start ts
5c51552 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
skip invalid meta file
7446357 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
summary restore log about size and kv-count
cc16a62 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
set service-safe-point with global-checkpoint-ts-1
4a449d9 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jul 11, 2022 •
edited

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • 3pointer
  • Leavrth

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 11, 2022
@joccau
Copy link
Member Author

joccau commented Jul 11, 2022

/draft

@joccau joccau changed the title log-backup: support calculating dynamic shift-start-ts when restoring default-cf kv-files WIP log-backup: support calculating dynamic shift-start-ts when restoring default-cf kv-files Jul 11, 2022
@ti-chi-bot ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 11, 2022
@sre-bot
Copy link
Contributor

sre-bot commented Jul 11, 2022 •
edited

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/9f7c35ffc4626278a91c69adfe0419c87dd945e6

@joccau joccau changed the title WIP log-backup: support calculating dynamic shift-start-ts when restoring default-cf kv-files WIP log-backup: support calculating dynamic shift-start-ts when restoring kv-files belonging to default-cf Jul 12, 2022
@joccau
make fmt
9e69096 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
Copy link
Member Author

joccau commented Jul 12, 2022

/run-all-tests

@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 12, 2022
3pointer
3pointer reviewed Jul 12, 2022
View reviewed changes
br/pkg/restore/client.go Outdated Show resolved Hide resolved
3pointer
3pointer reviewed Jul 12, 2022
View reviewed changes
br/pkg/task/stream.go Outdated Show resolved Hide resolved
@joccau
add test case
5a7bcee 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
deal comments in pr
f00e161 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 12, 2022
@joccau
fix typo
715e1a4 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
Copy link
Member Author

joccau commented Jul 12, 2022

/run-all-tests

@joccau joccau changed the title WIP log-backup: support calculating dynamic shift-start-ts when restoring kv-files belonging to default-cf log-backup: support calculating dynamic shift-start-ts when restoring kv-files belonging to default-cf Jul 12, 2022
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 12, 2022
@joccau
Copy link
Member Author

joccau commented Jul 12, 2022

/cc @YuJuncen @3pointer

@joccau
maintain test case
8c73379 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
Copy link
Member Author

joccau commented Jul 12, 2022

/cc @Leavrth

@ti-chi-bot ti-chi-bot requested a review from Leavrth July 12, 2022 09:56
@joccau
fix typo
35b67d6 
Signed-off-by: joccau <zak.zhao@pingcap.com>
3pointer
3pointer approved these changes Jul 12, 2022
View reviewed changes
Copy link
Contributor

@3pointer 3pointer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 12, 2022
@joccau
make build
9f7c35f 
Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau
Copy link
Member Author

joccau commented Jul 12, 2022

/run-all-tests

@joccau
Copy link
Member Author

joccau commented Jul 12, 2022

/run-unit-test

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 12, 2022
@joccau joccau merged commit 8ef47f2 into pingcap:6.1.0-pitr-dev Jul 12, 2022
joccau added a commit to joccau/tidb that referenced this pull request Jul 12, 2022
@joccau
log-backup: support calculating dynamic shift-start-ts when restoring…
2c13360 
… kv-files belonging to default-cf (pingcap#36089)

* print the size and kv count of truncated log

Signed-off-by: joccau <zak.zhao@pingcap.com>

* calculate shift start ts

Signed-off-by: joccau <zak.zhao@pingcap.com>

* skip invalid meta file

Signed-off-by: joccau <zak.zhao@pingcap.com>

* summary restore log about size and kv-count

Signed-off-by: joccau <zak.zhao@pingcap.com>

* set service-safe-point with global-checkpoint-ts-1

Signed-off-by: joccau <zak.zhao@pingcap.com>

* make fmt

Signed-off-by: joccau <zak.zhao@pingcap.com>

* add test case

Signed-off-by: joccau <zak.zhao@pingcap.com>

* deal comments in pr

Signed-off-by: joccau <zak.zhao@pingcap.com>

* fix typo

Signed-off-by: joccau <zak.zhao@pingcap.com>

* maintain test case

Signed-off-by: joccau <zak.zhao@pingcap.com>

* fix typo

Signed-off-by: joccau <zak.zhao@pingcap.com>

* make build

Signed-off-by: joccau <zak.zhao@pingcap.com>
@joccau joccau mentioned this pull request Jul 12, 2022
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@3pointer 3pointer 3pointer approved these changes

@Leavrth Leavrth Leavrth approved these changes

@YuJuncen YuJuncen Awaiting requested review from YuJuncen

Assignees
No one assigned
Labels
release-note-none size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@joccau @ti-chi-bot @sre-bot @3pointer @Leavrth