New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
log-backup: implement the checkpoint V3 #36114
Conversation
* Squashed commit of the following: commit a3b65e6cf6d96a56b1c4b6261aa0972555b2d0eb Merge: fdba35d03 df9b54b Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu Jun 23 11:32:25 2022 +0800 Merge branch 'master' of https://github.com/pingcap/tidb into checkpoint-v2 Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit fdba35d03c1074e54ab5d811fa923173f22a2c11 Author: Yu Juncen <yujuncen@pingcap.com> Date: Wed Jun 22 17:48:45 2022 +0800 added the tsheap struct, make abstuction over environment Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 11e07089775b3deb51db30b627b48684522bd916 Author: Yu Juncen <yujuncen@pingcap.com> Date: Wed Jun 22 17:47:31 2022 +0800 conn: make StoreManager a package Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit abf840cd8e40c47dae6ecedb4051a63e6e32a5c0 Author: Yu Juncen <yujuncen@pingcap.com> Date: Fri Jun 17 17:00:25 2022 +0800 implement basic get checkpoint Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 314015c3c24d8d76950b049195d90bb0e3bfe426 Author: Yu Juncen <yujuncen@pingcap.com> Date: Tue Jun 7 14:47:31 2022 +0800 fix the retry over wrapped errors Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 5c8e77a Author: Yu Juncen <yujuncen@pingcap.com> Date: Mon Jun 6 15:33:14 2022 +0800 adapt new checkpoint model Signed-off-by: Yu Juncen <yujuncen@pingcap.com> Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * move StoreManager to utils package Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * don't make streamhelper requires stream Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * integrated advancer to TiDB Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * added more metrics. Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * clear when task removed Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * add collector Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * fix collapse ranges Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * fix typo && make bucket larger Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * fix stuck when error Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * fix inconsistent ranges Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * added even more comments Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * address comments; added some comments Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * make linter happy Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * address comments Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * make clippy happy Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * guard on close store manager Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * added even more guards Signed-off-by: Yu Juncen <yujuncen@pingcap.com> * Update br/pkg/streamhelper/advancer.go Co-authored-by: 3pointer <qdlc2010@gmail.com> Co-authored-by: Zak Zhao <57036248+joccau@users.noreply.github.com> Co-authored-by: 3pointer <qdlc2010@gmail.com> Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
This reverts commit a38ea91.
please remove bazel file changes in this PR. let's focus on cherry-pick of https://github.com/pingcap/tidb/pull/35685/files |
Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/d6ca3ac1ee5c0384300a99515bc460fdbf1312bd |
Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
… checkpoint-v3
Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
/rebuild |
2 similar comments
/rebuild |
/rebuild |
Signed-off-by: Yu Juncen <yujuncen@pingcap.com>
/merge |
This pull request has been accepted and is ready to merge. Commit hash: d6ca3ac
|
TiDB MergeCI notify🔴 Bad News! New failing [1] after this pr merged.
|
What problem does this PR solve?
Issue Number: close #35164(cherry-pick from feature branch https://github.com/pingcap/tidb/pull/35685/files)
Problem Summary:
Generally, this PR implements the checkpoint V3, which is for solving the consistency problem checkpoint V2 meets.
For more details about the problems we have meet, check tikv/tikv#12715.
We have introduced a central node (embed in TiDB currently) for solving the problem: this central node pulls the checkpoint of each region, once it found that it have collected the checkpoint of the full keyspace, it advances the checkpoint. (So it named
advancer
).What is changed and how it works?
This PR(cherry-pick #35685):
streamhelper
, which contains the advancer type and client type.LogBackup
for the latest checkpoint of each region, collect them and advancing the global checkpoint.br log advancer
command.log-backup.enabled
to true.Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.