log-backup: added intervally resolve regions #14180

YuJuncen · 2023-02-08T03:15:21Z

Signed-off-by: hillium yujuncen@pingcap.com

What is changed and how it works?

Issue Number: Ref #13638

What's Changed:

This PR is the "resolved TS" part of #14023.

This PR added a “two phase” flush to log backup for reducing checkpoint lag.
Generally, we added a `MinTs` task, where resolve the regions and advance the `resolved_ts` in the checkpoint manager.
then, once we are doing flush, we would make current `resolved_ts` become `checkpoint_ts`.
This allows us to advance checkpoint_ts even the leader has gone. When the leader changes frequently, this can greatly reduce checkpoint lag.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test

Side effects

Performance regression
- Consumes more CPU
- Consumes more MEM

Release note

Make the checkpoint lag of PITR more stable when there are some leadership transforming.

Signed-off-by: hillium <yujuncen@pingcap.com>

ti-chi-bot · 2023-02-08T03:15:23Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

3pointer
hicqu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: hillium <yujuncen@pingcap.com>

Leavrth · 2023-02-23T02:59:31Z

components/backup-stream/src/endpoint.rs

+            RegionCheckpointOperation::PrepareMinTsForResolve => {
+                let min_ts = self.pool.block_on(self.prepare_min_ts());
+                let start_time = Instant::now();
+                try_send!(
+                    self.scheduler,
+                    Task::RegionCheckpointsOp(RegionCheckpointOperation::Resolve {
+                        min_ts,
+                        start_time
+                    })
+                );
+            }
+            RegionCheckpointOperation::Resolve { min_ts, start_time } => {
+                let sched = self.scheduler.clone();
+                try_send!(
+                    self.scheduler,
+                    Task::ModifyObserve(ObserveOp::ResolveRegions {
+                        callback: Box::new(move |mut resolved| {
+                            let t =
+                                Task::RegionCheckpointsOp(RegionCheckpointOperation::Resolved {
+                                    checkpoints: resolved.take_resolve_result(),
+                                    start_time,
+                                });
+                            try_send!(sched, t);
+                        }),
+                        min_ts
+                    })
+                );
+            }


It seems that RegionCheckpointOperation::Resolve is only scheduled by RegionCheckpointOperation::PrepareMinTsForResolve ?

Suggested change

RegionCheckpointOperation::PrepareMinTsForResolve => {

let min_ts = self.pool.block_on(self.prepare_min_ts());

let start_time = Instant::now();

try_send!(

self.scheduler,

Task::RegionCheckpointsOp(RegionCheckpointOperation::Resolve {

min_ts,

start_time

})

);

}

RegionCheckpointOperation::Resolve { min_ts, start_time } => {

let sched = self.scheduler.clone();

try_send!(

self.scheduler,

Task::ModifyObserve(ObserveOp::ResolveRegions {

callback: Box::new(move |mut resolved| {

let t =

Task::RegionCheckpointsOp(RegionCheckpointOperation::Resolved {

checkpoints: resolved.take_resolve_result(),

start_time,

});

try_send!(sched, t);

}),

min_ts

})

);

}

RegionCheckpointOperation::PrepareMinTsForResolve => {

let min_ts = self.pool.block_on(self.prepare_min_ts());

let start_time = Instant::now();

let sched = self.scheduler.clone();

try_send!(

self.scheduler,

Task::ModifyObserve(ObserveOp::ResolveRegions {

callback: Box::new(move |mut resolved| {

let t =

Task::RegionCheckpointsOp(RegionCheckpointOperation::Resolved {

checkpoints: resolved.take_resolve_result(),

start_time,

});

try_send!(sched, t);

}),

min_ts

})

);

}

We must reschedule the task to consume all the pending events after we updated the concurrency manager.

ti-chi-bot · 2023-02-23T03:14:44Z

@Leavrth: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: hillium <yujuncen@pingcap.com>

…to add-resolved-ts

3pointer · 2023-02-27T10:41:26Z

components/backup-stream/src/checkpoint_manager.rs

@@ -60,49 +64,59 @@ impl SubscriptionManager {
        while let Some(msg) = self.input.next().await {
            match msg {
                SubscriptionOp::Add(sub) => {
-                    self.subscribers.insert(Uuid::new_v4(), sub);
+                    let uid = Uuid::new_v4();
+                    info!("log backup adding new subscriber"; "id" => %uid);


will it cause too many logs?

Generally, one advancer in its lifetime should only subscribe once. 🤔

components/backup-stream/src/checkpoint_manager.rs

components/backup-stream/src/endpoint.rs

Signed-off-by: hillium <yujuncen@pingcap.com>

components/backup-stream/src/endpoint.rs

3pointer · 2023-03-09T08:02:18Z

src/config/mod.rs

@@ -2595,6 +2597,13 @@ impl BackupStreamConfig {
            );
            self.num_threads = default_cfg.num_threads;
        }
+        if self.max_flush_interval < ReadableDuration::secs(1) {


maybe also check min_ts_interval

Signed-off-by: hillium <yujuncen@pingcap.com>

YuJuncen · 2023-03-13T02:31:25Z

/test

ti-chi-bot · 2023-03-13T04:08:48Z

@YuJuncen: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

YuJuncen · 2023-03-13T06:50:03Z

/test
[2023-03-13T04:23:05.868Z] thread 'test::resolved_follower' panicked at 'called Option::unwrap()on aNone value', components/backup-stream/tests/mod.rs:545:84 🤔

YuJuncen · 2023-03-13T06:50:45Z

(Perhaps we need to add a callback for flush...)

Signed-off-by: hillium <yujuncen@pingcap.com>

YuJuncen · 2023-03-13T08:05:56Z

/test

hicqu · 2023-03-13T08:20:34Z

/merge

ti-chi-bot · 2023-03-13T08:20:35Z

@hicqu: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-03-13T08:20:38Z

This pull request has been accepted and is ready to merge.

Commit hash: 96a0b08

ti-chi-bot · 2023-03-13T08:24:03Z

In response to a cherrypick label: new pull request created to branch release-6.5: #14381.

ref tikv#13638 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2023-03-13T08:24:45Z

In response to a cherrypick label: new pull request created to branch release-6.6: #14382.

ti-chi-bot · 2023-03-13T08:25:26Z

In response to a cherrypick label: new pull request created to branch release-6.2: #14383.

ref tikv#13638 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2023-03-13T08:26:08Z

In response to a cherrypick label: new pull request created to branch release-6.3: #14384.

ref tikv#13638 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2023-03-13T08:26:50Z

In response to a cherrypick label: new pull request created to branch release-6.4: #14385.

ref tikv#13638 This PR added a “two phase” flush to log backup for reducing checkpoint lag. Generally, we added a `MinTs` task, where resolve the regions and advance the `resolved_ts` in the checkpoint manager. then, once we are doing flush, we would make current `resolved_ts` become `checkpoint_ts`. This allows us to advance checkpoint_ts even the leader has gone. When the leader changes frequently, this can greatly reduce checkpoint lag. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

ref #13638 This PR added a “two phase” flush to log backup for reducing checkpoint lag. Generally, we added a `MinTs` task, where resolve the regions and advance the `resolved_ts` in the checkpoint manager. then, once we are doing flush, we would make current `resolved_ts` become `checkpoint_ts`. This allows us to advance checkpoint_ts even the leader has gone. When the leader changes frequently, this can greatly reduce checkpoint lag. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: hillium <yujuncen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

added intervally resolve regions

b80f5e6

Signed-off-by: hillium <yujuncen@pingcap.com>

ti-chi-bot added needs-cherry-pick-release-6.5 release-note size/XL labels Feb 8, 2023

YuJuncen and others added 4 commits February 15, 2023 11:22

make clippy happy

23044e4

Signed-off-by: hillium <yujuncen@pingcap.com>

make clippy happy

8e33567

Signed-off-by: hillium <yujuncen@pingcap.com>

Merge branch 'master' into add-resolved-ts

75e24f0

Merge branch 'master' into add-resolved-ts

2eb9a14

YuJuncen requested a review from 3pointer February 22, 2023 03:52

Leavrth reviewed Feb 23, 2023

View reviewed changes

Leavrth approved these changes Feb 23, 2023

View reviewed changes

YuJuncen added 2 commits February 28, 2023 13:50

added some comments

c603e86

Signed-off-by: hillium <yujuncen@pingcap.com>

Merge branch 'add-resolved-ts' of https://github.com/YuJuncen/tikv in…

41b5990

…to add-resolved-ts

3pointer reviewed Feb 28, 2023

View reviewed changes

3pointer reviewed Mar 9, 2023

View reviewed changes

components/backup-stream/src/checkpoint_manager.rs Outdated Show resolved Hide resolved

components/backup-stream/src/endpoint.rs Outdated Show resolved Hide resolved

make min_ts interval configurable

6e4f144

Signed-off-by: hillium <yujuncen@pingcap.com>

ti-chi-bot added needs-cherry-pick-release-6.2 needs-cherry-pick-release-6.3 Type: Need cherry pick to release-6.3 needs-cherry-pick-release-6.4 needs-cherry-pick-release-6.6 labels Mar 9, 2023

3pointer reviewed Mar 9, 2023

View reviewed changes

components/backup-stream/src/endpoint.rs Outdated Show resolved Hide resolved

3pointer reviewed Mar 9, 2023

View reviewed changes

address comments

6144c3a

Signed-off-by: hillium <yujuncen@pingcap.com>

YuJuncen removed needs-cherry-pick-release-6.2 needs-cherry-pick-release-6.3 Type: Need cherry pick to release-6.3 needs-cherry-pick-release-6.4 needs-cherry-pick-release-6.6 labels Mar 9, 2023

3pointer approved these changes Mar 9, 2023

View reviewed changes

Merge branch 'master' into add-resolved-ts

f2c2a01

fix test

96a0b08

Signed-off-by: hillium <yujuncen@pingcap.com>

ti-chi-bot removed the status/can-merge Status: Can merge to base branch label Mar 13, 2023

ti-chi-bot added the status/can-merge Status: Can merge to base branch label Mar 13, 2023

ti-chi-bot merged commit 571e513 into tikv:master Mar 13, 2023
1 check passed

ti-chi-bot added this to the Pool milestone Mar 13, 2023

ti-chi-bot mentioned this pull request Mar 13, 2023

log-backup: added intervally resolve regions (#14180) #14381

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Mar 13, 2023

This is an automated cherry-pick of tikv#14180

0411b9a

ref tikv#13638 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Mar 13, 2023

log-backup: added intervally resolve regions (#14180) #14382

Closed

ti-chi-bot mentioned this pull request Mar 13, 2023

log-backup: added intervally resolve regions (#14180) #14383

Closed

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Mar 13, 2023

This is an automated cherry-pick of tikv#14180

5175612

ref tikv#13638 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Mar 13, 2023

log-backup: added intervally resolve regions (#14180) #14384

Closed

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Mar 13, 2023

This is an automated cherry-pick of tikv#14180

3fe9431

ref tikv#13638 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Mar 13, 2023

log-backup: added intervally resolve regions (#14180) #14385

Closed

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Mar 13, 2023

This is an automated cherry-pick of tikv#14180

a4c541f

ref tikv#13638 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

log-backup: added intervally resolve regions #14180

log-backup: added intervally resolve regions #14180

YuJuncen commented Feb 8, 2023 •

edited

ti-chi-bot commented Feb 8, 2023 •

edited

Leavrth Feb 23, 2023

YuJuncen Feb 23, 2023

Leavrth Feb 23, 2023

ti-chi-bot commented Feb 23, 2023

3pointer Feb 27, 2023

YuJuncen Mar 1, 2023

3pointer Mar 9, 2023

YuJuncen commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

YuJuncen commented Mar 13, 2023

YuJuncen commented Mar 13, 2023

YuJuncen commented Mar 13, 2023

hicqu commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

log-backup: added intervally resolve regions #14180

log-backup: added intervally resolve regions #14180

Conversation

YuJuncen commented Feb 8, 2023 • edited

What is changed and how it works?

Related changes

Check List

Release note

ti-chi-bot commented Feb 8, 2023 • edited

Leavrth Feb 23, 2023

Choose a reason for hiding this comment

YuJuncen Feb 23, 2023

Choose a reason for hiding this comment

Leavrth Feb 23, 2023

Choose a reason for hiding this comment

ti-chi-bot commented Feb 23, 2023

3pointer Feb 27, 2023

Choose a reason for hiding this comment

YuJuncen Mar 1, 2023

Choose a reason for hiding this comment

3pointer Mar 9, 2023

Choose a reason for hiding this comment

YuJuncen commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

YuJuncen commented Mar 13, 2023

YuJuncen commented Mar 13, 2023

YuJuncen commented Mar 13, 2023

hicqu commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

ti-chi-bot commented Mar 13, 2023

YuJuncen commented Feb 8, 2023 •

edited

ti-chi-bot commented Feb 8, 2023 •

edited