br: make more sane variables of ticking timeout (#39887)

[ti-chi-bot]

This is an automated cherry-pick of #39887

Signed-off-by: hillium yujuncen@pingcap.com

What problem does this PR solve?

Issue Number: close #39620 (again!)

Problem Summary:
This PR make a saner config for ticking timeout, and make some sentinels for prevent us doing insane things, they are:

• Uploading global checkpoint would check the old checkpoint then do the upload. (For now, we are not sure how to compare encoded bytes, and for now the keys are operated by exactly one node (elected by etcd), so the etcd conditional transaction isn't used.)
• The tick timeout now is tickDuration, so the tick won't be skipped due to blocking.
• The threshold now is 4 mins when subscription is available, 108s when not. So once we lost some subscriptions, the probability of breaking our RPO would be less.
• The tick is now split into two parts: optional part and important part. The former includes updating subscription and polling the checkpoints, which could be skipped and won't ruin things. The latter includes updating checkpoint and GC safepoint(not added yet), etc..

What is changed and how it works?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Fixed a bug that may cause log backup checkpoint lag increases to infinity when doing rolling update.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• 3pointer
• joccau

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[joccau]

LGTM

[joccau]

/merge

[ti-chi-bot]

@joccau: /merge in this pull request requires 2 approval(s).

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[joccau]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: bf50a83

[YuJuncen]

/merge

[ti-chi-bot]

@ti-chi-bot: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.