gc (ticdc): optimize the algorithm calculating gc safepoint

[charleszheng44]

What problem does this PR solve?

Issue Number: close #8403

What is changed and how it works?

gc (ticdc): Instead of keeping restarting error changefeed in 24 hours, put the error changefeed into failed state and calculate the ticdc global gc safepoint based on checkpoint ts of all changefeeds and give users 24 hours grace period to handle the failed changefeed.

e.g.,

Have two ChangeFeeds,
cf1(failed) with checkpointTs ts1
cf2(normal) with checkpointTs ts2

the global gc safepoint will be:
min(ts2, max(ts1, currentPDTs - 24 hours))

After this pr being merged:

• Following errors will be set to fail directly:

ErrExpressionColumnNotFound,
ErrExpressionParseFailed,
ErrSchemaSnapshotNotFound,
ErrSyncRenameTableFailed,
ErrChangefeedUnretryable,
ErrGCTTLExceeded, 
ErrSnapshotLostByGC,
ErrStartTsBeforeGC,

• all other errors will be retried no more than 90 mins
• when cal the gcsafepoint, changefeed has been failed for more than 24 hours will be ignored

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Questions

No

Will it cause performance regression or break compatibility?

No

Do you need to update user documentation, design documentation or monitoring documentation?

No

Release note

- only retry error changefeed for no more than 90 mins
- cleanup data of changefeed that has been failed for more than 24 hours

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• asddongmen
• sdojjy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[charleszheng44]

/run-verify

[charleszheng44]

/run-verify

[charleszheng44]

/run-kafka-integration-test

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 81009ed

[nongfushanquan]

/cherry-pick 7.1

[ti-chi-bot]

@nongfushanquan: cannot checkout 7.1: error checking out 7.1: exit status 1. output: error: pathspec '7.1' did not match any file(s) known to git

In response to this:

/cherry-pick 7.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[nongfushanquan]

/cherry-pick release-7.1

[ti-chi-bot]

@nongfushanquan: new pull request created to branch release-7.1: #8887.

In response to this:

/cherry-pick release-7.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[nongfushanquan]

/cherry-pick release-6.5

[ti-chi-bot]

@nongfushanquan: new pull request created to branch release-6.5: #9014.

In response to this:

/cherry-pick release-6.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.