sink(cdc): fix internal retry algothrim (#9530)

[ti-chi-bot]

This cherry-pick contains #9265 #9322, #9358, #9385 and #9530.

What problem does this PR solve?

Issue Number: close #9518 , ref #9272

What is changed and how it works?

📎

• Add changefeed warning and failed alert rules.
• Change the max retry duration for changefeed errors from 90 minutes to 30 minutes.
• Use warning state instead of error state for changefeeds. If a changefeed encounters an error, it will enter the warning state and retry for up to 30 minutes. If the retry is successful, it will resume normal operation. If not, it will become a failed state and block GC for gcttl time.
• Allow cdc cli changefeed list to display the error message when a changefeed is in warning state.

> ./cdc cli changefeed list
[
  {
    "id": "test1",
    "namespace": "default",
    "summary": {
      "state": "warning",
      "tso": 443594678570057753,
      "checkpoint": "2023-08-16 17:50:07.387",
      "error": {
        "time": "2023-08-16T18:04:37.930002+08:00",
        "addr": "127.0.0.1:8300",
        "code": "CDC:ErrProcessorUnknown",
        "message": "SinkManagerRunError"
      }
    }
  }
]

• Limit the sinkManager to a maximum of 30 minutes for retrying on sink errors.

Check List

Tests

• Unit test

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Limit the sinkManager to a maximum of 30 minutes for retrying on sink errors.

[asddongmen]

/test all

[asddongmen]

/retest

[asddongmen]

/retest

[asddongmen]

/hang

[asddongmen]

/retest

[asddongmen]

/retest

[hicqu]

/lgtm

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CharlesCheung96, hicqu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [CharlesCheung96,hicqu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[asddongmen]

/test all

[asddongmen]

/test all

[asddongmen]

/retest

[asddongmen]

/test all