server, dispatcher: improve node liveness self fence (#5106)#5600
server, dispatcher: improve node liveness self fence (#5106)#5600ti-chi-bot wants to merge 2 commits into
Conversation
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot
added
do-not-merge/hold
|
@3AceShowHand This PR has conflicts, I have hold it. |
|
@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements a node liveness self-fencing mechanism to improve server and dispatcher stability by introducing an etcd session watchdog that triggers a local fence upon session expiration. However, the current changes contain numerous unresolved merge conflict markers across several key files, including dispatcher_manager.go, dispatcher_manager_redo.go, dispatcher_orchestrator.go, and their corresponding test files. These conflicts must be resolved to ensure the code compiles and functions correctly.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| <<<<<<< HEAD | ||
| if e.IsRedoEnabled() { | ||
| e.redoSink.Close(removeChangefeed) | ||
| // FIXME: cleanup redo log when remove the changefeed | ||
| e.closeRedoMeta(removeChangefeed) | ||
| } | ||
| e.sink.Close(removeChangefeed) | ||
| ======= | ||
| if e.IsRedoEnabled() && e.redoSink != nil { | ||
| e.redoSink.Close() | ||
| } | ||
| if e.sink != nil { | ||
| e.sink.Close() | ||
| } | ||
| >>>>>>> f73e8dba2 (server, dispatcher: improve node liveness self fence (#5106)) |
There was a problem hiding this comment.
Unresolved merge conflict markers (<<<<<<< HEAD, =======, >>>>>>>) found in this file. Please resolve the conflicts to ensure the code compiles and functions correctly.
if e.IsRedoEnabled() && e.redoSink != nil {
e.redoSink.Close()
}
if e.sink != nil {
e.sink.Close()
}
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Cherry-pick conflicts appear resolved; removing the |
ti-chi-bot
removed
the
do-not-merge/hold
|
/test all |
|
@3AceShowHand: No jobs can be run with DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test pull-build-next-gen |
|
/test all |
|
@asddongmen: No jobs can be run with DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test pull-cdc-kafka-integration-heavy-next-gen |
|
/test pull-cdc-mysql-integration-light-next-gen |
|
@ti-chi-bot: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
3 participants
This is an automated cherry-pick of #5106
What problem does this PR solve?
Issue Number: close #5202
When a TiCDC capture loses its etcd session or lease, it can no longer prove that it still owns local dispatcher work. The previous shutdown path did not immediately fence local write paths, so a stale capture could still accept maintainer requests or continue writing downstream while failover was already in progress. This created a short but unsafe window for duplicate or out-of-date downstream writes before normal cleanup finished.
What is changed and how it works?
This PR adds a local fence path for session-done and lease-expired events. The server watches the etcd session, triggers local fencing before shutdown, and the dispatcher orchestrator stops accepting new maintainer requests. Dispatcher managers cancel local write paths immediately, close local dispatchers asynchronously, and continue cleanup without waiting for progress draining. The redo cleanup path is also guarded so partially initialized managers do not panic when fencing. New integration coverage simulates capture session loss and verifies the local fence behavior and downstream consistency.
Check List
Tests
/test allQuestions
Will it cause performance regression or break compatibility?
No. The new path only runs when a local capture loses its session or is being fenced.
Do you need to update user documentation, design documentation or monitoring documentation?
No.
Release note
Summary by CodeRabbit
New Features
Bug Fixes
Tests