Potential data inconsistency that is caused by multiple writer

[overvenus]

What did you do?

It's possible that TiCDC may have multiple nodes replicating the same table to the same downstream when something goes wrong. E.g., #9344.

We need a mechanism to protect access to downstream. AFAIK, there are two ways

1. Fencing token [1]. Although it has a negative impact on performance, some downstream may not work out.
2. Lease. Every node maintains a TTL, it renews on successful heartbeat. A processor must stop replicating ASAP if TTL expires. If the owner finds a processor doesn't heartbeat within one TTL, it considers that processor is down, and needs to wait for one more TTL before scheduling tables on that processor.

[1]: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html

What did you expect to see?

No data inconsistency.

What did you see instead?

Data inconsistency.

Versions of the cluster

All release

[lance6716]

#3737 (comment)

DM has met a similar problem before. And even though dm-worker has TTL and one worker knows its lease is expired, the exiting logic is not finished before the new worker started.

[overvenus]

#3737 (comment)

DM has met a similar problem before. And even though dm-worker has TTL and one worker knows its lease it expired, the exiting logic is not finished before the new worker started.

Good point, that is the issue of the TTL approach. It only does best the effort to prevent data inconsistency. "ASAP" may never be fast enough in the case of "processor pause".

[fubinzh]

/severity major

[nongfushanquan]

Timeout handling has been implemented. It will be activated when the actual event occurs.

[nongfushanquan]

/close

[ti-chi-bot]

@nongfushanquan: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.