restored in-flight creates a duplicate dispatcher in maintainer_failover_when_operator

[wk989898]

What did you do?

Ran tests/integration_tests/maintainer_failover_when_operator

What did you expect to see?

After failover the restored operators should converge: t6 / tableID 20 should end with exactly 1 replication after the restored merge completes, and no duplicate full-span dispatchers should run on the same table.

What did you see instead?

A newly created table (t4 / tableID 22) gets two full-span dispatchers on the same surviving node. One is cleaned up only after it starts writing, the cleanup races with a cloud-storage write and raises a retryable sink error, the whole changefeed is stopped/restarted, and the already-finished t6 merge is undone. After restart, t6 is re-added and auto-split into 2 spans (20 regions > region-threshold 10), so wait_for_table_replication_count ... eq 1 times out.

Timeline from logs

• 12:35:20.223 (old maintainer epoch 468956868881678341): t4 in-flight Create 154415... is stored on the surviving node 8301.
• 12:35:47.345 (new maintainer epoch 468956885016641547): restore merge operator succeeds, and in the same ms:
  maintainer_controller_bootstrap.go:384 logs
  found a working table that is not in initial table map, just ignore it,
  tableID=22.
• 12:35:47.328 / 12:35:47.951: node 8301 creates two dispatchers for the same t4 full span, tableID 22, identical start/end keys:
  • restored in-flight Create 154415...
  • new Create 693762... scheduled from the CREATE TABLE barrier under the
    new maintainer
• 12:35:49.599: maintainer detects the extra dispatcher:
  no span found, remove it, dispatcherID=154415....
  • 12:35:50.552: while that dispatcher is still being removed/flushed, its
    cloudstorage writer fails:
    CDC:ErrS3StorageAPI ... CDC_154415...json.tmp.<uuid>: no such file or directory
  • 12:35:50.607: maintainer treats it as ErrChangefeedRetryable, stops the
    changefeed, next retry 12:36:00.197.
  • 12:35:49.755: merge dispatcher operator finished for t6 — but the
    changefeed stop removes the merged dispatcher shortly after.
  • 12:36:00.557: on restart, tableID 20 is re-added and immediately split:
    split span by region count ... tableID: 20 ... spans=2 regionCount=20.
    Table 20 keeps 2 replications and the eq 1 check never passes.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

master `b09792d0b1dd1385a21ea095bfa6ca288cd0c3d4` (next-gen)

[wk989898]

Root-cause hypothesis

For a table whose CREATE DDL is still pending in the maintainer’s barrier while its
Create dispatcher is already in-flight on a surviving node:

• the new maintainer restores the in-flight Create into a working span but
  logs
  not in initial table map, just ignore it;
• the CREATE barrier later treats the table as absent and schedules another
  full-span Create;
• the old dispatcher is only removed by the no span found, remove it
  fallback after it is already Working/flushing, and that cleanup can race with its storage writer.

The result is a transient retryable changefeed restart caused by a scheduler/bootstrap
recovery issue, which in turn hides the otherwise successful merge convergence.

This looks like a variant of #5083 that is not fully covered by the epoch-fencing
fixes (#5434/#5435/#5436): here the stale request is not delayed over the
network;
it was already stored in a surviving dispatcher manager and is resumed during
bootstrap.

Related issues

• Duplicate dispatcher can be created during maintainer failover before orphan dispatcher drains #5083 Duplicate dispatcher can be created during maintainer failover before orphan dispatcher drains
• maintainer failover can leave in-flight operators inconsistent #4763 maintainer failover can leave in-flight operators inconsistent
• unstable integration test maintainer_failover_when_operator #6132 / unstable integration test maintainer_failover_when_operator #6074 / unstable test maintainer_failover_when_operator #4528 unstable test maintainer_failover_when_operator