non-reentrant DDLs can be executed twice, which can lead inconsistency

[hicqu]

What did you do?

Steps:

1. create a ticdc cluster with 2 cdc instances;
2. create a changefeed with table create table x(id int primary key, val int);
3. execute SQLs at upstream:
   1. insert into x (id, val) values (1, 1);
   2. truncate table x;
   3. insert into x (id, val) values (2, 2);
4. resign cdc owner;
5. consistency check on upstream and down stream should always pass.

What did you expect to see?

consistency check on upstream and down stream should always pass.

What did you see instead?

However, a test case shows that the consistency check can't pass after owner resign if

• the old owner had pushed checkpointTs=DDL(truncate).Ts and resolvedTs>checkpointTs to etcd, and
• insert into x (id, val) values (2,2) had already been executed before owner resign finished.

So why the inconsistency happens?

I think the reason is the new owner executes DDL(truncate) again.

Versions of the cluster

master branch.

[hicqu]

/severity major

[hicqu]

/remove-label may-affects-4.0
/remove-label may-affects-5.0
/remove-label may-affects-5.1
/remove-label may-affects-5.2
/remove-label may-affects-5.3
/remove-label may-affects-5.4
/remove-label may-affects-6.0
/remove-label may-affects-6.1
/remove-label may-affects-6.2

[hicqu]

/label effects-6.0
/label effects-6.1
/label effects-6.2

[ti-chi-bot]

@hicqu: The label(s) effects-6.0, effects-6.1, effects-6.2 cannot be applied. These labels are supported: duplicate, bug-from-internal-test, bug-from-user, affects-4.0, affects-5.0, affects-5.1, affects-5.2, affects-5.3, affects-5.4, affects-6.0, affects-6.1, affects-6.2, may-affects-4.0, may-affects-5.0, may-affects-5.1, may-affects-5.2, may-affects-5.3, may-affects-5.4, may-affects-6.0, may-affects-6.1, may-affects-6.2, needs-cherry-pick-release-4.0, needs-cherry-pick-release-5.0, needs-cherry-pick-release-5.1, needs-cherry-pick-release-5.2, needs-cherry-pick-release-5.3, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.0, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.2, question, release-blocker, wontfix, require-LGT1, require-LGT3.

In response to this:

/label effects-6.0
/label effects-6.1
/label effects-6.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[hicqu]

/label affects-6.0
/label affects-6.1
/label afects-6.2

[ti-chi-bot]

@hicqu: The label(s) afects-6.2 cannot be applied. These labels are supported: duplicate, bug-from-internal-test, bug-from-user, affects-4.0, affects-5.0, affects-5.1, affects-5.2, affects-5.3, affects-5.4, affects-6.0, affects-6.1, affects-6.2, may-affects-4.0, may-affects-5.0, may-affects-5.1, may-affects-5.2, may-affects-5.3, may-affects-5.4, may-affects-6.0, may-affects-6.1, may-affects-6.2, needs-cherry-pick-release-4.0, needs-cherry-pick-release-5.0, needs-cherry-pick-release-5.1, needs-cherry-pick-release-5.2, needs-cherry-pick-release-5.3, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.0, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.2, question, release-blocker, wontfix, require-LGT1, require-LGT3.

In response to this:

/label affects-6.0
/label affects-6.1
/label afects-6.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[hicqu]

/label affects-6.2

[hicqu]

/label affects-6.2