admin check failed with “Error 8223 (HY000): data inconsistency in table” after add index and inject ddl owner network partition

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、tidb_enable_dist_task='on'
2、run sysbench
3、add index for one of table
4、inject network partition between ddl owner and all other pods
5、admin check index

2. What did you expect to see? (Required)

admin check can success

3. What did you see instead (Required)

admin check failed：
admin check index sbtest1 index_test_1703552562599
Error 8223 (HY000): data inconsistency in table: sbtest1, index: index_test_1703552562599, handle: 31702184, index-values:"handle: 31702184, values: [KindString 42604366371-21451895812-27518569128-37799479491-91566181356-76146461753-53016366997-52301770304-09657853519-55786043329]" != record-values:"handle: 31702184, values: [KindString 96558301162-97928307451-01675937654-31540943557-87200715168-69587168424-27210814037-31751409618-31353104576-34317501676]"

operatorLogs:
[2023-12-26 09:02:42] ###### start adding index
alter table sbtest1 add index index_test_1703552562599 (c)
[2023-12-26 09:02:42] ###### wait for ddl job finish
[2023-12-26 09:06:43] ###### ddl job finished
select job_id, job_type, state from information_schema.ddl_jobs where query = 'alter table sbtest1 add index index_test_1703552562599 (c)'
jobId: 561, job type: add index /* ingest */, state: synced
add index done, it takes: 4m0.473890156s
[2023-12-26 09:06:43] ###### start admin check
admin check index sbtest1 index_test_1703552562599

tidb logs：
endless-ha-test-add-index-tps-5251454-1-745.zip

[2023/12/26 09:11:24.460 +08:00] [ERROR] [reporter.go:291] ["admin check found data inconsistency"] [conn=3877634872] [session_alias=] [table_name=sbtest1] [index_name=index_test_1703552562599] [row_id=31702184] [index="handle: 31702184, values: [KindString 42604366371-21451895812-27518569128-37799479491-91566181356-76146461753-53016366997-52301770304-09657853519-55786043329]"] [row="handle: 31702184, values: [KindString 96558301162-97928307451-01675937654-31540943557-87200715168-69587168424-27210814037-31751409618-31353104576-34317501676]"] [row_mvcc="{"decoded":{"429391636255211523":{"c":"42604366371-21451895812-27518569128-37799479491-91566181356-76146461753-53016366997-52301770304-09657853519-55786043329","k":"35159807","pad":"21788622147-59129949870-21609172825-73750306916-68396718180"},"446576145727488168":{"c":"96558301162-97928307451-01675937654-31540943557-87200715168-69587168424-27210814037-31751409618-31353104576-34317501676","k":"35159807","pad":"21788622147-59129949870-21609172825-73750306916-68396718180"}},"key":"7480000000000000F55F728000000001E3BCA8","mvcc":{"info":{"writes":[{"start_ts":446576145727488168,"commit_ts":446576145858560101,"short_value":"gAADAAAAAgMEBAB7ALYA/34YAjk2NTU4MzAxMTYyLTk3OTI4MzA3NDUxLTAxNjc1OTM3NjU0LTMxNTQwOTQzNTU3LTg3MjAwNzE1MTY4LTY5NTg3MTY4NDI0LTI3MjEwODE0MDM3LTMxNzUxNDA5NjE4LTMxMzUzMTA0NTc2LTM0MzE3NTAxNjc2MjE3ODg2MjIxNDctNTkxMjk5NDk4NzAtMjE2MDkxNzI4MjUtNzM3NTAzMDY5MTYtNjgzOTY3MTgxODA="},{"start_ts":429391636255211523,"commit_ts":429391636884094978,"short_value":"gAADAAAAAgMEBAB7ALYA/34YAjQyNjA0MzY2MzcxLTIxNDUxODk1ODEyLTI3NTE4NTY5MTI4LTM3Nzk5NDc5NDkxLTkxNTY2MTgxMzU2LTc2MTQ2NDYxNzUzLTUzMDE2MzY2OTk3LTUyMzAxNzcwMzA0LTA5NjU3ODUzNTE5LTU1Nzg2MDQzMzI5MjE3ODg2MjIxNDctNTkxMjk5NDk4NzAtMjE2MDkxNzI4MjUtNzM3NTAzMDY5MTYtNjgzOTY3MTgxODA="}]}},"regionID":44171}"] [index_mvcc="{"decoded":{"446576083376537788":{"handle":"31702184"}},"key":"7480000000000000F55F69800000000000004F013432363034333636FF3337312D32313435FF313839353831322DFF3237353138353639FF3132382D33373739FF393437393439312DFF3931353636313831FF3335362D37363134FF363436313735332DFF3533303136333636FF3939372D35323330FF313737303330342DFF3039363537383533FF3531392D35353738FF3630343333323900FE038000000001E3BCA8","mvcc":{"info":{"writes":[{"start_ts":446576083376537788,"commit_ts":446576083376537788,"short_value":"MA=="}]}},"regionID":138168}"] [stack="github.com/pingcap/tidb/pkg/util/logutil/consistency.(*Reporter).ReportAdminCheckInconsistent\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/logutil/consistency/reporter.go:290\ngithub.com/pingcap/tidb/pkg/executor.(*checkIndexWorker).HandleTask\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/executor.go:2685\ngithub.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool.(*WorkerPool[...]).handleTaskWithRecover\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool/workerpool.go:131\ngithub.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool.(*WorkerPool[...]).runAWorker.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool/workerpool.go:147\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/wait_group_wrapper.go:157"]

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v7.6.0-alpha
Edition: Community
Git Commit Hash: 07e9ece
Git Branch: heads/refs/tags/v7.6.0-alpha
UTC Build Time: 2023-12-25 11:44:42
GoVersion: go1.21.5
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2023-12-26T08:44:32.225+0800

[Lily2025]

/assign tangenta