admin check failed with “Error 8223 (HY000): data inconsistency in table” after kill one of tikv or tikv rolling restart during adding index

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、tidb_enable_dist_task='on'
2、run sysbench
3、add index for one of table
4、kill one of tikv
5、admin check index when adding index finish

tidb logs:
endless-ha-test-add-index-tps-7080337-1-324.tar.gz

2. What did you expect to see? (Required)

admin check success

3. What did you see instead (Required)

admin check failed(Error 8223 (HY000): data inconsistency in table: sbtest1, index: index_test_1709003362736, handle: 35087534, index-values:"handle: 35087534, values: [KindString 54150009827-63531154822-50885588575-38796535820-59831497760-15874376114-47900593283-86891964691-89920570946-88503431724]" != record-values:"handle: 35087534, values: [KindString 48350368445-67037925709-53068813339-93858222535-70857817307-21035926610-73758438108-82851278774-47393726843-33708297691]")
operatorLogs:
[2024-02-27 11:09:22] ###### start adding index
alter table sbtest1 add index index_test_1709003362736 (c)
[2024-02-27 11:09:22] ###### wait for ddl job finish
[2024-02-27 11:17:30] ###### ddl job finished
select job_id, job_type, state from information_schema.ddl_jobs where query = 'alter table sbtest1 add index index_test_1709003362736 (c)'
jobId: 449, job type: add index /* ingest */, state: synced
add index done, it takes: 8m7.626657145s
[2024-02-27 11:17:30] ###### start admin check
admin check index sbtest1 index_test_1709003362736

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v8.0.0-alpha
Edition: Community
Git Commit Hash: c1b70d5
Git Branch: heads/refs/tags/v8.0.0-alpha
UTC Build Time: 2024-02-26 11:45:04
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-02-27T10:57:36.218+0800

[Lily2025]

/assign tangenta

[Lily2025]

admin check failed after rolling restart tikv during adding index

admin check failed(Error 8223 (HY000): data inconsistency in table: sbtest1, index: index_test_1709662618059, handle: 42670109, index-values:"handle: 42670109, values: [KindString 79793922415-96432231064-49793791306-09392438963-57303495845-77266668293-27635913748-54072365834-99544797245-07250190896]" != record-values:"handle: 42670109, values: [KindString 06357938616-54130163307-25396091638-95567474086-75208163438-20698888865-07940411079-68156310255-39716096584-08207716142]")
operatorLogs:
[2024-03-06 02:16:58] ###### start adding index
ALTER TABLE sbtest1 ADD INDEX index_test_1709662618059(c)
[2024-03-06 02:16:58] ###### wait for ddl job finish
[2024-03-06 02:19:58] ###### ddl job finished
select job_id, job_type, state from information_schema.ddl_jobs where query = 'ALTER TABLE sbtest1 ADD INDEX index_test_1709662618059(c)'
jobId: 579, job type: add index /* ingest */, state: synced
add index done, it takes: 3m0.357638174s
[2024-03-06 02:19:58] ###### start admin check
admin check index sbtest1 index_test_1709662618059

Analysis from @tangenta
It appears that the MDL did not take effect and did not block the DDL advance, resulting in a very old transaction being committed

tidb version:
./tidb-server -V
Release Version: v8.0.0-alpha
Edition: Community
Git Commit Hash: 5ac8a5b
Git Branch: heads/refs/tags/v8.0.0-alpha
UTC Build Time: 2024-03-04 11:47:38
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-03-05T20:02:37.101+0800