*: redirect the DML writes to a temp index during creating index

[tangenta]

What problem does this PR solve?

Issue Number: ref #35983

Problem summary:

Previously, the new index records backfill are done by a reorg worker with multiple transactions (DDL txn). When the index schema state has entered delete-only/write-only, according to the online schema change, the DML (DELETE/INSERT/UPDATE) statements can write new records the index (DML txn).

If the same keys are written in both DDL txn and DML txn, one of them will be rollbacked because of the write conflict. This could make either the create-index DDL inefficient or introduce high latency to the DML.

What is changed and how it works?

The old implementation:

rows    = [1, 2, 3]
-- CREATE INDEX
new_idx = [nil, nil, nil] <- [1, 2, 3]  (DDL txn)
-- UPDATE 1 -> 4
new_idx = [nil, nil, nil] <- [4, nil, nil] (DML txn)

This PR provides a new index creating process. We can redirect the DML txn writes to a temporary index once the creating index's schema state has entered delete-only:

rows    = [1, 2, 3]
-- CREATE INDEX
new_idx = [nil, nil, nil] <- [1, 2, 3]  (DDL txn)
-- UPDATE 1 -> 4
new_idx_tmp = [nil, nil, nil] <- [4, nil, nil] (DML txn)

Because the index IDs are different, no write conflicts can happen. After the DDL txns finished, it requires a new txn to merge the temp index record to the origin index.

rows =    [1, 2, 3]
-- CREATE INDEX BACKFILL
new_idx = [nil, nil, nil] <- [1, 2, 3]  (DDL txn)
-- UPDATE 1 -> 4
new_idx_tmp = [nil, nil, nil] <- [4, nil, nil] (DML txn)
-- CREATE INDEX MERGE
new_idx = [1, 2, 3] <- [4, nil, nil]  (DDL-merge txn)

Thus, the whole add index process needs to reorganize the data twice. To control the progress and make sure the data consistency, here we introduce the new field BackfillState to model.IndexInfo. It contains the following values:

• Inapplicable: this DDL job doesn't use the merge process.
• Running: the first backfill work is running. The index's write and delete operations are redirected to a temporary index.
• ReadyToMerge: the first backfill worker is finished. The index's write and delete operations are copied to a temporary index, aka "double write". This state is used to make sure that all the TiDB instances are aware of the copy during the merge.
• Merging: the merging worker is running. The index's write and delete operations are copied to a temporary index.

The tuple (SchemaState, BackfillState) reflects the current status of creating index.

As a result, the index's schema state changes are as follows:

1. (none, backfill_state_inapplicable)
2. (delete-only, backfill_state_running)
3. (write-only, backfill_state_running)
4. (write-reorg, backfill_state_running)
5. (write-reorg, backfill_state_ready_to_merge)
6. (write-reorg, backfill_state_merging)
7. (public, backfill_state_inapplicable)

There are also changes for GC-delete range: before this PR, ADD INDEX jobs only insert the index ID to gc_delete_range table if it is rollbacked or cancelled. After this PR, we always insert the temporary index ID to gc_delete_range because TiDB could fallback to the old backfill process in extreme cases.

Note that the DML writes to temporary index are handled specially. For example, delete operations are converted to writes like key -> "delete". In the merging stage, TiDB convert these special records back to delete operations.

Check List

Tests

• Unit test
• Integration test
  • The realtikvtest/addindextest is enabled with set global tidb_ddl_enable_fast_reorg = on;.
  • Two specific tests with DDL hooks are added.
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• wjhuang2016
• xiongjiwei

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[Benjamin2037]

LGTM

[ti-chi-bot]

@Benjamin2037: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

LGTM

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[zimulala]

LGTM

[tangenta]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: ead1325

[sre-bot]

TiDB MergeCI notify

🔴 Bad News! New failing [1] after this pr merged.
These new failed integration tests seem to be caused by the current PR, please try to fix these new failed integration tests, thanks!

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci-tidb/integration-ddl-test	🟥 failed 1, success 5, total 6	23 min	New failing
idc-jenkins-ci-tidb/common-test	🔴 failed 1, success 10, total 11	34 min	Existing failure
idc-jenkins-ci-tidb/integration-common-test	🔴 failed 2, success 15, total 17	8 min 5 sec	Existing failure
idc-jenkins-ci/integration-cdc-test	🟢 all 37 tests passed	32 min	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	5 min 31 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	4 min 23 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	4 min 2 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	2 min 59 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	2 min 45 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed