Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ddl: support concurrent ddl #32169

Merged
merged 27 commits into from Jul 20, 2022
Merged

ddl: support concurrent ddl #32169

merged 27 commits into from Jul 20, 2022

Conversation

hawkingrei
Copy link
Member

@hawkingrei hawkingrei commented Feb 8, 2022 •
edited by xiongjiwei

Signed-off-by: Weizhen Wang wangweizhen@pingcap.com
Signed-off-by: xiongjiwei xiongjiwei1996@outlook.com
Signed-off-by: wjhuang2016 huangwenjun1997@gmail.com

What problem does this PR solve?

Issue Number: ref #32031

this is a big PR and we split it into many commits, every commit almost has a single purpose. I will introduce them briefly, you may need to reference the doc #33629

  • init ddl tables
    create tidb_ddl_job, tidb_ddl_reorg, tidb_ddl_history tables with raw meta write, these 3 tables is use to replace the ddl job queue and reorg and history hash table. you can see this part in doc

  • setup concurrent ddl env and add ddl worker pool
    this commit adds the ddl worker pool definition, the ddl job manager will find a job and ship it to a worker in the worker pool.
    Also, this commit provides a sessionctx wrapper, only use in ddl relate. it just wraps begin, commit and execute.

  • add ddl manager to handle ddl job
    this commit implements the ddl manager, which is used for

    • find a runnable ddl job
    • ship the job to the worker

    you can ref the doc. Also, it adds a function HandleDDLJob which will do the ddl job, and it is a replacer of HandleDDLJobQueue. The last thing of this commit is adding schemaVersionManager to update the schema version in a new txn, it prevents the txn conflict with the schema version key.

  • reorg handler for concurrent ddl
    just implements the partner of the reorg information.

  • manage ddl jobs for concurrent ddl
    same as above, the partner of add job, delete job and many other related to history job

  • change ddl interface caller
    because many of the functions need a session now, we just change the caller

  • add metrics for concurrent ddl
    add metrics

  • migrate ddl between table and queue
    support switch between the old and new ddl framework, migrate the existing ddl job between queue and table

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Release note

None

@hawkingrei hawkingrei added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 8, 2022
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Feb 8, 2022 •
edited

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • wjhuang2016
  • zimulala

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue release-note-none size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Feb 8, 2022
@sre-bot
Copy link
Contributor

sre-bot commented Feb 8, 2022

Please follow PR Title Format:

  • pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

  • *: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

@hawkingrei hawkingrei force-pushed the concurrent_ddl_workspace branch 2 times, most recently from f064240 to c8495c6 Compare February 9, 2022 06:54
@sre-bot
Copy link
Contributor

sre-bot commented Feb 9, 2022

Please follow PR Title Format:

  • pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

  • *: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

1 similar comment
@sre-bot
Copy link
Contributor

sre-bot commented Feb 9, 2022

Please follow PR Title Format:

  • pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

  • *: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

@hawkingrei hawkingrei changed the title [DNM] implemant concurrent ddl [DNM] ddl: implemant concurrent ddl Feb 9, 2022
@hawkingrei hawkingrei force-pushed the concurrent_ddl_workspace branch 18 times, most recently from c3c2de0 to fc3f2f1 Compare February 10, 2022 06:46
@hawkingrei
Copy link
Member Author

/hold

@ti-chi-bot ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 20, 2022
@sre-bot
Copy link
Contributor

sre-bot commented Jul 20, 2022

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/e542bccbcdb00c88a77b18ef4ee3225b3e50298f

@sre-bot
Copy link
Contributor

sre-bot commented Jul 20, 2022

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/2ff012c648b0118b6604516afd4f0f5ecd1c985f

@wjhuang2016
Copy link
Member

/unhold

@ti-chi-bot ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 20, 2022
@wjhuang2016 wjhuang2016 added the status/LGT3 The PR has already had 3 LGTM. label Jul 20, 2022
@ti-chi-bot ti-chi-bot removed the status/LGT3 The PR has already had 3 LGTM. label Jul 20, 2022
@wjhuang2016
Copy link
Member

/run-build

@wjhuang2016
Copy link
Member

/run-mysql-test

@wjhuang2016 wjhuang2016 changed the title ddl: implement concurrent ddl ddl: support concurrent ddl Jul 20, 2022
@wjhuang2016 wjhuang2016 reopened this Jul 20, 2022
@wjhuang2016
Copy link
Member

/run-unit-test

1 similar comment
@wjhuang2016
Copy link
Member

/run-unit-test

@wjhuang2016
Copy link
Member

/run-unit-test

@sre-bot
Copy link
Contributor

sre-bot commented Jul 20, 2022

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/2e4b1d1df3498a4ffa1363d9aa6ae848fdd3c964

@hawkingrei hawkingrei merged commit ab513bf into pingcap:master Jul 20, 2022
@hawkingrei hawkingrei deleted the concurrent_ddl_workspace branch July 20, 2022 13:20
@sre-bot
Copy link
Contributor

sre-bot commented Jul 20, 2022

TiDB MergeCI notify

🔴 Bad News! New failing [2] after this pr merged.
These new failed integration tests seem to be caused by the current PR, please try to fix these new failed integration tests, thanks!

CI Name Result Duration Compare with Parent commit
idc-jenkins-ci/integration-cdc-test 🟥 failed 34, success 2, total 36 1 hr 11 min New failing
idc-jenkins-ci-tidb/integration-common-test 🟥 failed 2, success 9, total 11 48 min New failing
idc-jenkins-ci-tidb/common-test 🟢 all 12 tests passed 19 min Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2 🟢 all 28 tests passed 7 min 11 sec Existing passed
idc-jenkins-ci-tidb/integration-ddl-test 🟢 all 6 tests passed 6 min 42 sec Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1 🟢 all 26 tests passed 6 min 12 sec Existing passed
idc-jenkins-ci-tidb/tics-test 🟢 all 1 tests passed 6 min 2 sec Existing passed
idc-jenkins-ci-tidb/mybatis-test 🟢 all 1 tests passed 3 min 44 sec Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test 🟢 all 1 tests passed 3 min 14 sec Existing passed
idc-jenkins-ci-tidb/plugin-test 🟢 build success, plugin test success 4min Existing passed

@3pointer 3pointer mentioned this pull request Jul 21, 2022
Merged
12 tasks
This was referenced Aug 24, 2022
Merged
Closed
@YuJuncen YuJuncen mentioned this pull request Oct 18, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@overvenus overvenus overvenus left review comments

@3pointer 3pointer 3pointer left review comments

@xiongjiwei xiongjiwei xiongjiwei left review comments

@zimulala zimulala zimulala approved these changes

@XuHuaiyu XuHuaiyu XuHuaiyu approved these changes

@wjhuang2016 wjhuang2016 wjhuang2016 approved these changes

@tangenta tangenta tangenta approved these changes

Assignees
No one assigned
Labels
release-note-none size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet