Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support CREATE, DROP statements in ApplySchema and online DDL #7083

Merged
merged 9 commits into from
Dec 9, 2020
Merged

Support CREATE, DROP statements in ApplySchema and online DDL #7083

merged 9 commits into from
Dec 9, 2020

Conversation

shlomi-noach
Copy link
Contributor

@shlomi-noach shlomi-noach commented Nov 29, 2020 •
edited
Loading

With this PR, we support CREATE TABLE and DROP TABLE statements to run as Online DDL. Right now this new functionality is only supported by vtctl ApplySchema, and not yet by vtgate queries.

e.g. you may vtctl ApplySchema -ddl_strategy "gh-ost" -sql "create table t(id int primary key); drop table t2" commerce

CREATE and DROP statements submitted this way are not executed immediately. Instead, they are scheduled, and pushed to the tablets through the Online DDL mechanism; tablets will run those queries with their own schedule/queue flow.

Reasons for this PR:

  • Generalizing the online DDL statements to more than just ALTER
  • Be able to group together multiple schema changes under same context (see also OnlineDDL: request_context/migration_context #7082), and have them executed more-or-less together, regardless of DDL type
  • In next steps: implicitly converting DROP TABLE statements to RENAME, as per Tracking issue: safe, lazy, managed DROP TABLE #6689, making DROP TABLE truly online DDL.
    • Note: this is still TBD, and we should break multi-table DROP statements into distinct single-table statements.

@shlomi-noach
Support CREATE, DROP statements in ApplySchema and online DDL
d839167 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Copy link
Contributor Author

shlomi-noach commented Nov 29, 2020 •
edited
Loading

  • TODO: endtoend tests

@shlomi-noach
fixed unit test
90452df 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
added CREATE/DROP online DDL endtoend tests
496f28c 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Copy link
Contributor Author

I'm considering how to set a more general ddl_strategy, since "gh-ost" and "pt-osc" obviously do not apply to CREATE and DROP statements.

Current line of thought:

  • if -ddl_strategy is set to pt-osc or gh-ost, then this is online DDL, and CREATE and DROP and statements should go online.
  • introduce -ddl_strategy "online". It's a more general purpose setting. For CREATE and DROP statements it makes sense.
    For ALTER statements it's a bit more vague and I'm thinking whenever we have online strategy then vitess should dynamically pick gh-ost or pt-osc. I'll keep thinking about this.

@shlomi-noach
trigger early next invocation of onMigrationCheckTick upon:
2f06eb6 
- migration completion
- migration failure
- new INSERT to schema_migrations
throttle invocation of onMigrationCheckTick() to at most 1/sec

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
triggering initial tick early on
a1ac2cf 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
TODO comment
e167fa9 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach marked this pull request as ready for review November 29, 2020 19:31
@shlomi-noach shlomi-noach requested a review from a team November 29, 2020 19:31
@shlomi-noach
Copy link
Contributor Author

Ready for review

@shlomi-noach
Copy link
Contributor Author

General context: #6926

This was referenced Dec 2, 2020
Open
Merged
@shlomi-noach
merge upstream/master
d4fb510 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach mentioned this pull request Dec 3, 2020
Merged
@shlomi-noach
Copy link
Contributor Author

Documentation PR: vitessio/website#623

@shlomi-noach
Merge branch 'master' into online-ddl-create-drop-online
93bc2ed
@shlomi-noach
goimports
b62c71d 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Copy link
Contributor Author

🙏 @vitessio/ps-vitess request for review

rohit-nayak-ps
rohit-nayak-ps approved these changes Dec 9, 2020
View reviewed changes
Copy link
Contributor

@rohit-nayak-ps rohit-nayak-ps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@shlomi-noach shlomi-noach merged commit 7703097 into vitessio:master Dec 9, 2020
@shlomi-noach shlomi-noach deleted the online-ddl-create-drop-online branch December 9, 2020 15:50
@askdba askdba added this to the v9.0 milestone Dec 10, 2020
@shlomi-noach shlomi-noach mentioned this pull request Dec 23, 2020
Merged
7 tasks
@rafael rafael mentioned this pull request Mar 17, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rohit-nayak-ps rohit-nayak-ps rohit-nayak-ps approved these changes

@sougou sougou Awaiting requested review from sougou

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v9.0
Development

Successfully merging this pull request may close these issues.
3 participants
@shlomi-noach @rohit-nayak-ps @askdba