Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Online DDL: support INSTANT DDL special plan, flag protected (flag undocumented for now) #10402

Merged
merged 12 commits into from
Jun 2, 2022

Conversation

shlomi-noach
Copy link
Contributor

@shlomi-noach shlomi-noach commented Jun 1, 2022

Description

#10315 introduced fast partition rotation, and with that, it paved the way for more "special Online DDL plans".

This PR adds (partial) support for Instant DDL. Some breakdown:

  • Instant DDL is applicable in 8.0, and depending on the specific version
  • Strategy must be vitess (aka online)
  • We currently support instant DDL for ADD COLUMN and DROP COLUMN statements
  • An ALTER TABLE can be eligible to Instant DDL if each and every ALTER option is eligible to Instant DDL for the specific backend MySQL version
  • Current logic is aware of recent 8.0.29 changes where ADD/DROP COLUMN are Instant-able for all column
  • An Online DDL that uses an Instant DDL operation is non-revertible
  • This is flag-protected by undocumented --fast-over-revertible ddl strategy.

v14 is due to be released in the next couple weeks. We will NOT officially document this flag in v14. This means the functionality is disabled by default and undocumented, hence we're able to iterate and make changes.

Usage example:

vtctlclient ApplySchema --skip_preflight  --ddl_strategy "vitess --allow-zero-in-date --fast-over-revertible" --sql "alter table t add column i int not null" commerce

Breakdown:

  • We use --fast-over-revertible strategy flag which indicates "we prefer Instant DDL if possible, and are aware it makes the migration unrevertible"
  • Our ALTER statement is eligible for Instant DDL on all MySQL 8.0.* versions. Vitess will auto-analyze this capability.

Another example:

vtctlclient ApplySchema --skip_preflight  --ddl_strategy "vitess --allow-zero-in-date --fast-over-revertible" --sql "alter table t add column i int not null after id, drop column c" commerce

The above is only eligible to run as Instant DDL on MySQL >= 8.0.29. On 8.0.21, for example, Vitess will run it as normal Online DDL.

Related Issue(s)

#6926

Checklist

  • "Backport me!" label has been added if this change should be backported
  • Tests were added or are not required
  • Documentation was added or is not required

Deployment Notes

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
…ctions within existing flavor functions

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
… 'instant-ddl' special plan, where applicable, given CreateTable, AlterTable and server version

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
… or non-revertible

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: Query Serving release notes labels Jun 1, 2022
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach marked this pull request as ready for review June 1, 2022 07:44
@github-actions
Copy link
Contributor

github-actions bot commented Jun 1, 2022

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has the correct release notes label. release notes none should only be used for PRs that are so trivial that they need not be included.
  • If a new flag is being introduced, review whether it is really needed. The flag names should be clear and intuitive (as far as possible), and the flag's help should be descriptive.

Bug fixes

  • There should be at least one unit or end-to-end test.
  • The Pull Request description should either include a link to an issue that describes the bug OR an actual description of the bug and how to reproduce, along with a description of the fix.

Non-trivial changes

  • There should be some code comments as to why things are implemented the way they are.

New/Existing features

  • Should be documented, either by modifying the existing documentation or creating new documentation.
  • New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • vtctl command output order should be stable and awk-able.

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
…le' in CREATE/DROP

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Copy link
Contributor

@dbussink dbussink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like how straightforward this looks.

@shlomi-noach
Copy link
Contributor Author

To be followed up by a PR that refactors a few structs and functions into designated and more appropriate packages.

@shlomi-noach shlomi-noach merged commit afd2340 into vitessio:main Jun 2, 2022
@shlomi-noach shlomi-noach deleted the online-ddl-instant-ddl-plan branch June 2, 2022 04:10
dbussink pushed a commit that referenced this pull request Jun 4, 2022
…documented for now) (#10402) (#733)

* refactor: serverVersionAtLeast and fillFlavour externalized and public

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* publish GetFlavor and ServerVersionAtLeast; refactor to use these functions within existing flavor functions

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* AnalyzeInstantDDL: a public and declarative function that generates a 'instant-ddl' special plan, where applicable, given CreateTable, AlterTable and server version

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* handle 'instant-ddl' special plan

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* utility: GetMySQLVersion

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* 'require' over 'assert'

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* add INSTANT DDL test for mysql80 and non mysql80; validate revertible or non-revertible

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* test: default value for column

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* simplify edit iteration

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

* couple more tests to validate lack of impact on '--fast-over-revertible' in CREATE/DROP

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Query Serving Type: Enhancement Logical improvement (somewhere between a bug and feature)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants