Proposed `schemadiff` blog post #1435

shlomi-noach · 2023-04-04T07:37:31Z

Titled "'schemadiff: in-memory schema diffing, normalization, validation and manipulation'"

Randomly scheduled for April 15 (which is on a weekend) -- we'll change that per blog release schedule.

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

netlify · 2023-04-04T07:40:25Z

✅ Deploy Preview for vitess ready!

Name	Link
🔨 Latest commit	`67fead4`
🔍 Latest deploy log	https://app.netlify.com/sites/vitess/deploys/643e13254e06fc00085edcec
😎 Deploy Preview	https://deploy-preview-1435--vitess.netlify.app/blog
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

shlomi-noach · 2023-04-04T07:40:57Z

Blog preview: ~~https://deploy-preview-1435--vitess.netlify.app/blog/2023-04-15-schemadiff/~~ https://deploy-preview-1435--vitess.netlify.app/blog/2023-04-24-schemadiff/

dbussink · 2023-04-04T12:52:01Z

content/en/blog/2023-04-15-schemadiff.md

+However, there are disadvantages, as well:
+
+- Complex expressions are left, well, complex. A `CREATE VIEW ...` statement is really just a long SQL text. Which tables or views are referenced in our `VIEW` definition? Which columns?
+- Not everything is normalized and formalized. We know `int(10)` and `int(11)` are really the same. But as MySQL goes, the column type is different. Contract this with `varchar(10)` and `varchar(11)`, where the type is truly different. It still takes some higher level logic to understand what is a real diff and what isn't.


Maybe good to call out here that this is deprecated for MySQL 8.0 and link to that it doesn't mean anything anymore? Feels like it's maybe something people don't really all know yet since I see ints with lengths still in so many places.

Added link to https://dev.mysql.com/worklog/task/?id=13127 ; it's the best I could find that talks not only about ZEROFILL but also about precision.

dbussink · 2023-04-04T12:53:37Z

content/en/blog/2023-04-15-schemadiff.md

+
+Vitess is a sharding and infrastructure framework running on top of MySQL, and masquerades as a MySQL server to route queries into relevant shards. It thus obviously must be able to parse MySQL's SQL dialect. [`sqlparser`](https://github.com/vitessio/vitess/tree/main/go/vt/sqlparser) is the Vitess library that does so.
+
+`sqlparser` utilizes a [classic yacc file](https://github.com/vitessio/vitess/blob/main/go/vt/sqlparser/sql.y) to parse SQL into an Abstract Syntax Tree (AST), with `golang` structs generated and populated by the parser. For example, a SQL `CREATE TABLE` statement is parsed into a [`CreateTable`](https://github.com/vitessio/vitess/blob/157857a4c5ee6dcb44e7de08894db8334750746e/go/vt/sqlparser/ast.go#L509-L517) instance:


Maybe useful to also link to https://en.wikipedia.org/wiki/Yacc here?

dbussink · 2023-04-04T12:55:04Z

content/en/blog/2023-04-15-schemadiff.md

+There's a few things to normalize here. Inputs can come in different shapes and sizes, and still mean the same thing. Even MySQL itself presents different output for textual columns based on which version the table was created in. In the above table definition, we can normalize the following:
+
+- The canonical way to declare a `primary key` is as an index definition, not as part of the column definition.
+- `int(12)` is just an `int`. `12` does not matter. Interestingly, some ORMs do have special treatment for `int(1)`, as an indication to a boolean value. `schemadiff` accommodates that.


The example has int(11) but you talk about int(12) here.

dbussink · 2023-04-04T12:55:45Z

content/en/blog/2023-04-15-schemadiff.md

+- The canonical way to declare a `primary key` is as an index definition, not as part of the column definition.
+- `int(12)` is just an `int`. `12` does not matter. Interestingly, some ORMs do have special treatment for `int(1)`, as an indication to a boolean value. `schemadiff` accommodates that.
+- It looks like `i` is `NULL`able (because it does not say `not null`). Which means its default value is `null` even if we don't explicitly say that.
+- `v`'s collation, and thereby character set, agree with the table's collation. It can be removed.


Should we indicate that this is a choice we made for schemadiff to have it be simpler for the user to read. Vs. MySQL, which is often very verbose here in show create table?

Done, in next paragraph.

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2023-04-13T12:03:07Z

I will make minor edits to the code samples per vitessio/vitess#12551 and vitessio/vitess#12885

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

deepthi

I like it! Very well-structured and informative.
Lots of wording / re-phrasing / grammar suggestions though.

content/en/blog/2023-04-15-schemadiff.md

deepthi · 2023-04-16T19:40:05Z

@shlomi-noach can you rename the file and change the date so that we can plan to publish this on April 24, Monday? I assume all reviews can be addressed by then.

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2023-04-17T05:16:45Z

@deepthi thank you, I believe I've addressed all of your comments. File is renamed and edited to be published on April 24th.

The new preview link is now: https://deploy-preview-1435--vitess.netlify.app/blog/2023-04-24-schemadiff/

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2023-04-17T05:36:44Z

Updated code samples to reflect changes merged in vitessio/vitess#12885

deepthi · 2023-04-17T20:53:15Z

@deepthi thank you, I believe I've addressed all of your comments. File is renamed and edited to be published on April 24th.

The new preview link is now: https://deploy-preview-1435--vitess.netlify.app/blog/2023-04-24-schemadiff/

The old file still needs to be deleted, there are now 2 files in this PR.
Rest LGTM, so I'm going to approve. You can plan to publish it next Monday.

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2023-04-18T03:49:03Z

there are now 2 files in this PR.

That's bizzarre. I used git mv. Not sure how that file was left over. anyway, deleted.

shlomi-noach · 2023-04-18T03:50:17Z

You can plan to publish it next Monday.

Does that mean the PR can be merged any time before Monday April 24, or does it need to be merged on the 24th?

deepthi · 2023-04-21T00:27:30Z

You can plan to publish it next Monday.

Does that mean the PR can be merged any time before Monday April 24, or does it need to be merged on the 24th?

If you merge it before Apr 24, it won't appear on the website right away. The first website deploy on or after Apr 24 will publish it.
Since we can't guarantee that there will be a PR merge to trigger a website deploy on that exact day (and we don't want to have to go into netlify and manually trigger a deployment), the best thing to do is to push the green button on Apr 24.

shlomi-noach · 2023-04-24T07:31:23Z

Published URL: https://vitess.io/blog/2023-04-24-schemadiff/

shlomi-noach added 8 commits July 4, 2022 08:58

blog post: schemadiff

2a6ce67

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

more progress

c7c91b5

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Merge branch 'prod' into blog-schemadiff

c8b1509

push date

b1cdee1

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

towards schemadiff blog

3d83457

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

ongoing schemadiff blog content

49c6340

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

content complete

d230b71

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

remove duplicate intros

c793246

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

dbussink reviewed Apr 4, 2023

View reviewed changes

fixes per review

65ae137

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach requested a review from a team April 13, 2023 05:10

shlomi-noach added 4 commits April 16, 2023 14:30

typo

aba25e4

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

grammar

493f4a1

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

minor improvements

2977b59

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

minor rewording

ccf2e41

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

deepthi reviewed Apr 16, 2023

View reviewed changes

shlomi-noach and others added 10 commits April 17, 2023 06:49

Update content/en/blog/2023-04-15-schemadiff.md

5c42346

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

1b9bdc5

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

c1bce4d

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

2cd216e

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

3108f95

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

25b9468

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

25af072

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

73bb726

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

b74b856

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

0410fa4

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach and others added 16 commits April 17, 2023 06:54

Update content/en/blog/2023-04-15-schemadiff.md

2ba8282

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

5263293

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

97657ce

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

5046389

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

15e888d

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Update content/en/blog/2023-04-15-schemadiff.md

83faa47

Co-authored-by: Deepthi Sigireddi <deepthi@planetscale.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

fixes per review

0e2af5c

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

grammar: SQL diff

3e7966e

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

grammar

50db759

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

grammar

384a9bc

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

fixes per review

fa40f71

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

better example

ecd4104

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

clarify foreign key sentence

e31edb4

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

s/resolves/computes

9bd5f21

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

publish date

f563279

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

publish date

45dcd88

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

adapt to changes in vitessio/vitess#12885

1f72607

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

deepthi approved these changes Apr 17, 2023

View reviewed changes

removed old entry

67fead4

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach marked this pull request as ready for review April 20, 2023 12:55

shlomi-noach requested a review from a team April 20, 2023 14:28

shlomi-noach merged commit 0c86aad into prod Apr 24, 2023

shlomi-noach deleted the blog-schemadiff branch April 24, 2023 07:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposed `schemadiff` blog post #1435

Proposed `schemadiff` blog post #1435

shlomi-noach commented Apr 4, 2023

netlify bot commented Apr 4, 2023 •

edited

Loading

shlomi-noach commented Apr 4, 2023 •

edited

Loading

dbussink Apr 4, 2023

shlomi-noach Apr 4, 2023

dbussink Apr 4, 2023

shlomi-noach Apr 4, 2023

dbussink Apr 4, 2023

shlomi-noach Apr 4, 2023

dbussink Apr 4, 2023

shlomi-noach Apr 4, 2023

shlomi-noach commented Apr 13, 2023

deepthi left a comment

deepthi commented Apr 16, 2023

shlomi-noach commented Apr 17, 2023

shlomi-noach commented Apr 17, 2023

deepthi commented Apr 17, 2023

shlomi-noach commented Apr 18, 2023

shlomi-noach commented Apr 18, 2023 •

edited

Loading

deepthi commented Apr 21, 2023

shlomi-noach commented Apr 24, 2023


		Vitess is a sharding and infrastructure framework running on top of MySQL, and masquerades as a MySQL server to route queries into relevant shards. It thus obviously must be able to parse MySQL's SQL dialect. [`sqlparser`](https://github.com/vitessio/vitess/tree/main/go/vt/sqlparser) is the Vitess library that does so.

		`sqlparser` utilizes a [classic yacc file](https://github.com/vitessio/vitess/blob/main/go/vt/sqlparser/sql.y) to parse SQL into an Abstract Syntax Tree (AST), with `golang` structs generated and populated by the parser. For example, a SQL `CREATE TABLE` statement is parsed into a [`CreateTable`](https://github.com/vitessio/vitess/blob/157857a4c5ee6dcb44e7de08894db8334750746e/go/vt/sqlparser/ast.go#L509-L517) instance:

Proposed schemadiff blog post #1435

Proposed schemadiff blog post #1435

Conversation

shlomi-noach commented Apr 4, 2023

netlify bot commented Apr 4, 2023 • edited Loading

✅ Deploy Preview for vitess ready!

shlomi-noach commented Apr 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shlomi-noach commented Apr 13, 2023

deepthi left a comment

Choose a reason for hiding this comment

deepthi commented Apr 16, 2023

shlomi-noach commented Apr 17, 2023

shlomi-noach commented Apr 17, 2023

deepthi commented Apr 17, 2023

shlomi-noach commented Apr 18, 2023

shlomi-noach commented Apr 18, 2023 • edited Loading

deepthi commented Apr 21, 2023

shlomi-noach commented Apr 24, 2023

Proposed `schemadiff` blog post #1435

Proposed `schemadiff` blog post #1435

netlify bot commented Apr 4, 2023 •

edited

Loading

shlomi-noach commented Apr 4, 2023 •

edited

Loading

shlomi-noach commented Apr 18, 2023 •

edited

Loading