Skip to content

MySQL Source Versioning using FULL binlog_row_metadata SS-71 SS-67#35598

Merged
patrickwwbutler merged 10 commits intoMaterializeInc:mainfrom
patrickwwbutler:patrick/mysql-sv
Apr 7, 2026
Merged

MySQL Source Versioning using FULL binlog_row_metadata SS-71 SS-67#35598
patrickwwbutler merged 10 commits intoMaterializeInc:mainfrom
patrickwwbutler:patrick/mysql-sv

Conversation

@patrickwwbutler
Copy link
Copy Markdown
Contributor

@patrickwwbutler patrickwwbutler commented Mar 23, 2026

Implements a simple source versioning support for MySQL sources that relies on the upstream database setting binlog_row_metadata = FULL, guaranteeing that binlog row events will contain column names, making it quite easy to handle changes to upstream schemas.

Updates docs, and creates a new test for this capability, and updates other tests with expected behavior.

@github-actions
Copy link
Copy Markdown
Contributor

Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone.

PR title guidelines

  • Use imperative mood: "Fix X" not "Fixed X" or "Fixes X"
  • Be specific: "Fix panic in catalog sync when controller restarts" not "Fix bug" or "Update catalog code"
  • Prefix with area if helpful: compute: , storage: , adapter: , sql:

Pre-merge checklist

  • The PR title is descriptive and will make sense in the git log.
  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).

@patrickwwbutler patrickwwbutler changed the title MySQL Source Versioning MySQL Source Versioning using FULL binlog_row_metadata Mar 26, 2026
@patrickwwbutler patrickwwbutler marked this pull request as ready for review April 2, 2026 14:27
@patrickwwbutler patrickwwbutler requested review from a team as code owners April 2, 2026 14:27
@patrickwwbutler patrickwwbutler changed the title MySQL Source Versioning using FULL binlog_row_metadata MySQL Source Versioning using FULL binlog_row_metadata SS-71 SS-67 Apr 2, 2026
@patrickwwbutler patrickwwbutler requested a review from a team April 2, 2026 14:40
@patrickwwbutler patrickwwbutler requested a review from a team as a code owner April 2, 2026 18:02
Copy link
Copy Markdown
Contributor

@martykulma martykulma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great @patrickwwbutler ! I had some comments, the most important of which is providing the reasoning for the conditional treatment of row in decoding.rs.

Comment thread src/mysql-util/src/decoding.rs Outdated
Comment thread src/mysql-util/src/decoding.rs Outdated
Comment thread src/mysql-util/src/decoding.rs Outdated
Comment thread src/mysql-util/src/decoding.rs Outdated
Comment thread src/mysql-util/src/desc.rs Outdated
Comment thread test/mysql-cdc/upstream-schema-changes.td Outdated
@patrickwwbutler patrickwwbutler merged commit 011b3b5 into MaterializeInc:main Apr 7, 2026
122 checks passed
bosconi added a commit that referenced this pull request Apr 7, 2026
binlog-backward-compat.td (from #35598) sets binlog_row_metadata to
MINIMAL but never restores it. When sharded together with
upstream-schema-changes.td (also from #35598), the purification check
added in #35839 rejects CREATE TABLE FROM SOURCE because the MySQL
server is still in MINIMAL mode.

Failure seen in https://buildkite.com/materialize/test/builds/120130#019d6892-80d9-4ae8-b919-814f200dc7ba

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bosconi added a commit that referenced this pull request Apr 7, 2026
binlog-backward-compat.td (from #35598) sets binlog_row_metadata to
MINIMAL but never restores it. When sharded together with
upstream-schema-changes.td (also from #35598), the purification check
added in #35839 rejects CREATE TABLE FROM SOURCE because the MySQL
server is still in MINIMAL mode.

Failure seen in https://buildkite.com/materialize/test/builds/120130#019d6892-80d9-4ae8-b919-814f200dc7ba

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bosconi added a commit that referenced this pull request Apr 7, 2026
Fixes merge skew between #35598 and #35839.

`binlog-backward-compat.td` (from #35598) sets `binlog_row_metadata` to
MINIMAL but never restores it. When sharded together with
`upstream-schema-changes.td` (also from #35598), the purification check
added in #35839 rejects `CREATE TABLE FROM SOURCE` because the MySQL
server is still in MINIMAL mode.

Failure seen in
https://buildkite.com/materialize/test/builds/120130#019d6892-80d9-4ae8-b919-814f200dc7ba

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mgree pushed a commit to mgree/materialize that referenced this pull request Apr 13, 2026
…aterializeInc#35598)

Implements a simple source versioning support for MySQL sources that
relies on the upstream database setting `binlog_row_metadata = FULL`,
guaranteeing that binlog row events will contain column names, making it
quite easy to handle changes to upstream schemas.

Updates docs, and creates a new test for this capability, and updates
other tests with expected behavior.
mgree pushed a commit to mgree/materialize that referenced this pull request Apr 13, 2026
Fixes merge skew between MaterializeInc#35598 and MaterializeInc#35839.

`binlog-backward-compat.td` (from MaterializeInc#35598) sets `binlog_row_metadata` to
MINIMAL but never restores it. When sharded together with
`upstream-schema-changes.td` (also from MaterializeInc#35598), the purification check
added in MaterializeInc#35839 rejects `CREATE TABLE FROM SOURCE` because the MySQL
server is still in MINIMAL mode.

Failure seen in
https://buildkite.com/materialize/test/builds/120130#019d6892-80d9-4ae8-b919-814f200dc7ba

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants