docs: document source behavior on upstream schema changes and destructive operations by claude[bot] · Pull Request #37842 · MaterializeInc/materialize

claude · 2026-07-23T20:19:12Z

Requested by Pranshu Maheshwari · Slack thread

Motivation

A customer asked what happens to a Materialize source when the upstream table undergoes schema changes or destructive operations. This documents that behavior on the primary Postgres, MySQL, and SQL Server ingestion pages.

Description

Added a single reusable include — a new parameterized Hugo shortcode doc/user/layouts/shortcodes/upstream-schema-change-behavior.html — describing add-column / drop-column / constraint changes / error-state operations, and wired it into the three _index.md primary pages (postgres, mysql, sql-server) under a new ## Upstream schema changes and destructive operations section placed after ## Considerations. The shortcode takes a connector param and is called as {{% upstream-schema-change-behavior connector="..." %}}; source-versioning links are built from the connector so each page links to its own guide. This keeps a single source of truth for the prose (modeled on the existing source-versioning-disambiguation.html "parameterize by variant" pattern) rather than three copies.

PUBLICATION and REPLICA IDENTITY operations, and EXCLUSION constraints, are gated with {{ if eq (.Get "connector") "postgres" }} to render only on the Postgres page, since MySQL (GTID binlog) and SQL Server (CDC capture instances) have no such concepts.

The "Changing a column's data type" error-state bullet now renders per-connector, since the behavior differs by upstream. Postgres errors on any type change except changing it to text. MySQL only errors when the new type maps to a different Materialize type. Changes within the same Materialize type (for example tinyint ↔ smallint, the text/tinytext/mediumtext/longtext family, or adjusting bit(n) precision) do not error, and appending values to an existing enum is safe while removing enum values errors. SQL Server errors on any ALTER COLUMN. This was clarified by the MySQL and SQL Server SME.

Resolved caveat:

Whether a truncation error case applies to SQL Server CDC. Confirmed by the SQL Server SME (Marty Kulma): SQL Server prevents TRUNCATE against a CDC-enabled upstream table, which fails with error 4711 ("Cannot truncate table ... because it is published for replication or enabled for Change Data Capture"). The truncate-error path is therefore not reachable on SQL Server, so the "Truncating a table" bullet is intentionally omitted from the SQL Server page. It stays on the Postgres and MySQL pages, which both allow the operation and have documented truncation restrictions.

Verification

Built the site locally with the pinned Hugo extended v0.152.2 (hugo from doc/user). Build succeeded with exit 0, 604 pages rendered, no template/shortcode errors. Grepped the generated public/ output per connector to confirm the gating:

ingest-data/postgres/index.html: PUBLICATION, REPLICA IDENTITY, and EXCLUSION all present; rendered constraint line reads "foreign key and CHECK and EXCLUSION constraints"; DROP TABLE bullet includes the ALTER PUBLICATION ... DROP TABLE / DROP PUBLICATION clause.
ingest-data/mysql/index.html and ingest-data/sql-server/index.html: PUBLICATION, REPLICA IDENTITY, and EXCLUSION all absent (0 occurrences); constraint line reads "foreign key and CHECK constraints".
All three pages render the new section and link to their own source-versioning guide (/ingest-data/{connector}/source-versioning/#handle-upstream-column-{addition,drop}).

martykulma · 2026-07-23T21:47:11Z

SQL server prevents users from running truncate against the upstream table when CDC is enabled, so that path is not reachable. If a user attempt to truncate a CDC enabled table, the command fails with error 4711 - Cannot truncate table '%.*ls' because it is published for replication or enabled for Change Data Capture.

claude · 2026-07-23T21:51:21Z

Thanks — updated: dropped the TRUNCATE bullet from the SQL Server page since CDC-enabled tables can't be truncated (error 4711). It stays on the Postgres and MySQL pages, which do allow the operation. Pushed in ccabc87.

Generated by Claude Code

maheshwarip · 2026-07-23T22:21:46Z

Read through all the documentation carefully and validate that it makes sense. For instance, the page for MySQL now has two sections for schema changes. Think about how to incorporate this gracefully @claude . Bear in mind the legacy syntax and the new syntax.

claude · 2026-07-23T22:22:26Z

Claude finished @maheshwarip's task in 2m 42s —— View job

Reviewing documentation for duplication issues

Read all changed files and existing content
Identify duplicate sections
Fix duplication gracefully
Validate all three pages look correct

Finding: All three pages (Postgres, MySQL, SQL Server) already had a ### Schema changes section inside ## Considerations. That section covers the legacy CREATE SOURCE ... FOR ... (subsource-based) syntax — both its compatible-change bullets and its "Incompatible schema changes" note are about recreating subsources via ALTER SOURCE...ADD SUBSOURCE. The new PR added ## Upstream schema changes and destructive operations for current behavior, resulting in two schema-change sections on each page with no clear distinction.

Fix (pushed in cb0d018): Renamed ### Schema changes → ### Schema changes (legacy source syntax) in:

doc/user/content/headless/mysql-considerations.md
doc/user/content/headless/sql-server-considerations.md
doc/user/data/postgres_source_details.yml (the postgres-considerations block)

Readers now see clearly that the Considerations block covers the old subsource-based syntax, while the new top-level section covers current CREATE TABLE FROM SOURCE behavior. No content was removed.
· Branch

maheshwarip · 2026-07-23T22:23:47Z

@claude please also expand the "Type changes" into its own subheading, much like "Add a column" and "Drop a column"

claude · 2026-07-23T22:26:05Z

Claude finished @maheshwarip's task in 56s —— View job

Tasks

Read current shortcode structure
Expand "Type changes" into its own ### Changing a column's data type subheading
Commit and push

Done. Pushed in 060b9c7. The type-change bullet is now a proper ### Changing a column's data type section sitting between ### Changing constraints and ### Operations that require recreating the table. Each connector gets its own prose:

Postgres: errors on any type change except to text
MySQL: errors only when the new type maps to a different Materialize type; safe cases (same-MZ-type casts, enum appends) listed in a follow-on paragraph
SQL Server: any ALTER COLUMN errors
·

kay-kim

Before I do a little reorg, wanted to clarify some points mentioned in the pr.

kay-kim · 2026-07-28T14:25:13Z

+Materialize ignores foreign key and `CHECK`{{ if $is_postgres }} and `EXCLUSION`{{ end }}
+constraints today, so you can change them with no impact to Materialize. Adding
+a new `UNIQUE` or `NOT NULL` constraint also has no impact. Dropping a `UNIQUE`
+or `NOT NULL` constraint isn't supported today.


What does isn't supported today mean? We tend to be more specific in other places.
Also, if it was nullable when we created the source/table, and then later you add these constraints and remove these constraints?

kay-kim · 2026-07-28T14:31:37Z

+
+{{ if $is_postgres -}}
+Changing a column's data type puts the affected table into an error state,
+unless you change it to `text`. Ingestion for that table stops, and you must


By your "unless you change it to text", are you saying that if upstream table used to have some binary column B, and I alter table alter column B text upstream, that we'll ingest it?

kay-kim · 2026-07-28T14:35:57Z

+puts the affected table into an error state. Ingestion for that table stops,
+and you must drop and recreate the affected table in Materialize to resume.
+
+Changes within the same Materialize type do not error. For example, `tinyint`


Probably could word this sentence better to better compare with the previous paragraph.

kay-kim · 2026-07-28T14:51:24Z

+- Renaming a column
+- Renaming a table or moving it to a different schema
+{{ if $is_postgres }}- Setting a table's replica identity to anything other than `FULL` (`ALTER TABLE ... REPLICA IDENTITY`)
+{{ end }}{{ if ne $connector "sql-server" }}- Truncating a table (`TRUNCATE`)


Should we just mention a workaround for truncate, that you can just do a delete from table

kay-kim · 2026-07-31T12:10:11Z

+Adding a `UNIQUE` constraint does not affect ingestion. Dropping a `UNIQUE`
+constraint puts the affected table into an error state.
+
+SQL Server does not allow dropping a `PRIMARY KEY` from a table while change data
+capture is enabled on it. A primary key that existed when Materialize began
+ingesting the table therefore cannot be dropped upstream.


updated per #37960

kay-kim · 2026-07-31T12:10:26Z

+### Removing a capture instance
+
+SQL Server allows up to two capture instances to exist for a table at once.
+Materialize ingests from one of them.
+
+Removing the capture instance that Materialize is using causes ingestion to
+stall. Removing a capture instance that Materialize is not using does not affect
+ingestion.


Updated per #37960

…tive operations

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XYKBA8c2X2KsdFqhMQQDN7

Rename the existing '### Schema changes' headings inside the Considerations includes to '### Schema changes (legacy source syntax)' so that readers are not confused by two schema-change sections on each ingestion page: the legacy-subsource-focused block in Considerations now clearly signals its scope, while the new '## Upstream schema changes and destructive operations' section covers current CREATE TABLE FROM SOURCE behavior. Co-authored-by: Pranshu Maheshwari <maheshwarip@users.noreply.github.com>

Move the per-connector column-type-change details out of the "Operations that require recreating the table" bullet list and into a dedicated "### Changing a column's data type" section, consistent with the "### Adding a column" and "### Dropping a column" subheadings. Co-authored-by: Pranshu Maheshwari <maheshwarip@users.noreply.github.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XYKBA8c2X2KsdFqhMQQDN7

- incorporate legacy into the shortcode. - clarify actions.

…tate)

martykulma

🐟

kay-kim force-pushed the docs/upstream-schema-change-behavior branch from c65a0ba to 4f38df4 Compare July 27, 2026 19:37

kay-kim reviewed Jul 28, 2026

View reviewed changes

kay-kim force-pushed the docs/upstream-schema-change-behavior branch from a4f37aa to 696837b Compare July 31, 2026 12:04

kay-kim reviewed Jul 31, 2026

View reviewed changes

claude and others added 10 commits July 31, 2026 08:11

docs: document source behavior on upstream schema changes and destruc…

b56f08a

…tive operations

docs: drop forward-looking roadmap statement per feedback

c4610ee

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XYKBA8c2X2KsdFqhMQQDN7

docs: omit truncate bullet on SQL Server (CDC blocks TRUNCATE)

98a0a07

docs: make column type-change behavior per-connector

def0487

docs: fix dangling #schema-changes anchors after legacy-section rename

d7d23c6

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XYKBA8c2X2KsdFqhMQQDN7

updates (#37960)

45abc38

- incorporate legacy into the shortcode. - clarify actions.

additional contraints for sql server

c4f4018

incorporate shortcode in other pages

6cb9285

kay-kim force-pushed the docs/upstream-schema-change-behavior branch from 696837b to 6cb9285 Compare July 31, 2026 12:11

kay-kim marked this pull request as ready for review July 31, 2026 12:12

kay-kim requested a review from a team as a code owner July 31, 2026 12:12

kay-kim approved these changes Jul 31, 2026

View reviewed changes

docs: use consistent "error state" wording (per SME, stall == error s…

607b1b5

…tate)

martykulma approved these changes Jul 31, 2026

View reviewed changes

martykulma merged commit 651dde6 into main Jul 31, 2026
12 checks passed

martykulma deleted the docs/upstream-schema-change-behavior branch July 31, 2026 13:27

kay-kim mentioned this pull request Aug 7, 2026

docs: tweak snapshotting #38116

Open

Uh oh!

Conversation

claude Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

Verification

Uh oh!

martykulma commented Jul 23, 2026

Uh oh!

claude Bot commented Jul 23, 2026

Uh oh!

maheshwarip commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviewing documentation for duplication issues

Uh oh!

maheshwarip commented Jul 23, 2026

Uh oh!

claude Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

Uh oh!

kay-kim left a comment

Choose a reason for hiding this comment

Uh oh!

kay-kim Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

kay-kim Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

kay-kim Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

kay-kim Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

kay-kim Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

kay-kim Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

martykulma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

claude Bot commented Jul 23, 2026 •

edited

Loading

maheshwarip commented Jul 23, 2026 •

edited

Loading

claude Bot commented Jul 23, 2026 •

edited

Loading

claude Bot commented Jul 23, 2026 •

edited

Loading