catalog: convert mz_materialized_views into a view over the catalog by teskje · Pull Request #35819 · MaterializeInc/materialize

teskje · 2026-04-01T10:27:06Z

This PR converts the mz_materialized_views builtin table into a materialized view over the catalog.

It includes some pre-work to make this possible:

Introduces a constant builtin view mz_builtin_materialized_views that exposes... builtin materialized views. The mz_materialized_views query joins against that view to augment the catalog contents (which only include user MVs) with all the builtin MVs.
Introduces two new internal SQL functions:
- parse_catalog_create_sql, to extract information from the MV create_sql
- redact_sql, to derived redacted_create_sql from the MV create_sql

Alternatives to having mz_builtin_materialized_views as a constant builtin view:

Make it a builtin table that gets updated during bootstrap. We want to move away from builtin tables as they complicate having concurrent envds, so this seems like a step in the wrong direction. Builtin views pose no issues for concurrent envds.
Make it a builtin table function. This doesn't work because SQL funcs are defined in mz-expr, while builtins are defined in mz-catalog, and the dependency relationship is the wrong way round. We could introduce a way to add table functions that get evaluated during planning time, but afaict that infrastructure doesn't exist yet.
Insert builtins into the catalog so we can simply treat them like user MVs. This might be nice but is also a lot of work, and might also complicate 0dt upgrades by adding more state to the catalog that will need to be migrated.

TODO

The sqlsmith nightly errors need to be fixed by excluding redact_sql: postgres: exclude the redact_sql function sqlsmith#6

Motivation

Part of SQL-118

github-actions · 2026-04-01T10:27:16Z

Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone.

PR title guidelines

Use imperative mood: "Fix X" not "Fixed X" or "Fixes X"
Be specific: "Fix panic in catalog sync when controller restarts" not "Fix bug" or "Update catalog code"
Prefix with area if helpful: compute: , storage: , adapter: , sql:

Pre-merge checklist

The PR title is descriptive and will make sense in the git log.
This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).

This commit adds the first constant builtin view exposing builtin objects to the catalog. Specifically, `mz_builtin_materialized_views` reports what builtin materialized views exist. It is required to define `mz_materialized_views` as a view over the catalog, since the catalog does not contain builtin objects. Further builtin views exposing other builtin object types will be added as needed to support converting more builtin tables.

teskje · 2026-04-01T13:53:57Z

src/expr/src/scalar/func/impls/jsonb.rs

+// TODO: This function isn't parsing JSONB and therefore shouldn't live in the `jsonb` module.
+//       Consider moving all the `parse_catalog_*` functions into their own module.


Didn't want to include this in this PR because of the noise. I'll make a separate code movement PR.

teskje · 2026-04-01T13:55:31Z

src/expr/src/scalar/func/unary.rs


 use mz_repr::{Datum, InputDatumType, OutputDatumType, ReprColumnType, RowArena, SqlColumnType};

+use crate::scalar::func::RedactSql;


I put this in mz_expr::scalar::func so it's next to pretty_sql, which is the only similar function we have, afaict. But now it's the only unary func that doesn't live in mz_expr::scalar::func::impls. Should I move it?

I'd say it's ok. We can move it later if needed.

teskje · 2026-04-01T13:56:49Z

test/0dt/mzcompose.py

-
-        new_mz_tables_gid = c.sql_query(
-            "SELECT id FROM mz_tables WHERE name = 'mz_tables'",
-            service="mz_new",
-        )[0][0]
-        new_mv_gid = c.sql_query(
-            "SELECT id FROM mz_materialized_views WHERE name = 'mv'",
-            service="mz_new",
-        )[0][0]
-        assert new_mz_tables_gid == mz_tables_gid
-        assert new_mv_gid == mv_gid
-        # mz_internal.mz_storage_shards won't update until this instance becomes the leader
-


Had to remove this part because mz_materialized_views is now a builtin MV and becomes unreadable in read-only mode after a replacement migration. I think this is fine, the checks here didn't add much anyway.

teskje · 2026-04-01T13:59:07Z

src/catalog/src/builtin/builtin.rs

+            .with_key(vec![0])
+            .with_key(vec![2])
+            .with_key(vec![4])
+            .with_key(vec![6])


The optimizer decides these keys based on the concretes values in the VALUES list. It determines that name, definition and create_sql values are keys because they are unique... which isn't wrong!

Yeah, these are somewhat surprising keys, but I'd say it's ok. (At some point, we'll hopefully introduce more rigorous testing for the keys of builtins.)

teskje · 2026-04-01T15:48:58Z

Green nightly (except that sqlsmith stubles over redact_sql): https://buildkite.com/materialize/nightly/builds/15939#_

This commit adds two new internal sqlfuncs to support converting `mz_materialzied_views` into a view over the catalog. * `parse_catalog_create_sql` parses a create_sql string and returns information extracted from it as JSON * `redact_sql` redacts a given SQL statement

ggevay

LGTM!

src/catalog/src/builtin.rs

ggevay · 2026-04-02T12:27:12Z

src/catalog/src/builtin/builtin.rs

+            .with_key(vec![0])
+            .with_key(vec![2])
+            .with_key(vec![4])
+            .with_key(vec![6])


Yeah, these are somewhat surprising keys, but I'd say it's ok. (At some point, we'll hopefully introduce more rigorous testing for the keys of builtins.)

ggevay · 2026-04-02T12:27:13Z

test/0dt/mzcompose.py

-
-        new_mz_tables_gid = c.sql_query(
-            "SELECT id FROM mz_tables WHERE name = 'mz_tables'",
-            service="mz_new",
-        )[0][0]
-        new_mv_gid = c.sql_query(
-            "SELECT id FROM mz_materialized_views WHERE name = 'mv'",
-            service="mz_new",
-        )[0][0]
-        assert new_mz_tables_gid == mz_tables_gid
-        assert new_mv_gid == mv_gid
-        # mz_internal.mz_storage_shards won't update until this instance becomes the leader
-


ggevay · 2026-04-02T12:27:16Z

src/expr/src/scalar/func/unary.rs


 use mz_repr::{Datum, InputDatumType, OutputDatumType, ReprColumnType, RowArena, SqlColumnType};

+use crate::scalar::func::RedactSql;


I'd say it's ok. We can move it later if needed.

ggevay · 2026-04-02T12:27:17Z

src/expr/src/scalar/func/impls/jsonb.rs

+// TODO: This function isn't parsing JSONB and therefore shouldn't live in the `jsonb` module.
+//       Consider moving all the `parse_catalog_*` functions into their own module.


teskje force-pushed the mz_materialized_views-view branch 2 times, most recently from f27e0e4 to 0ae57d9 Compare April 1, 2026 13:28

teskje marked this pull request as ready for review April 1, 2026 13:52

teskje requested review from a team as code owners April 1, 2026 13:52

teskje requested a review from ggevay April 1, 2026 13:52

teskje commented Apr 1, 2026

View reviewed changes

teskje force-pushed the mz_materialized_views-view branch from 0ae57d9 to ddfb2e6 Compare April 2, 2026 08:24

ggevay approved these changes Apr 2, 2026

View reviewed changes

catalog: convert mz_materialized_views into a view over the catalog

2b78c03

teskje force-pushed the mz_materialized_views-view branch from ddfb2e6 to 2b78c03 Compare April 2, 2026 12:44

teskje mentioned this pull request Apr 2, 2026

postgres: exclude the redact_sql function MaterializeInc/sqlsmith#6

Open

		// TODO: This function isn't parsing JSONB and therefore shouldn't live in the `jsonb` module.
		// Consider moving all the `parse_catalog_*` functions into their own module.


		use mz_repr::{Datum, InputDatumType, OutputDatumType, ReprColumnType, RowArena, SqlColumnType};

		use crate::scalar::func::RedactSql;

Conversation

teskje commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Motivation

Uh oh!

github-actions bot commented Apr 1, 2026

PR title guidelines

Pre-merge checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teskje commented Apr 1, 2026

Uh oh!

ggevay left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

teskje commented Apr 1, 2026 •

edited

Loading