Add system.masking_policies introspection in open-source builds

[alexey-milovidov]

SHOW MASKING POLICIES rewrote the query into SELECT name FROM system.masking_policies, but that system table did not exist in open-source builds, so the statement failed with a confusing UNKNOWN_TABLE error.

Following review feedback, instead of making SHOW MASKING POLICIES throw a cloud-only error, this adds the system.masking_policies table and introspection to open-source builds (reusing the existing SHOW MASKING POLICIES grant), so the query returns an empty result. Masking policies can still only be created, altered, dropped and applied in ClickHouse Cloud — the manipulation interpreters remain cloud-only and continue to throw SUPPORT_IS_DISABLED. In open-source builds the table is therefore always empty (the new system table intentionally does not depend on the cloud-only MaskingPolicy entity implementation, so it links; ClickHouse Cloud populates the same table).

• New system.masking_policies system table.
• SHOW MASKING POLICIES returns an empty result instead of erroring.
• CREATE / ALTER / DROP MASKING POLICY remain cloud-only (SUPPORT_IS_DISABLED).

Closes: #101116
Related: #102785

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Added the system.masking_policies table and SHOW MASKING POLICIES introspection to open-source builds. Masking policies themselves remain a ClickHouse Cloud feature, so the table is empty in open-source builds, but introspection queries no longer fail with an error.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

Version info

• Merged into: 26.7.1.383 (included in 26.7 and later)

[clickhouse-gh]

Workflow [PR], commit [d8039c9]

Summary: ✅

• Performance Comparison: Performance dashboard

---

AI Review

Summary

This PR adds system.masking_policies to open-source builds, keeps it empty there, makes SHOW MASKING POLICIES work through the system table, and preserves SUPPORT_IS_DISABLED for masking-policy manipulation statements. I did not find any remaining correctness issues in the current diff; the prior inline findings are addressed in the code now under review.

Final Verdict

Status: ✅ Approve

[alexey-milovidov]

Instead, let's do it entirely differently.

We should have a system.masking_policies table, all the grants, and introspection, just as in ClickHouse Cloud.
But the manipulation of masking policies and their implementation remains cloud-only.

So the query will not throw an exception.

[alexey-milovidov]

@groeneai, the only CI failure here is unrelated to this PR (which only adds the system.masking_policies system table and an access grant). The Stateless tests (arm_asan_ubsan, azure, sequential) shard crashed with a segmentation fault during a vertical merge — the backtrace is entirely in MergeTree merge code (createMergeTreeSequentialSource → MergeTask::VerticalMergeStage::executeVerticalMergeForAllColumns), surfacing as a Server died on test 03716_mutations_parts_in_progress_rmt:

https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=108030&sha=979dbca2c4ac1445522bfce011f4d28e900b5399&name_0=PR&name_1=Stateless%20tests%20%28arm_asan_ubsan%2C%20azure%2C%20sequential%29

This is a recurring master-wide crash flake: the same MergeTreeSequentialSource segfault hits master directly (PR=0 on 2026-06-09, 06-11, 06-19) and many unrelated PRs (106222, 102215, 108000, 100930, 107060, 105419, 107064), and 03716_mutations_parts_in_progress_rmt itself fails sporadically on master (PR=0 on 2026-06-24) and on unrelated PRs (64315, 107338, 77210). Related open tracking issues: #107487 (Server died), #107488 / #106370 (Segmentation fault). Could you investigate and provide a fix in a separate PR (or link an existing one)?

[groeneai]

@alexey-milovidov Confirmed: not caused by this PR. The exact backtrace (readStringUntilEOFInto -> String::append reading part metadata under VerticalMergeStage::createPipelineForReadingOneColumn -> createMergeTreeSequentialSource -> MergeTreeReaderWide::prefetchForColumn) appears once in 45 days, only on this PR's run, 0 on master, 0 on other PRs. It is an innocent allocation faulting on a heap already corrupted earlier on the same MergeMutate thread, so the metadata-read frame is just the victim, not the bug.

This is the chronic MergeTree merge heap-corruption family (tracking issues #107487, #107488, #106370). The existing fix is your own #107451 plus follow-up #107457 (COW-safe IColumn::mutate on the merge output and read/build paths), which hardened the confirmed free site (MergeTreeSequentialSource::generate shrinkToFit). It is incomplete though: the family still hits master after both merged:

• read-side SIGSEGV in MergeTreeReaderCompact::readData (STID 1760-433a), master 2026-06-19, a descendant of both fixes: https://s3.amazonaws.com/clickhouse-test-reports/json.html?REF=master&sha=50faa6ff8b5968e5ba89034f768c952b2ff75038&name_0=MasterCI&name_1=Stateless%20tests%20%28arm_asan_ubsan%2C%20azure%2C%20sequential%29
• instrumented ASan with 25 ClickHouse frames in the compact-writer free path (MergeTreeSink::finishDelayedChunk -> ~CompressedWriteBuffer), master 2026-06-29: https://s3.amazonaws.com/clickhouse-test-reports/json.html?REF=master&sha=af19657538ab6e6605767fbfba0402179de12b0b&name_0=MasterCI&name_1=Stateless%20tests%20%28amd_asan_ubsan%2C%20db%20disk%2C%20distributed%20plan%2C%20sequential%2C%201%2F2%29

The instrumented twin (STID 2046-4f7a) shows the bug is single-threaded on MergeMutate: the column buffer is allocated by PODArray::resize in the reader, freed by a realloc on the merge-source path, and still read by the writer's serialize memcpy. So a residual free site remains on that path after the COW-safe sweep. It is not pinpointable from static analysis alone, the reports carry no usable coredump (build-id mismatch), and ~10 local repro attempts (local and remote-FS disks, horizontal and vertical merge, tiny merge blocks, concurrent readers) did not trigger it.

I am not opening a separate speculative PR. The obvious guard (a use_count-aware mutate at the shrinkToFit site) is exactly what #107451 already does, and it did not fully resolve the family, so a second blind guard at the residual site would not be backed by a repro. I will capture a coredump on the next instrumented (2046-*) hit to name the residual aliased holder, then follow up with a targeted fix and regression test. Issues #107487 / #107488 / #106370 should stay open.

[alexey-milovidov]

The only CI failure on the latest commit (d8039c9b) is unrelated to this PR.

This PR only adds the system.masking_policies system table, an access grant, and SHOW MASKING POLICIES introspection. The sole failing check is Stateless tests (amd_tsan, parallel, 2/2), failing on the test 03227_distinct_dynamic_types_json_paths, which exercises the distinctJSONPaths / distinctJSONPathsAndTypes / distinctDynamicTypes aggregate functions over JSON and Dynamic columns through remote('127.0.0.{1,2,3}', …) distributed queries. That test is not touched by this PR and has no path to the masking-policies code.

It is a chronic flake. Over the last 90 days the CIDB shows it failing on unrelated PRs on different sanitizer builds:

• 2026-06-30 — this PR (Add system.masking_policies introspection in open-source builds #108030), Stateless tests (amd_tsan, parallel, 2/2)
• 2026-06-29 — Fix flaky 04410_primes_source_cancellation timeout on amd_msan #108837 "Fix flaky 04410_primes_source_cancellation timeout on amd_msan", same Stateless tests (amd_tsan, parallel, 2/2)
• 2026-06-10 — Add min_by and max_by aggregate aliases #105712 "Add min_by and max_by aggregate aliases", Stateless tests (arm_asan_ubsan, azure, parallel)

The failure is a "return code" failure (the test process exits non-zero), not a deterministic reference mismatch: the captured output diverges in the Remote if section — distinctJSONPathsIf returns a1/a2 correctly, then the following distinctJSONPathsAndTypesIf / distinctDynamicTypesIf produce empty results and the run terminates before the Remote group by section. This is the signature of a transient exception on one of the remote('127.0.0.{1,2,3}', …) distributed long-test queries under loaded sanitizer CI, not a behavioral change.

Report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=108030&sha=d8039c9b66da56bce9943ef1c3d4f30ef3a14097&name_0=PR&name_1=Stateless%20tests%20%28amd_tsan%2C%20parallel%2C%202%2F2%29

All other checks are green (every build, the other stateless shards, all three flaky-check shards, stress, BuzzHouse, AST fuzzer, integration and performance jobs), there are no unresolved review threads, and the AI review verdict is Approve.

@groeneai, could you investigate this 03227_distinct_dynamic_types_json_paths flake (link above) and provide a fix in a separate PR? If a fix is already in progress, please link it here.

[alexey-milovidov]

Re-ran the only failing check (Stateless tests (amd_tsan, parallel, 2/2), failing on the unrelated flake 03227_distinct_dynamic_types_json_paths) to clear the red — the rerun is in progress. CIDB confirms this is a chronic low-frequency flake hitting unrelated PRs on different sanitizer builds (#108030 amd_tsan 2026-06-30, #108837 amd_tsan 2026-06-29, #105712 arm_asan_ubsan 2026-06-10), with no path from the masking-policies changes to that distinctJSONPaths/distinctDynamicTypes distributed test; there is no dedicated tracking issue, so it remains with @groeneai as flagged above.

I re-reviewed the diff this run and it is sound and complete: the new system.masking_policies table is always empty in open-source builds, the implicit SELECT grant is derived from SHOW MASKING POLICIES, DROP MASKING POLICY (including IF EXISTS) consistently throws SUPPORT_IS_DISABLED, the docs drop the cloud-only badge, and both tests (04489 introspection, 04490 restricted-user access) cover the behavior. The 04489_ prefix now also appears on master (04489_max_threads_auto_parsing_compat), but duplicate numeric prefixes are normal and tolerated (there is no CI uniqueness check), so no renumbering is needed. The branch is behind master but the merge base is from today and the red is not master-fixable, so I did not re-merge.

[groeneai]

Confirmed ours, and it is not related to the masking-policies change.

The two amd_tsan hits (#108030 today, #108837 yesterday) are the connection-pool desync (issue #93018):

Code: 102. Unexpected packet from server 127.0.0.2:9000 (expected TablesStatusResponse, got ProfileInfo): While executing Remote.

thrown from the distributed remote('127.0.0.{1,2,3}', ...) queries in this test (e.g. the distinctJSONPathsAndTypesIf(...) line). CIDB shows 0 desync hits in the 29 days before 2026-06-29, then a spike starting ~17:54Z that bisects to #108630 (commit b681776). That is exactly the regression already fixed by the in-flight #108854. So this needs no change to 03227_distinct_dynamic_types_json_paths itself; it clears once #108854 lands, and the rerun here is already green.

The third run you cited, #105712 (arm_asan_ubsan, azure), is a different cause: Code 241 host-OOM (RSS 53.69 GiB), not this family.

0 failures on master in 30d. We are tracking #108854 to verify across the affected distributed tests.

[clickhouse-gh]

LLVM Coverage Report

Metric	Baseline	Current	Δ
Lines	85.40%	85.50%	+0.10%
Functions	92.70%	92.70%	+0.00%
Branches	77.70%	77.80%	+0.10%

Changed lines: Changed C/C++ lines covered: 41/42 (97.62%) · Uncovered code

Full report · Diff report

[alexey-milovidov]

Resolved the CH Inc sync failure — the sync PR ClickHouse/clickhouse-private#61645 was CONFLICTING against private master. I merged private master into sync-upstream/pr/108030 and reconciled the masking-policy code with the Cloud implementation:

• StorageSystemMaskingPolicies.{h,cpp} (add/add conflict): kept the Cloud implementation (real fillData from findAll<MaskingPolicy> plus backupData/restoreDataFromBackup) instead of the open-source empty stub, so the table still populates and supports backup in Cloud. The upstream doc comment is preserved.
• attachSystemTables.cpp: attached system.masking_policies unconditionally (dropped the private #if CLICKHOUSE_CLOUD gate) to match this PR's intent of exposing the table in open-source builds. Safe in Cloud — the MaskingPolicy entity compiles in every configuration in that repository.
• InterpreterDropAccessEntityQuery.cpp: this PR's unconditional SUPPORT_IS_DISABLED rejection of DROP MASKING POLICY merged cleanly but would have broken Cloud (Cloud routes DROP MASKING POLICY through this generic interpreter and actually performs the drop). Guarded it with #if !CLICKHOUSE_CLOUD so the drop keeps working in Cloud; verified it still compiles in the Cloud config (CLICKHOUSE_CLOUD=1).
• tests/queries-no-cloud-tests.txt: skipped 04489_masking_policies_introspection in Cloud runs, since it asserts the open-source-only SUPPORT_IS_DISABLED behavior of CREATE/ALTER/DROP MASKING POLICY (all supported in Cloud). 04490_masking_policies_restricted_user is left running in Cloud as a cross-check.

The sync PR is now MERGEABLE and the private CI is running. All public checks were already green (AI review Approve, no unresolved threads).

[clickhouse-gh]

📊 Cloud Performance Report

🔴 AI verdict: degradation — 3 query(s) regressed out of 38 analysed

This PR only adds the system.masking_policies introspection table and its implicit SELECT grant (and rejects DROP MASKING POLICY in open-source builds) — it does not touch any query-execution code. The two ClickBench deltas (Q30 +18.75%, Q31 +16.05%) and the TPC-H Q16 -50% 'improvement' are off-path and have been downgraded to run-to-run variance. The three remaining TPC-H join regressions (Q4 +220%, Q7 +797%, Q8 +29%) are large enough to stand on the numbers alone, but since the diff cannot plausibly cause them, verify the benchmark build and environment before treating them as a real effect of this change.

clickbench

⚠️ 2 inconclusive

Flagged queries (2 of 43)

	Query	Verdict	Baseline median (ms)	PR median (ms)	Change	q-value	Hint
⚠️	30	not_sure	240	285	+18.8%	<0.0001	cpu: PR only adds masking-policy access control; Q30's execution path is untouched. Below override threshold — run-to-run var
⚠️	31	not_sure	308	358	+16.1%	<0.0001	cpu: Same off-path story as Q30: masking-policy plumbing can't affect this SELECT. Below override threshold — variance.

_{q-value = BH-FDR adjusted p; smaller is stronger evidence. MIRAI flags a query when q < fdr_q (default 0.10) — the value the verdict is based on.}

tpch_adapted_1_official

🔴 3 regressed · ⚠️ 1 inconclusive

Flagged queries (4 of 22)

	Query	Verdict	Baseline median (ms)	PR median (ms)	Change	q-value	Hint
🔴	4	regression	326	1042	+219.6%	<0.0001	join_operator: +220% clears the override so it stands. Diff is access-control only; verify benchmark build/environment before acting.
🔴	7	regression	71	637	+797.2%	<0.0001	join_operator: +797% multi-table join stands on the numbers. Diff cannot touch execution — check environment/build parity.
🔴	8	regression	168	216	+28.6%	<0.0001	join_operator: +29% multi-way join stands via override. Diff is masking-policy only; confirm this isn't environment noise.
⚠️	16	not_sure	100	50	-50.0%	<0.0001	join_operator: -50% 'improvement' but PR only touches masking-policy introspection, can't speed this up — variance.

_{q-value = BH-FDR adjusted p; smaller is stronger evidence. MIRAI flags a query when q < fdr_q (default 0.10) — the value the verdict is based on.}

Debug info

• StressHouse run: 30c38023-f27f-451b-951e-c7c6b66cca36
• MIRAI run: 0e34b492-b7d4-4064-b770-755a51566314
• PR check IDs:
  • clickbench_616559_1782907830
  • clickbench_616565_1782907830
  • clickbench_616575_1782907830
  • tpch_adapted_1_official_616582_1782907830
  • tpch_adapted_1_official_616595_1782907830
  • tpch_adapted_1_official_616606_1782907830

[alexey-milovidov]

Fixed the CH Inc sync failure (it flipped from pending to tests failed after the private CI finished).

Root cause: the sync run went red across ~12 private stateless shards (plus the flaky and stress jobs) on 04489_masking_policies_introspection. That test asserts the open-source-only behavior that CREATE / ALTER / DROP MASKING POLICY throw SUPPORT_IS_DISABLED, but the private build supports masking policies, so those { serverError SUPPORT_IS_DISABLED } assertions fail deterministically.

The earlier reconciliation only added the test to tests/queries-no-cloud-tests.txt, but that list is consulted only when clickhouse-test runs with --cloud. The regular private stateless jobs ("meta in keeper", "s3 storage", …) do not pass --cloud; they run with --private (the default in the private clickhouse-test, set_defaults(private=True)), so the cloud list never applied. The should_skip_test check for private_skip_list runs before the flaky/--test-runs logic, so it also covers the flaky-check and stress jobs.

Fix: added 04489_masking_policies_introspection to tests/queries-no-private-tests.txt on the sync branch (ClickHouse/clickhouse-private#61645, commit bdd4fc455ba). The private CI is re-running now. 04490_masking_policies_restricted_user was already passing in the private build (it only needs the grant + empty-result introspection) and keeps running.

The other reds on the sync run were unrelated known flakes: test_distributed_cache::test_idle_client_ttl and test_refreshable_mv_no_multi_read::test_refreshable_mv_attach_without_multi_read (Cloud-only integration flakes), and the Stress test Hung check (#107941, which the private CI itself already marks as flaky).

[alexey-milovidov]

The CH Inc sync run against the fix commit bdd4fc45 ("Skip 04489_masking_policies_introspection in the private build") finished, and the earlier 04489 stateless failures are gone across all ~12 private stateless shards — the tests/queries-no-private-tests.txt fix worked.

The four remaining reds are all chronic private/Cloud flakes, none touching the masking-policies code (this PR only adds the system.masking_policies table, a grant, and SHOW MASKING POLICIES introspection):

• Integration tests (amd_asan_ubsan, db disk, 1/7) and (…, old analyzer, 1/7) — both fail on test_refreshable_mv_no_multi_read/test.py::test_refreshable_mv_attach_without_multi_read, a refreshable-MV test with no path to masking policies. Tracked as flaky: ClickHouse/clickhouse-private#62697 ("Flaky test: …test_refreshable_mv_attach_without_multi_read").
• Stress test (amd_tsan, SharedCatalog, meta in keeper) — S3_ERROR No such key thrown, a Cloud object-storage stress flake.
• Stress test (amd_msan, SharedCatalog, meta in keeper) — Cannot start clickhouse-server, tracked as ClickHouse/clickhouse-private#61090.

I re-ran the four failed private jobs to clear the red. Public CI is fully green (0 failures), the AI review is ✅ Approve, and there are no unresolved review threads.