feat(datasets): add partition filter mapping model, migration and validation

[hughhhh]

---

Stack (review bottom-up):

1. feat(datasets): add partition filter mapping model, migration and validation #43757 — model, migration, serialization, validation
2. feat(datasets): mirror filters onto the partition column at query time #43758 — query rewrite + evaluator
3. feat(datasets): add partition mapping preview endpoint and editor plumbing #43759 — preview endpoint + editor plumbing
4. docs: document partition filter mapping #43760 — docs

SUMMARY

First of a four-PR stack adding partition filter mapping. This one adds the storage and the save-time validation; nothing reads the mapping yet.

Datasets on Hadoop-family engines are often partitioned on a technical column — an epoch integer, a lowercased region key — that no analyst would ever filter on. Unless a query carries a predicate on that column the engine scans every partition, and today the only workaround is hand-writing the predicate as custom SQL in a virtual dataset, which pushes a performance concern onto every chart author and takes the dataset out of the physical/syncable path.

Storage. Four columns, following the always_filter_main_dttm / currency_code_column precedent for "a dataset-level setting that names a column":

Column	Purpose
tables.partition_column	the physical partition column
tables.partition_mapped_column	explicit override; NULL follows main_dttm_col
table_columns.partition_value_transform	the :value expression
table_columns.partition_transform_is_monotonic	gates range mirroring (see PR 2)

Effective mapped column is partition_mapped_column or main_dttm_col, which makes "re-pointing the default datetime column moves the mapping with it, unless overridden" fall out of the model rather than needing code.

The monotonic flag is NOT NULL DEFAULT false rather than a nullable tri-state, matching normalize_columns — a nullable boolean invites if x: bugs where None and False need distinguishing and don't get it.

A JSON blob inside tables.extra was considered and rejected: extra is a user-editable free-text box in the dataset editor, buildExtraJsonObject rebuilds column.extra from a hardcoded key list on every save, and there is no validation layer for it today. Every comparable setting shipped in the last year chose a real column.

Validation runs in two tiers, because the PRD wants a mapping to "stay inactive until it parses" while some errors must still hard-block:

• Blocks the save: unknown columns, a column mapped onto itself, Jinja in the transform, non-deterministic functions.
• Saves, mapping inactive: unparseable transform, transform missing :value, no transform.

Two details worth a look:

• The self-mapping check validates the effective mapped column. Checking only the explicit override misses the case an owner actually hits — pointing partition_column at the column that is already main_dttm_col.
• SQLStatement.get_niladic_functions is added because the denylist can't be purely name-based: on Hive and Impala unix_timestamp() means "now" while unix_timestamp(x) — the canonical transform for this feature — is pure. Note sqlglot's Hive dialect already resolves the zero-arg form to CURRENT_TIMESTAMP; this is the backstop for dialects that don't normalize.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A — no UI in this PR.

TESTING INSTRUCTIONS

pytest tests/unit_tests/connectors/sqla/partition_mapping_test.py \
       tests/unit_tests/datasets/partition_mapping_serialization_test.py \
       tests/unit_tests/sql/parse_tests.py \
       tests/unit_tests/commands/dataset/update_test.py \
       tests/unit_tests/datasets/commands/export_test.py

Migration, verified on sqlite in both directions:

superset db upgrade     # adds all four columns to tables/table_columns + both _version shadow tables
superset db downgrade 1072de5ed955

The serialization tests assert the mapping survives every layer it passes through (export_fields, the data payload, PUT/import-v1 schemas, the API column lists) — a field missing from any one of them is dropped silently, which is the failure mode they exist to catch.

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags: PARTITION_FILTER_MAPPING (off by default)
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided — four ADD COLUMNs with no backfill and no index; effectively instant on Postgres/MySQL, no downtime expected
• Introduces new feature or API
• Removes existing feature or API

[netlify]

✅ Deploy Preview for superset-docs-preview ready!

Name	Link
🔨 Latest commit	b37eefd
🔍 Latest deploy log	https://app.netlify.com/projects/superset-docs-preview/deploys/6a9b871cbad9e1000881c6d4
😎 Deploy Preview	https://deploy-preview-43757--superset-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[codecov]

Codecov Report

❌ Patch coverage is 90.30303% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.82%. Comparing base (1358543) to head (f21009f).

Files with missing lines	Patch %	Lines
superset/commands/dataset/update.py	68.42%	8 Missing and 4 partials ⚠️
superset/connectors/sqla/partition_mapping.py	94.11%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #43757      +/-   ##
==========================================
+ Coverage   79.81%   79.82%   +0.01%     
==========================================
  Files        2913     2914       +1     
  Lines      170326   170491     +165     
  Branches    39485    39514      +29     
==========================================
+ Hits       135945   136094     +149     
- Misses      31859    31869      +10     
- Partials     2522     2528       +6     

Flag	Coverage Δ
hive	37.62% <35.15%> (-0.01%)	⬇️
mysql	57.31% <43.03%> (-0.03%)	⬇️
postgres	57.33% <43.03%> (-0.04%)	⬇️
presto	39.49% <35.15%> (-0.01%)	⬇️
python	84.22% <90.30%> (+0.01%)	⬆️
sqlite	57.04% <43.03%> (-0.03%)	⬇️
unit	75.22% <90.30%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bito-code-review]

Code Review Agent Run #e33814

Actionable Suggestions - 0

Additional Suggestions - 4

• superset/datasets/schemas.py - 1

  • load_default clobbers persisted flag · Line 112-112
    `load_default=False` injects `False` into every deserialized column dict when the client omits the field. In `DatasetDAO._upsert_columns`, `setattr(col, key, value)` then overwrites a previously-persisted `True` on any partial column update (e.g. renaming a column), silently disabling range mirroring. Use `allow_none=True` to preserve the persisted value, matching `partition_value_transform`.
• superset/commands/dataset/update.py - 1

  • Unparseable transform blocks save · Line 452-468
    The docstring (lines 402-406) says an unparseable transform is Tier-2 and "deliberately let[s] the save through." But `validate_stored_expression` raises `SupersetSecurityException` for any unparseable non-Jinja expression (models.py 1010-1021), and `parse_skeleton` leaves no Jinja, so an unparseable transform always blocks the save. Guard with `is_parseable(transform, database.backend)` to honor the documented Tier-2 behavior.
• superset/dashboards/schemas.py - 1

  • CWE-200: Column Metadata Leak · Line 375-377
    These new fields are serialized by `DashboardDatasetSchema` but not added to `DASHBOARD_DATASET_INACCESSIBLE_FIELDS` in `superset/dashboards/api.py`. `_serialize_dashboard_dataset` strips that tuple for callers lacking datasource access, so `partition_column`/`partition_mapped_column`/`partition_filter_mapping` now leak column metadata to unauthorized users, unlike the other column fields. Add them to the tuple. ([CWE-200](https://cwe.mitre.org/data/definitions/200.html))
• superset/migrations/versions/2026-09-04_00-00_a7f3c2e91d84_add_partition_filter_mapping.py - 1

  • Stale migration docstring · Line 31-31
    The docstring says `Revises: 1072de5ed955`, but the actual `down_revision` is `7e2c9a4f1b83` (line 42). `1072de5ed955` is revised by a different migration (`39097d124752`), so this docstring is stale and misrepresents the chain. Align it with the real `down_revision` to avoid confusion for readers/tooling.

Review Details

• Files reviewed - 18 · Commit Range: f21009f..f21009f

  • superset/commands/dataset/update.py
  • superset/config.py
  • superset/connectors/sqla/models.py
  • superset/connectors/sqla/partition_mapping.py
  • superset/daos/dataset.py
  • superset/dashboards/schemas.py
  • superset/datasets/api.py
  • superset/datasets/schemas.py
  • superset/mcp_service/common/schema_discovery.py
  • superset/migrations/versions/2026-09-04_00-00_a7f3c2e91d84_add_partition_filter_mapping.py
  • superset/sql/parse.py
  • superset/superset_typing.py
  • tests/integration_tests/datasets/commands_tests.py
  • tests/unit_tests/commands/dataset/update_test.py
  • tests/unit_tests/connectors/sqla/partition_mapping_test.py
  • tests/unit_tests/datasets/commands/export_test.py
  • tests/unit_tests/datasets/partition_mapping_serialization_test.py
  • tests/unit_tests/sql/parse_tests.py
• Files skipped - 2

  • docs/static/feature-flags.json - Reason: Filter setting
  • docs/static/resources/openapi.json - Reason: Filter setting
• Tools

  • MyPy (Static Code Analysis) - ✔︎ Successful
  • Astral Ruff (Static Code Analysis) - ✔︎ Successful
  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful

---

Bito Usage Guide

Commands

Type the following command in the pull request comment and save the comment.

• /review - Manually triggers an incremental AI Review.

• /review full - Manually triggers a full AI Review.

• /pause - Pauses automatic reviews on this pull request.

• /resume - Resumes automatic reviews.

• /resolve - Marks all Bito-posted review comments as resolved.

• /abort - Cancels all in-progress reviews.

Refer to the documentation for additional commands.

Configuration

This repository uses Superset You can customize the agent settings here or contact your Bito workspace admin at evan@preset.io.

Documentation & Help

• Customize agent settings
• Review rules
• General documentation
• FAQ

AI Code Review powered by

[sadpandajoe]

A nonblank transform that fails validation (for example, one without :value) is deliberately saved as inactive, but this summary marks it active anyway. Should the summary use the same validation result so the Explore indicator does not report a mapping that will never mirror filters?

[bito-code-review]

The current implementation deliberately separates validation into two tiers to ensure that non-blocking issues (like a missing :value placeholder or an unparseable transform) do not prevent the dataset from being saved. This design choice, as documented in the code, allows users to save a "half-written" transform without losing their other edits, leaving the mapping inactive until it is fully valid.

While the summary could theoretically use the same validation result to reflect the mapping's active status, the current approach prioritizes user experience by allowing partial saves. The Explore indicator's behavior is consistent with this design, as it reports the configuration as it is stored, even if the mapping is currently inactive due to validation issues.

superset/connectors/sqla/partition_mapping.py

if not contains_value_placeholder(transform):
        return [
            MappingValidationIssue(
                field=field,
                message=_(
                    "The value transform must contain the :value placeholder, "
                    "which stands for the filter value being mirrored."
                ),
                blocking=False,
            )
        ]