Skip to content

Filter pushdown should work with multi-ref filters#2021

Merged
shangyian merged 4 commits intoDataJunction:mainfrom
shangyian:multi-ref-filter-rewrite
Apr 19, 2026
Merged

Filter pushdown should work with multi-ref filters#2021
shangyian merged 4 commits intoDataJunction:mainfrom
shangyian:multi-ref-filter-rewrite

Conversation

@shangyian
Copy link
Copy Markdown
Collaborator

@shangyian shangyian commented Apr 18, 2026

Summary

Builds on the earlier alias/substring fixes. Three correctness changes plus one refactor:

  • Multi-reference predicates rewrite fully. Filters like a.x = 1 OR b.y = 2 now rewrite every matching dim ref instead of stopping after the first. If any ref resolves to a column the target CTE doesn't expose, the whole filter stays on the outer query rather than pushing a partial rewrite with an unresolved name behind.
  • Set-operation CTE bodies refuse pushdown. Transforms whose body is UNION/INTERSECT/EXCEPT have per-arm projections that can differ, and _inject_filter_into_where only mutates the first arm's WHERE. Pushing would under-filter the other arms.
  • Bare column refs in filters now raise. parse_dimension_ref rejects references without a node prefix (e.g., status = 'X') with a clear error directing the user to the fully-qualified form (v3.order_details.status = 'X'). Before, bare refs were silently routed to whichever parent node happened to expose a matching column, which is ambiguous in multi-parent builds.

The rewriter itself moved from regex substitution to a two-pass AST walk that swaps Subscript and Column nodes, then serializes once.

Test Plan

  • PR has an associated issue: #
  • make check passes
  • make test shows 100% unit test coverage

Deployment Plan

@netlify
Copy link
Copy Markdown

netlify Bot commented Apr 18, 2026

Deploy Preview for thriving-cassata-78ae72 canceled.

Name Link
🔨 Latest commit 5244e19
🔍 Latest deploy log https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/69e407dba453620008bb0900

@shangyian shangyian changed the title Fix filter pushdown producing invalid SQL in several edge cases Filter pushdown should work with multi-ref filters Apr 18, 2026
@shangyian shangyian marked this pull request as ready for review April 19, 2026 07:17
@shangyian shangyian merged commit f612237 into DataJunction:main Apr 19, 2026
17 checks passed
@shangyian shangyian deleted the multi-ref-filter-rewrite branch April 19, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant