Skip to content

v3 SQL: walk every source-table reference when pushing dim filters#2166

Merged
shangyian merged 5 commits into
DataJunction:mainfrom
shangyian:primary-arm-only
May 22, 2026
Merged

v3 SQL: walk every source-table reference when pushing dim filters#2166
shangyian merged 5 commits into
DataJunction:mainfrom
shangyian:primary-arm-only

Conversation

@shangyian
Copy link
Copy Markdown
Collaborator

@shangyian shangyian commented May 22, 2026

Summary

Direct dim-link pushdown in v3 only injects the filter into the primary select of a transform's source body. When the same source table is referenced multiple times (e.g., a pattern where the inner subquery rejoins the source for an FK lookup), the secondary reference is left unfiltered.

There are two pushdown paths in v3:

  • Indirect path (dimensions.py's _register_pushdown_into_upstream): runs when the filter resolves via an upstream link rather than a direct one. Uses _resolve_pushdown_targets, which searches through the entire parsed source body and injects into every reference's enclosing select.
  • Direct path (cte.py's _resolve_pushdown_filters_for_cte): runs when the parent transform has a direct dim link. Only looks at the CTE's primary select projection and injects into its where clause. Doesn't walk nested subqueries.

Counter-intuitively, adding a direct dim link to a transform makes pushdown less thorough because it short-circuits the upstream path that does the full walk. Modelers see the regression appear when they add what they think is a more-specific link.

The fix is to extend the direct path in cte.py to do the same find_all(ast.Table) walk that the indirect path does. After the primary-select rewrite succeeds.

Test Plan

  • PR has an associated issue: #
  • make check passes
  • make test shows 100% unit test coverage

Deployment Plan

@netlify
Copy link
Copy Markdown

netlify Bot commented May 22, 2026

Deploy Preview for thriving-cassata-78ae72 canceled.

Name Link
🔨 Latest commit 7376d79
🔍 Latest deploy log https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/6a10caede618890008c0b124

@shangyian shangyian changed the title Revert to asymmetric arm pushdown v3 SQL: walk every source-table reference when pushing dim filters May 22, 2026
@shangyian shangyian marked this pull request as ready for review May 22, 2026 23:42
@shangyian shangyian merged commit 9d8d8b6 into DataJunction:main May 22, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant