feat: close gaps 1, 2, 8 — struct dot-access, dedup qualify, WHERE filter lineage#65
Merged
Conversation
Add WHERE filter lineage tracking: columns referenced in WHERE clauses now produce where_filter edges to all non-star output columns. Subquery columns within WHERE are excluded from the outer query's predicates. - Add WherePredicateInfo dataclass and where_predicates to QueryUnit - Add is_where_filter and where_condition fields to ColumnEdge - Extract WHERE column refs in query parser (skipping subquery subtrees) - Create where_filter edges in lineage builder - Preserve WHERE metadata in pipeline cross-query edge copies - Fix trace_forward BFS to treat nodes as terminals when all outgoing targets are already visited (prevents cycles from breaking traversal)
When sqlglot parses `after.id` as Column(table="after", name="id") and "after" cannot be resolved as a table, alias, or unit in scope, the new struct fallback emits a lineage edge with nested_path=".id" and access_type="struct", using the first base table from the dependency chain as the source table. Includes recursive base table resolution for CDC-like subquery patterns.
…n (Gap 2) Detect the common dedup pattern (ROW_NUMBER() OVER (...) AS rn in subquery + WHERE rn = 1 in outer query) and promote it to qualify_info on the outer unit. Supports EQ, LTE, LT comparisons against ranking functions (ROW_NUMBER, RANK, DENSE_RANK, NTILE). Adds ranking_window_columns to QueryUnit model for cross-unit metadata propagation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
where_filteredges for all column refs in WHERE clauses, targeting all non-star output columns. NewWherePredicateInfodataclass and_extract_where_columnshelper that skips subquery subtrees.after.id) — when a table qualifier is unresolvable, recursively walk the unit dependency chain to find base tables and emit struct edges withnested_pathandaccess_type="struct". Also wired into_resolve_qualify_column.qualify_infofrom subquery-based dedup patterns (ROW_NUMBER() ... AS rn+WHERE rn = 1/<=/<). Newranking_window_columnsfield onQueryUnitand_promote_dedup_qualify_if_applicableparser method.Closes all 3 remaining open gaps from the CDC/SCD pipeline stress test.
Test plan
test_where_filter_lineage.py(simple, compound, subquery exclusion, SELECT *, dataclass fields)test_struct_dot_access.py(single-table, CDC subquery, bracket regression, multi-table JOIN, empty fallback)test_subquery_dedup_qualify.py(EQ/LTE/LT promotion, non-ranking rejection, explicit QUALIFY preservation)🤖 Generated with Claude Code