feat(parser): resolve lateral flatten lineage (phantom_alias 23→1 models) - #138
Merged
Conversation
A Snowflake `LATERAL FLATTEN(<variant_expr>) f` (and `TABLE(FLATTEN(...)) f`) is a pseudo-relation, so a downstream `f.value` / `f.value:path` could not be tied to any upstream and was declared `phantom_alias` (honest, but the lineage edge was dropped). Resolve it instead: `f.value` derives from the column(s) inside `<variant_expr>`, so attribute the downstream column to those real upstream columns — a genuine `derived` edge. - Build a per-query flatten-alias -> flattened-expression map and resolve each flattened expression's source columns through the same CTE/alias/forward-ref machinery used for projections, evaluated in the flatten's own SELECT scope (so an unqualified `flatten(order_ids)` binds to that SELECT's FROM table, and a same-select derived `flatten(parse_json(raw):items)` follows the forward reference to its real upstream column). - Expand nested flatten transitively (`flatten(outer.value)` where `outer` is itself a flatten alias), with a visited-set cycle guard. - Preserve the honesty tail: a flatten over an untraceable expression (a literal array, a non-flatten `lateral (subquery)`) still emits the `phantom_alias` marker — no fabrication. Measured over a real Snowflake dbt project (1504 models): partial-edges models 116 -> 97; models carrying a `phantom_alias` marker 23 -> 1 (the one remaining is a genuinely-unresolvable literal-array flatten). No model gained a `fabricated_column` marker; pivot/star/unexpandable constructs untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Teach the parser to resolve
LATERAL FLATTENlineage instead of dropping it. ALATERAL FLATTEN(<variant_expr>) fis a pseudo-relation, so a downstreamf.value/f.value:pathcouldn't be tied to an upstream and was (correctly, since #135) declaredphantom_alias— honest, but the edge was lost. This attributesf.valueto the real columns inside<variant_expr>.How
sql_parser_utils.py—get_flatten_alias_nodes()yields(alias, flattened_expr, enclosing_select)forexp.Lateral(Explode)andexp.TableFromRows(Explode), unwrapping theinput =>kwarg or positional arg.sql_parser.py—_build_flatten_alias_sourcesresolves each flattened expression through the same CTE/alias/forward-ref machinery used for projections (soflatten(parse_json(content):transactions)follows the forward ref to its real source column rather than fabricating a same-name column on the base relation);_expand_flatten_sourceshandles nested flatten transitively with a cycle guard.derivededge); otherwise the honestphantom_aliasmarker is still emitted.Impact (real analytics manifest+catalog, 1504 models,
--adapter snowflake)phantom_aliasmarker: 23 → 1 (remaining one is a genuinely-unresolvable literal-array flattenflatten([60, 69, ...])).fabricated_column; pivot_output / unexpandable_star untouched — no wrong edges introduced.lateral (subquery)still emitphantom_aliaswith no fabrication.Tests
Added flatten-resolution unit tests (qualified alias,
input => …:pathvariant/cast →derived, nested flatten,table(flatten(...)), unresolvable literal-array → honest marker, non-flatten lateral subquery → honest marker); updated the #135 tests whose flatten now correctly resolves. 624 unit tests pass. (integration/e2e error at collection on the pre-existing dbt-env issue; the non-dbt ones pass.)Version not bumped (CI owns it).
🤖 Generated with Claude Code