Skip to content

fix(optimizer): keep columns a recursive CTE reads from itself [CLAUDE] - #7899

Merged
georgesittas merged 1 commit into
tobymao:mainfrom
isaka1022:fix/pushdown-projections-recursive-cte
Jul 20, 2026
Merged

fix(optimizer): keep columns a recursive CTE reads from itself [CLAUDE]#7899
georgesittas merged 1 commit into
tobymao:mainfrom
isaka1022:fix/pushdown-projections-recursive-cte

Conversation

@isaka1022

Copy link
Copy Markdown
Contributor

Fixes optimizer.optimize crashing (or silently producing invalid SQL) on a self-referencing recursive CTE when the enclosing query doesn't select every CTE column.

Repro on current main:

from sqlglot import parse_one
from sqlglot.optimizer import optimize

sql = """
WITH RECURSIVE t AS (
  SELECT id, link FROM graph WHERE id = 1
  UNION ALL
  SELECT g.id, g.link FROM graph AS g, t WHERE g.id = t.link
)
SELECT id FROM t
"""
optimize(parse_one(sql, read="postgres"), schema={"graph": {"id": "INT", "link": "INT"}}, dialect="postgres")
# KeyError: 'link'  (merge_subqueries.py)

Root cause: _traverse_ctes gives the CTE body a throwaway branch scope to resolve the recursive term's self-reference. pushdown_projections records the self-reference's column usage (t.link above) against that throwaway scope, so it is lost, and the projections the recursive term still joins on get pruned from the CTE. merge_subqueries then crashes on the missing projection; depending on the query the pruning can also survive silently as invalid SQL. Remapping the usage to the real scope wouldn't help on its own, because the reversed-postorder pass visits the union before the recursive term, making the self-reference a back-edge.

Fix: treat self-referencing recursive CTEs conservatively and skip pruning their projections, mirroring the existing DISTINCT guard in the same loop. merge_subqueries._is_recursive and pushdown_predicates already special-case recursive CTEs the same way; pushdown_projections was the one rule missing the guard.

Added a regression test that asserts the optimized CTE keeps the column its recursive term joins on (it fails with a KeyError without the fix).

🤖 Generated with Claude Code

pushdown_projections records column usage of a recursive CTE's
self-reference against the throwaway scope that _traverse_ctes creates
for the recursive term, so that usage is lost and projections the CTE
still joins on get pruned. The corrupted CTE then crashes
merge_subqueries with a KeyError (or silently generates invalid SQL).

Guard self-referencing recursive CTEs with SELECT_ALL, mirroring the
existing DISTINCT guard; merge_subqueries and pushdown_predicates
already special-case recursive CTEs the same way.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@georgesittas georgesittas self-assigned this Jul 20, 2026

@georgesittas georgesittas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, LGTM, but I think we can improve this by checking what projections are actually referenced to avoid the over-conservative block.

I'll take it to the finish line myself.

@georgesittas
georgesittas merged commit 199547a into tobymao:main Jul 20, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants