Skip to content

fix: Optimize projections in recursive CTEs#22476

Open
nuno-faria wants to merge 3 commits into
apache:mainfrom
nuno-faria:recursive_cte_optimize_projections
Open

fix: Optimize projections in recursive CTEs#22476
nuno-faria wants to merge 3 commits into
apache:mainfrom
nuno-faria:recursive_cte_optimize_projections

Conversation

@nuno-faria
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Optimize projections for the static and recursive terms of recursive CTEs just like regular subqueries. The previous implementation optimized projections based on the outer columns, which could cause bugs.

This new implementation still ensures that #16684 remains fixed.

What changes are included in this PR?

  • Updated optimize_projections to be applied to the static and recursive terms of a recursive query.
  • Updated and added tests.

Are these changes tested?

Yes.

Are there any user-facing changes?

Maybe if the user relied on projections being pushed down from the outer query to the recursive CTE, but this can be fixed by removing those unnecessary projections directly in the CTE.

}

#[test]
fn recursive_cte_with_unused_columns() -> Result<()> {
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test was very similar to the now named recursive_cte_outer_projection_pushdown.

Comment on lines +845 to +861
06)------Cross Join:
07)--------Projection:
08)----------Filter: recursive_cte.val < Int64(2)
09)------------TableScan: recursive_cte projection=[val]
10)--------SubqueryAlias: sub_cte
11)----------EmptyRelation: rows=1
physical_plan
01)RecursiveQueryExec: name=recursive_cte, is_distinct=false
02)--ProjectionExec: expr=[1 as val]
03)----PlaceholderRowExec
04)--ProjectionExec: expr=[2 as val]
05)----CrossJoinExec
06)------CoalescePartitionsExec
07)--------FilterExec: val@0 < 2
07)--------FilterExec: val@0 < 2, projection=[]
08)----------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
09)------------WorkTableExec: name=recursive_cte
10)------ProjectionExec: expr=[2 as val]
11)--------PlaceholderRowExec
10)------PlaceholderRowExec
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Projection is now optimized here.

Comment on lines +1207 to +1213
03)----TableScan: closure projection=[start, end]
04)----Projection: l.start, r.end
05)------Inner Join: l.end = r.start
06)--------SubqueryAlias: l
07)----------TableScan: trans projection=[start, end]
08)--------SubqueryAlias: r
09)----------TableScan: closure projection=[start, end]
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

@github-actions github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels May 23, 2026
@Omega359
Copy link
Copy Markdown
Contributor

This PR tests the recursive CTE projection fix for UNION ALL, but not for recursive UNION where an outer-unselected column still affects distinctness. This matters because pruning that column would collapse distinct rows incorrectly. The test is below.

# Keep columns that are not selected by the outer query, but still affect
# recursive UNION distinctness.
query I
with recursive t(k, v) as (
  select 1 k, 10 v
  union
  select 2, 10 from t where v = 10
)
select v
from t
order by 1;
----
10
10

Co-authored-by: Bruce Ritchie <bruce.ritchie@gmail.com>
@nuno-faria
Copy link
Copy Markdown
Contributor Author

Thanks @Omega359. I've added the union test as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimize projections causes recursive CTEs to fail Projection pushdown on Recursive CTEs with nested subqueries

2 participants