fix: don't duplicate volatile expressions when pushing projection into file scan#23395
Open
fordN wants to merge 1 commit into
Open
fix: don't duplicate volatile expressions when pushing projection into file scan#23395fordN wants to merge 1 commit into
fordN wants to merge 1 commit into
Conversation
…o file scan A volatile expression (e.g. `random()`, `uuid()`) aliased once in a subquery and referenced multiple times must be evaluated once and reused. The physical projection-pushdown rule merged the outer projection into the file `DataSourceExec`, inlining the aliased volatile expression at every reference site and turning one call into N independent calls, so the references diverged. This regressed in 52.0.0 and reproduces on file scans (parquet/csv) but not in-memory tables. Guard `FileScanConfig::try_swapping_with_projection`: decline the merge when a volatile expression in the scan's existing projection is referenced by the incoming projection, reusing the existing `is_volatile()` utility -- the same gate the physical `ProjectionPushdown` and `FilterPushdown` rules already apply. Deterministic expressions still merge freely. Closes apache#23220.
github-actions
Bot
added
sqllogictest
Author
xudong963
approved these changes
Jul 9, 2026
Comment on lines
+676
to
+689
| let mut referenced = vec![false; inner_exprs.len()]; | ||
| for proj_expr in outer_exprs { | ||
| for col in collect_columns(&proj_expr.expr) { | ||
| let idx = col.index(); | ||
| if idx < referenced.len() { | ||
| referenced[idx] = true; | ||
| } | ||
| } | ||
| } | ||
|
|
||
| referenced | ||
| .iter() | ||
| .enumerate() | ||
| .any(|(idx, &referenced)| referenced && is_volatile(&inner_exprs[idx].expr)) |
Member
There was a problem hiding this comment.
That is safe but imprecise. A better implementation would count references with expr.apply(...), like try_collapse_projection_chain already does and block only when a volatile inner expression is referenced more than once.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
The reported bug showed that when volatile (non-deterministic) expressions are referenced multiple times in an outer function the query results can be incorrect.
To produce correct results, a volatile expression (e.g.
random(),uuid()) aliased once in a subquery and referenced multiple times must be evaluated once and reused. Since 52.0.0 this evaluation pattern for volatile functions has been broken; the physical projection-pushdown rule merges the outer projection into the fileDataSourceExec, inlining the aliased volatile expression at each reference site. Instead of being evaluated once, the volatile expression is evaluated N times, so the references diverge:This was correct in 51.0.0 and regressed in 52.0.0/53.0.0. It reproduces on file scans (Parquet/CSV) but not in-memory tables, and was surfaced downstream in Ibis. Worth noting, #10337 appears to report the same class of bug but a different cause (I'll try to look into that issue soon as well).
What changes are included in this PR?
FileScanConfig::try_swapping_with_projectionnow declines to merge a projection into the file source when the merge would inline a volatile expression that the incoming projection references. The check reuses the existingis_volatile()utility fromdatafusion_physical_expr_common— the same volatility gate the physicalProjectionPushdownandFilterPushdownrules already apply — so file-source projection merging is now consistent with them. Deterministic expressions still merge freely.The guard blocks when a volatile inner expression is referenced at least once (not only more than once), because a single outer expression can itself duplicate the value (e.g.
r + r).Are these changes tested?
Yes:
file_scan_config.rs:r + r) → blocked;projection_pushdown.sltassertingx = yfor the aliased-random()pattern over a Parquet scan.datafusion-sqllogictestsuite passes.Are there any user-facing changes?
No API changes! Query results for the affected pattern are corrected (a volatile expression aliased in a subquery is no longer duplicated by projection pushdown into a file scan). Physical plans for such queries now retain a
ProjectionExecabove the scan rather than inlining the volatile expression into theDataSourceExecprojection.