Project record batches to avoid filtering unused columns #18329
+336
−110
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
10 tasks
rluvaton
reviewed
Oct 28, 2025
Comment on lines
+92
to
+112
| let mut used_column_indices = HashSet::<usize>::new(); | ||
| let mut collect_column_indices = |expr: &Arc<dyn PhysicalExpr>| { | ||
| expr.apply(|expr| { | ||
| if let Some(column) = expr.as_any().downcast_ref::<Column>() { | ||
| used_column_indices.insert(column.index()); | ||
| } | ||
| Ok(TreeNodeRecursion::Continue) | ||
| }) | ||
| .expect("Closure cannot fail"); | ||
| }; | ||
|
|
||
| if let Some(e) = &self.expr { | ||
| collect_column_indices(e); | ||
| } | ||
| self.when_then_expr.iter().for_each(|(w, t)| { | ||
| collect_column_indices(w); | ||
| collect_column_indices(t); | ||
| }); | ||
| if let Some(e) = &self.else_expr { | ||
| collect_column_indices(e); | ||
| } |
There was a problem hiding this comment.
Looks like in projection.rs we have something similar (not pub though)
datafusion/datafusion/physical-plan/src/projection.rs
Lines 900 to 912 in 6eb8d45
There was a problem hiding this comment.
I'd recommend you use ProjectionExprs from https://github.com/apache/datafusion/blob/main/datafusion/physical-expr/src/projection.rs, if there's manipulations that you think might be duplicated elsewhere or useful to have abstracted feel free to propose adding them
There was a problem hiding this comment.
@rluvaton I don't think I'll be able to access code in physical_plan since that would create a dependency loop. Should I turn this into a public util function in physical_expr and use that from physical_plan?
@adriangb It's not clear to me how I could make use of ProjectionExprs. update_expr is I think the closest to what this code is trying to do, but what I need is a very limited version of it where I'm only renumbering columns.
There was a problem hiding this comment.
There's ProjectionExprs::column_indices which is pub and similar to the non-pub collect_column_indices referenced above. I haven't reviewed this PR in detail but there may be other helper bits that you can use and generally it would be nice if we coalesce projection manipulation into ProjectionExprs because I feel like there's a lot of duplicate code in random places right now (obviously needs to be balanced with keeping the API surface area on ProjectionExprs reasonable).
datafusion/datafusion/physical-expr/src/projection.rs
Lines 314 to 325 in e9431fc
rluvaton
reviewed
Oct 28, 2025
rluvaton
reviewed
Oct 28, 2025
rluvaton
reviewed
Oct 28, 2025
rluvaton
reviewed
Oct 28, 2025
|
Can you please add tests for each eval method and when used all columns and not also check when there are duplicate columns, columns with different name but same index and so on and can you please provide benchmark results |
Co-authored-by: Raz Luvaton <16746759+rluvaton@users.noreply.github.com>
|
@rluvaton I've added some SLTs which cover what you asked for
|
|
Benchmark results so far. I'll do another run with all the lookup table ones, but those take much longer to complete. |
|
🤖 |
|
🤖: Benchmark completed Details
|
|
Benchmark results confirm, I think, the waste in filtering unused columns and that the gains can be significant. I am still a bit worried about the heavy handedness of the approach I implemented here. Does the gain in execution speed cost too much during planning? Or is there maybe a simpler way to achieve the same end result? The second version I made of this was more like an optimizer pass. Instead of narrowing the record batch inside the CaseExpr, a “project record batch” expr node was wrapped around it. That has the benefit of not needing the duplicate expr tree. The downside was that this becomes visible in the physical expr tree. Much less an internal implementation detail that way. I was experimenting with an actual |
At the logical level this would be possible though, but there's no Perhaps at the logical-to-physical translation point? Although you need insight into the internals of |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
SELECT *, CASE ... END#18056Rationale for this change
When
CaseExprneeds to evaluate aPhysicalExprfor a subset of the rows of the inputRecordBatchit will first filter the record batch using a selection vector. This filter steps filters all columns of theRecordBatch, including ones that may not be accessed by thePhysicalExpr. For wide (many columns) record batches and narrow expressions (few column references) it can be beneficial to project the record batch first to reduce the amount of wasted filtering work.What changes are included in this PR?
This PR attempts to reduce the amount of time spent filtering columns unnecessarily by reducing the columns of the record batch prior to filtering. Since this renumbers the columns, it is also required to derive new versions of the
when,then, andelseexpressions that have corrected column references.To make this more manageable the set of child expressions of a
caseexpression are collected in a new struct namedCaseBody. The projection logic derives a projection vector and a projectedCaseBody.This logic is only used when the number of used columns (the length of the projection vector) is less than the number of columns of the incoming record batch.
Certain evaluation methods in
casedo not perform any filtering. These remain unchanged and will never perform the projection logic since this is only beneficial when filtering of record batches is required.Are these changes tested?
Are there any user-facing changes?
No