Move error check from pipeline fixer to pipeline checker #5938
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mustafasrepo
changed the title
comphead
approved these changes
Apr 10, 2023
| LIMIT 5".to_string(), | ||
| cases: vec![Arc::new(test1), Arc::new(test2)], | ||
| error_operator: "Window Error".to_string() | ||
| error_operator: "Sort Error".to_string() |
Contributor
There was a problem hiding this comment.
lgtm, thanks for taking this, why the error message changed to Sort Error, I dont see the respective logic change..
Contributor
There was a problem hiding this comment.
I think this happens because when the task of erroring out is left to PipelineChecker, which is the last rule, sorts are already in-place. While checking, the algorithm first encounters the sort (before the window), which is is pipeline-breaking. Hence the error messsage.
Contributor
|
Thank you @mustafasrepo @ozankabak and @comphead for the review |
korowa
pushed a commit
to korowa/arrow-datafusion
that referenced
this pull request
Apr 13, 2023
* Add window reversal sub rule to pipeline fixer. * update test * Remove an unnecessary clone, avoid object construction by using mutable binding * Propagate error to pipeline checker --------- Co-authored-by: Mehmet Ozan Kabak <ozankabak@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #.
Rationale for this change
Sometimes
PipelineFixerwill receive executors that cannot run on unbounded data. In these cases currently, we generate error. However, during optimization stages these pipeline breaking executors may be removed, may be changed, etc. Hence,PipelineFixershouldn't give pre-mature decisions.For instance, If window expression is in the form
SUM(inc_col) OVER(ORDER BY ts DESC ROWS BETWEEN 1 PRECEDING AND UNBOUNDED FOLLOWING) as sum1it would be calculated withWindowAggExec. However, it can be converted to the equivalent formSUM(inc_col) OVER(ORDER BY ts ASC ROWS BETWEEN UNBOUNDED PRECEDING AND 1 FOLLOWING) as sum1Second form can be calculated with
BoundedWindowAggExecwithout breaking the pipeline (given that its input is already ordered withts ASC).If source is unbounded, during optimization we may convert expression in the form 1 to in the form 2 which is not breaking pipeline.
What changes are included in this PR?
With this PR,
PipelineFixerdoesn't generate an error, when some executor cannot be run unbounded data (Those cases may be fixed during optimization.) If those executors cannot be fixed, we generate error inPipelineCheckeranyway.Util codes to construct test table are moved from
window.rsto undertest_util/mod.rsto be able to use utils in other files.Are these changes tested?
Yes
test_source_sorted_unbounded_sourceis added to check this behaviour.Are there any user-facing changes?