Return errors properly from RepartitionExec #521
Conversation
| Ok(()) | ||
| }); | ||
|
|
||
| // In a separate task, wait for each input to be done |
There was a problem hiding this comment.
This is the actual code change (to check for return value in another task). Otherwise the rest of this PR is tests
| }) | ||
| } | ||
|
|
||
| /// Waits for `input_task` which is consuming one of the inputs to |
There was a problem hiding this comment.
I wonder if it might be slightly clearer to push the body of the main task into a fallible function, and to then handle propagating any error it returns within the spawned task? i.e. rather than propagating the error through the JoinHandle, make the task that is spawned onto tokio infallible and handle its errors internally??
Edit: I guess the advantage with this approach would be that you could propagate panics as well...
There was a problem hiding this comment.
I agree the approach you describe would be clearer (and avoid needing a separate task) 👍
The reason I did not pull the main body out into its own function was mostly "trying to keep the diff small" (or perhaps my own laziness wanting to avoid having to figure out all the types of the arguments that got captured),
Perhaps that would be a good follow on PR (there is a lot of messiness / duplication for updating counters which I would also kind of like to fix too)
Codecov Report
@@ Coverage Diff @@
## master #521 +/- ##
==========================================
+ Coverage 76.07% 76.09% +0.01%
==========================================
Files 156 156
Lines 26750 26858 +108
==========================================
+ Hits 20351 20438 +87
- Misses 6399 6420 +21
Continue to review full report at Codecov.
|
alamb
force-pushed
the
alamb/repartition_swallows_errors
branch
from
76a1d96 to
fda9122
Compare
|
Rebased |
## Which issue does this PR close? Closes apache#521 and apache#665. ## What changes are included in this PR? This PR normalize the right hand of div and rem ## How are these changes tested? Test updated
* Add support for window function bindings * Refactor PyWindow to be moved to Expr submodule * Adjust Expr module imports, moved WindowFrame to expr * Update src/expr/window.rs --------- Co-authored-by: Andy Grove <andygrove73@gmail.com>
5 participants
Which issue does this PR close?
Closes #437
Rationale for this change
Errors should be returned rather than incorrect results (empty stream). This bug threw me for a loop while debugging something in IOx. See more details on #437
Also, since this operator gets added automatically by the repartition operator, it is likely to affect many/most plans in DataFusion
What changes are included in this PR?
Properly propagate errors from input, and tests for same