Please sign in to comment.
[SPARK-25139][SPARK-18406][CORE][BRANCH-2.3] Avoid NonFatals to kill …
…the Executor in PythonRunner ## What changes were proposed in this pull request? Python uses a prefetch approach to read the result from upstream and serve them in another thread, thus it's possible that if the children operator doesn't consume all the data then the Task cleanup may happen before Python side read process finishes, this in turn create a race condition that the block read locks are freed during Task cleanup and then the reader try to release the read lock it holds and find it has been released, in this case we shall hit a AssertionError. We shall catch the AssertionError in PythonRunner and prevent this kill the Executor. ## How was this patch tested? Hard to write a unit test case for this case, manually verified with failed job. Closes #24670 from rezasafi/branch-2.3. Authored-by: Xingbo Jiang <email@example.com> Signed-off-by: Dongjoon Hyun <firstname.lastname@example.org>
- Loading branch information...
Showing with 20 additions and 16 deletions.