Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
[SPARK-25921][Follow Up][PySpark] Fix barrier task run without BarrierTaskContext while python worker reuse #23435
What changes were proposed in this pull request?
It's the follow-up PR for #22962, contains the following works:
How was this patch tested?
New UT in test_taskcontext.py
It's a bug that worker reuse loses efficacy caused by the unexpected return of checking the end of stream logic in python worker, I'll give another PR and JIRA tomorrow to fix it, this PR will continue after the problem fix.
referenced this pull request
Jan 5, 2019
This followup was not merged into branch-2.4 although the main PR went into branch-2.4 due to conflicts. Since it's rather stylic changes, I think it's okay not to backport this followup.
To reduce the diff between master and branch-2.4, we can backport it too if anyone thinks it should.