-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-7044] [SQL] Fix the deadlock in script transformation #5625
Conversation
Test build #30728 has finished for PR 5625 at commit
|
// and keep the collector as remain in the main thread. | ||
// otherwise it will causes deadlock if the data size greater than | ||
// the pipeline / buffer capacity. | ||
future { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we just create a new thread to run this? seems wrong to use a global shared thread pool
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @rxin, I think you are right, updated!
885685a
to
5ec1dd2
Compare
Test build #30847 has finished for PR 5625 at commit
|
Merging in master & branch-1.3. |
Actually this doesn't merge cleanly into 1.3. Do you mind submitting a pull request for that branch? Thanks. |
Author: Cheng Hao <hao.cheng@intel.com> Closes apache#5625 from chenghao-intel/transform and squashes the following commits: 5ec1dd2 [Cheng Hao] fix the deadlock issue in ScriptTransform
Author: Cheng Hao <hao.cheng@intel.com> Closes apache#5625 from chenghao-intel/transform and squashes the following commits: 5ec1dd2 [Cheng Hao] fix the deadlock issue in ScriptTransform
No description provided.