-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-19486][CORE](try 3) Investigate using multiple threads for task serialization #17139
Conversation
a06f8c8
to
6874d1e
Compare
Test build #73775 has finished for PR 17139 at commit
|
Test build #73776 has finished for PR 17139 at commit
|
Test build #73769 has finished for PR 17139 at commit
|
Test build #73771 has finished for PR 17139 at commit
|
6874d1e
to
4d9b666
Compare
Test build #73848 has finished for PR 17139 at commit
|
ping @kayousterhout @squito |
Why is the time improvement so much larger here than in the other PR? |
Added the multi-threaded code for serialization |
Can you also post the time differences for some smaller jobs (e.g., 100 tasks, 1000 tasks, 10K tasks) to get a sense of how this varies with size? |
@kayousterhout The test report has been updated. |
4d9b666
to
8fbe15c
Compare
Test build #74270 has finished for PR 17139 at commit
|
@witgo Are you still working on this? |
@jiangxb1987 ,Yes do you have any questions? |
Please bring this PR up to date, then maybe some guy can review this. :) |
This should probably just be closed |
We are closing it due to inactivity. please do reopen if you want to push it forward. Thanks! |
What changes were proposed in this pull request?
See https://issues.apache.org/jira/browse/SPARK-19486
In the case of stage has a lot of tasks, this PR can improve the scheduling performance of
15%The test code:
and
spark-defaults.conf
file:The test results are as follows
How was this patch tested?
Existing tests.