New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-17396][core] Share the task support between UnionRDD instances. #14985
Conversation
This removes the private task support in each UnionRDD and replaces it with a shared pool on the UnionRDD object.
Test build #65012 has finished for PR 14985 at commit
|
Looks good to me as a minimal change. It's not hard to use the Java executors here too if that would be preferable but would require more rewrite and some more code. |
Test build #3252 has finished for PR 14985 at commit
|
Jenkins retest this please |
Just retesting to see what that 'does not merge cleanly' is about. |
LGTM as well. This seems like a reasonable fix. |
I can also rebase on master. I'm surprised it isn't merging cleanly, it's a tiny patch to code that hardly ever changes. |
I think it merges fine, because github says so. I don't know why Jenkins reported that it didn't merge but that it succeeded. |
Test build #65160 has finished for PR 14985 at commit
|
## What changes were proposed in this pull request? Share the ForkJoinTaskSupport between UnionRDD instances to avoid creating a huge number of threads if lots of RDDs are created at the same time. ## How was this patch tested? This uses existing UnionRDD tests. Author: Ryan Blue <blue@apache.org> Closes #14985 from rdblue/SPARK-17396-use-shared-pool. (cherry picked from commit 6ea5055) Signed-off-by: Sean Owen <sowen@cloudera.com>
Merged to master/2.0 |
## What changes were proposed in this pull request? Share the ForkJoinTaskSupport between UnionRDD instances to avoid creating a huge number of threads if lots of RDDs are created at the same time. ## How was this patch tested? This uses existing UnionRDD tests. Author: Ryan Blue <blue@apache.org> Closes apache#14985 from rdblue/SPARK-17396-use-shared-pool.
What changes were proposed in this pull request?
Share the ForkJoinTaskSupport between UnionRDD instances to avoid creating a huge number of threads if lots of RDDs are created at the same time.
How was this patch tested?
This uses existing UnionRDD tests.