-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-20920][SQL] ForkJoinPool pools are leaked when writing hive tables with many partitions #18216
Conversation
Test build #77773 has finished for PR 18216 at commit
|
Test build #77774 has finished for PR 18216 at commit
|
Test build #3779 has finished for PR 18216 at commit
|
The change looks good. Shall we add a test for it? |
Ah, I remember that it is a bit hard to automatically test it. |
The fix looks straightforward as previous fix. LGTM. Should be better if we can manually verify it like before. cc @cloud-fan |
LGTM, can we add a manual test like #18100 ? |
@viirya / @cloud-fan what do you mean here by a manual test? |
OK, I tried this as a test:
Before the change you can see that it does look like one thread is left from each of several ForkJoinPools. After the change, the result is |
Thanks @srowen. I think it looks good. |
…bles with many partitions ## What changes were proposed in this pull request? Don't leave thread pool running from AlterTableRecoverPartitionsCommand DDL command ## How was this patch tested? Existing tests. Author: Sean Owen <sowen@cloudera.com> Closes #18216 from srowen/SPARK-20920. (cherry picked from commit 7b7c85e) Signed-off-by: Sean Owen <sowen@cloudera.com>
Merged to master/2.2/2.1 |
…bles with many partitions ## What changes were proposed in this pull request? Don't leave thread pool running from AlterTableRecoverPartitionsCommand DDL command ## How was this patch tested? Existing tests. Author: Sean Owen <sowen@cloudera.com> Closes #18216 from srowen/SPARK-20920. (cherry picked from commit 7b7c85e) Signed-off-by: Sean Owen <sowen@cloudera.com>
…bles with many partitions ## What changes were proposed in this pull request? Don't leave thread pool running from AlterTableRecoverPartitionsCommand DDL command ## How was this patch tested? Existing tests. Author: Sean Owen <sowen@cloudera.com> Closes apache#18216 from srowen/SPARK-20920.
…bles with many partitions Don't leave thread pool running from AlterTableRecoverPartitionsCommand DDL command Existing tests. Author: Sean Owen <sowen@cloudera.com> Closes apache#18216 from srowen/SPARK-20920. (cherry picked from commit 7b7c85e) Signed-off-by: Sean Owen <sowen@cloudera.com>
What changes were proposed in this pull request?
Don't leave thread pool running from AlterTableRecoverPartitionsCommand DDL command
How was this patch tested?
Existing tests.