New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-18507][SQL] HiveExternalCatalog.listPartitions should only call getTable once #15978
Conversation
Test build #68993 has finished for PR 15978 at commit
|
I'll give this patch a try. Thanks. |
This patch resolves the performance problem I was seeing. Thank you! |
Lgtm. Should we also add a metric on the number if hive calls and check that in the table perf stats suite to prevent future regression? |
Oops, that was my fault. Thanks merging into master 2.1 |
…l getTable once ## What changes were proposed in this pull request? HiveExternalCatalog.listPartitions should only call `getTable` once, instead of calling it for every partitions. ## How was this patch tested? N/A Author: Wenchen Fan <wenchen@databricks.com> Closes #15978 from cloud-fan/perf. (cherry picked from commit 702cd40) Signed-off-by: Andrew Or <andrewor14@gmail.com>
@cloud-fan can you make a patch for 2.0? |
(Actually never mind, not my fault! :p) |
…l getTable once ## What changes were proposed in this pull request? HiveExternalCatalog.listPartitions should only call `getTable` once, instead of calling it for every partitions. ## How was this patch tested? N/A Author: Wenchen Fan <wenchen@databricks.com> Closes apache#15978 from cloud-fan/perf.
…l getTable once ## What changes were proposed in this pull request? HiveExternalCatalog.listPartitions should only call `getTable` once, instead of calling it for every partitions. ## How was this patch tested? N/A Author: Wenchen Fan <wenchen@databricks.com> Closes apache#15978 from cloud-fan/perf.
What changes were proposed in this pull request?
HiveExternalCatalog.listPartitions should only call
getTable
once, instead of calling it for every partitions.How was this patch tested?
N/A