HIVE-25800: Improvement in loadDynamicPartitions() to not load all partitions from HMS for managed table #2868
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rtitions from HMS for managed table HIVE-20661 added an improvement in loadDynamicPartitions() api in Hive.java to not add partitions one by one in HMS. It used to fetch all the existing partitions for a table from HMS and compare that with dynamic partitions list to decide old and new partitions to be added to HMS (in batches). The call to fetch all partitions has introduced a performance regression for tables with large number of partitions (of the order of 100K). This is fixed for external tables in HIVE-25178. However for ACID tables there is an open Jira HIVE-25187. Until we have an appropriate fix in HIVE-25187,we can skip fetching all partitions. Instead, in the threadPool which loads each partition individually,call getPartition() to check if the partition already exists in HMS or not. This will introduce additional getPartition() call for every partition to be loaded dynamically but does not fetch all existing partitions for a table anymore. Change-Id: I1308d51d56d77aae2c8378e153d002b1d13f7cc1
|
@lcspinter @rbalamohan @nrg4878 : Please review and provide your feedback. |
added 2 commits
December 14, 2021 10:39
…rtitions from HMS for managed table HIVE-20661 added an improvement in loadDynamicPartitions() api in Hive.java to not add partitions one by one in HMS. It used to fetch all the existing partitions for a table from HMS and compare that with dynamic partitions list to decide old and new partitions to be added to HMS (in batches). The call to fetch all partitions has introduced a performance regression for tables with large number of partitions (of the order of 100K). This is fixed for external tables in HIVE-25178. However for ACID tables there is an open Jira HIVE-25187. Until we have an appropriate fix in HIVE-25187,we can skip fetching all partitions. Instead, in the threadPool which loads each partition individually,call getPartition() to check if the partition already exists in HMS or not. This will introduce additional getPartition() call for every partition to be loaded dynamically but does not fetch all existing partitions for a table anymore. Change-Id: I1308d51d56d77aae2c8378e153d002b1d13f7cc1
Change-Id: I9b76082fd8846ad390da2bc3218b151f11d9e394
|
@kgyrtkirk : Please review and provide your feedback. |
kgyrtkirk
added
tests unstable
tests pending
and removed
tests pending
tests unstable
labels
|
The following test is failing and it seems unrelated to the patch. I have triggered the run again: |
lcspinter
approved these changes
Dec 16, 2021
|
Fix has been merged. Please close the PR |
|
Thank you @lcspinter for the approval and @nrg4878 for merging it. Commit merged to master: 63c6d8b |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
HIVE-20661 added an improvement in loadDynamicPartitions() api in Hive.java to not add partitions one
by one in HMS. It used to fetch all the existing partitions for a table from HMS and compare that with
dynamic partitions list to decide old and new partitions to be added to HMS (in batches). The call to
fetch all partitions has introduced a performance regression for tables with large number of
partitions (of the order of 100K).
This is fixed for external tables in HIVE-25178. However for ACID tables there is an open Jira HIVE-25187.
Until we have an appropriate fix in HIVE-25187,we can skip fetching all partitions. Instead, in the
threadPool which loads each partition individually,call getPartition() to check if the partition already
exists in HMS or not. This will introduce additional getPartition() call for every partition to be loaded
dynamically but does not fetch all existing partitions for a table anymore.
Why are the changes needed?
Does this PR introduce any user-facing change?
No
How was this patch tested?
Since it is an improvement in existing logic, therefore relying on existing tests.