[#1974] fix(hive): Hive table listPartitions reduces HMS RPC calls #1975
Conversation
cxzl25
changed the title
| return IntStream.range(0, partitionNames.size()) | ||
| .mapToObj(i -> fromHivePartition(partitionNames.get(i), partitions.get(i))) | ||
| List<String> partCols = | ||
| table.buildPartitionKeys().stream().map(FieldSchema::getName).collect(Collectors.toList()); |
There was a problem hiding this comment.
If the partition keys of the underlying Hive table were changed, table.buildPartitionKeys() will get an error result.
There was a problem hiding this comment.
Hive table partition keys will not change unless the table is dropped and then created.
| .map( | ||
| partition -> | ||
| fromHivePartition( | ||
| FileUtils.makePartName(partCols, partition.getValues()), partition)) |
There was a problem hiding this comment.
Is it necessary to ensure that partCols and partition.getValues() have corresponding orders?
There was a problem hiding this comment.
| @@ -41,7 +41,7 @@ public class TestHiveTableOperations extends MiniHiveMetastoreService { | |||
| private static Partition existingPartition; | |||
|
|
|||
| @BeforeAll | |||
| private static void setup() { | |||
| public static void setup() { | |||
There was a problem hiding this comment.
Why change the access modifier to public?
There was a problem hiding this comment.
When running related test UT, IDEA has a prompt, so it is modified by the way.
Here is a non-necessary modification, I can revert.
There was a problem hiding this comment.
LGTM. Could you please review this when you have time? @jerryshao @FANNG1 Thank you!
|
LGTM |
jerryshao
added
need backport
…1975) ### What changes were proposed in this pull request? Use HMS's `listPartitions` RPC to replace the two RPC calls of `listPartitionNames` and `getPartitionsByNames`. ### Why are the changes needed? Fix: #1974 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? exist UT
What changes were proposed in this pull request?
Use HMS's
listPartitionsRPC to replace the two RPC calls oflistPartitionNamesandgetPartitionsByNames.Why are the changes needed?
Fix: #1974
Does this PR introduce any user-facing change?
No
How was this patch tested?
exist UT