Skip to content

[Fix](multi catalog)Return all partition values while reading hive table. #19434

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
morningman merged 1 commit into from
May 10, 2023
Merged

[Fix](multi catalog)Return all partition values while reading hive table. #19434

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions fe/fe-common/src/main/java/org/apache/doris/common/Config.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1679,6 +1679,14 @@ public class Config extends ConfigBase {
@ConfField(mutable = false, masterOnly = false)
public static long max_hive_partition_cache_num = 100000;

@ConfField(mutable = false, masterOnly = false, description = {"Hive表到分区名列表缓存的最大数量。",
"Max cache number of hive table to partition names list."})
public static long max_hive_table_catch_num = 1000;

@ConfField(mutable = false, masterOnly = false, description = {"获取Hive分区值时候的最大返回数量，-1代表没有限制。",
"Max number of hive partition values to return while list partitions, -1 means no limitation."})
public static short max_hive_list_partition_num = -1;

/**
* Max cache loader thread-pool size.
* Max thread pool size for loading external meta cache
Expand Down
2 changes: 1 addition & 1 deletion fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ public HiveMetaStoreCache(HMSExternalCatalog catalog, Executor executor) {
}

private void init() {
partitionValuesCache = CacheBuilder.newBuilder().maximumSize(Config.max_hive_partition_cache_num)
partitionValuesCache = CacheBuilder.newBuilder().maximumSize(Config.max_hive_table_catch_num)
.expireAfterAccess(Config.external_cache_expire_time_minutes_after_access, TimeUnit.MINUTES)
.build(CacheLoader.asyncReloading(
new CacheLoader<PartitionValueCacheKey, HivePartitionValues>() {
Expand Down
3 changes: 2 additions & 1 deletion fe/fe-core/src/main/java/org/apache/doris/datasource/hive/PooledHiveMetaStoreClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,8 @@ public class PooledHiveMetaStoreClient {
private static final Logger LOG = LogManager.getLogger(PooledHiveMetaStoreClient.class);

private static final HiveMetaHookLoader DUMMY_HOOK_LOADER = t -> null;
private static final short MAX_LIST_PARTITION_NUM = 10000;
// -1 means no limit on the partitions returned.
private static final short MAX_LIST_PARTITION_NUM = Config.max_hive_list_partition_num;

private Queue<CachedClient> clientPool = new LinkedList<>();
private final int poolSize;
Expand Down