[Enhancement](multi-catalog) Make meta cache batch loading concurrently.#21471
[Enhancement](multi-catalog) Make meta cache batch loading concurrently.#21471morningman merged 6 commits intoapache:masterfrom
Conversation
6602348 to
03ae40a
Compare
|
run buildall |
|
run buildall |
2 similar comments
|
run buildall |
|
run buildall |
|
TeamCity pipeline, clickbench performance test result: |
|
TeamCity pipeline, clickbench performance test result: |
5f31fc7 to
ddedcbf
Compare
|
run buildall |
|
TeamCity pipeline, clickbench performance test result: |
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
ddedcbf to
d1d52cc
Compare
|
run buildall |
|
(From new mechine)TeamCity pipeline, clickbench performance test result: |
|
TeamCity pipeline, clickbench performance test result: |
…ly. (#21471) I will enhance performance about querying meta cache of hms tables by 2 steps: **Step1** : use concurrent batch loading for meta cache **Step2** : execute some other tasks concurrently as soon as possible **This pr mainly for step1 and it mainly do the following things:** - Create a `CacheBulkLoader` for batch loading - Remove the executor of the previous async cache loader and change the loader's type to `CacheBulkLoader` (We do not set any refresh strategies for LoadingCache, so the previous executor is not useful) - Use a `FixedCacheThreadPool` to replace the `CacheThreadPool` (The previous `CacheThreadPool` just log warn infos and will not throw any exceptions when the pool is full). - Remove parallel streams and use the `CacheBulkLoader` to do batch loadings - Change the value of `max_external_cache_loader_thread_pool_size` to 64, and set the pool size of hms client pool to `max_external_cache_loader_thread_pool_size` - Fix the spelling mistake for `max_hive_table_catch_num`
Proposed changes
I will enhance performance about querying meta cache of hms tables by 2 steps:
Step1 : use concurrent batch loading for meta cache
Step2 : execute some other tasks concurrently as soon as possible
This pr mainly for step1 and it mainly do the following things:
CacheBulkLoaderfor batch loadingCacheBulkLoader(We do not set any refresh strategies for LoadingCache, so the previous executor is not useful)FixedCacheThreadPoolto replace theCacheThreadPool(The previousCacheThreadPooljust log warn infos and will not throw any exceptions when the pool is full).CacheBulkLoaderto do batch loadingsmax_external_cache_loader_thread_pool_sizeto 64, and set the pool size of hms client pool tomax_external_cache_loader_thread_pool_sizemax_hive_table_catch_numFurther comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...