[fix](catalog) disable FileSystem Cache to avoid too many fs cache #21283
Conversation
| Path path = new Path(finalLocation); | ||
| URI uri = path.toUri(); | ||
| if (uri.getScheme() != null) { | ||
| updateJobConf("fs." + uri.getScheme() + ".impl.disable.cache", "true"); |
There was a problem hiding this comment.
s3 scheme: s3 and s3a
cos scheme: cosn
There was a problem hiding this comment.
It does need to be same logic as in FileInputFormat.setInputPaths(), which is to use schema is file location.
|
PR approved by anyone and no changes requested. |
|
run buildall |
|
TeamCity pipeline, clickbench performance test result: |
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
…ache#21283) When creating a new hive catalog or refresh the hive catalog, it will refresh the HiveMetaStore cache. And it will call "FileInputFormat.setInputPaths()". In this method, it will create a new FileSystem instance and store it in FileSystem's cache. So if refresh catalog frequently, there will be too many FileSystem instances in cache, causing OOM. This PR disable the FileSystem Cache.
…ache#21283) When creating a new hive catalog or refresh the hive catalog, it will refresh the HiveMetaStore cache. And it will call "FileInputFormat.setInputPaths()". In this method, it will create a new FileSystem instance and store it in FileSystem's cache. So if refresh catalog frequently, there will be too many FileSystem instances in cache, causing OOM. This PR disable the FileSystem Cache.
…1283) When creating a new hive catalog or refresh the hive catalog, it will refresh the HiveMetaStore cache. And it will call "FileInputFormat.setInputPaths()". In this method, it will create a new FileSystem instance and store it in FileSystem's cache. So if refresh catalog frequently, there will be too many FileSystem instances in cache, causing OOM. This PR disable the FileSystem Cache.
5 participants
Proposed changes
When creating a new hive catalog or refresh the hive catalog, it will refresh the HiveMetaStore cache.
And it will call "FileInputFormat.setInputPaths()".
In this method, it will create a new FileSystem instance and store it in FileSystem's cache.
So if refresh catalog frequently, there will be too many FileSystem instances in cache, causing OOM.
This PR disable the FileSystem Cache.
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...