[HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue#7105
Conversation
|
@danny0405 please help review this : ) |
| Path path = new Path(directory); | ||
| FileSystem fileSystem = path.getFileSystem(new Configuration()); | ||
| FileSystem fileSystem = FSUtils.getFs(path, fs.getConf()); | ||
| RemoteIterator<LocatedFileStatus> itr = fileSystem.listFiles(path, true); |
There was a problem hiding this comment.
fs is already a FileSystem, why new an instance here ?
There was a problem hiding this comment.
For context.flatMap is not thread safe, three is many thread in context.flatMap may operate same fs if not create new one in directory -> {} function.
There was a problem hiding this comment.
-> may operate same fs
What's the problem here, the fs instance is cached BTW, so you would still fetch the same fs instance each time.
There was a problem hiding this comment.
So we just remove path.getFileSystem(new Configuration());, and rename fs ouside to filesystem in lambda，right?
There was a problem hiding this comment.
@danny0405 Hi, I have fixed it, please help review again ~
There was a problem hiding this comment.
What bug is fixed here then ? Or simplify a code refactoring.
…atePartitionPathSelector and BootstrapUtils not consistent issue
|
@danny0405 is this still a bug or we can close the PR? |
I kind of think it makes sense, we better to unify the instantiation of the fs objects. |
|
Sounds good. @CTTY could you revamp this PR given you have worked on |
|
I've assigned the JIRA ticket to myself. FileSystemBackedTableMetadata, FSUtils, and BootStrapUtils in this PR have already been refactored. I'll post a new PR to replace FS usages in DatePartitionPathSelector and related classes later |
|
Sounds good. Closing this stale PR. |
6 participants
Change Logs
Make getFileSystem way unify to org.apache.hudi.common.fs.FSUtils#getFs(org.apache.hadoop.fs.Path, org.apache.hadoop.conf.Configuration)
Impact
none
Risk level (write none, low medium or high below)
none
Documentation Update
none
Contributor's checklist