[HUDI-7336] Introduce new HoodieStorage abstraction #10567
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yihua
force-pushed
the
HUDI-7336-hoodie-storage-abstraction
branch
from
18e8ccc
to
ca3cb83
Compare
danny0405
reviewed
Jan 26, 2024
hudi-io/src/main/java/org/apache/hudi/io/storage/HoodieStorage.java
Outdated
Show resolved
Hide resolved
hudi-io/src/main/java/org/apache/hudi/io/storage/HoodieStorage.java
Outdated
Show resolved
Hide resolved
yihua
changed the title
Yes, it's mainly based on hadoop's |
danny0405
approved these changes
Jan 27, 2024
yihua
force-pushed
the
HUDI-7336-hoodie-storage-abstraction
branch
from
ea050a0
to
c5ebd50
Compare
yihua
changed the title
|
Azure CI is green. |
4 tasks
yihua
added a commit
that referenced
this pull request
Feb 27, 2024
This commit introduces `HoodieStorage` abstraction and Hudi's counterpart classes for Hadoop File System classes (`org.apache.hadoop.fs.`[`FileSystem`, `Path`, `PathFilter`, `FileStatus`]) to decouple Hudi's implementation from Hadoop classes, so it's much easier to plugin different file system implementation.
yihua
added a commit
that referenced
this pull request
Apr 18, 2024
…mmon module (#10591) This commit makes the changes to replace most `FileSystem`, `Path`, and `FileStatus` usage with `HoodieStorage`, `StoragePath` and `StoragePathInfo` (introduced in #10567, renamed in #10672) in `hudi-common` module, to remove dependency on Hadoop FS abstraction which is not essential to most Hudi core read and write logic. This commit still keeps using the Hadoop FileSystem-based implementation under the hood. A follow-up PR will make `HoodieStorage` and I/O implementation pluggable.
yihua
added a commit
that referenced
this pull request
May 15, 2024
…mmon module (#10591) This commit makes the changes to replace most `FileSystem`, `Path`, and `FileStatus` usage with `HoodieStorage`, `StoragePath` and `StoragePathInfo` (introduced in #10567, renamed in #10672) in `hudi-common` module, to remove dependency on Hadoop FS abstraction which is not essential to most Hudi core read and write logic. This commit still keeps using the Hadoop FileSystem-based implementation under the hood. A follow-up PR will make `HoodieStorage` and I/O implementation pluggable.
yihua
added a commit
that referenced
this pull request
May 15, 2024
…mmon module (#10591) This commit makes the changes to replace most `FileSystem`, `Path`, and `FileStatus` usage with `HoodieStorage`, `StoragePath` and `StoragePathInfo` (introduced in #10567, renamed in #10672) in `hudi-common` module, to remove dependency on Hadoop FS abstraction which is not essential to most Hudi core read and write logic. This commit still keeps using the Hadoop FileSystem-based implementation under the hood. A follow-up PR will make `HoodieStorage` and I/O implementation pluggable.
yihua
added a commit
that referenced
this pull request
May 15, 2024
…mmon module (#10591) This commit makes the changes to replace most `FileSystem`, `Path`, and `FileStatus` usage with `HoodieStorage`, `StoragePath` and `StoragePathInfo` (introduced in #10567, renamed in #10672) in `hudi-common` module, to remove dependency on Hadoop FS abstraction which is not essential to most Hudi core read and write logic. This commit still keeps using the Hadoop FileSystem-based implementation under the hood. A follow-up PR will make `HoodieStorage` and I/O implementation pluggable.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
This PR introduces
HoodieStorageabstraction and Hudi's counterpart classes for Hadoop File System classes (org.apache.hadoop.fs.[FileSystem,Path,PathFilter,FileStatus]) to decouple Hudi's implementation from Hadoop classes, so it's much easier to plugin different file system implementation. Detailed changes include:HoodieStorageinterface: the counterpart class for Hadoop'sFileSystem. This provides all I/O APIs on files and directories on storage, such asopen,read, etc. This can also contain storage layer optimizations like caching, federated storage layout, hot/cold storage separation, etc. This needs to be implemented based on particular systems.HoodieHadoopStorageimplemenetsHoodieStoragewith Hadoop'sFileSystem.HoodieLocation: the counterpart class for Hadoop'sPath. We migrate and simply path parsing logic in this class.HoodieLocationFilterinterface: the counterpart class for Hadoop'sPathFilter.HoodieFileStatus: the counterpart class for Hadoop'sFileStatus. This keeps the location, length, isDirectory, and modification which are used by Hudi.This is part of the effort to provide Hudi storage abstraction and decouple
hudi-commonfrom hadoop dependencies. For reference, the single big-change PR can be found here: #10360.Impact
No impact as this PR does not have the integration.
Risk level
none
Documentation Update
N/A
Contributor's checklist