[Draft] Master phase1 fs interface refactoring#61907
Closed
morningman wants to merge 5 commits intoapache:masterfrom
Closed
[Draft] Master phase1 fs interface refactoring#61907morningman wants to merge 5 commits intoapache:masterfrom
morningman wants to merge 5 commits intoapache:masterfrom
Conversation
### What problem does this PR solve? Issue Number: N/A Problem Summary: Before splitting filesystem implementations into independent Maven modules (Phase 3), several compile-time couplings must be eliminated. This commit completes all Phase 0 prerequisite decoupling tasks: - P0.1: Introduce FsStorageType enum in fe-foundation (zero-dep module) to replace StorageBackend.StorageType (Thrift-generated) in PersistentFileSystem. Add FsStorageTypeAdapter for bidirectional Thrift conversion. Update all subclasses and callers (Repository, BackupJob, RestoreJob, CloudRestoreJob). - P0.2: Add IOException-based default bridge methods to ObjStorage interface (checkObjectExists, getObjectChecked, putObjectChecked, deleteObjectChecked, deleteObjectsChecked, copyObjectChecked, listObjectsChecked). Add ObjStorageStatusAdapter for Status→IOException conversion. Zero changes to existing implementations. - P0.3: Decouple SwitchingFileSystem from ExternalMetaCacheMgr via new FileSystemLookup functional interface. FileSystemProviderImpl passes a lambda. - P0.4: Extract MultipartUploadCapable interface from ObjFileSystem, removing the forced abstract method. S3FileSystem and AzureFileSystem implement it. HMSTransaction now uses instanceof check instead of ObjFileSystem cast. - P0.5: Introduce FileSystemDescriptor POJO for Repository metadata serialization, replacing direct PersistentFileSystem subclass serialization. Migrate GsonUtils to string-based Class.forName() reflection for legacy format backward compat, removing 7 compile-time imports of concrete filesystem classes. - P0.6: Add FileSystemSpiProvider interface skeleton in fs/spi/ as the future ServiceLoader contract for Phase 3 module split. ### Release note None ### Check List (For Author) - Test: No need to test (pure refactor; all changes are backward compatible; three successful FE builds verified during development) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Maven build cache was caching checkstyle:check results and emitting 'Skipping plugin execution (cached)' even when sources had changed. Two fixes: 1. Add check/checkstyle/ to global cache input so that changes to checkstyle rules (checkstyle.xml, suppressions.xml, etc.) correctly invalidate all module caches. 2. Mark the checkstyle:check execution (id: validate) as runAlways in executionControl so it is never skipped regardless of cache state. Checkstyle is a quality gate and must always execute. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ObjFileSystem: remove unused Map import (was used by the now-removed abstract completeMultipartUpload method) - GsonUtils: fix CustomImportOrder violations - LogManager/Logger imports were inserted before com.google.* imports; move them after all com.* imports in correct lexicographical order Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…pache#61862) ## Summary This PR completes **Phase 0** of the [FE filesystem SPI refactoring](apache#61860) — removing compile-time couplings that would otherwise prevent splitting filesystem implementations into independent Maven modules in later phases. ## Changes ### P0.1 — FsStorageType enum migration - Introduce `FsStorageType` enum in `fe-foundation` (zero-dependency module) to replace Thrift-generated `StorageBackend.StorageType` in `PersistentFileSystem` - Add `FsStorageTypeAdapter` in `fe-core` for bidirectional Thrift↔FsStorageType conversion - Update all subclasses and callers: `Repository`, `BackupJob`, `RestoreJob`, `CloudRestoreJob` ### P0.2 — ObjStorage IOException bridge - Add `IOException`-based `default` bridge methods to `ObjStorage` interface - Add `ObjStorageStatusAdapter` for `Status→IOException` conversion ### P0.3 — SwitchingFileSystem decoupling - Introduce `FileSystemLookup` functional interface - Decouple `SwitchingFileSystem` from `ExternalMetaCacheMgr` ### P0.4 — MultipartUploadCapable interface - Extract `MultipartUploadCapable` interface from `ObjFileSystem` - `S3FileSystem` and `AzureFileSystem` implement it; `HMSTransaction` uses `instanceof` check ### P0.5 — GsonUtils compile-time decoupling - Introduce `FileSystemDescriptor` POJO for `Repository` metadata serialization - `GsonUtils` removes 7 compile-time concrete class imports, uses `Class.forName()` reflection ### P0.6 — FileSystemSpiProvider skeleton - Add `FileSystemSpiProvider` interface in `fs/spi/` ### Build - Fix Maven build cache incorrectly skipping `checkstyle:check` - Fix checkstyle violations (unused import, import order) ## Testing - FE build: ✅ Checkstyle: 0 violations ✅ Closes part of apache#61860 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tem API and value objects ### What problem does this PR solve? Issue Number: N/A Problem Summary: The existing FileSystem interface uses Status-based return values, bare String paths, and Hadoop-dependent RemoteFile objects throughout the FE codebase, making it hard to test and impossible to isolate from Hadoop at the module boundary. Phase 1 introduces the new clean IOException-based FileSystem API with typed Location value objects, while preserving full backward compatibility via LegacyFileSystemApi. ### Release note None ### Check List (For Author) - Test: Manual build verification (./build.sh --fe) passes with zero errors - Behavior changed: No (all existing code paths preserved via LegacyFileSystemApi) - Does this need documentation: No New files: - Location.java: immutable URI value object replacing bare String paths - FileEntry.java: immutable file/dir descriptor replacing Hadoop-dependent RemoteFile - FileIterator.java: lazy Closeable iterator interface for directory listing - LegacyFileSystemApi.java: @deprecated copy of old FileSystem interface (Status-based) - LegacyFileSystemAdapter.java: abstract bridge implementing new FileSystem via legacy* methods - LegacyToNewFsAdapter.java: wraps any LegacyFileSystemApi as new FileSystem - MemoryFileSystem.java: in-memory FileSystem for unit testing Modified files: - FileSystem.java: replaced with new clean IOException-based interface - PersistentFileSystem, LocalDfsFileSystem, SwitchingFileSystem: implements LegacyFileSystemApi - FileSystemProvider, FileSystemLookup: return LegacyFileSystemApi - DorisInputFile, DorisOutputFile: add location() method; deprecate path() - HdfsInputFile, HdfsOutputFile, HdfsInputStream: use Location instead of ParsedPath - ParsedPath: @deprecated + toLocation() conversion method - RemoteFile: @deprecated + toFileEntry()/fromFileEntry() conversion methods - RemoteFiles, RemoteFileRemoteIterator: @deprecated - All callers updated to use LegacyFileSystemApi type Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
morningman
requested review from
CalvinKirs,
dataroaring,
gavinchou and
w41ter
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 26411 ms |
TPC-DS: Total hot run time: 168525 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.