[refactor](fe) Phase 0: Prerequisite decoupling for filesystem SPI#61861
Closed
morningman wants to merge 3 commits intomasterfrom
Closed
[refactor](fe) Phase 0: Prerequisite decoupling for filesystem SPI#61861morningman wants to merge 3 commits intomasterfrom
morningman wants to merge 3 commits intomasterfrom
Conversation
### What problem does this PR solve? Issue Number: N/A Problem Summary: Before splitting filesystem implementations into independent Maven modules (Phase 3), several compile-time couplings must be eliminated. This commit completes all Phase 0 prerequisite decoupling tasks: - P0.1: Introduce FsStorageType enum in fe-foundation (zero-dep module) to replace StorageBackend.StorageType (Thrift-generated) in PersistentFileSystem. Add FsStorageTypeAdapter for bidirectional Thrift conversion. Update all subclasses and callers (Repository, BackupJob, RestoreJob, CloudRestoreJob). - P0.2: Add IOException-based default bridge methods to ObjStorage interface (checkObjectExists, getObjectChecked, putObjectChecked, deleteObjectChecked, deleteObjectsChecked, copyObjectChecked, listObjectsChecked). Add ObjStorageStatusAdapter for Status→IOException conversion. Zero changes to existing implementations. - P0.3: Decouple SwitchingFileSystem from ExternalMetaCacheMgr via new FileSystemLookup functional interface. FileSystemProviderImpl passes a lambda. - P0.4: Extract MultipartUploadCapable interface from ObjFileSystem, removing the forced abstract method. S3FileSystem and AzureFileSystem implement it. HMSTransaction now uses instanceof check instead of ObjFileSystem cast. - P0.5: Introduce FileSystemDescriptor POJO for Repository metadata serialization, replacing direct PersistentFileSystem subclass serialization. Migrate GsonUtils to string-based Class.forName() reflection for legacy format backward compat, removing 7 compile-time imports of concrete filesystem classes. - P0.6: Add FileSystemSpiProvider interface skeleton in fs/spi/ as the future ServiceLoader contract for Phase 3 module split. ### Release note None ### Check List (For Author) - Test: No need to test (pure refactor; all changes are backward compatible; three successful FE builds verified during development) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Maven build cache was caching checkstyle:check results and emitting 'Skipping plugin execution (cached)' even when sources had changed. Two fixes: 1. Add check/checkstyle/ to global cache input so that changes to checkstyle rules (checkstyle.xml, suppressions.xml, etc.) correctly invalidate all module caches. 2. Mark the checkstyle:check execution (id: validate) as runAlways in executionControl so it is never skipped regardless of cache state. Checkstyle is a quality gate and must always execute. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ObjFileSystem: remove unused Map import (was used by the now-removed abstract completeMultipartUpload method) - GsonUtils: fix CustomImportOrder violations - LogManager/Logger imports were inserted before com.google.* imports; move them after all com.* imports in correct lexicographical order Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR completes Phase 0 of the FE filesystem SPI refactoring — removing compile-time couplings that would otherwise prevent splitting filesystem implementations into independent Maven modules in later phases.
Changes
P0.1 — FsStorageType enum migration
FsStorageTypeenum infe-foundation(zero-dependency module) to replace Thrift-generatedStorageBackend.StorageTypeinPersistentFileSystemFsStorageTypeAdapterinfe-corefor bidirectional Thrift↔FsStorageType conversionRepository,BackupJob,RestoreJob,CloudRestoreJobP0.2 — ObjStorage IOException bridge
IOException-baseddefaultbridge methods toObjStorageinterface:checkObjectExists,getObjectChecked,putObjectChecked,deleteObjectChecked,deleteObjectsChecked,copyObjectChecked,listObjectsCheckedObjStorageStatusAdapterforStatus→IOExceptionconversion (mapsNOT_FOUND→FileNotFoundException)S3ObjStorage/AzureObjStorageimplementationsP0.3 — SwitchingFileSystem decoupling
FileSystemLookupfunctional interfaceSwitchingFileSystemto acceptFileSystemLookupinstead ofExternalMetaCacheMgrFileSystemProviderImplto pass a lambda:key -> extMetaCacheMgr.getFsCache().getRemoteFileSystem(key)P0.4 — MultipartUploadCapable interface
MultipartUploadCapableinterface fromObjFileSystem, removing the forcedabstract completeMultipartUploadmethodS3FileSystemandAzureFileSystemnow explicitlyimplements MultipartUploadCapableHMSTransactionupdated to useinstanceof MultipartUploadCapablecheck instead ofObjFileSystemcastP0.5 — GsonUtils compile-time decoupling
FileSystemDescriptorPOJO forRepositorymetadata serialization, replacing directPersistentFileSystemsubclass serializationRepositorygainsfs_descriptorfield (new format) with backward-compatible reading of oldstoragefieldGsonUtilsremoves 7 compile-time imports of concrete filesystem classes; replaces withClass.forName()string-based reflection (gracefulClassNotFoundExceptionhandling)P0.6 — FileSystemSpiProvider skeleton
FileSystemSpiProviderinterface infs/spi/as the future ServiceLoader contract for Phase 3 module splitBuild fix
checkstyle:check(runAlways+ include checkstyle config files in cache input)Testing
./build.sh --fe✅Related
Closes part of #61860