Skip to content

[refactor](fe) Phase 0: Prerequisite decoupling for filesystem SPI#61861

Closed
morningman wants to merge 3 commits intomasterfrom
branch-fs-spi
Closed

[refactor](fe) Phase 0: Prerequisite decoupling for filesystem SPI#61861
morningman wants to merge 3 commits intomasterfrom
branch-fs-spi

Conversation

@morningman
Copy link
Copy Markdown
Contributor

Summary

This PR completes Phase 0 of the FE filesystem SPI refactoring — removing compile-time couplings that would otherwise prevent splitting filesystem implementations into independent Maven modules in later phases.

Changes

P0.1 — FsStorageType enum migration

  • Introduce FsStorageType enum in fe-foundation (zero-dependency module) to replace Thrift-generated StorageBackend.StorageType in PersistentFileSystem
  • Add FsStorageTypeAdapter in fe-core for bidirectional Thrift↔FsStorageType conversion
  • Update all subclasses and callers: Repository, BackupJob, RestoreJob, CloudRestoreJob

P0.2 — ObjStorage IOException bridge

  • Add IOException-based default bridge methods to ObjStorage interface: checkObjectExists, getObjectChecked, putObjectChecked, deleteObjectChecked, deleteObjectsChecked, copyObjectChecked, listObjectsChecked
  • Add ObjStorageStatusAdapter for Status→IOException conversion (maps NOT_FOUNDFileNotFoundException)
  • Zero changes to existing S3ObjStorage/AzureObjStorage implementations

P0.3 — SwitchingFileSystem decoupling

  • Introduce FileSystemLookup functional interface
  • Refactor SwitchingFileSystem to accept FileSystemLookup instead of ExternalMetaCacheMgr
  • Update FileSystemProviderImpl to pass a lambda: key -> extMetaCacheMgr.getFsCache().getRemoteFileSystem(key)

P0.4 — MultipartUploadCapable interface

  • Extract MultipartUploadCapable interface from ObjFileSystem, removing the forced abstract completeMultipartUpload method
  • S3FileSystem and AzureFileSystem now explicitly implements MultipartUploadCapable
  • HMSTransaction updated to use instanceof MultipartUploadCapable check instead of ObjFileSystem cast

P0.5 — GsonUtils compile-time decoupling

  • Introduce FileSystemDescriptor POJO for Repository metadata serialization, replacing direct PersistentFileSystem subclass serialization
  • Repository gains fs_descriptor field (new format) with backward-compatible reading of old storage field
  • GsonUtils removes 7 compile-time imports of concrete filesystem classes; replaces with Class.forName() string-based reflection (graceful ClassNotFoundException handling)

P0.6 — FileSystemSpiProvider skeleton

  • Add FileSystemSpiProvider interface in fs/spi/ as the future ServiceLoader contract for Phase 3 module split

Build fix

  • Fix Maven build cache incorrectly skipping checkstyle:check (runAlways + include checkstyle config files in cache input)
  • Fix 2 checkstyle violations introduced by this refactoring (unused import, import order)

Testing

  • FE build passes: ./build.sh --fe
  • Checkstyle: 0 violations ✅
  • All changes are backward compatible; no behavior change

Related

Closes part of #61860

morningman and others added 3 commits March 30, 2026 12:02
### What problem does this PR solve?

Issue Number: N/A

Problem Summary: Before splitting filesystem implementations into independent
Maven modules (Phase 3), several compile-time couplings must be eliminated.
This commit completes all Phase 0 prerequisite decoupling tasks:

- P0.1: Introduce FsStorageType enum in fe-foundation (zero-dep module) to
  replace StorageBackend.StorageType (Thrift-generated) in PersistentFileSystem.
  Add FsStorageTypeAdapter for bidirectional Thrift conversion. Update all
  subclasses and callers (Repository, BackupJob, RestoreJob, CloudRestoreJob).

- P0.2: Add IOException-based default bridge methods to ObjStorage interface
  (checkObjectExists, getObjectChecked, putObjectChecked, deleteObjectChecked,
  deleteObjectsChecked, copyObjectChecked, listObjectsChecked). Add
  ObjStorageStatusAdapter for Status→IOException conversion. Zero changes to
  existing implementations.

- P0.3: Decouple SwitchingFileSystem from ExternalMetaCacheMgr via new
  FileSystemLookup functional interface. FileSystemProviderImpl passes a lambda.

- P0.4: Extract MultipartUploadCapable interface from ObjFileSystem, removing
  the forced abstract method. S3FileSystem and AzureFileSystem implement it.
  HMSTransaction now uses instanceof check instead of ObjFileSystem cast.

- P0.5: Introduce FileSystemDescriptor POJO for Repository metadata serialization,
  replacing direct PersistentFileSystem subclass serialization. Migrate GsonUtils
  to string-based Class.forName() reflection for legacy format backward compat,
  removing 7 compile-time imports of concrete filesystem classes.

- P0.6: Add FileSystemSpiProvider interface skeleton in fs/spi/ as the future
  ServiceLoader contract for Phase 3 module split.

### Release note

None

### Check List (For Author)

- Test: No need to test (pure refactor; all changes are backward compatible;
  three successful FE builds verified during development)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Maven build cache was caching checkstyle:check results and emitting
'Skipping plugin execution (cached)' even when sources had changed.
Two fixes:

1. Add check/checkstyle/ to global cache input so that changes to
   checkstyle rules (checkstyle.xml, suppressions.xml, etc.) correctly
   invalidate all module caches.

2. Mark the checkstyle:check execution (id: validate) as runAlways in
   executionControl so it is never skipped regardless of cache state.
   Checkstyle is a quality gate and must always execute.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ObjFileSystem: remove unused Map import (was used by the now-removed
  abstract completeMultipartUpload method)
- GsonUtils: fix CustomImportOrder violations - LogManager/Logger imports
  were inserted before com.google.* imports; move them after all com.* imports
  in correct lexicographical order

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Mar 30, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman morningman closed this Mar 30, 2026
@morningman morningman deleted the branch-fs-spi branch March 30, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants