[filesystem] Add JuiceFS filesystem support

[weizuo93]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

JuiceFS is a high-performance distributed file system designed for cloud-native scenarios, which can be connected to various object storage services at the underlying layer. Some enterprises adopt JuiceFS as a unified access layer for various object storage services. It ships with a Hadoop-compatible Java SDK, making it well-suited to serve as remote storage for snapshots of primary-key tables and tiered log segments of log tables within Fluss.

Solution

Add a new filesystem plugin module fluss-fs-juicefs under fluss-filesystems/, registered under the jfs:// URI scheme, bridging Fluss to the JuiceFS Hadoop SDK.

• Module: fluss-filesystems/fluss-fs-juicefs
• Scheme: jfs://
• Underlying SDK: io.juicefs:juicefs-hadoop
• Distribution: Not bundled in the default binary distribution; installed manually into ${FLUSS_HOME}/plugins/juicefs/, consistent with the COS / OBS / Azure plugins.

Design / Implementation Overview

The plugin is intentionally thin and reuses HadoopFileSystem from fluss-fs-hadoop, mirroring the structure of the existing OSS / COS plugins.

1. JuiceFsPlugin implements FileSystemPlugin

• getScheme() returns "jfs".
• create(URI, Configuration) builds a Hadoop Configuration from Fluss config, applies JuiceFS defaults, and instantiates the JuiceFS Hadoop FileSystem via reflection (FileSystem.newInstance) so that Fluss has no hard compile-time dependency on io.juicefs.*.
• Forwards any Fluss configuration entry whose key starts with fs.jfs. or juicefs. to the Hadoop Configuration.
• Injects sensible defaults when the user does not set them:
  • fs.jfs.impl = io.juicefs.JuiceFileSystem
  • fs.jfs.impl.disable.cache = false

2. JuiceFsFileSystem extends HadoopFileSystem

• Wraps the JuiceFS Hadoop FileSystem.
• Overrides obtainSecurityToken() to return an empty placeholder token, because JuiceFS handles authentication locally (see below).

3. SPI registration via META-INF/services/org.apache.fluss.fs.FileSystemPlugin.

4. Shading / packaging consistent with the other filesystem plugins (depends on fluss-fs-hadoop-shaded, fluss-fs-hadoop, and the juicefs-hadoop SDK).

5. Tests: A JuiceFsPluginTest covering scheme registration, prefix-based config forwarding (fs.jfs.*, juicefs.*, and unrelated keys), default-value injection (and non-override of user values), and null-config safety.

Configuration Example

Minimum required configuration in server.yaml:

# The dir that used to be as the remote storage of Fluss                                                                                                                                                                                                 
remote.data.dir: jfs://<your-volume-name>/path/to/remote/storage                                                                                                                                                                                         
                                                                                                                                                                                                                                                              
# JuiceFS metadata engine address of the pre-created volume, e.g. redis://<host>:<port>/<db>                                                                                                                                                                                                                        
juicefs.meta: <your-meta-url> 

Any other fs.jfs.* / juicefs.* keys documented in the JuiceFS Hadoop Java SDK docs can be added to server.yaml and will be transparently forwarded, e.g.:

juicefs.cache-dir: /data*/jfscache                                                                                                                                                                                                                       
juicefs.cache-size: 1024                                                                                                                                                                                                                                 
juicefs.access-log: /tmp/juicefs.access.log                                                                                                                                                                                                              

Authentication Model

Unlike the OSS / S3 / COS plugins, Fluss does not perform any STS or delegation-token exchange for JuiceFS:

• The JuiceFS client on each node authenticates directly against its metadata engine and backing object storage, using credentials embedded when the JuiceFS volume was formatted (or supplied via juicefs.access-key / juicefs.secret-key overrides where applicable).
• Deployment implication: every Fluss process that accesses remote storage (CoordinatorServer, TabletServer, and every client) must have (a) the plugin installed, (b) the same juicefs.* configuration, and (c) network reachability to the JuiceFS metadata engine and object storage.

References

• JuiceFS Hadoop Java SDK: https://juicefs.com/docs/community/hadoop_java_sdk
• JuiceFS GitHub: https://github.com/juicedata/juicefs

Willingness to contribute

• I'm willing to submit a PR!