查看中文版

StoreFS - Distributed S3-Compatible Storage System

Index

• Overview
• Installation and Deployment
• Management Console
• User Roles and Groups
• S3 API
• Admin API
• s3file CLI
• Monitoring
• MCP for AI Agent
• Quick Start
• Technical Support
• License

Overview

StoreFS is a distributed S3-compatible storage system implemented in Go language, using gossip protocol for cluster membership management and communication. The system supports dynamic node management, data distribution, and fault tolerance functions, providing users with high-performance, scalable object storage services. This project uses Claude Code to automatically generate all codes and documentation.

Core Features

• S3-Compatible API: Compatible with AWS S3 API, supporting AWS CLI and other S3 tools
• Distributed Architecture: Node discovery and communication through gossip protocol
• Dynamic Scalability: Supports adding/removing nodes freely without downtime
• High-Performance Storage: Optimized storage engine supporting multiple storage media
• RDMA Acceleration: Get, put, and multipart upload all support RDMA for high-throughput object transfer
• Fault Tolerance: Data automatically recovers when nodes fail
• Load Balancing: Requests are automatically distributed to available nodes
• Web Management Console: Provides an intuitive web interface to manage users, policies, buckets, and objects
• Multi-Language Support: Management console supports Chinese and English
• Security: Supports object versioning (keeps historical versions and prevents accidental deletion), object locking (WORM model with Governance and Compliance modes), bucket-level AES-256-CTR encryption, and SSE-C (client-provided encryption keys) for data protection.
• Gzip Compression: PutObject and Multipart Upload support gzip-compressed request bodies via Content-Encoding header.
• s3file CLI: Interactive S3FS mode for browsing objects like a local file system, with Ctrl+C cancellation support.

Core Concepts

• User: System user with a unique identity. Each user has a role (user, group_admin, or super_admin) and usually belongs to a group. Users authenticate using Access Key (AK) and Secret Key (SK).

• Role: Determines the user's management scope and storage access behavior. Normal users manage their own resources, group administrators manage users and resources in their group, and super administrators manage global system configuration but do not directly access S3 object data.

• Group: Organizes users for delegated administration and shared policy defaults. A group can define a default policy that is applied when creating buckets without explicitly selecting a policy.

• Policy: Defines a user's access permissions to buckets and objects. Policies can precisely control user operations such as read, write, list bucket contents, delete objects, etc.

• Bucket: Container for storing objects. Each bucket has a unique name, and users can create, delete, and manage objects within buckets. Buckets can be configured with access policies to control which users can access them.

• Versioning: Bucket-level configuration that keeps historical versions of objects. When objects are overwritten or deleted, new versions or delete markers are created, allowing recovery to previous versions.

• Object Lock: Bucket-level WORM (Write Once, Read Many) configuration supporting two lock modes:

  • Governance Mode: Users with special permissions can overwrite or delete locked objects
  • Compliance Mode: No user can overwrite or delete locked objects until the retention period expires
  • Supports default retention policies automatically applied to newly uploaded objects

• Encryption: Bucket-level AES-256-CTR encryption (default: ON). Every object uploaded to an encrypted bucket is automatically encrypted with a unique AES-256 key generated per object. Encryption can be enabled or disabled at bucket creation or update time via the management console or admin API.

Cluster Architecture

A StoreFS cluster consists of multiple nodes that communicate through the gossip protocol:

• Dynamic Node Management: Supports adding/removing nodes freely without downtime
• Data Distribution: Object data is distributed to multiple nodes according to policies
• Fault Tolerance: Data automatically recovers when nodes fail
• Load Balancing: Requests are automatically distributed to available nodes

Installation and Deployment

1. Configuration File Details

StoreFS uses a YAML format configuration file (config.yaml). Here is a detailed explanation of the configuration items:

cluster:
  name: mycluster              # Cluster name, all nodes must use the same name
  db:                          # Database configuration (using StarRocks as metadata storage)
    host: "127.0.0.1"          # Database host address
    port: 9030                 # MySQL query port
    user: "root"               # Database username
    password: ""               # Database password
    database: "mydb"           # Database name
    timeout: 10s               # Connection timeout
  node:                        # Current node configuration
    name: node1                # Node name, must be unique
    num: 1                     # Node number, must be unique
    ip: 127.0.0.1              # Node IP address
    port: 7946                 # Reuse port. Admin REST API, admin web console, and node communication port (gossip protocol) all use this port
    internal_port: 17946       # Internal port for file operations between nodes
    disks:                     # Node disk configuration
      - path: /path/to/disk1   # Disk path
        weight: 1              # Disk weight for data distribution strategy
      - path: /path/to/disk2
        weight: 1
    s3:                        # S3 API configuration
      host: 127.0.0.1          # S3 API host address
      port: 8901               # S3 API port
  seeds:                       # Cluster seed node list (for node discovery)
    - 127.0.0.1:7946
    - 127.0.0.1:7947
    - 127.0.0.1:7948

2. Physical Machine/Cloud Virtual Machine Deployment

Step 1: Deploy Database

StoreFS uses StarRocks as metadata storage, so StarRocks needs to be deployed first:

# Download and start StarRocks (single-node deployment)
url: https://www.starrocks.io/download/community/index.html

tar -xzf StarRocks-<versiion>.tar.gz
cd StarRocks-<version>

# Start FE (Frontend)
./fe/bin/start_fe.sh --daemon

# Start BE (Backend)
./be/bin/start_be.sh --daemon

# Initialize metadata (using MySQL client to connect)
mysql -h db -P9030 -uroot < /init.sql

Step 2: Prepare Configuration File

Create a configuration file for each node (such as config1.yaml, config2.yaml, etc.), ensuring that node.name and node.num are unique for each node.

Step 3: Start StoreFS Nodes

# Download the StoreFS binary file for the corresponding platform
e.g., storefs_linux_x86_64

# Start node 1
./storefs_linux_x86_64 -config config1.yaml

# Start node 2 (in another terminal)
./storefs_linux_x86_64 -config config2.yaml

# Start node 3 (in another terminal)
./storefs_linux_x86_64 -config config3.yaml

3. Linux Binary Versions

Two Linux binary versions are available:

storefs_linux (Standard Version)

• Requires: No special dependencies! Works on any Linux system
• Features: Normal S3 functionality only
• Use when: You don't need RDMA or target system lacks libibverbs

storefs_linux_rdma (RDMA-Enabled Version)

• IMPORTANT: Will CRASH immediately if libibverbs is not installed on target system!
• Requires: libibverbs must be installed on target system
• Features: Full RDMA support for get, put, and multipart upload + normal S3 functionality
• Use when: You need high-performance RDMA data transfers

How to check for libibverbs:

ldconfig -p | grep libibverbs

If you see output, you have libibverbs and can use storefs_linux_rdma!

4. Docker Compose Deployment

StoreFS provides Docker Compose deployment for quickly starting a 3-node cluster:

# Prepare directories for Docker Volumes
./createDirs.sh

# Start Docker Compose
docker-compose up -d

Docker Compose will automatically start:

• 1 StarRocks database container
• 3 StoreFS node containers
• Port mapping: node1(7946/8901), node2(7947/8902), node3(7948/8903)

# Stop Docker Compose
docker-compose stop

# Clear Docker Compose containers
docker-compose down
rm -rf configs/db-init/

4. RDMA Support (Linux Only)

StoreFS supports RDMA (Remote Direct Memory Access) for high-performance data transfers, including get, put, and multipart upload. RDMA bypasses the operating system kernel and TCP/IP stack to achieve extremely low-latency and high-throughput data transfers.

Key features of RDMA support:

• GetObject, PutObject, and multipart UploadPart all support RDMA
• RDMA READ for put and multipart upload operations (server reads from client memory directly)
• RDMA WRITE for get operations (server writes to client memory directly)
• WebSocket control channel for RDMA connection setup
• Zero-copy data transfer
• Support for both hardware RDMA and Soft-RoCE (software emulation)

Note: RDMA support is Linux-only. It does not work on macOS, Windows, or other operating systems.

For detailed documentation about RDMA setup, configuration, and usage, please refer to:

• RDMA Documentation - Detailed RDMA documentation in English

Management Console

Management Console Introduction

StoreFS provides a Vue.js-based web management console located in the web directory. The console provides an intuitive user interface for managing users, policies, buckets, and objects.

Access Method

Visit http://localhost:7946/console with the default administrator account:

• Username: admin
• Password: admin123
• Role: super_admin (super admin)

Features

Function Module	Description	Screenshot
User Management	Create/edit/delete users, manage access keys	Login, UserList, GroupList
Policy Management	Create/edit/delete policies, configure permission rules	PolicyList
Bucket Management	Create/edit/delete buckets, configure access policies, toggle encryption on/off	BucketList
		CreateVersionBucket
Object Management	Upload/download/delete objects, preview file contents	ObjectList, ObjectInfo
Object Versioning Management	Upload/download/delete objects, preview file contents	ObjectVersionList, VersionList
Multipart Management	Complete/abort	MultipartList, MultipartInfo, MultipartFragmentInfo
Node Management	View node status, add/remove nodes	NodeList
Internationalization	Switch languages	Internationalization

User Roles and Groups

Overview

StoreFS uses role-based access control together with user groups. A user belongs to a group and receives one of three roles: user, group_admin, or super_admin. Groups make it possible to delegate administration by team or tenant, and each group can configure a default policy for buckets created by users in that group.

Role Permissions

Role	Management Console / Admin API Permissions	S3 API and Object Data Permissions	Group Scope
user	Manage own buckets, objects, profile, and access keys	Can access buckets owned by the user according to bucket policy	Own account and own buckets
group_admin	Manage users, buckets, objects, and group settings within the same group; can assign user and group_admin roles in the group	Can access buckets owned by users in the same group according to bucket policy	Current group only
super_admin	Manage all groups, users, policies, buckets, nodes, and global system resources; can create or assign super_admin users	Cannot directly access S3 API buckets or read/write object data; should delegate data operations to regular or group admin users	Global system scope

Note: Bucket policies still control the detailed S3 actions, such as read, write, list, and delete. Roles define the management boundary and the bucket ownership scope that a user can operate on.

S3 API

Overview

StoreFS implements the core functions of the S3 API, compatible with AWS S3 clients and tools. You can use AWS CLI, S3 SDK, or other tools that support the S3 protocol to interact with StoreFS.

Implemented API Interfaces

For detailed API interface documentation, please refer to: S3 API Documentation

Main implemented API interfaces include:

• Bucket Operations: Create bucket, list buckets, delete bucket
• Object Operations: Upload object, download object, delete object, list objects
• Multipart Operations: Create multipart upload, upload part, complete multipart upload, abort multipart upload, list parts, list multipart uploads
• Versioning Operations: Get bucket versioning status, set bucket versioning status
• Object Lock Operations: Get bucket object lock configuration, get object retention configuration

Public URI Reading

StoreFS supports public read access for objects in buckets marked as public. When a bucket is set to public, objects can be accessed directly via HTTP GET requests without authentication.

Enabling Public Access

A bucket can be marked as public during creation or updated later through:

• Admin API: Set isPublic field to true in bucket create/update requests
• Management Console: Toggle the "Public" switch in bucket settings

Public URI Formats

Objects in public buckets can be accessed using either path-style or virtual-hosted-style URIs:

Path-style:

http://<s3-host>:<s3-port>/<bucket-name>/<object-key>

Example: http://127.0.0.1:8901/my-bucket/documents/report.pdf

Virtual-hosted-style:

http://<bucket-name>.<s3-host>:<s3-port>/<object-key>

Example: http://my-bucket.127.0.0.1:8901/documents/report.pdf

How It Works

When StoreFS receives a GET request for an object:

1. If the bucket is public, it serves the object without requiring authentication
2. If the bucket is not public, it falls back to normal authentication checks

Public access only applies to GET requests for objects. All other operations (upload, delete, list, etc.) still require proper authentication and authorization.

Admin API

Overview

StoreFS provides a set of RESTful Admin APIs for managing the system's users, policies, buckets, and nodes. These APIs are mainly used for web management consoles and automated operations.

Implemented API Interfaces

For detailed API interface documentation, please refer to: Admin API Documentation

Main implemented API interfaces include:

• Authentication: Login, logout, change password
• User Management: Create/delete users, modify user information, manage access keys
• Policy Management: Create/delete policies, modify policy content
• Bucket Management: Create/delete buckets, modify bucket attributes, list bucket contents
• Object Management: Manage objects in buckets, get object metadata
• Node Management: View node status, add/delete nodes

s3file CLI

Overview

s3file is a command-line tool for interacting with S3-compatible storage services, supporting both interactive and silent modes. It works with StoreFS, MinIO, AWS S3, and all S3-compatible services.

Features

• Interactive Shell Mode: Navigate S3 storage like a local file system
• Silent Mode: Execute commands programmatically
• Multi-Provider Support: Works with StoreFS, MinIO, AWS S3, and all S3-compatible services
• Pagination Support: Browse large directories with ease
• Command History: Navigate through previous commands
• Auto-completion: Tab completion for commands
• Wildcard Support: Use * and ? for fuzzy matching

Documentation

For detailed documentation, please refer to: s3file CLI Documentation

Monitoring

StoreFS provides a comprehensive monitoring and alerting system based on the Prometheus + Grafana + Alertmanager stack.

Key Features

• Metrics Collection: Each StoreFS node exposes a /metrics endpoint with system-level metrics (CPU, memory, disk, network), operation counters (object upload/download, multipart, fragment), and Go runtime metrics.
• Hot Bucket Detection: Real-time top-K hot bucket tracking using a sliding window algorithm (2-minute window), covering uploads, downloads, upload parts, and multipart completes.
• Pre-built Grafana Dashboards: Two pre-configured dashboards are included — a single-node detailed view and a cluster-wide summary view.
• Alert Rules: Pre-defined Prometheus alert rules for node down, disk usage, CPU/memory thresholds, and goroutine count.
• Notification Channels: Alertmanager configuration with templates for Slack, Email, and Webhook notifications (all disabled by default — easy to enable).

Documentation

For detailed information, please refer to: Monitoring Guide

MCP for AI Agent

StoreFS provides an MCP (Model Context Protocol) server that enables AI assistants — primarily Claude Code — to manage the cluster through natural language. Instead of remembering API endpoints and request formats, you can simply describe what you want to do.

Key Features

• Natural Language Management: Manage users, groups, buckets, policies, and objects by chatting with the AI
• 40+ Tools: Six groups of tools covering cluster management, user administration, storage policies, bucket operations, object management, and S3 data operations
• Automatic Language Detection: Responses auto-switch between English and Chinese based on your input
• Secure Authentication: Bearer token for admin operations, AWS Signature V4 for S3 data operations
• File Operations: Upload, download, and copy files directly through natural language commands

Prerequisites

• Node.js >= 18
• StoreFS v0.3.7 or above

Documentation

For detailed information, please refer to: MCP Server Guide

Quick Start

1. Connect Using s3file CLI

# Start interactive mode (connects to localhost:8901 by default)
s3file

# List all buckets
s3file --silent --command 'buckets'

# Create bucket and upload file using silent mode
s3file --silent --command 'mb mybucket' --command 'cd s3://mybucket' --command 'upload localfile.txt remote.txt'

# Download file using silent mode
s3file --silent --command 'cd s3://mybucket' --command 'download remote.txt localfile.txt'

For more detailed usage, please refer to: s3file CLI Documentation

2. Connect Using AWS CLI

# Configure AWS CLI
aws configure --profile storefs
AWS Access Key ID [None]: <your-ak>
AWS Secret Access Key [None]: <your-sk>
Default region name [None]: us-east-1
Default output format [None]: json

# List all buckets
aws s3 ls --endpoint-url http://127.0.0.1:8901 --profile storefs

# Create bucket
aws s3 mb s3://mybucket --endpoint-url http://127.0.0.1:8901 --profile storefs

# Upload file
aws s3 cp localfile.txt s3://mybucket/ --endpoint-url http://127.0.0.1:8901 --profile storefs

# Download file
aws s3 cp s3://mybucket/localfile.txt . --endpoint-url http://127.0.0.1:8901 --profile storefs

3. Use Management Console

1. Visit http://localhost:7946/console
2. Log in with the default administrator account (username: admin, password: admin123, role: super_admin)
3. Create users, policies, and buckets
4. Manage your storage resources

Technical Support

If you encounter problems while using StoreFS, please refer to:

1. FAQ Documentation - Frequently Asked Questions
2. Troubleshooting - Common Problem Troubleshooting
3. GitHub Issues - Submit Issue Reports

License

You can use and distribute this software freely, but you need to retain the original author's copyright notice and license information.