Feature Request: Multi-Writer Support or Docker Volume Plugin for Distributed Containers (Docker Swarm)

[NobinKhan]

Hi,
First of all, thank you for the great project. I’m evaluating ZeroFS for use inside a Docker Swarm cluster to provide distributed storage for stateful containers.

🎯 What I am trying to achieve

I want to run a distributed application across 3 Docker Swarm nodes, where:

• The same zero bucket is mounted on all nodes
• Containers can read/write data regardless of which node they run on
• All nodes have access to the same consistent data

This is the classic “distributed shared volume” scenario similar to GlusterFS/BeeGFS/MooseFS.

🧪 What I tested

• A single ZeroFS instance connected to a bucket → works fine
• Mounting the same bucket from two different clients (read/write) → also works
• Running multiple ZeroFS instances connected to the same bucket → fails

After your reply clarifying that ZeroFS currently supports:
✔ one read/write instance
✔ multiple read-only instances
✘ no multi-read/write concurrency

… I now understand why my Swarm tests failed.

❗ Feature Request

Is there any plan to support either of the following?

1. Multi-Writer Support

Allow multiple ZeroFS instances to mount the same backend storage in read/write mode simultaneously.

This would unlock many distributed scenarios:

• Docker Swarm shared volumes
• High-availability stateful applications
• Edge cluster synchronization

2. Docker Volume Plugin

A native Docker Volume Plugin (or CSI driver) that manages ZeroFS mounts across nodes, similar to:

docker volume create -d zerofs -o bucket=mybucket sharedvol

This would automatically handle:

• Authentication
• Mounting
• Failover
• Node-level access

💡 Why this matters

ZeroFS could become a very appealing alternative to GlusterFS / MooseFS / BeeGFS for people who want:

• Simpler setup
• S3-like backend compatibility
• Strong consistency
• Distributed read/write workloads

Right now, the single-writer limitation is the only blocker that prevents using ZeroFS in a highly available container environment.

🙏 Request

• Is multi-writer support in your roadmap?
• Would you consider developing a Docker volume plugin / CSI driver?
• Are there technical limitations preventing this?

Any guidance would be greatly appreciated.
Thank you!

[Barre]

Allow multiple ZeroFS instances to mount the same backend storage in read/write mode simultaneously.

This is not planned as of now.

Right now, the single-writer limitation is the only blocker that prevents using ZeroFS in a highly available container environment.

You can already achieve that by deploying ZeroFS in a active/passive fashion with a proxy on top.

[matzeeable]

Hey! 🙂

Just found this repository via X and it looks really promising. My first thought was "cool, can this be used for HA applications" and directly came accross this thread. If I understand it correctly, we can mount multiple ZeroFS instances to multiple S3 backends, but not multiple ZeroFS instances to the same S3. We are also using Kubernetes and Docker Swarm and having multiple stateful applications mounting on a NFS, but it is a bit of overhead to configure and maintain.

@Barre can you please share some insights if and how we could cover this case?

You can already achieve that by deploying ZeroFS in a active/passive fashion with a proxy on top.

If I understand correctly, you mean something like a Failover mechanism on another dedicated VM/node?

[Barre]

Hey! 🙂

Just found this repository via X and it looks really promising. My first thought was "cool, can this be used for HA applications" and directly came accross this thread. If I understand it correctly, we can mount multiple ZeroFS instances to multiple S3 backends, but not multiple ZeroFS instances to the same S3. We are also using Kubernetes and Docker Swarm and having multiple stateful applications mounting on a NFS, but it is a bit of overhead to configure and maintain.

@Barre can you please share some insights if and how we could cover this case?

You can already achieve that by deploying ZeroFS in a active/passive fashion with a proxy on top.

If I understand correctly, you mean something like a Failover mechanism on another dedicated VM/node?

Hi,

I have something in the works that implements replication of in-memory state as well as automatic failover. Hang tight!

[matzeeable]

Hey @Barre!

Thanks for your fast reply.

I'm “unfortunately” very curious and like what you're building here, since it would solve one of the biggest problems (from our perspective) in distributed systems in a simple and elegant way. Do you have a diagram or Mermaid file I could look at so I can better visualize how ZeroFS would integrate into a Kubernetes/Docker Swarm service in the future?

[Barre]

Hey @Barre!

Thanks for your fast reply.

I'm “unfortunately” very curious and like what you're building here, since it would solve one of the biggest problems (from our perspective) in distributed systems in a simple and elegant way. Do you have a diagram or Mermaid file I could look at so I can better visualize how ZeroFS would integrate into a Kubernetes/Docker Swarm service in the future?

The idea is to not introduce a second coordination layer (that s3 already is with if-*-match) headers.

What I settled on (for now...) looks like this:

graph TB
    subgraph CL[Clients]
        M["FUSE mount<br/>(multi-target ninep-client)"]
        F["RPC FailoverClient"]
    end

    subgraph LD[Leader]
        S["9P + RPC servers<br/>lease-gated"]
        FS["Filesystem + DedupCache"]
        WC["Write coordinator<br/>(commit-then-apply)"]
        RS["Replicator sender<br/>+ heartbeat sender"]
        S --> FS --> WC --> RS
    end

    subgraph SB["Standby (watching, db not yet open)"]
        RR["ReplicationReceiver (gRPC)"]
        TBUF["TailBuffer + DedupCache"]
        HBW["Heartbeat watcher<br/>+ takeover timer"]
        RR --> TBUF
        RR --> HBW
    end

    DB[("Shared object store<br/>SlateDB data db<br/>writer-epoch = CAS fence")]

    M -->|adopt serving leader| S
    F -->|adopt serving leader| S
    RS -->|"(1) ship batch, await durable ack"| RR
    RS -->|heartbeats| HBW
    WC ==>|"(2) apply locally, then ack client"| DB
    FS -.->|"lease from DbStatus (Fenced = step down)"| DB
    WC -.->|"fsync = flush to store"| DB

Loading

Failover sequence:

sequenceDiagram
    autonumber
    participant C as Client
    participant A as Leader A
    participant B as Standby B
    participant S as Shared store

    Note over A,B: Steady state — Connected, semi-sync
    C->>A: write
    A->>B: ship batch (commit-then-apply)
    B->>B: buffer in TailBuffer
    B-->>A: durable ack
    A->>S: apply locally (not yet flushed)
    A-->>C: ack (write returns)
    A-)B: heartbeats (reset takeover timer)

    Note over A: Leader A crashes
    A--xB: heartbeats stop
    B->>B: gap exceeds takeover_ttl

    Note over B,S: Takeover
    B->>S: open db as writer (CAS bumps epoch, fences A)
    B->>S: replay TailBuffer + flush (no acked write lost)
    B->>B: acquire lease, serve 9P

    Note over C,B: Client reroute
    C-xA: next op times out (A dead)
    C->>C: re-probe + leader_check
    C->>B: retry (idempotent / stable op-id, deduped)
    B-->>C: ack (serves committed data incl. replayed tail)

Loading

[Barre]

@matzeeable This got released in the latest version: https://www.zerofs.net/docs/high-availability