[Idea/RFC] Shared-storage backend for cubecow (Ceph RBD) — cluster-durable sandbox disks

[lf4096]

We run long-lived agent sandboxes: each instance owns a persistent workspace, stays mostly resident on one node (paused when idle, resumed on demand), and is created/migrated infrequently. Two requirements keep surfacing that node-local storage can't meet:

1. Disk state must survive node failure — losing a machine must not lose the instance's accumulated environment and files.
2. An instance must be cold-startable on another machine afterwards, without data copying.

Sketch — what would change and what wouldn't:

• A second cubecow backend behind cow.backend.kind = "rbd" — the Engine trait explicitly anticipates sibling backends. Per-sandbox writable layers, committed template/snapshot rootfs and data volumes become RBD images (activate → rbd map).
• An opt-in control-plane mode for shared-storage deployments: committed templates become cluster-global, scheduling/routing no longer pinned to the data-holding node, capacity watermarks read pool fill.
• Per-sandbox block-device disks opened with O_DIRECT (the host page cache duplicates the guest's own and dies with the unmap); shared read-only base devices keep buffered I/O.
• Untouched: the default reflink path, the pmem base-image mechanism (base stays node-local and lazily pulled, preserving the DAX/page-cache sharing win), the hypervisor.

Node failure then degrades to: re-map the image on any healthy node and cold boot from the crash-consistent disk. This looks complementary to the reliability items already on the roadmap (sandbox failover, node drain with migration) — an opt-in deployment mode rather than a replacement for the local fast path.

Questions for the maintainers:

1. Is a second in-tree cubecow backend acceptable in principle, or would you prefer this shipped behind a storage-plugin interface?
2. How does a shared-storage mode sit with your plans for node-failure handling and sandbox failover — would you take it as an opt-in deployment mode?
3. RBD splits cubecow's single snapshot concept into read-only snapshots and writable clones. The existing trait works as-is (map every snapshot to a clone) at the cost of doubled clone-chain depth; an explicit writable flag on create_snapshot would enable native mapping (version points as real rbd snap, half the chain depth). Would you take such a trait change, or prefer the zero-change mapping?
4. Any known constraints on krbd exclusive-lock interactions (snap-create / resize against mapped images)?

If the direction sounds right, we'd like to help drive it.

[kinwin-ustc]

Is a second in-tree cubecow backend acceptable in principle, or would you prefer this shipped behind a storage-plugin interface?

I believe a second shared-storage-based cubecow backend needs to be implemented, but it would be best to abstract it into a generic interface that allows for easy implementation of different shared-storage backends.

How does a shared-storage mode sit with your plans for node-failure handling and sandbox failover — would you take it as an opt-in deployment mode?

Of course, this is part of our plan. We will utilize shared storage to implement volume semantics, enable cross-node pause and resume, and achieve cross-node exception recovery from the sandbox.

RBD splits cubecow's single snapshot concept into read-only snapshots and writable clones. The existing trait works as-is (map every snapshot to a clone) at the cost of doubled clone-chain depth; an explicit writable flag on create_snapshot would enable native mapping (version points as real rbd snap, half the chain depth). Would you take such a trait change, or prefer the zero-change mapping?

In our design, each disk clone can exist independently, without any snapshot chain. Ideally, clones based on shared storage should also operate in this manner.

Any known constraints on krbd exclusive-lock interactions (snap-create / resize against mapped images)?

I am not familiar with Ceph, perhaps we can discuss this issue in more specific scenario details

[lf4096]

Thanks for confirming the direction 🙏. Quick update: we've built a working PoC on a real Ceph cluster (two nodes) and validated the core pieces — volume create/map, cross-node cold migration, crash consistency, O_DIRECT, and exclusive-lock/fencing behavior. A few directions I'd like to align on:

1) Generic interface — Agreed. cubecow already has a backend-agnostic Engine trait (reflink and rbd as sibling implementations), which can serve as this seam directly. We'd lean toward landing RBD on Engine first and factoring the shared-storage-common parts into a dedicated layer once a second shared-storage backend appears (so we don't fit the abstraction to a single implementation) — but if you'd rather define the shared-storage sub-interface up front, we're happy to align on that with you. Which would you prefer?

2) Cross-node pause/resume and failover — Good to hear this is on your roadmap; it's exactly what we're hoping to build on. We've validated cross-node cold recovery at the disk layer, and cross-node resume looks like a natural extension (the memory artifact goes on shared storage too). Since how you approach cross-node resume and failover shapes what the storage layer needs to provide, we'd like to understand your current thinking there — it likely affects some of our design choices, and we're glad to share what we've validated so far.

3) Clone model — Following your "each clone independent, no chain" design, both mappings below give independent, chain-free clones; they differ only in how a committed template is stored, and we'd like your call:

• (a) uniform — every derived disk, including a committed template, becomes an independent clone (a copy). Simplest, and needs no trait change. It also keeps rbd behaviorally consistent with the existing reflink backend (independent, writable objects; no special read-only/zombie handling).
• (b) cheap template commits — a committed template stays as a near-free O(1) snapshot, and only actually-writable copies become independent clones. More space/time efficient, but needs the small writable flag we floated so the engine can tell the two apart.

Either works on our side — which fits your model better?

4) krbd exclusive-lock / fencing — We've worked through the Ceph-side behavior in the PoC, so we're happy to cover that part. One thing worth surfacing, since it touches your failover plans: dual-write safety isn't something the storage layer can fully guarantee on its own. RBD gives the tools — an exclusive-lock, and fencing a stale owner during failover — but -o exclusive can't be left on during normal operation: it breaks map-then-resize and snapshotting a running image, both of which cube does on live volumes. So from what we saw, the robust path looks like a control-plane lease plus fencing at failover — but that's really a control-plane call, so we'd like to talk it through with you: whether an approach like that fits your design, and how the fencing step would tie into your failover flow.

One more thing we'd like to understand: you mentioned volume semantics as part of the shared-storage plan — how do you see this cubecow shared-storage backend relating to the CubeVolume work in #470 (same layer, or complementary)? We'd like to make sure we build in the right place.

Is anyone already building a shared-storage backend for cubecow? We'd like to avoid duplicating effort. Once we've aligned on (1) the interface and (3) the clone model, how would you prefer the contribution to land — one PR, or split by component?

[kinwin-ustc]

1. Generic interface — Agreed. cubecow already has a backend-agnostic Engine trait (reflink and rbd as sibling implementations), which can serve as this seam directly. We'd lean toward landing RBD on Engine first and factoring the shared-storage-common parts into a dedicated layer once a second shared-storage backend appears (so we don't fit the abstraction to a single implementation) — but if you'd rather define the shared-storage sub-interface up front, we're happy to align on that with you. Which would you prefer?

I think we can implement it first, and then optimize the architecture here based on subsequent developments.

2. Cross-node pause/resume and failover — Good to hear this is on your roadmap; it's exactly what we're hoping to build on. We've validated cross-node cold recovery at the disk layer, and cross-node resume looks like a natural extension (the memory artifact goes on shared storage too). Since how you approach cross-node resume and failover shapes what the storage layer needs to provide, we'd like to understand your current thinking there — it likely affects some of our design choices, and we're glad to share what we've validated so far.

We are preparing to implement cross-node pause and resume based on S3 (object storage), just like E2B and AgentSubstrate do. Of course, I think it would be best if the backend shared storage implementation was pluggable, and could be based on various shared storage systems, such as NFS, S3, Ceph, JuiceFS, etc. The first step is to upload and download the corresponding snapshot from the shared storage to the local, and the best way is to implement lazy loading directly based on the remote shared storage.

3. Clone model — Following your "each clone independent, no chain" design, both mappings below give independent, chain-free clones; they differ only in how a committed template is stored, and we'd like your call:
   (a) uniform — every derived disk, including a committed template, becomes an independent clone (a copy). Simplest, and needs no trait change. It also keeps rbd behaviorally consistent with the existing reflink backend (independent, writable objects; no special read-only/zombie handling).
   (b) cheap template commits — a committed template stays as a near-free O(1) snapshot, and only actually-writable copies become independent clones. More space/time efficient, but needs the small writable flag we floated so the engine can tell the two apart.
   Either works on our side — which fits your model better?

I think the best approach is not to change the core logic or the semantics of the framework; it's better to implement it very independently. For the Cow implementation of shared storage, we can divide it into two categories: The first is shared storage that natively supports flat snapshots (completely independent data block indexes, shared actual data blocks, similar to XFS reflink and LVM thin clones); the second type is shared storage that natively does not support flat snapshots (simulated through full copy or chain merging).

krbd exclusive-lock / fencing — We've worked through the Ceph-side behavior in the PoC, so we're happy to cover that part. One thing worth surfacing, since it touches your failover plans: dual-write safety isn't something the storage layer can fully guarantee on its own. RBD gives the tools — an exclusive-lock, and fencing a stale owner during failover — but -o exclusive can't be left on during normal operation: it breaks map-then-resize and snapshotting a running image, both of which cube does on live volumes. So from what we saw, the robust path looks like a control-plane lease plus fencing at failover — but that's really a control-plane call, so we'd like to talk it through with you: whether an approach like that fits your design, and how the fencing step would tie into your failover flow.

I think a more appropriate approach would be to ensure disk exclusivity within the Cube's work flow, without relying on any assumptions about the storage layer.

how do you see this cubecow shared-storage backend relating to the CubeVolume work in #470 (same layer, or complementary)? We'd like to make sure we build in the right place.

I believe Cubecow focuses more on cross-node design for snapshots/rollbacks/cloning/pauses/resumes within the sandbox lifecycle, balancing storage and runtime efficiency. Volume, on the other hand, provides persistent storage decoupled from the sandbox lifecycle. While their focuses differ, they likely rely on the same shared storage. Could some standardization be achieved in the pluggable design (similar to CSI) of the shared storage backend?

Is anyone already building a shared-storage backend for cubecow?

As far as I know, not yet.

Once we've aligned on (1) the interface and (3) the clone model, how would you prefer the contribution to land — one PR, or split by component?

The best approach is one PR per feature, which can be fully compiled and tested, with a complete test report attached.

[lf4096]

@kinwin-ustc Thanks for the detailed answers earlier.

Following the "implement the backend first, refine the architecture later" direction, I've built the RBD backend and opened a draft PR: #990

What this does:

• Opt-in RBD backend: enabled with cow.backend.kind = "rbd"; the default stays reflink, untouched, with no effect when rbd isn't enabled.
• Snapshot: the uniform flattened clone we aligned on earlier (independent, no parent chain, no Engine trait change).
• O_DIRECT for rbd disks: the pmem base and reflink file-backed disks stay buffered.
• Verified on real Ceph (microceph squid + krbd): full engine lifecycle, sandbox cold start with the rbd volume opened O_DIRECT, and block-level cross-node cold migration with byte-identical data.

This PR is only the storage backend; cross-node orchestration (lease / fencing / shared-storage) and memory snapshot / warm start belong to the cross-node/failover roadmap and are left out here.

Two things I'd like to ask:

1. Is there a rough timeline for the cross-node interface (lease / fencing / shared-storage)? Once it lands, we'd be glad to build out the rbd/Ceph side on top of it — and to feed back from the storage side while it takes shape.
2. For writable-layer data (volume / rootfs) backup / cross-node transfer, is there any plan to support incremental (diff) down the line? Full transfer/storage gets expensive with large data, so I'd like to understand the direction here.

[kinwin-ustc]

Is there a rough timeline for the cross-node interface (lease / fencing / shared-storage)? Once it lands, we'd be glad to build out the rbd/Ceph side on top of it — and to feed back from the storage side while it takes shape.

We will support Volume capabilities(https://e2b.dev/docs/volumes) in the upcoming 0.6.0, providing a separate persistent storage abstraction decoupled from the sandbox lifecycle, and offering CSI-like custom backend plugin capabilities.

For writable-layer data (volume / rootfs) backup / cross-node transfer, is there any plan to support incremental (diff) down the line? Full transfer/storage gets expensive with large data, so I'd like to understand the direction here.

Pursuing the ultimate in cost and performance is our core objective, so to achieve these capabilities, we will definitely use the most reasonable method.

[lf4096]

Thanks, that answers both. We'll wait for the 0.6.0 Volume / plugin interface work, and refine this PR further to align with it once it lands.

[lf4096]

Following up on the incremental-diff point from earlier in this thread: I've open-sourced that piece as cowdiff — a reflink-aware incremental binary diff (Go library + CLI).

It works directly against cubecow's reflink layout, since volumes and snapshots are FICLONE siblings: FIEMAP finds the diverged extents, so a diff reads only what actually changed, and apply restores a byte-exact copy on any node. Two real-workload data points from an ubuntu:24.04 sandbox on cubecow (4 GiB volume): a fresh apt-get install of 11 common tools (~316 MB changed) exports in ~1.9s vs ~4.7s by full content comparison; afterwards, installing one more small package on the now ~300 MB-allocated volume (~1 MB changed) exports in ~0.3s vs ~6.8s (~22×). Diff cost tracks the size of the change, not the size of the volume — while a content scan pays for the whole volume every time.

Could be a ready-made building block for incremental snapshot export / cross-node distribution when the Volume work lands.