[RFC] CubeVolume — Persistent Storage for CubeSandbox

[ls-ggg]

[RFC] CubeVolume — Persistent Storage for CubeSandbox

Status (2026-07): Implemented on branch coolli/volume-plugin-framework. Below is the actual high-level approach (the earlier "SDK owns the backend" draft is folded at the end).

Docs: Volume Plugin Development · 中文 · COS examples

---

Implemented approach (high level)

Goal: e2b-compatible Volume API with pluggable storage backends. The platform owns lifecycle, identity, and refcount; plugins own backend create/destroy and host mount/unmount.

Layering

Layer	Role
SDK / HTTP	cubesandbox >= 0.6.0 (Volume, Sandbox.create(volume_mounts=...)) or raw REST
CubeAPI	e2b-shaped /volumes CRUD; sandbox create carries volumeMounts
CubeMaster	Persist t_cube_volume; run Controller hooks (Create / Destroy); cross-node refcount; reject delete while in use
Cubelet	Run Node hooks (Attach / Detach); local BoltDB refcount; bind-mount hostPath into the microVM via virtiofs
Volume Plugin	Backend-specific work (COS / NFS / ...); credentials stay in plugin-local config and are not passed through the API stack on Attach

Dual-role plugin model (CSI-like)

POST /volumes          -> CubeMaster -> Controller.Create
DELETE /volumes/:id    -> CubeMaster -> Controller.Destroy   (only if refcount == 0)

Sandbox create + mount -> Cubelet -> Node.Attach  -> hostPath -> virtiofs
Sandbox destroy        -> Cubelet -> Node.Detach

Plugins support binary (fork + CLI / stdout JSON) and rpc (gRPC). The same four hooks either way; driver is selected from config before the hook runs.

Mount path (sketch)

1. User: Volume.create("my-data"), then Sandbox.create(volume_mounts={"/workspace": vol})
2. CubeAPI -> CubeMaster: resolve the Volume record, inject mounts / plugin-volume-* annotations
3. Cubelet: Attach; plugin returns a hostPath under volume_plugin_base_dir, then bind into the sandbox
4. I/O inside the sandbox uses the normal filesystem (sb.files.*); there is no Volume content REST API in the platform today

Refcount (two layers)

• Per-node (Cubelet BoltDB): share one host mount across sandboxes on the same node (refCount == 0 means first Attach / last Detach on that node)
• Cross-node (MySQL t_cube_volume.refcount): number of nodes holding the Volume; DELETE /volumes returns 409 while non-zero

What changed vs. the earlier RFC

Earlier RFC	Actual
SDK talks to the backend directly; CubeAPI has no Volume endpoints	CubeAPI owns /volumes CRUD; plugins do the backend work
Built-in NFS/COS in Cubelet + stdin/stdout only	Plugin framework (binary + rpc); COS as the reference (examples/volume/cos/{binary,rpc})
Full VolumeInfo (including credentials) flows on sandbox create	Credentials stay in plugin-local config; create/get may return an optional token only
Direct file R/W on a Volume without a sandbox	Not implemented — mount into a sandbox and use the files API
Snapshot / Clone / Rollback	Not in this release

Minimal SDK usage

from cubesandbox import Sandbox, Volume

vol = Volume.create("my-data")  # omit driver -> first volume_plugins entry
with Sandbox.create(volume_mounts={"/workspace": vol}) as sb:
    sb.files.write("/workspace/hello.txt", "from volume")
Volume.destroy(vol.volume_id)

Requires platform >= 0.6.0 and cubesandbox >= 0.6.0. The official e2b Python SDK is not supported against CubeSandbox.

Docs & examples

• Architecture / protocol: docs/guide/volume-plugin.md
• COS walkthrough: examples/volume/cos/
• Branch: coolli/volume-plugin-framework

---

Original RFC draft (historical — superseded by the implementation above)

Goal

A storage-backend-agnostic Volume API. Different backends (COS/S3, NFS, local disk) each derive from a common abstract config and implement a shared interface.

Core Concept

A Volume outlives any individual sandbox. Data is retained after a sandbox is destroyed. The same Volume can be mounted into multiple sandboxes.

Original layering (superseded)

Layer	Responsibility
SDK / CubeVolume	Owns VolumeBackend; builds complete VolumeInfo
CubeAPI	Forwards VolumeInfo to CubeMaster on sandbox create only
CubeMaster	Forwards VolumeInfo to Cubelet on sandbox create only
Cubelet	Selects plugin by backend_type; mounts into sandbox

The implemented design moved Volume CRUD into CubeAPI/CubeMaster and split Controller/Node plugin hooks — see the summary above.

[jamesxia20181001]

· Does the E2B-compatible API shape make sense to you?
Yes. Meanwhile, ensure Cubelet process failover or upgrade does not disrupt volume access of running sandboxes. Extended APIs can be defined in the future for integration with external plugins.
· Any backend you'd like to see prioritized?
NFS, s3fs
· Any concerns about the hostdir reference implementation?
Can Volumes support Snapshot, Clone and Rollback, either performed independently or together with associated sandboxes?

[ls-ggg]

Can Volumes support Snapshot, Clone and Rollback, either performed independently or together with associated sandboxes?

Yes, Snapshot, Clone, and Rollback are supported. However, consistency of the backend filesystem state at the time of the operation is the responsibility of the user

[yanyuezhuo]

Hi maintainers, thanks for sharing this design.

I’m interested in helping with the Persistent Storage for CubeSandbox and would like to understand the current development status before starting work.

Could you clarify:

1. Has this design been accepted as the direction for implementation?
2. Is there already an active branch, PR, or owner working on this?
3. Which parts are still open for contribution?
   • SDK VolumeBackend abstraction
   • CubeAPI / CubeMaster volume_mounts forwarding
   • Cubelet VolumePlugin interface
   • built-in NFS/COS/local implementations
   • external plugin protocol
4. Are there any API or architecture decisions that still need discussion before coding?

If there is no active implementation yet, I’d be happy to start with a small scoped PR, for example defining the shared VolumeInfo / VolumeMount data structures and wiring the sandbox creation path first.

Please let me know the preferred starting point and any contribution guidelines I should follow.

[honkiko]

Current E2B template only has 1 container.
But multiple containers per sandbox is allowed, both for cubelet API (Cubelet/api/services/cubebox/v1/cubebox.proto) and cubemaster API (CubeMaster/pkg/service/httpservice/cube/cubeboxutil.go).

For better flexibility, it's better to allow multiple volume spec inside sandbox spec, and multiple volumeMount spec inside container spec. The VolumeInfo API described above doesn't fit into case of multiple containers.