Support sandbox platforms running on Substrate

[ronlv10]

Background

One of the potential consumers today with large scale are evaluation and RL frameworks like Harbor and NeMo Gym. These run agents inside execution sandbox environments, and already integrate with several of these platforms: Harbor ships backends for roughly twenty-five providers, e2b and OpenSandbox among them, each implementing a single interface of start, stop, exec, and file upload and download.

Over the last couple of days I looked at what it would take to run a Harbor or NeMo Gym style platform on Substrate. This issue writes up what those platforms need from a compute backend, and the gaps I hit.

What these platforms are

A sandbox platform (like e2b, OpenShell, OpenSandbox) gives an SDK caller a fresh enforced environment: filesystem restrictions, controlled egress, credentials injection. Over that it exposes an API: run a command, stream output, download and upload files, open a shell.

These platforms let users bring their own images, and inject the sandbox binary at execution time without requiring the user's image to be rebuilt. This aligns with Harbor-style systems, which need to run many image types (100+), one per benchmark environment.

A few examples I looked on:

Platform	Isolation	Source
Vercel Sandbox	Firecracker microVM	Closed
e2b	Firecracker microVM	Apache 2.0
NVIDIA OpenShell	OCI container	Apache 2.0
OpenSandbox (Alibaba)	OCI container	Apache 2.0

I would focus on the two container-based fully open-sourced platforms, OpenShell and OpenSandbox.

There are certainly more than these, and this list is not meant to be exhaustive. Happy to hear from the community about others worth considering.

Both already support Kubernetes as a pluggable compute backend. Adding Substrate alongside it is the goal this issue is framing.

How they work on Kubernetes

Both follow the same shape: the user brings an arbitrary OCI image, and the platform delivers its own binary into that image at pod-creation time.

OpenShell

A single supervisor binary runs as the container's first process and spawns the agent as a child. Everything the platform enforces lives in that supervisor process: policy decisions, a loopback proxy for egress, credential injection, etc. Because the supervisor owns the child, it applies filesystem and syscall restrictions before the agent ever starts.

Supervisor binary delivery: on modern k8s clusters the supervisor's own OCI image is mounted into the pod as a read-only image volume; on older ones an init container copies the binary into a shared volume. Either way the pod's container command is overridden to point at the delivered binary.

The supervisor container runs as root and is granted capabilities like NET_ADMIN and NET_RAW for network setup, and CHOWN to prepare paths for the sandbox user. The supervisor drops privileges for the child.

The supervisor holds an outbound connection to the control plane and receives exec, file, and policy operations over it.

OpenSandbox

A separate daemon, execd, implements the API surface directly: command execution with streaming output, interactive shell sessions over an ssh-like channel, and file operations. It is delivered by an init container that copies it out of a dedicated image into a shared volume, and the container command is overridden to a bootstrap script that starts execd, then launches the user's workload as a child.

Egress is handled by a second container. A sidecar carrying the network policy joins the pod, holds NET_ADMIN, and installs the rules. The workload container has that same capability explicitly dropped. Since all containers in a pod share one network namespace, the workload's traffic necessarily traverses the sidecar's rules, and the workload has no privilege to inspect or remove them.

The pattern

Reduced to essentials, both need two things from their compute backend:

1. Deliver a binary into an arbitrary user image and run it as the container process

2. Give enforcement components privileges the workload does not have

The API for exec, upload and download is not a separate requirement. The injected binary implements it. This is aligned with the comment on #185, which separates exec-into-actor for debugging from the API an execution environment offers, and suggests the latter need not live in Substrate itself.

Gaps identified

1. No way to deliver a binary into an actor

ActorTemplate containers take an image, command, args, env, and volume mounts. Volumes exist, but every source starts empty. Nothing carries content in. The only way to get a binary into an actor today is to have it in the image, which means asking users to rebuild theirs.

2. No per-container capabilities

Every container in an actor receives the same fixed capability set. OpenSandbox's egress model depends on the asymmetry between a privileged sidecar and an unprivileged workload, and OpenShell's supervisor needs elevated privileges to install its controls.

Options to close each gap

Delivering the binary

• An image volume: a volume source naming an OCI image, mounted read-only into the actor. Something similar to the Kubernetes image volume model.

• An init container: a phase that runs before the actor's containers, so a user can copy content into a shared volume.

Capabilities

• Per-container securityContext with capability add on ActorTemplate.

Related work in Substrate

• #185 exec channel into a running actor

• #264 an API to push files into an actor

• #126 transparent egress policy and credential injection

Looking for feedback on the gaps and the options above. Happy to open a separate issue per option we agree on.

[ronlv10]

I know that Julian Gutierrez Oschmann (@juli4n) worked on how to build "E2B-like" code execution sandbox solution in substrate
and also Tim Hockin (@thockin) and rogeroger (@rogeroger-yu) had discussion around this in #185

happy to hear your thoughts!

[BenTheElder]

An image volume: a volume source naming an OCI image, mounted read-only into the actor. Something similar to the Kubernetes image volume model.

That seems ~fine. It will add a bit of complexity but we can add another RO mount. The behavior is pretty predictable, and there are probably other use-cases.

An init container: a phase that runs before the actor's containers, so a user can copy content into a shared volume.

We'd also need something like an emptyDir scratch dir between the init container and the working containers

This potentially adds a lot of lifecycle complexity / confusion we should be careful to communicate clearly about (e.g. init probably shouldn't run again on resume but you might assume or want that).

Per-container securityContext with capability add on ActorTemplate.

I do think we need this eventually, but I think we should be careful about how we expand this API, communicate defaults, etc.

There's also some awkward things where these have evolved a little in the ecosystem over time ...

[BenTheElder]

This smells like two feature requests packed into one issue, support for something e2b like is sort of it's own thing that may or may not require these specific features depending on how we implement it. There are other ways you could solve these problems (e.g. a separate component could patch the binary into an additional image layer as an image pull proxy).

[ronlv10]

That seems ~fine. It will add a bit of complexity but we can add another RO mount. The behavior is pretty predictable, and there are probably other use-cases.

An init container: a phase that runs before the actor's containers, so a user can copy content into a shared volume.

We'd also need something like an emptyDir scratch dir between the init container and the working containers

This potentially adds a lot of lifecycle complexity / confusion we should be careful to communicate clearly about (e.g. init probably shouldn't run again on resume but you might assume or want that).

Just to make sure I was clear - I think one of them should be sufficient for now. I thought that the init container would be a more powerful tool that users could leverage for more use cases, but it sounds like you think an image volume makes more sense for now?

This smells like two feature requests packed into one issue, support for something e2b like is sort of it's own thing that may or may not require these specific features depending on how we implement it. There are other ways you could solve these problems (e.g. a separate component could patch the binary into an additional image layer as an image pull proxy).

I agree. I wanted to get feedback on:

1. Whether we agree these platforms are good potential consumers of Substrate, and that we should invest in enabling this use case.
2. Which of the options above are valid solutions, so we can open an issue per feature.

[BenTheElder]

My take:

1. Whether we agree these platforms are good potential consumers of Substrate, and that we should invest in enabling this use case.

I do. But I can't exactly speak for the project on that overall.

2. Which of the options above are valid solutions, so we can open an issue per feature.

Personally I'd rather add read-only image volumes for binary injection than add slow and complex startup ordering (init containers) with confusing semantics. I think that's going to be easier to build performantly and with clear behavior.

[zmberg]

Thanks Ron Lev (@ronlv10) for the thorough write-up. This resonates strongly with what we're building.

I'm from the Alibaba Cloud Container Service team. We develop and maintain OpenKruise Agents — a CNCF incubation-stage project under the OpenKruise family. It is a native Kubernetes sandbox orchestration system that manages the full lifecycle of sandbox instances (create, checkpoint-based pause/resume, in-place upgrade, reclaim) on top of OCI containers, with first-class support for Virtual Kubelet and ECI integration.

• Current scale & usage: OpenKruise Agents currently powers production workloads for multiple RL training and serving customers. A single cluster supports hundreds of thousands of concurrent sandbox instances (Sandbox CRD, agents.kruise.io/v1alpha1). Core capabilities include sandbox warm pools, checkpoint-based hibernation and restoration, and in-place image updates.
• E2B protocol compatibility: We are compatible with the E2B protocol, so upstream applications can switch seamlessly. In essence, OpenKruise Agents is a native Kubernetes agents sandbox solution, similar in category to the OpenSandbox mentioned in the issue.

Interest in Substrate: We are internally exploring Substrate as an additional backend beyond containers. Specifically, we plan to integrate directly with Substrate's interfaces at the sandbox-manager (E2B service layer) level, rather than replacing the underlying runtime. The two gaps you identified resonate strongly with us:

• Gap 1 (Binary injection): We currently use init containers to inject helper binaries into users' OCI images, providing process- and filesystem-related capabilities. If Substrate could support init containers or image volume mechanisms, our injection layer could migrate smoothly.
• Gap 2 (Sidecar container support): We need sidecar container capabilities. Currently we have csi-sidecar handling on-demand volume mounts with Fuse (OSS/NAS storage), and traffic-proxy handling L4/L7 traffic control (via iptables rule modifications). These sidecar containers require basic isolation from the workload container, each holding different privileges. If Substrate supports sidecar containers and isolation mechanisms, our traffic control and storage mount model can migrate smoothly.

The RL scenario vs. actorTemplate model dilemma. In our RL scenarios, customers have hundreds of thousands of images, each containing different SWE-bench examples. These sandboxes have two main requirements:

• High-throughput concurrency and fast startup — the ability to spin up large numbers of sandbox instances in a short time.
• Filesystem checkpoint + fork — During evaluation, agents explore different paths via tree search. At specific checkpoints, a filesystem state snapshot is taken, and multiple sandbox instances are forked from that snapshot to explore different branches. This avoids re-initializing the environment from the image for each branch, significantly boosting throughput. Notably, this scenario doesn't necessarily require full process-level memory snapshots — filesystem-level snapshot and fork is the key.

Currently, Substrate requires creating an actorTemplate per image, which means users would need to create hundreds of thousands of actorTemplates — not a friendly experience. Moreover, for RL scenarios, what customers truly need is fast startup speed + filesystem fork capability, not full VM-level checkpoint/restore. We haven't yet figured out how to reconcile this massive-image + high-throughput + filesystem-fork pattern with Substrate's current actorTemplate model. We'd love to hear the community's thoughts on this.

We'd love to have more communication with the community to facilitate better integration and iteration. We're also very willing to contribute our experience and code during the actual implementation process.