[Proposal] Pluggable ateom backend: abstract the checkpoint/restore interface beyond gVisor

[yummypeng]

We are an OS infrastructure team building system-level optimizations for Agent workloads (high-density scheduling, process/filesystem checkpointing, structured CLI). We are evaluating Agent Substrate as a potential upstream dependency for actor lifecycle management and want to understand the project's stance on supporting alternative checkpoint/restore backends.

Current State

From reading the codebase, I see that:

1. The ateom proto (internal/proto/ateompb/ateom.proto) already hints at extensibility:

   // Ateom is the interface to control a single gVisor (or, in the future microVM)
   // guest inside a worker pod.
   

2. The component is named cmd/ateom-gvisor (not just cmd/ateom), suggesting the naming convention anticipates alternative implementations.

3. However, the proto messages include gVisor-specific fields (e.g., runsc_path in all three request types), and the WorkerPool CRD's ateomImage field is the only configuration point for selecting a backend.

4. The newly proposed Actor State Machine ([Feature] Implementation of Actor State Machine: Lifecycle and Transitions #119) introduces PAUSED vs SUSPENDED states with different durability/performance trade-offs — this is orthogonal to the backend question but has implications for it (e.g., a microVM backend might have different PAUSED semantics than gVisor).

Motivation: Why Alternative Backends Matter

Different deployment environments have different constraints and existing infrastructure:

1. CRIU + container runtime: For environments already running containerd/CRI-O, CRIU-based checkpoint/restore integrates with existing infrastructure without adding a new runtime dependency. The Linux kernel has supported CRIU for 10+ years, and containerd has native checkpoint APIs. This makes it a natural fit for teams that want actor lifecycle management without changing their container runtime stack.

2. Firecracker/microVM: For strong multi-tenancy isolation requirements where gVisor's syscall-interception model isn't sufficient, microVM snapshots (Firecracker's PUT /snapshot/create) provide VM-level checkpoint/restore.

Proposal

I'd like to discuss formalizing the ateom backend interface. Concretely:

Phase 1 — Proto cleanup (minimal change):

Generalize runsc_path to a runtime_config oneof (or make it optional and backend-specific), so alternative ateom implementations don't need to accept gVisor-specific fields:

message RuntimeConfig {
  oneof backend {
    GVisorConfig gvisor = 1;
    CRIUConfig criu = 2;
    MicroVMConfig microvm = 3;
  }
}

message GVisorConfig {
  string runsc_path = 1;
}

Phase 2 — ateom interface contract:

Document the behavioral contract that any ateom implementation must satisfy:

• RunWorkload: Start a new workload from OCI spec, report ready
• CheckpointWorkload: Capture full state (process + filesystem) to snapshot_uri_prefix, return to available state
• RestoreWorkload: Restore from snapshot_uri_prefix, report ready

This contract already exists implicitly; making it explicit enables third-party implementations.

Phase 3 — Reference alternative implementation (future):

We would be interested in contributing an ateom-criu implementation that uses containerd's checkpoint API.

Questions for Maintainers

1. Is supporting alternative ateom backends aligned with the project's near-term goals, or is gVisor intended to remain the sole implementation for the foreseeable future?
2. Would you accept a proto change that generalizes runsc_path, or do you prefer keeping the proto gVisor-specific and handling multi-backend at a different layer?
3. Is there any internal design doc or roadmap for the "microVM" path mentioned in the proto comment?

Related

• [Feature] Implementation of Actor State Machine: Lifecycle and Transitions #119 — Actor State Machine (the PAUSED/SUSPENDED distinction has implications for backend choice)

[EItanya]

I agree with this 100%, and I'd love to get to the bottom of where this extensibility layer lies. I just create #110 and #122 which try to generalize the networking within the existing ateom-gvisor component, but maybe the extensibility layer should actually be the proto itself. Can't wait to hear more from everyone.

[ahmedtd]

I would consider all of the APIs in flux right now. Some form of microvm-based actors is on our roadmap for alpha (end of June). I think it's currently undecided whether we will have one ateom that supports multiple runtimes, or one type of ateom for each type of runtime.

These runtime-specific things will ultimately need to be represented in the ActorTemplate, and plumbed all the way down (just like runsc_version). I am assuming that criu has some similar versioning constraints (though I guess maybe that's tied to the version of the host kernel?)

[BenTheElder]

I would consider all of the APIs in flux right now.

And we're clear on that in the README currently:

Agent Substrate is currently in VERY early development. It is not ready for production use, and the APIs are almost guaranteed to change. We are not making any guarantees about backward compatibility at this stage, and everything in this project may be changed.

https://github.com/agent-substrate/substrate#status-and-compatibility

See #123 for MicroVM

These runtime-specific things will ultimately need to be represented in the ActorTemplate, and plumbed all the way down (just like runsc_version).

And I'm a bit wary of making them extremely generic at this stage, versus fleshing out what we need for uVM + gVisor.

I am assuming that criu has some similar versioning constraints (though I guess maybe that's tied to the version of the host kernel?)

criu also may not be a very comparable alternative ...

We would be interested in contributing an ateom-criu implementation that uses containerd's checkpoint API.

AIUI, this allows you to snapshot/restore a particular container, but a core aspect of substrate is that the worker pods can be re-used in-place, otherwise we would suggest eg Agent Sandbox. IE the next actor may have a different container image, while using the same worker. Unless I'm missing something the containerd snapshot API doesn't support this.

This also doesn't offer any level of sandboxing, right? I think that's a bit of a mixed message versus the other backend(s).

Currently, we also make ~no requirements on the host beyond standard kubernetes. uVM will require KVM and nested virt but we will avoid depending on the host container runtime.

[yummypeng]

Thank you Benjamin Elder (@BenTheElder) for the clarification on worker pod reuse. This is a critical point I had overlooked.

You're absolutely right that containerd's checkpoint API doesn't support the in-place actor swapping model that Substrate uses. After revisiting this more carefully, I now understand the distinction:

• Substrate's model: the Worker Pod is a long-lived host, and ateom manages multiple independent sandboxes (each with its own rootfs from a different OCI image) within that pod. Actors come and go while the pod stays.
• CRIU + containerd checkpoint: captures an entire container's state (process tree + mount namespace + rootfs diff) as a unit. Restore requires the same rootfs layout, so you cannot swap in a different container image during restore.

So CRIU cannot serve as a drop-in replacement for gVisor in the "multiplex different actors on one worker pod" scenario. I hadn't fully appreciated that the pod reuse / actor swapping capability is structurally tied to having a nested isolation layer (gVisor sandbox or microVM) that provides independent rootfs per actor.

Regarding the sandboxing concern, also a fair point. CRIU alone provides no security isolation boundary, which would make it a fundamentally different trust model compared to gVisor or microVM backends.

I see that #123 has further discussion on MicroVM support and the pluggable ateom interface. I'll shift my attention to following the progress there.

Thanks again for taking the time to explain this. Learned a lot from this exchange.

[thockin]

Aside from the APIs being massively in flux (they are), we're still figuring out the overall architecture - just this week we have a new idea which might deliver even better performance, but we need to prove it. So the short answer is that YES we want it to be pluggable, and we need at least 2 implementations to keep us honest, but it's a LITTLE premature to pin down.

To your questions:

1. Is supporting alternative ateom backends aligned with the project's near-term goals, or is gVisor intended to remain the sole implementation for the foreseeable future?

We will, 100%, support some flavor of microVM. It is an open question exactly which one, and a longer term decision about whether we maintain in in this repo or in a separate repo.

2. Would you accept a proto change that generalizes runsc_path, or do you prefer keeping the proto gVisor-specific and handling multi-backend at a different layer?

All such implementation details will need to be removed. What I do not think we want is to enumerate all the different backends into the proto. I think we want generic proto + opaque extensions.

3. Is there any internal design doc or roadmap for the "microVM" path mentioned in the proto comment?

We're working on it. I know Taahir Ahmed (@ahmedtd) and Benjamin Elder (@BenTheElder) are discussing it. See #123.

[thockin]

The issues you cited around proto being gvisor-centric is real -- should we leave this issue open but re-title it to cover that specifically?

[BenTheElder]

The issues you cited around proto being gvisor-centric is real -- should we leave this issue open but re-title it to cover that specifically?

I think we might naturally do this as part of uVM, I personally consider "the proto is gvisor-centric" to be a bugfix as part of uVM, and the shape of the solution to be somewhat dictated by the uVM direction (Which we're actively working on).

SGTM to leave this open.

So the short answer is that YES we want it to be pluggable, and we need at least 2 implementations to keep us honest, but it's a LITTLE premature to pin down.

Yeah, we'll have a better idea what the pattern is when we have 2 implementations we're happy with. And we're currently fleshing out the direction for the second implementation.

[...] just this week we have a new idea which might deliver even better performance, but we need to prove it.

This little tangent will come first, but I expect we will have more concrete plans for both very rapidly.