[Feature] Implementation of Actor State Machine: Lifecycle and Transitions

[dberkov]

Issue Description

This issue tracks the formal implementation of the actor state machine within the substrate.

To achieve high density and efficiency, we must distinguish between different "resting" states for actors - specifically focusing on the trade-offs between resume speed, data durability, and resource costs. The implementation must specifically address a potential NIC saturation problem. A simplistic approach of uploading full memory and filesystem images over the network during every suspend/resume cycle would quickly overwhelm the host VM's network interface card (NIC). Consequently, the state machine implementation must prioritize keeping data on the local host to maintain required performance and density.

The "story" of an actor follows a journey from active execution through various stages of dormancy or failure, driven by both user actions and system timeouts.

Actor State Definitions

State	Description	Durability & Performance
RUNNING	The actor is currently active and executing on a worker.	Active CPU/RAM usage.
PAUSED	The actor is offloaded from the CPU, with snapshots stored locally on the node VM.	Fast resume; lower durability (data lost if node fails); cheaper (local, already provisioned storage, no NIC); we can optimize this (e.g., save to peers opportunistically (TODO add issue link)).
SUSPENDED	The actor is offloaded from the CPU, and snapshots are persisted to durable storage.	Higher SLO/durability (e.g., saved to blob storage); expensive (external storage and NIC); requires retention policies; slower resume due to data transfer; can be symbolically tagged.
CRASHED	A failure state reached if the host node fails while an actor is Running or Paused.	Requires manual recovery or revert to a known good state.

Snapshot layers

---
config:
  treemap:
    showValues: false
---
treemap-beta
"Actor's process"
    "memory" : 40
    "local storage"
        "rootfs"
            "homedir" : 20
            "oci bundle" : 80:::noborder

classDef noborder stroke-width:0px;

Loading

• memory - a snapshot of an actor's process memory. A snapshot cannot be used for restoration if the OCI image of the actor has been changed (e.g., upgraded).
• rootfs - an entire boot disk snapshot with all files. A snapshot cannot be used for restoration if the OCI image of the actor has been changed (e.g., upgraded).
• homedir - files under the home directory. This can be used for restoration even if the OCI image has been changed.

Actor Lifecycle Story

---
title: Actor State (`ate actor ...`)
---
stateDiagram-v2
direction LR

classDef crushedEvent fill:#f00,font-weight:bold,stroke-width:2px

[*] --> Suspended : create --from

Suspended --> Running : resume
Suspended --> Suspended : commit --tag | revert --to
Suspended --> [*] : delete

Running --> Running : commit --remain [--tag]
Running --> Paused : pause [--timer]
Running --> Suspended : commit [--tag] | revert --to
Running --> Crashed : [VM crashed | ateom crashed | etc...]

Paused --> Running : resume 
Paused --> Suspended : commit [--tag] | revert --to | (auto-commit-timeout)
Paused --> Paused : commit --remain [--tag]
Paused --> Crashed  : [VM crashed | ateom crashed | etc...]
Paused --> [*] : delete

Crashed --> Suspended : revert --to | dump | commit --tag
Crashed --> [*] : delete

class Crashed crushedEvent
class end crushedEvent

Loading

1. Create a new actor

• Action: ate actor create [--from]
• Mechanism: The system creates a new actor from either the default configuration or a tagged snapshot.
• Transition: NONE → SUSPENDED

2. Active Execution to Temporary Pause

When a user needs to temporarily free up vCPU resources but expects a quick return, the actor moves to a PAUSED state.

• Action: ate actor pause [--timer]
• Mechanism: The system takes a snapshot and keeps it on the local node. If an actor remains paused for an extended period and --timer is provided, the system escalates it to a SUSPENDED state to ensure data safety and free up local node storage.
• Transition: RUNNING → PAUSED

3. Active Execution to Long-term Suspension

When a user needs to free up vCPU resources and does not expect a quick return, or needs to ensure data safety.

• Action: ate actor commit [--tag]
• Mechanism: The system takes a snapshot and moves it to durable, persistent storage. A tag can be attached to the snapshot state, allowing the future creation of a new actor from this state (fork) or reverting the current actor to this state.
• Transition: RUNNING → SUSPENDED

4. Resuming Operations

Actors can be brought back to a RUNNING state from either dormancy stage.

• Action: ate actor resume
• From Paused: The scheduler prefers the original node to avoid data transfer.
• From Suspended: The system copies the latest snapshots to a node where a new available worker is running.
• Transition: PAUSED/SUSPENDED → RUNNING

5. Handling Failures

If the underlying node fails or ateom crashes while the actor's state is only stored locally (Running or Paused), the actor enters the CRASHED state.

• Recovery: Users can use ate actor revert --tag to abandon the lost local data and resume from the last known durable snapshot.
• Debugging: A user can run ate actor dump to transition an actor to the SUSPENDED state by dumping the current in-memory state while the node VM is still alive, allowing for future debugging.
• Transition: RUNNING/PAUSED → CRASHED

Actor’s Storage Snapshots configuration

snapshotConfig:
  onPause: (none | homedir | process)
  onCommit: (none | homedir | process)

The substrate manages three distinct but interrelated data types: homedir, rootfs, and process (memory+rootfs).

Specific system events necessitate the discarding of certain data layers - a process termed "devolving." For instance, updating an actor's OCI image invalidates both rootfs and memory, whereas a gVisor version update permits rootfs reuse while requiring a memory purge.

• None:
  • Use Case: Ideal for ephemeral, high-speed startup requirements.
  • Note: For components like MCP, this effectively disables suspension.
• Homedir only:
  • Use Case: Suitable for fast-starting agents capable of autonomous state recovery.
  • Persistence: This layer is designed to never devolve.
• Process + Homedir:
  • Mechanism: Represents a comprehensive full process snapshot.
  • Use Case: Critical for slow-starting actors with significant RAM-resident state (e.g., actors with slow bootstrap entry point and/or actors with state in RAM).
  • Devolution Path: Reverts to rootfs + homedir on hardware/gVisor changes, and homedir only on image upgrades.

Failure Modes

SUSPENDED State Details

• Resume: If the actor snapshot is missing, the system transitions it to the CRASHED state. Should the snapshot fail checksum validation or prove unusable, the actor moves to CRASHED.
• Tagging: Existing tags trigger an error during assignment unless the -f flag is employed.
• Revert Operations: Attempts to revert to a non-existent commit or tag result in a system error.

RUNNING State Details

• Asynchronous Events (Containers & Kernel): Internal container failures force a transition to CRASHED, though the system attempts to preserve the rootfs and home directory. A gVisor kernel failure results in a CRASHED state without any data preservation.
• Asynchronous Events (Workers): If the worker (ateom) crashes, the actor moves to CRASHED while saving the essential filesystem state, regardless of recovery speed. Permanent worker loss triggers the CRASHED state with metadata preservation. API-driven deletions put workers in a "deleting" phase until all actors are cleared, mirroring the autoscaler's scale-down logic.
• Asynchronous Events (Host Nodes): Host node failures lead to a CRASHED state; filesystem data is retained for future recovery. Prolonged node downtime necessitates a transition to CRASHED, saving the current rootfs. Should a node never return, a controller is required to finalize the move to CRASHED.
• Commit: Failures during snapshotting return an error, leaving the actor RUNNING. Persistent storage upload errors maintain the RUNNING state, though critical failures may result in a CRASHED status.
• Revert To: Errors are thrown if the target commit identifier or tag is missing.
• Pause: If snapshot creation fails, the actor remains in the RUNNING state and reports an error.

PAUSED State Details

• Asynchronous Events: Node crashes result in a transition to CRASHED; however, since the state was local, it remains preserved if the node returns. Extended node failures follow the same CRASHED logic with the pre-saved local state. Total node loss requires controller intervention to move the actor to the CRASHED state.
• Commit: If the underlying local data is missing, the actor enters the CRASHED state. Checksum failures on local data also trigger a transition to CRASHED. Persistent storage upload failures return an API error while maintaining the PAUSED status.
• Revert To: If the target tag is invalid, an API error is returned and the state remains PAUSED.
• Resume: Actor data missing from local storage results in a move to CRASHED. Corruption detected via checksum leads to a CRASHED state.

CRASHED Recovery Options

• Snapshot Dump: Execute ate actor dump <aid> <tag> to persist filesystem state (excluding memory) to a new tag; transitions to SUSPENDED at the prior commit.
• State Commitment: Use ate actor commit <aid> to push the current head state to SUSPENDED.
• Reversion: Running ate actor revert abandons the failed local data and returns the actor to a SUSPENDED state based on the previous stable commit.

[dims]

So, we are committed to the noun-first restructure of the CLI UX? Let's call that out explicitly please.

Some bits here seem to be gvisor specific, separable homedir/rootfs layers etc. true?

[dberkov]

Davanum Srinivas (@dims)

Some bits here seem to be gvisor specific, separable homedir/rootfs layers etc. true?

Could you please clarify what do you mean by gVisor specific. Both microVM & gVisor have memory, rootfs and homedir. The proposal for homedir snapshot, is as part of the resume, use the clean rootfs, override the homedir with content of snapshot and start bootstrap of the entry point.

[dims]

Dmitry Berkovich (@dberkov) a couple of statements made me pause to ask the question:

• whereas a gVisor version update permits rootfs
• Reverts to rootfs + homedir on hardware/gVisor changes

There wasn't a reference to microvm ( https://www.qemu.org/docs/master/system/i386/microvm.html ) either.

As long as gvisor works with one additional alternative as well, i feel comfortable. thanks!

[dims]

Dmitry Berkovich (@dberkov) looking deeper a couple of other things stood out

ate actor dump has two contradictory prose?

Which one is true? "the current in-memory state" or "filesystem state (excluding memory)."

• A user can run ate actor dump to transition an actor to the SUSPENDED state by dumping the current in-memory state
• Execute ate actor dump <aid> <tag> to persist filesystem state (excluding memory)

Is SUSPENDED → CRASHED is missing from the diagram?

the Failure Modes → SUSPENDED section explicitly routes failed resumes there.

revert always demotes to SUSPENDED

revert --to v1 from RUNNING throws away your live process and drops you to cold durable storage — so you then have to resume (another transfer) to use the state you reverted to?

commit means three unrelated operations

• RUNNING / PAUSED: snapshot live or local state → durable (real work, uses NIC).
• SUSPENDED commit --tag: the actor is already durable and not running, so there's nothing to snapshot — this is purely attach a tag (metadata).
• CRASHED commit: "push the current head to durable," which is nearly identical to revert --to .

[thockin]

This should be considered a proposal, not a set-in-stone design, but we need to drive fast on this one.

I marked this P0 since I think we need to do a minimal impl ASAP to prove that it works. It really rests on a few assumptions:

1. "local storage" exists and is meaningful (not often true on cloud VMs)
2. "local storage" is actually faster / less expensive than networked blob storage
3. We can do effective management of local snapshots (treat as a cache)

The state diagram shows a lot of edges not detailed here. That's OK, but as we get to implementation we should sort which edges are the most important and which can come later.

So, we are committed to the noun-first restructure of the CLI UX? Let's call that out explicitly please.

I think those are orthogonal, this doc needs to express the ideas SOMEHOW. IMO noun-first makes more sense for these, otherwise we end up with "ate commit" and "ate pause" and "ate resume" and "ate revert" and "ate dump" -- not terrible, but it pollutes the "top level" CLI namespace.

Some bits here seem to be gvisor specific, separable homedir/rootfs layers etc. true?

It should not be gvisor specific. I'm presuming any implementation of an ateom should be able to mount a directory/volume distinct from the rootfs -- is that not true for some impl?

As long as gvisor works with one additional alternative as well, i feel comfortable. thanks!

ACK - we need a microVM impl ASAP to keep us all honest.

ate actor dump has two contradictory prose?
Which one is true? "the current in-memory state" or "filesystem state (excluding memory)."

• A user can run ate actor dump to transition an actor to the SUSPENDED state by dumping the current in-memory state
• Execute ate actor dump to persist filesystem state (excluding memory)

I think dump should mean "as much as possible". If the machine has died and the state is lost, all we can do is revert back to some previous commit. If the process segfaulted, there's no useful memory snapshot, but there SHOULD BE rootfs and homedir.

If you buy the snapshotConfig part, here's how I think of it. none < homedir < process. The onCommit and onPause configs can be different, but onCommit is always <= onPause. So concretely: You can set onPause: process and onCommit: homedir, meaning we store the larger full-process snapshot AND homedir when pausing locally, but only the homedir when committing to permanent storage (which makes scheduling back to the same node very important). IN the context of dump it means "recover as much as possible, up to the onCommit config". If successful, that can then be used (I guess via revert, which is a little weird?).

revert always demotes to SUSPENDED

Yeah, this is a little unusual, but I think we should see if RUNNING --revert--> RUNNING really needed -- it seems like it will add complexity that will rarely be needed.

commit means three unrelated operations

• RUNNING / PAUSED: snapshot live or local state → durable (real work, uses NIC).
• SUSPENDED commit --tag: the actor is already durable and not running, so there's nothing to snapshot — this is purely attach a tag (metadata).
• CRASHED commit: "push the current head to durable," which is nearly identical to revert --to .

Logically we have two operations - "commit" and "tag" which sometimes are used together and sometimes not. commit [--tag] means you can think of the combination as an atomic operation. We could describe it as ate commit and ate tag (non-atomic) or we could describe ate tag and ate commit [--tag] and 2 ways of achieving the same result. I think there's value in having less verbs, but not if it is confusing. Note, we haven't describe this at the API layer yet. To me (and I am biased), commit --tag makes sense -- it means "commit any dirty state (including none) and add a tag". Calling commit on a SUSPENDED state should be a no-op (there's no dirty state, by definition). I am happy to be argued down -- I don't feel strongly.

The CRASHED -> SUSPENDED edge is genuinely weird. I am uncomfortable (for now) being too opinionated about error recovery, and I think we should leave it to the caller (for now). At this layer of a system, we don't know what's going on inside, so some folks will want to do quarantine and post-mortem, some will want to just try again.

We could do CRASHED --resume--> RUNNING, but it's not clear to me if that is useful. Heck we could make that automatic, but it's not obviously correct. Forcing CRASHED to go back to SUSPENDED seems like the baseline requirement, and then we can explore if we need richer handling.

But here's where it is tricky, IMO:

Think of a given actor as a time-series of N committed snapshots (where N is governed by some retention policy). Any commit can be tagged, and tagged commits are subject to different retention policies. At any time you can "fork" an actor into a new actor with create --from=<actor:tagname> or "revert" an actor to a previous state with revert (RUNNING->SUSPENDED) or revert --to=<tag>. There's an implicit "HEAD" or "latest" tag.

But again, CRASHED is weird.

I want to be able to save the CRASHED state, but MAYBE not keep it. If commit pushes the crashed state into the sequence as HEAD, retention policy might discard the previous state. We could either:
a) Force the caller to create a temporary tag for HEAD, commit the CRASHED state, inspect it, then decide if they should keep it or revert --to <tmp>, then finally delete the tmp tag
b) Force the caller to commit the CRASHED state to a new tag which DOES NOT replace HEAD, inspect it, then decide if they should keep it (revert --to <crashed>) or delete the tag and let the commit get GC'ed

The current diagram describes (b) but I could see some variation of (a). Input welcome.

[dims]

This should be considered a proposal, not a set-in-stone design, but we need to drive fast on this one.

Sounds good. Just pointing out things as we go along.

[thockin]

I hit the button too soon. Updated the comment - sorry for the book.

[thockin]

I don't like the generated diagram, here's my manual version (from https://docs.google.com/drawings/d/13ZFdJhmOfX1RaNInjMfs6xjzmPQlmY5BiBEPgV0lFJY/edit which is shared with ate-dev mailing list since I can't share to "world" on corp account :)