Add basic per-actor external volume flow by msau42 · Pull Request #405 · agent-substrate/substrate

Michelle Au (msau42) · 2026-07-08T01:01:24Z

Initial part of #232.

Extends the ActorTemplate API to add support for per-actor external volumes and adds control plane and atelet hooks following the Actor lifecycle.

Callouts to volume operations are abstracted with a Volume interface. Right now, only a mock volume plugin for testing has been implemented, but we will add CSI support next.

Tests pass
Appropriate changes to documentation are included in the PR - Not going to update documentation until we add CSI support.

Michelle Au (msau42) · 2026-07-08T15:46:21Z

+	Capacity resource.Quantity `json:"capacity" protobuf:"bytes,1,opt,name=capacity"`
+	// storageClassName refers to the StorageClass to create the volume from.
+	// +required
+	StorageClassName string `json:"storageClassName" protobuf:"bytes,2,opt,name=storageClassName"`


open for discussion: can we reference k8s objects or should we create our own api?

I don't think there's a cleaner way. In theory we could have an ateapi level api which just acts as an indirect, but I don't see a reason at this moment.

Michelle Au (msau42) · 2026-07-08T15:48:09Z

  map<string, string> match_labels = 1;
 }

+message ExternalVolume {


discuss: should ExternalVolume be its own standalone object rather than embedded fields into the Actor?

Right now ExternalVolume is tied to an Actor but we are also thinking about the scenario where you want to share a volume across multiple actors.

I vote 👍

We all know how tricky volumes can be in k8s, with RWO, RWX, RO, mounts on the same node being virtually RWX but not ACTUALLY RWX, etc. If we make this a first-class resource, do we need to track the same sorts of things (e.g. NFS can be RWX, blockdev can't)?

Yes, we will need to add the notion of access modes and enforce that shared volumes can only be RWX or ROX.

should ExternalVolume be its own standalone object rather than embedded fields into the Actor?

It can be standalone message, as we have other standalone messages.
About standalone object? Do you mean customer will do "kubectl ate create external_volume"? We dont have requirements to have it today. So lets open it for discussion when it will be required.

Yeah I mean standalone object. It's primarily for the shared volume use case when you want to import an existing volume that already exists.

Michelle Au (msau42) · 2026-07-08T15:56:14Z

 	}
 	err = s.persistence.CreateActor(ctx, actor)
 	if err != nil {
+		// Cleanup created volumes if DB write fails


Discuss: In general, how should we approach errors when an operation only partially succeeds? Should we rollback all the partially completed operations? What happens if the rollback fails?

Or we leave the system in the partial state and require users to cleanup by calling Delete? Even then, if deletion fails and Create is called again, we could still have partially leftover resources from a previous invocation.

My current thoughts:

If there is a pair of "Create" and "Delete" operations, then we can leave the partial state and clean it up on the Delete. This does imply we will have to add some "Creating" and possibly "Deleting" state to Actor before we start creating resources for it. I already have marked that as a todo.

For operations like "Resume", there isn't a corresponding delete operation if the resume failed. It's possible resume could be called on a different node. That means we could have partial state left on the first node. In those cases we can try to rollback the state as much as possible but that would still be a best effort operation. This has a few implications:

We probably need to have some process on the node that monitors oprhaned resources and tries to clean them up and/or alert

We need to make sure these orphaned resources don't get reused if a new actor with the same name gets created. This potentially implies that we should have some sort of actor uuid and be using that in our actor directories.

The current implementation does not handle error in deleteActorVolume.

Might be we need introduce "FAILED" state. So the delete volume will be cleaned in this state.

We dont have this problem today, since during create we just create record in suspended state.

Yeah I am thinking of 2 main changes to CreateActor:

Add a new "CREATING" state and persist that to DB before starting to create volumes.

Then if CreateVolume fails, we can introduce a FAILED state.

User has to call DeleteActor in order to trigger deletion of volumes and Actor DB entry

Tim Hockin (thockin)

I didn't get too far into the reading of this yet, but wanted to send a couple questions before EOD

Tim Hockin (thockin) · 2026-07-09T00:19:39Z

+	Capacity resource.Quantity `json:"capacity" protobuf:"bytes,1,opt,name=capacity"`
+	// storageClassName refers to the StorageClass to create the volume from.
+	// +required
+	StorageClassName string `json:"storageClassName" protobuf:"bytes,2,opt,name=storageClassName"`


I don't think there's a cleaner way. In theory we could have an ateapi level api which just acts as an indirect, but I don't see a reason at this moment.

Tim Hockin (thockin) · 2026-07-09T00:27:07Z

+
+// VolumePlugin abstracts storage operations.
+type VolumePlugin interface {
+	CreateVolume(ctx context.Context, name string, capacity string, storageClass string) (volumeID string, err error)


Do you see this as "kubernetes PVC is the plugin", or "CSI is the plugin", or "NFS is the plugin" ?

What I really want to understand is what credentials are granted to the API server to create and destroy volumes, and what additional layers are in between. E.g. if I need to create a million k8s PVCs and PVs, that's trouble, but if I need to give ateapi binaries the ability to delete EBS volumes in the same context as k8s, that's also trouble.

My thought is that substrate will directly call CSI grpc APIs (so ateapi needs to be authorized to communicate with CSI drivers). We will not be creating PVC objects, although we will need to have a substrate API reference to the volume IDs (which I have added in this PR).

Michelle Au (msau42) · 2026-07-09T01:09:45Z

 			Type:        "bind",
-			Source:      ateompath.DurableDirVolumeMountPoint(atespace, actorID, vm.GetName()),
+			Source:      srcPath,
+			Options:     []string{"bind", "rw"},


double check this. durabledir didn't set any options.

Tim Hockin (thockin)

I read this. It seems broadly correct. I am still worried about permissions, but not enough to block this.

I swear I had another comment, and it was important, but I forgot what it was. Oh well, Friday. :)

Tim Hockin (thockin) · 2026-07-10T15:30:46Z

+	// for each actor. The volume only lives as long as the actor and is deleted
+	// when the actor is deleted.
+	// +optional
+	ExternalVolumeTemplate *ExternalVolumeTemplate `json:"externalVolumeTemplate,omitempty" protobuf:"bytes,3,opt,name=externalVolumeTemplate"`


Do we need the word "Template" in here? All of these are templates

Open to naming, I am trying to distinguish between volumes that will trigger dynamic provisioning, and volumes that won't. So for shared volumes, for instance, I am thinking we could have something like:

volumes: externalVolumeTemplate: - name: my-own-volume storageClassName: class1 externalVolume: - name: shared-volume volumeName: someExistingVolume (in the same atespace) # This volume is already created and populated with data through other means

Eitan Yarmush (EItanya) · 2026-07-12T19:58:51Z

+	// for each actor. The volume only lives as long as the actor and is deleted
+	// when the actor is deleted.
+	// +optional
+	ExternalVolumeTemplate *ExternalVolumeTemplate `json:"externalVolumeTemplate,omitempty" protobuf:"bytes,3,opt,name=externalVolumeTemplate"`


What exactly does External mean here, External to substrate? If we are just wrapping a StorageClass anyway should we name it PersistentVolumeSource to be closer to the source material?

It's not REALLY a PV in k8s, it's just reusing k8s storage classes

Open to naming :)

But agree with Tim that this doesn't have quite the same semantics as PVs in k8s (and even the "persistent" part in k8s is misleading sometimes). At least with the per-actor volumes, the volume is definitely not persistent in the traditional sense (it only lives as long as the actor).

By "External", I'm trying to distinguish with the sandbox-backed filesystems, like rootfs and durabledir. The volumes in this case are backed by storage systems external to substrate (ie nfs, remote block, pfs, distributed fs, etc).

Eitan Yarmush (EItanya) · 2026-07-12T19:59:53Z

  map<string, string> match_labels = 1;
 }

+message ExternalVolume {


I vote 👍

Benjamin Elder (BenTheElder)

[AI First Pass Experiment]

Benjamin Elder (BenTheElder) · 2026-07-22T22:58:52Z

+	return mounted
+}
+
+func actorVolumeID(ref *ateapipb.ObjectRef, volumeName string) string {


🟡 🤖 The hyphen join is ambiguous: atespace, actor name, and volume name may all contain hyphens, so distinct actors can produce the same ID (actor a in atespace x-y vs actor y-a in atespace x). This string is the idempotency name passed to CreateVolume, and CSI dedupes by name — colliding actors would silently share one storage volume. The existing TODO about actor UUID would fix it; an unambiguous encoding of the three parts would too.

I plan to address this as a folowup (I already have a todo) as it will require adding a "CREATING" status to Actor and persisting that before calling CreateVolume so that the Actor will have a UID by the time we create the volumes.

Benjamin Elder (BenTheElder) · 2026-07-22T22:58:52Z

+	}
+
+	// Delete associated volumes (TODO: best effort?)
+	s.deleteActorVolumes(ctx, req.GetActor(), actor.GetActorVolumes())


🟡 🤖 Two issues with volume deletion here. Volumes are never detached first — for an actor deleted while running or paused, CSI DeleteVolume on an attached volume fails. And since failures are only logged before the record is deleted below, the volume leaks with no record pointing at it. Consider detaching first (reuse detachActorVolumes) and deleting the record only after volume deletion succeeds, or persisting a deletion intent for background retry.

This is an interesting question. While the state machine here (#119) does allow delete from SUSPENDED, PAUSED and CRASHED states, I believe for pause/crash, there's probably still some cleanup that has to happen from atelet, and that workflow is currently missing today. Dmitry Berkovich (@dberkov) do we plan to add this workflow?

Agree with Benjamin Elder (@BenTheElder) , something is broken here. From moment we started deletion -> we cannot just ignore the error or keep it in suspended state.
I think we need split DELETE in 2 phases:

move to DELETING first

do actual delete all sub-resources before deleting the record from redis.

I am making a change to make volume deletion a hard failure for Actor Deletion so that we do not delete the Actor record until all volumes are successfully deleted.

However my question is about transitioning from PAUSE or CRASH directly to DELETE. Right now this just directly tries to Delete without telling the workers. I think we are missing some workflow to atelet to cleanup the nodes.

Dmitry Berkovich (dberkov) · 2026-07-23T02:02:29Z

+	ExternalVolumeTemplate *ExternalVolumeTemplate `json:"externalVolumeTemplate,omitempty"`
+}
+
+type ExternalVolumeTemplate struct {


nit:have you considered to call it "ExternalVolumeSource" rather then template?

As mentioned in #405 (comment), I am trying to think about how to add the "import existing volume" scenario for the shared volume use case in the future. So in the API I want to distinguish between "this will be a new volume created per actor" vs "we're accessing an existing volume that will be shared".

Dmitry Berkovich (dberkov) · 2026-07-23T02:14:24Z

 	}
 	err = s.persistence.CreateActor(ctx, actor)
 	if err != nil {
+		// Cleanup created volumes if DB write fails


The current implementation does not handle error in deleteActorVolume.

Might be we need introduce "FAILED" state. So the delete volume will be cleaned in this state.

We dont have this problem today, since during create we just create record in suspended state.

Dmitry Berkovich (dberkov) · 2026-07-23T02:16:41Z

+	}
+
+	// Delete associated volumes (TODO: best effort?)
+	s.deleteActorVolumes(ctx, req.GetActor(), actor.GetActorVolumes())


Agree with Benjamin Elder (@BenTheElder) , something is broken here. From moment we started deletion -> we cannot just ignore the error or keep it in suspended state.
I think we need split DELETE in 2 phases:

move to DELETING first

do actual delete all sub-resources before deleting the record from redis.

Dmitry Berkovich (dberkov) · 2026-07-23T02:33:31Z

+
+// TODO: Replace with actual volume plugin search
+func getVolumePlugin() volume.VolumePluginControlPlane {
+	return globalVolumePlugin


this is what actually was I was missing when I reviewed pkg/api/v1alpha1/actortemplate_types.go.

How the storage plugins will be registered in the substrate instance?

The CSI integration will include plugin discovery. We're still POC-ing CSI support right now but the current thought is to add a new CRD to register the CSI endpoints to use for substrate.

Dmitry Berkovich (dberkov) · 2026-07-23T02:37:53Z

+	ref := &ateapipb.ObjectRef{Atespace: state.Actor.GetMetadata().GetAtespace(), Name: state.Actor.GetMetadata().GetName()}
+	for _, vol := range getMountedActorVolumes(ctx, ref, state.Actor.GetActorVolumes(), state.ActorTemplate) {
+		slog.InfoContext(ctx, "Attaching volume to node", slog.String("volume_id", vol.GetStorageVolumeId()), slog.String("node", node))
+		err := getVolumePlugin().AttachVolume(ctx, vol.GetStorageVolumeId(), node)


what does attachment supposed to do? Attach volume to node VM? Is this code supposed to be executed on the atelet?

This depends on the actual volume driver implementation but it's essentially "anything you need to do to make the volume accessible on that node". For block storage, that would be attaching the device to the node. For file storage, that could be configuring network policies to allow the node access to the volume. This is run from the CP (ateapi)

The disadvantage to run this code in control plane, it makes resume operation slower. By running this code in atelet, this code can be executed in parallel with code that downloads images from GCS or download OCI image from artifcat registry.

The main issue is security. In the case of block storage, we don't want to give nodes permissions to attach any volume, and in the case of filers, we don't want to give the node permissions to set ACLs for itself.

In the future, we can explore optimizations to make this a non-blocking operation.

Dmitry Berkovich (dberkov) · 2026-07-23T02:43:53Z

+type VolumePluginControlPlane interface {
+	CreateVolume(ctx context.Context, name string, capacity string, storageClass string) (volumeID string, err error)
+	DeleteVolume(ctx context.Context, volumeID string) error
+	AttachVolume(ctx context.Context, volumeID string, node string) error


what information about node do you need for attach? Is it just node name? Do you need more information, like IP, zone, etc?

The CSI spec call only takes node id as input. The driver is responsible for interpreting that and getting any additional metadata about that node that they need.

Dmitry Berkovich (dberkov) · 2026-07-23T02:46:41Z

+}
+
+func (s *AttachVolumesStep) Execute(ctx context.Context, input *ResumeInput, state *ResumeState) error {
+	if state.Actor.GetAteomPodNamespace() == "" {


could you please take a look to

substrate/cmd/ateapi/internal/controlapi/crash.go

Line 50 in 9e2a669

func crashActor(ctx context.Context, st store.Interface, atespace, actorName string) error {

. Zoe recently added CRASH state.

I took a look, right now it just updates DB Actor state. I don't see anything to add here wrt volumes at the moment. Like I mentioned in #405 (comment), I think we are missing some workflow to cleanup the worker/node when a PAUSED or CRASHED actor gets deleted.

Dmitry Berkovich (dberkov) · 2026-07-23T02:53:02Z

+  - name: external-data
+    externalVolumeTemplate:
+      capacity: 1Gi
+      storageClassName: standard


out of curiosity what exactly it will do today, from what storage class data will be exeucted?

It's not used right now, but once we add CSI driver support, we will extract the driverName and the opaque driver-specific properties to pass to volume creation (things like tier, performance parameters, etc)

This adds a new externalVolumeTemplate where users specify the capacity, storage driver and storage properties they would like via StorageClass. When an actor is created, we will also provision a volume for the actor according to the template. The volume is deleted when the actor is deleted.

Adds external volumes to the Actor object and integrates control plane Actor workflows with the corresponding volume operations: CreateActor -> CreateVolume DeleteActor -> DeleteVolume ResumeActor -> AttachVolume Pause/Suspend Actor -> DetachVolume

Volume operations are integrated into the actor workflows: - Run/Restore: MountVolume - Checkpoint: UnmountVolume Volumes are mounted under the actor directory and passed as bind mounts to OCI.

This allows for different storage backends to be supported. Right now it only contains the mock volume plugin, which is only intended for testing purposes right now.

It makes the file counter path configurable so we can test it against different volume types. And adds an optional file verification step to test the presence of a file on an existing volume.

Michelle Au (msau42) · 2026-07-23T20:03:54Z

Here's the items I would like to followup in separate PRs to limit the size of this one, and to help unblock the CSI integration work;

naming of ExternalVolume API also taking into account potential future enhancement to support imported shares volumes (Add basic per-actor external volume flow #405 (comment), Add basic per-actor external volume flow #405 (comment))
Handling of partial volume creation failure during CreateActor (Add basic per-actor external volume flow #405 (comment))
Using actor uid in the volume name (depends on the above) (Add basic per-actor external volume flow #405 (comment))
Volume cleanup coordination/handling between node and CP (Add basic per-actor external volume flow #405 (comment))

Michelle Au (msau42) commented Jul 8, 2026

View reviewed changes

Michelle Au (msau42) force-pushed the mock-mount-dev branch from 3a4a327 to ccc7a2d Compare July 8, 2026 17:05

Michelle Au (msau42) changed the title ~~Add basic external volume flow~~ Add basic per-actor external volume flow Jul 8, 2026

Michelle Au (msau42) force-pushed the mock-mount-dev branch 2 times, most recently from c2286ea to 96fa815 Compare July 8, 2026 22:50

Haven Xia (HavenXia) reviewed Jul 9, 2026

View reviewed changes

Comment thread cmd/ateapi/internal/controlapi/volumes.go Outdated

Tim Hockin (thockin) reviewed Jul 9, 2026

View reviewed changes

Michelle Au (msau42) commented Jul 9, 2026

View reviewed changes

Zheng Lammerts (zlammerts-svg) added this to Substrate Milestones (go/substrate-milestones) Jul 9, 2026

github-project-automation Bot moved this to Todo in Substrate Milestones (go/substrate-milestones) Jul 9, 2026

Zheng Lammerts (zlammerts-svg) removed this from Substrate Milestones (go/substrate-milestones) Jul 9, 2026

Michelle Au (msau42) force-pushed the mock-mount-dev branch 4 times, most recently from 4890a34 to ffc96b8 Compare July 10, 2026 18:02

Julian Gutierrez Oschmann (juli4n) reviewed Jul 10, 2026

View reviewed changes

Comment thread pkg/api/v1alpha1/actortemplate_types.go Outdated

Tim Hockin (thockin) reviewed Jul 11, 2026

View reviewed changes

Eitan Yarmush (EItanya) reviewed Jul 12, 2026

View reviewed changes

Michelle Au (msau42) force-pushed the mock-mount-dev branch 3 times, most recently from 33fbae5 to ec684cf Compare July 22, 2026 21:46

Dmitry Berkovich (dberkov) self-requested a review July 22, 2026 22:13

Michelle Au (msau42) force-pushed the mock-mount-dev branch from ec684cf to 90ad7f1 Compare July 22, 2026 22:46

Benjamin Elder (BenTheElder) reviewed Jul 22, 2026

View reviewed changes

Dmitry Berkovich (dberkov) reviewed Jul 23, 2026

View reviewed changes

Comment thread pkg/api/v1alpha1/actortemplate_types.go Outdated

Dmitry Berkovich (dberkov) reviewed Jul 23, 2026

View reviewed changes

Michelle Au (msau42) force-pushed the mock-mount-dev branch 3 times, most recently from e627884 to 18a4ca4 Compare July 23, 2026 05:07

Michelle Au (msau42) force-pushed the mock-mount-dev branch from 18a4ca4 to 3afc69b Compare July 23, 2026 17:04

Michelle Au (msau42) added 8 commits July 23, 2026 19:47

ateapi handling of external volumes.

49471cf

Adds external volumes to the Actor object and integrates control plane Actor workflows with the corresponding volume operations: CreateActor -> CreateVolume DeleteActor -> DeleteVolume ResumeActor -> AttachVolume Pause/Suspend Actor -> DetachVolume

atelet handling of external volumes.

883ac3c

Volume operations are integrated into the actor workflows: - Run/Restore: MountVolume - Checkpoint: UnmountVolume Volumes are mounted under the actor directory and passed as bind mounts to OCI.

Allow ateom to see host mounts

f53df1d

Volume plugin interface and mock volume implementation.

698da05

This allows for different storage backends to be supported. Right now it only contains the mock volume plugin, which is only intended for testing purposes right now.

Update counter demo with external volume using mock volume plugin.

f0ddb9e

It makes the file counter path configurable so we can test it against different volume types. And adds an optional file verification step to test the presence of a file on an existing volume.

Make delete volume failure a hard failure in DeleteActor

9a619ba

Make external volumes optioanl to counter demo

7a527fb

Michelle Au (msau42) force-pushed the mock-mount-dev branch from 3afc69b to 7a527fb Compare July 23, 2026 19:53

Uh oh!

Conversation

Michelle Au (msau42) commented Jul 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michelle Au (msau42) Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Tim Hockin (thockin) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Tim Hockin (thockin) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michelle Au (msau42) Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michelle Au (msau42) Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michelle Au (msau42) Jul 8, 2026 •

edited

Loading

Michelle Au (msau42) Jul 21, 2026 •

edited

Loading

Michelle Au (msau42) Jul 23, 2026 •

edited

Loading

Michelle Au (msau42) Jul 23, 2026 •

edited

Loading

Michelle Au (msau42) Jul 24, 2026 •

edited

Loading