Proposal: Fold `atelet` into `ateom`

[yuval-k]

Background

Today the actor lifecycle is split across two tightly coupled components:

• atelet: a node-level DaemonSet that the control plane talks to
• ateom: the worker component inside the worker pod that actually runs, checkpoints, and restores actors

To create a single worker, the control plane today coordinates two RPCs:
one to atelet and one to ateom-gvisor. The two processes also share
state through a host bind mount at /run/ateom-gvisor so they can hand
off snapshot files.

Problem

This split presents three structural issues:

1. Two-component coordination. Every worker-lifecycle operation is a
   distributed transaction across atelet and ateom. Failures and
   partial states have to be reconciled by callers, and upgrades have to
   keep the two binaries version-compatible. Debugging means reading two
   sets of logs and reasoning about the handoff between them.

2. Backend lock-in. The split assumes the gVisor model (a node agent
   plus an in-sandbox helper). Adding a different worker backend
   (Firecracker, for example) will be harder as we will need to build the support for it in 2 components.

3. Shared host /run mount is a blast radius. The atelet ↔ ateom
   handoff requires a host bind mount on /run/ateom-gvisor. A
   misbehaving sandbox that fills that directory can exhaust /run on
   the node and take down every other pod on it. With per-pod state
   (no host mount), one bad sandbox only takes itself down.

Proposal

Remove atelet and consolidate its responsibilities into ateom
(running per-worker-pod), exposing a single control-plane-facing
interface. Concretely:

• Worker lifecycle RPCs (create / start / suspend / restore / destroy)
  become a single call to the per-pod agent.
• The backend (gVisor today, others later) lives behind an interface
  inside ateom; new backends plug in there.
• Snapshot/restore state stays inside the worker pod's own filesystem —
  no host mount needed.
• In the future, we can potentially even standardtize the api that ateom exposes to allow out-of-tree ateoms.

Currently it is not possible due to how the ateom does networking, but once #110 is in, we can implement this proposal.

[ahmedtd]

Benjamin Elder (@BenTheElder)

I think we are trending towards keeping atelet for a few reasons:

• Even once ateom has the ability to run a normal gRPC over TCP server, it may be prohibitive for ate-api-server to maintain open gRPC connections to all ateoms. Sending the traffic through atelet would reduce the number of outbound gRPC connections by a factor of the number of pods per node (commonly 110 or 256).
• Similarly, for high-scale scheduling, we may need atelet to act as an aggregation point for statistics and which ateoms are free.

I don't think the current division of work between atelet and ateom is particularly gVisor specific. atelet downloads everything (including the virtualization binaries), and ateom executes them. Firecracker or cloud hypervisor will have to be structured very similarly, because they have the same problem of snapshots not being guaranteed to be portable forwards and backwards across hypervisor versions.

It's very possible that we may change the division of work between atelet and ateom. However, even in a world where ateom does the image pulls, some sort of host volume is going to be needed in order to share the image pull cache among different ateoms (although, different flavors of ateoms might use different host volumes).

[BenTheElder]

Agreed with all of #128 (comment)

Shared host /run mount is a blast radius. The atelet ↔ ateom
handoff requires a host bind mount on /run/ateom-gvisor. A
misbehaving sandbox that fills that directory can exhaust /run on
the node and take down every other pod on it. With per-pod state
(no host mount), one bad sandbox only takes itself down.

This is addressable without removing atelet, and we definitely expect atelet to evolve.

[thockin]

Even once ateom has the ability to run a normal gRPC over TCP server,

You say this as if it is solved - is it? Last I heard this was still a problem. As I understood it, please tell me if I am getting it wrong:

In order to "talk to" the ateom over IP, it needs to consume a socket to listen() on. That socket is then unavailable for the actor itself (gvisor or uVM or whatever) to listen on. Best case it would get an "in use" error. Worst case it would appear to succeed in the sandbox, but in-bound traffic would not actually reach the sandbox (stopping at ateom).

Is that right? If so, is there a reasonable solution? Atelet is the proverbial "another level of indirection" here.

I suppose it would be OK to claim a single port, but that sort of intrusion into the actor's space is something I would prefer to avoid.

Another possible option would be to have multiple IPs per pod (no standard k8s API for that).

Or we could intercept traffic and look at L7 (e.g. host headers) toi decide if ateom should eat the packet or forward it.

Or... ?

[ahmedtd]

I think we are much closer to a solution with the veth PR (#110). Right now, it just unconditionally forwards port 80 into the workload, but I think we should have a list of listening ports declared in the ActorTemplate. We will then be free to claim some likely-unused port (1, 2, etc) for ateom to run a server.

This also opens the door to running more than one workload per ateom --- it could work Borg-style, where each workload just says that it needs N ports, and ateom doles out free inbound ports to each workload.

Even if we retain the UDS for inbound ateom communication, so that ateom doesn't consume a listening port, having ateom able to make outbound connections without stepping on the workload is a significant quality-of-life improvement

[thockin]

IMO declaring ports is problematic - agents are very general, so they may not know what ports they use. OTOH we need to do wakeups on SOME port.

it could work Borg-style, where each workload just says that it needs N ports, and ateom doles out free inbound ports to each workload.

Oh please no. The ip-per-port model was born as a reaction to that mess.

Outbound is less of an issue because they are usually ephemeral ports, but they CAN be explicitly chosen by a client.

One of the things we can do (slow path) is feed requests back to kubernetes on what we need to make this stuff work better. Perhaps there needs to be a way to request multiple pod IPs.

[yuval-k]

• Even once ateom has the ability to run a normal gRPC over TCP server, it may be prohibitive for ate-api-server to maintain open gRPC connections to all ateoms. Sending the traffic through atelet would reduce the number of outbound gRPC connections by a factor of the number of pods per node (commonly 110 or 256).
• Similarly, for high-scale scheduling, we may need atelet to act as an aggregation point for statistics and which ateoms are free.

These reasons still doesn't mean the logic should be split. The daemonset can be a point of aggregation, and all the logic can be in the worker pod. I.e. the proposal above does not contradict these goals.

It's very possible that we may change the division of work between atelet and ateom. However, even in a world where ateom does the image pulls, some sort of host volume is going to be needed in order to share the image pull cache among different ateoms (although, different flavors of ateoms might use different host volumes).

What's the advantage of the atelet pulling the OCI images vs something like OCI volumes?
There's a lot that goes into image management - and it would be nice to re-leverage kubelet for this, so all the image lifecycle (pulling, pull-secrets, caching, GC) is handled for us.