RFC: External execution plane — K8s as coordination plane only

[AlexsJones]

I've been looking into whether Kubernetes is the right long-term substrate for agent orchestration, prompted by Modal's Scaling to 1 million concurrent sandboxes in seconds. Before we commit to a direction I'd like the community's thoughts, including on a fairly radical option at the end.

The thesis

Split the coordination plane from the execution plane. Kubernetes stays as the coordination plane: CRDs, RBAC, admission, policy, GitOps, observability. The execution plane (where agent sandboxes actually run) becomes pluggable, and nothing strongly consistent sits on the per-run critical path. Networking learned this lesson years ago with the control plane/data plane split; this is the same idea applied to orchestration.

Why (the short version of the evidence)

• Measured control-plane ceiling. The Dirigent paper (SOSP '24) measured Knative-on-K8s sustaining ~2 sandbox cold starts/second before API-server saturation, versus ~300/s average (8,000/s bursts) in production serverless traces. Notably, even bare Pod create/delete (which is our one-Job-per-AgentRun regime) degrades under concurrent churn. The root cause is architectural: every creation is tens of synchronous read-modify-write RPCs through the API server to etcd.
• The SLOs don't cover us. K8s's 5s p99 pod-startup SLO explicitly excludes image pulls, init containers, and autoscaling; the API-latency SLO excludes CRDs entirely. AgentRun has no upstream latency guarantee at all.
• The alternative is demonstrated. Modal creates 1M concurrent sandboxes in under a minute with sub-500ms median starts, using no durable store on the creation path, gVisor memory snapshots (~2.5× faster cold starts), and lazy FUSE-mounted images.
• But this isn't "abandon K8s." KubeDirect (2026) shows a modified K8s with an API-server bypass gets close to Dirigent-class performance. The problem is etcd on the hot path, not K8s as a policy and coordination surface.

What this could look like for sympozium

Today the controller is the mechanism: it creates a Job per AgentRun, polls status on a 5s requeue, and scrapes the result from pod logs. The proposal is to invert that. AgentRun becomes a desired-state record, dispatch and status flow over a fast path (we already run NATS for the ipc-bridge), and the controller writes status back lazily.

flowchart LR
  subgraph K8s coordination plane
    CR[AgentRun CR - spec is intent] --> C[controller - policy and validation]
    C -->|lazy batched status| CR
  end
  C -->|publish work item| Q[(NATS JetStream - work queue and status stream)]
  Q -->|claim - pull model| NA[node-agent - any cloud or bare metal]
  NA -->|status, result, heartbeat| Q
  Q -->|consume| C
  NA --> S1[Firecracker / gVisor / runc sandbox]

Loading

Node agents dial out and pull work, GitHub-runner style. There are no inbound networking assumptions, so the execution plane is cloud-agnostic by construction: EC2, bare metal, homelab. The hard parts are the usual distributed-systems list: idempotent dispatch and fencing, orphan reconciliation in both directions (leases/heartbeats), and status write-back discipline. The hardest part, since we treat agents as adversarial, is that policy enforcement moves into the node agent. NetworkPolicy and SecurityContext don't exist off-cluster, so the agent has to enforce isolation and attest to it.

The radical option I'm weighing

We already have a seam: the controller branches between a Job and an agent-sandbox Sandbox CR. The incremental path is promoting that branch to an executor interface, with pods as one backend among several.

The stronger version: agent workloads are never reconciled by our controller at all. Pods stop being the default execution primitive. Every run executes on the execution plane (in-cluster runners or a remote fleet), and Kubernetes holds only intent, policy, and the durable record. The controller's job shrinks from managing pods to enforcing policy and keeping the record.

Questions for you

1. Does anyone actually hit the churn regime? The measured degradation starts at ~25–100 concurrent creations. What burst sizes do your deployments see? If real-world agent churn is low, this is premature optimization. Telemetry welcome.
2. Incremental (executor interface, pods remain default) or radical (execution-plane-native, pods demoted or dropped): which would you want to operate?
3. If you run agent platforms on other substrates (Firecracker, E2B, Fly machines, Cloudflare containers, Kata/gVisor-on-K8s), what does your stack actually look like, and what broke?
4. What's the minimum isolation bar an execution backend should be required to attest, given agents run adversarial (prompt-injectable) code?
5. Losing kubectl get pods visibility for runs: dealbreaker, or fine if an aggregated API (metrics-server-style, served from memory) shows live runs?

Nothing is decided here; this is a research direction, and counter-evidence is as welcome as agreement. If you've scaled K8s past these ceilings with boring mitigations (warm pools, image streaming, Kata RuntimeClasses), I especially want to hear about it.

[AlexsJones]

I also want to add that this is solving for a scale I'm not hitting myself yet. But I do expect that anyone using sympozium for ensemble scheduling in production would run into these limits, and the same goes for k8s gang scheduling at this scale.

[AlexsJones]

Some observations from the code as it stands today, since the repo already contains early evidence on several of these questions:

The incremental path has quietly started. KSiig's #302 (TaskModeHandler registry, currently approved and awaiting merge) is structurally the executor-interface seam this RFC describes: it decouples what a task is from how the pod runs it. Promoting that registry from "task mode" to "execution backend" is a much smaller step after #302 lands than before it. If the incremental option wins, #302 is effectively its first commit.

On Q1 (does anyone hit the churn regime): it's worth separating two costs the current design pays. API-server churn at 25–100 concurrent creations is the scale ceiling, but the 5s status-requeue poll and scraping results out of pod logs are latency and fragility costs paid at n=1. The recent sidecar-driven iterations surfaced exactly the failure modes the RFC's distributed-systems list predicts, at single-run scale: a /ipc/done signal that can be lost mid-LLM-call, and torn request files being dropped permanently (both flagged in review on the now-closed #306). That cuts both ways: the fast-path plumbing needs the idempotency/lease discipline anyway, and lazy status write-back plus an explicit work-item protocol would pay off well before anyone hits Dirigent-class burst rates.

On Q4 (minimum isolation bar): one concrete consequence of pull-model node agents. Today, tool policy is cluster-side config (the #303 edit_file review made the point that allow-list deployments must opt into each tool explicitly). Off-cluster, that allow-list has to travel with the work item and be enforced at the sandbox boundary, because NetworkPolicy/SecurityContext no longer exist there. So a reasonable minimum attestation set is: sandbox runtime and version (gVisor/Firecracker/runc), the egress policy actually applied, and the hash of the tool allow-list the agent enforced, all signed so the coordination plane can refuse results from an executor that can't prove its policy. That also gives the audit trail a place to live even when execution is remote.

On Q2: operating experience argues for the incremental shape with a hard rule borrowed from the radical one: nothing new should learn to depend on pod internals (log-scraping is the cautionary tale here; every consumer of it becomes a migration blocker later). Pods as one registered backend keeps kubectl visibility for the people who want it while the NATS path earns trust.

@KSiig — given #302 is effectively the seam in question, your view on where the registry boundary should sit (task mode vs execution backend) would be valuable here.

🤝 Handled by Alex's Repo Steward — replies reviewed by @AlexsJones.

Learn more or run your own: https://github.com/AlexsJones/repo-steward

[KSiig]

Strong +1 on the direction, though for a different reason than the one the RFC leads with. Some detailed thoughts, roughly grouped as: why I want this, how I think it has to ship, and what I think is unresolved on trust.

(Code references below are against main at 21f9171.)

Why I want this: the non-Kubernetes fleet, not the scaling

Personally I don't foresee hitting the API-server ceiling anytime soon. The Dirigent number is real but it's not my bottleneck. What I actually want is the thing that falls out of a pull-based execution plane: the ability to run agents on machines that can't accept inbound connections.

Concretely, something we've considered internally is hosting a Mac Studio in our office running local models for non-time-sensitive jobs. Today that's technically possible — spec.model.baseURL explicitly supports it, Ollama is even named in the field's doc comment — but only if the cluster can reach the Mac. That means either exposing it (out of the question) or standing up a tunnel that gives our cluster a route into the office network.

That tunnel is the problem for us. We're a compliance company and EU data sovereignty is a big part of our posture, so a managed mesh VPN with a non-EU-operated coordination plane isn't something we can adopt casually — and self-hosting the coordination server, or hand-rolling WireGuard, is ongoing operational burden we'd rather not own just to reach one machine.

A pull model removes the need entirely by inverting the direction. The Mac dials out to the control plane, outbound only. The office firewall stays default-deny inbound, there's no route from cloud into LAN, and no mesh VPN in scope at all. That's not a convenience difference, it's a different security posture — and it's the same reason GitHub Actions runners are pull-based.

The honest trade: this makes the control plane the publicly-reachable component instead of the office network. I think that's clearly the right trade — the control plane is already internet-facing and hardened, an office LAN isn't — but worth stating explicitly rather than glossing.

So my vote on Q1 is: pitch this as execution portability, not throughput. The scaling numbers are a nice secondary benefit, but "run agents anywhere, including behind NAT, without inbound access or a third-party network overlay" is the thing that unlocks new deployments.

Q2: incremental only — and I don't think #302 is the path

I'd only support an incremental implementation, and I want to push back on #302 being the seam.

TaskModeHandler's entire contract is expressed in pod vocabulary (internal/controller/taskmodes/taskmodes.go:84-90):

ConfigureAgentContainer(task *sympoziumv1alpha1.TaskSpec, agentEnv *[]corev1.EnvVar) error
AdjustSidecars(task *sympoziumv1alpha1.TaskSpec, sidecars []SidecarContext) ([]SidecarAdjustment, error)

corev1.EnvVar, and a SidecarAdjustment whose only fields are OverrideCommand []string and AddEnv []corev1.EnvVar. The most a handler can express is "set this env var on the agent container" and "change this sidecar's command." There's no return value that means "don't build containers at all — publish a work item instead."

It's also structurally in the wrong place. Both existing backends converge on buildContainers, and the task-mode hook runs inside it (agentrun_controller.go:2390):

reconcilePending             → buildJob (:2260)       → buildContainers (:2345) → applyTaskModeToAgentContainer
reconcilePendingAgentSandbox → buildSandboxCR (:381)  → buildContainers (:207)  → applyTaskModeToAgentContainer

So TaskModeHandler sits below the execution fork, which is exactly why it composes with pods and Sandbox CRs alike. That's what I wanted it to do — it answers "what runs inside the workload." An executor answers "is there a workload at all." Promoting one into the other would keep the name and discard the design.

The seam that already exists

I think the incremental path is the Sandbox CR backend, which already does most of what's being asked for:

• spec.agentSandbox.enabled is a per-AgentRun, off-by-default flag (api/v1alpha1/agentrun_types.go:64).
• It forks the reconcile at agentrun_controller.go:388: reconcilePendingAgentSandbox creates Sandbox/SandboxClaim CRs (agents.x-k8s.io) instead of a batchv1.Job.
• CheckAgentSandboxCRDs (agentrun_sandbox.go:860) degrades gracefully when the CRDs aren't present.
• EnsureWarmPool (:529) already handles warm-pool lifecycle for that backend.
• SympoziumPolicy already governs the choice at admission — validateAgentSandbox in internal/webhook/policy_enforcer.go enforces mutual exclusion with sidecar sandbox, rejects mode: server, honours AgentSandboxPolicy.Required, and validates spec.agentSandbox.runtimeClass against AllowedRuntimeClasses.

That's a second execution backend, a per-run opt-in, capability detection, warm pools, and admission policy over both backend selection and isolation runtime — all shipped, all in main. Generalising spec.agentSandbox.enabled into something like spec.executor: {pods|agentSandbox|external} and adding external as a third case is additive to a fork that already exists. It also means the experimental opt-in comes for free, because that's already how agentSandbox ships.

Which brings me to the strong version of my Q2 position: pods as one of several registered backends, yes — but external execution needs to land as an experimental, off-by-default backend from day one. I realise that's "an experimental feature inside a beta system," but sympozium is at a reasonably stable point now and there are a lot of moving parts here. I've watched architectural reworks live on a branch that drifts from main until it's unmergeable, and the rework dies with it. Shipping behind a flag in main avoids that entirely, and the agentSandbox precedent proves it's achievable here.

What the first PR actually is

One caveat on my own proposal: the existing second backend is only partially abstracted, and the boundary it stops at is exactly the one an external plane needs.

extractResultFromPod (agentrun_controller.go:4120) reads results by calling Pods().GetLogs() on Status.PodName, tailing 500 lines of the agent container. The Sandbox backend reuses it verbatim, and to make that work it assigns Status.PodName = sandboxName (agentrun_sandbox.go:301) with a comment saying precisely why: "The upstream agent-sandbox controller creates a pod with the same name as the Sandbox CR. Update podName on the AgentRun status so that extractResultFromPod can read logs from it."

So the "second backend" still depends on there being a pod in the same cluster whose logs the controller can scrape. An external node agent can't satisfy that. Same story for phase observation: refineSandboxPhaseFromPod (:684) does a live pod GET.

The first real unit of work therefore isn't "add a backend," it's give results, phase, and logs a backend-neutral contract. Full executor surface, for scoping:

Responsibility	Today
Submit work	buildJob / buildSandboxCR
Observe phase	refineSandboxPhaseFromPod — live pod GET, 5s requeue
Extract result	extractResultFromPod — GetLogs() on Status.PodName
Stream logs	K8s log API
Cancel / delete	ownerReferences cascade

buildJob is only the first row. I'd also expect Status.PodName to become something backend-neutral (Status.ExecutionRef?) as part of that, since it's already being abused as a generic handle.

Getting started has to be trivial, or the fleet never materialises

My biggest adoption worry: if standing up your own worker node is a full-day or multi-day project, almost nobody will do it, and the portability benefit stays theoretical.

I'd propose an explicit acceptance bar rather than a vague aspiration:

One copy-pasteable command, one join token as the only argument, under five minutes to a first successful run, working behind NAT with no inbound ports.

A single static binary or a docker run line. If the design can't hit that, the fleet story doesn't land regardless of how good the internals are. Worth deciding up front, because it constrains the protocol.

Q4: this silently downgrades an isolation control that is currently enforced

This is my main technical concern, and I think it's sharper than "how do we attest runtimes," because the control already exists.

Today spec.agentSandbox.runtimeClass selects the isolation runtime (gVisor, Kata) and maps to the Sandbox CR's runtimeClassName. SympoziumPolicy.SandboxPolicy.AgentSandboxPolicy can mark agent-sandbox Required and constrain AllowedRuntimeClasses, and the webhook enforces it at admission. Crucially, that enforcement is real: the field is honoured by the kubelet and CRI, which we control. A run that says runtimeClass: gvisor genuinely gets gVisor.

Move execution to an external node agent and that same field becomes self-reported. The admission webhook keeps validating the string against AllowedRuntimeClasses and keeps approving it, while nothing verifies the node honoured it. Nothing stops an operator from claiming Firecracker and running runc, which would put work that policy says requires kernel-level isolation into a shared-kernel container — with the policy check still passing and looking green. That's an existing security control being quietly converted into an honour system, which I think is worse than not having the control, because it reads as enforced.

I went looking for a cryptographic fix and I don't think one exists for this specific claim. Signing keys live on hardware the operator controls, so a malicious operator signs whatever they like. The adjacent primitives don't close it: TPM/measured boot attests the boot chain, not which runtime a userspace daemon picked later; SPIFFE/SPIRE gives solid identity but says nothing about behaviour. The only real answer is confidential computing (SEV-SNP / TDX), where the CPU vendor signs a measurement of the guest — which would mean Firecracker-on-SNP specifically, not Firecracker generally.

So I'd suggest inverting it: stop trying to attest the runtime, attest the operator.

• A node's self-reported runtime is a routing hint, never a security boundary.
• Policy reads "this AgentRun may only dispatch to nodes in trust tier X."
• The tier is assigned by the cluster admin at registration time, out of band — not claimed by the node in its heartbeat.
• Unknown nodes default to the lowest tier and can only receive work explicitly marked as untrusted-safe.
• For the external executor, AllowedRuntimeClasses should either be rejected outright at admission or explicitly documented as advisory-only, so it can't look enforced when it isn't.

The nice part is this extends AgentSandboxPolicy rather than inventing a parallel mechanism — the policy CRD and webhook hook are already there.

That also answers the registration question, and my answer is: not an open self-registration endpoint. Admin-issued join token (short-TTL, single-use, carrying the trust tier) → node exchanges it for a client cert → mTLS thereafter. That's the k3s / Nomad / Buildkite-agent model and it's what people already expect. PKCE is the wrong tool — it's for public OAuth clients with a human in the loop, and a node agent is neither.

One prerequisite: trust tiers only work if there's a routing mechanism. If dispatch is "any node pulls anything," tiers are unimplementable. Does the design include node pools/labels plus a selector on the AgentRun? If not I think that's a bigger structural gap than the attestation question itself.

Two things I didn't see in the RFC

Credential blast radius. This may be bigger than runtime attestation. Today provider API keys are K8s Secrets mounted into a pod in our cluster. Dispatch work to a node agent someone else operates and we're shipping live credentials to a machine we don't control — and work items already carry tool allow-lists, so they're carrying config that implies credential access.

The pattern I'd draw from is Claworc's virtual keys: real credentials stay in the control plane and never enter the worker; each run gets a synthetic key, and LLM traffic flows through the control plane, which injects the real credential and records per-instance token usage and cost. Same shape as Vault dynamic secrets or Actions OIDC → short-lived cloud creds. Bonus: per-node usage attribution falls out for free.

The tension worth naming: this puts the control plane back on the data path for the fattest flow in the system — bandwidth becomes O(tokens), not O(sandboxes). Survivable, since an LLM call already costs seconds and a proxy hop is noise, but it argues for the broker being a separate stateless egress component rather than living in the controller. And it needs an exemption path for locally-hosted models, otherwise it defeats the data-locality argument above — I specifically don't want my Mac Studio's local inference traffic routed out through a cloud control plane and back.

Version skew. Right now the controller pins the agent-runner image tag, so controller/runner skew is impossible by construction. A node agent is software someone else installed and won't upgrade on our schedule. That needs a wire protocol version, a declared compatibility window, and an answer for "node agent is three versions behind" — reject or degrade? Cheap to design in now, miserable to retrofit.

Q5: fine with losing kubectl get pods, with conditions

Genuinely fine with it, if:

1. The TUI/webview shows equivalent information, and it's exportable via the sympozium CLI. The CLI part matters more than it sounds — it's what makes iterative agent-driven testing of AgentRuns possible (Claude Code driving runs and reading results in a loop).
2. Logs stream, not just fetch-after-the-fact. kubectl logs -f is a live tail, and that's most of the debugging value. Note the current result path already isn't a stream — extractResultFromPod tails 500 lines after the fact — so if the external plane inherits that shape, we lose live debugging entirely. Please make streaming an explicit requirement rather than an implied one.
3. Pass-through metrics and logs — OTLP export to a user-supplied collector, so users handle their own data rather than depending on how sympozium designs its views. There's already OTel wiring in internal/eventbus to build on.

Smaller open questions

• What happens to spec.agentSandbox if executor becomes the general fork? Deprecated, aliased, or left alone? Cheap now, painful later.
• Warm pools. EnsureWarmPool exists for the Sandbox backend. Does the external plane need an equivalent, or do node agents keep their own warm processes? This is where the sub-500ms number actually lives, and the throughput argument is hollow without an answer.
• NATS JetStream isn't new infrastructure here, which is worth saying explicitly — charts/sympozium/templates/nats-deployment.yaml already runs nats --jetstream, nats.enabled defaults to true, and internal/eventbus/nats.go already uses pull consumers with AckExplicitPolicy. But the current config is tuned for notifications, not dispatch: streamConfig() sets Retention: LimitsPolicy with MaxAge: 24h and Replicas: 1, the Deployment is replicas: 1, and storage falls back to emptyDir when persistence is off. ensureStream's own comment says it must be safe to call repeatedly "after a NATS recreate the stream no longer exists" — fine for events, not fine when it's how agents get launched. If execution moves onto it, that implies clustered NATS with R3 streams, WorkQueuePolicy retention on dispatch subjects, and deliberate AckWait tuning, since ack timeout becomes the lease duration that decides how fast a dead node's work gets re-dispatched versus how likely we are to double-run a slow one.

[AlexsJones]

Thanks for the depth here, this is exactly the response the RFC needed.

You've convinced me on the seam. I pitched #302's registry as the executor seam, but your trace is right: TaskModeHandler runs inside buildContainers, below the execution fork, and its contract only speaks pod vocabulary. The fork that matters is the one agentSandbox already made, and generalising an existing per-run opt-in into spec.executor: {pods|agentSandbox|external} is a much smaller ask than a new abstraction. #302 stays what it is: the answer to "what runs inside the workload".

Also agreed that the first PR is the boundary work, not a new backend. Results, phase, and logs all assume a same-cluster pod today (extractResultFromPod tailing Status.PodName, refineSandboxPhaseFromPod doing a live GET), and the Sandbox backend only gets away with it by aliasing the pod name. A backend-neutral Status.ExecutionRef plus a results/phase contract would be useful even if the external plane never ships, which is a good property for a first PR to have.

Trust tiers over runtime attestation makes sense to me. Your question about whether AllowedRuntimeClasses should be advisory-only or rejected at admission for external is the one I most want to get right, since it's the difference between documenting a control and quietly downgrading one. That, and the credential broker trade-off (control plane on an O(tokens) data path, with a local-model exemption so your Mac Studio traffic never leaves the office), are direction calls I'd rather sit with than settle in a thread. They'll go into the next revision of the RFC body, along with the routing/node-pool gap you named, which I agree is a prerequisite for tiers to be implementable at all.

On the acceptance bar: adopted as written. One command, one join token, five minutes, works behind NAT.

The JetStream analysis (LimitsPolicy/24h/R1 is notification tuning, not dispatch tuning; AckWait becomes the lease duration) is too useful to leave buried in a comment thread. It's going into the RFC's failure-modes section, credited to you.

[KSiig]

It seems to me that we're fairly aligned on the direction this should go in. From here, I think it's mostly a matter of defining the milestones in which the new execution plane should be implemented. I'll leave that to you as the maintainer, but happy to provide input if needed!

Of course that's barring input from anyone else.

[AlexsJones]

I'll put this on X to get a bit more visibility on it

[KSiig]

Sounds good. I really think this is a cool feature, and could be the thing that truly positions Sympozium as something unique in the industry. It already is, no other K8s-native LLM scheduling projects that I can find. But, combining that with external execution planes just brings it up an extra notch.