Right-size actor sandboxes to declared ActorTemplate resource limits - #679
Right-size actor sandboxes to declared ActorTemplate resource limits#679Chenyi Wang (chw120) wants to merge 3 commits into
Conversation
0e66671 to
c1f9f03
Compare
|
This PR looks like it implements exactly what we proposed, but I think the premise was flawed (my bad!) I think maybe instead what we should actually do is:
In the current state we will wind up heavily pinning to the worker it first ran on but maybe not considering it when scheduling again? cc Julian Gutierrez Oschmann (@juli4n) 3) should be reusable from the current state, but I think 1/2 might need to replace the downward API approach. |
Implements BenTheElder's 3-step redesign (PR agent-substrate#679), replacing the downward-API pod-sizing approach: 1. ActorTemplate.Spec.Resources declares per-actor compute limits, baked into the immutable spec (so it can seed microVM snapshots). 2. Scheduling gates worker selection on advertised capacity: the syncer publishes each worker's ateom-container limits as Worker capacity, and the scheduler skips workers whose capacity is below the actor's limits (zero constraint or zero capacity = unconstrained, graceful fallback). 3. Limits flow over the actor RPCs (ateapi -> atelet -> ateom) via new cpu_milli/memory_bytes fields on Run/Restore/RunWorkload/RestoreWorkload requests, applied to the OCI spec by internal/sizing. internal/sizing loses env Discover(); sizes now come from RPC via FromLimits(), and its PodSize type is renamed SandboxSize (the size is no longer derived from the pod). atecontroller no longer injects resource-fieldRef env. docs/api-guide.md is updated: the WorkerPool resources section documents worker capacity (the scheduling envelope), and a new ActorTemplate spec.resources section documents actor right-sizing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
c1f9f03 to
6f01be6
Compare
Actors previously ran in sandboxes sized to the whole node; there was no way to declare how much CPU/memory a given actor should get. This adds an explicit, immutable sizing knob on the ActorTemplate and plumbs it all the way into the sandbox's OCI spec, for both the gVisor and micro-VM runtimes. - API: ActorTemplate.spec.resources (*corev1.ResourceRequirements). The Limits size the sandbox and are baked into the immutable spec; the CRD and generated code are regenerated accordingly. - internal/sizing: new SandboxSize value (FromLimits / VCPUs / ApplyToOCISpec). ApplyToOCISpec writes CPU quota+period and the memory limit onto the OCI spec and is a no-op when neither dimension is set, so 0 means "unconstrained". - Plumbing: ateapi reads the template limits (actorResourceLimits → tmpl.Spec.Resources), supplies CpuMilli/MemoryBytes over the actor RPCs (ateapi → atelet → ateom), and ateom applies them — gVisor via the cgroup leaf (runsc --cpu-num-from-quota provisions the sentry vCPU count), micro-VM via the guest spec. Proto messages carry the two fields. - Scheduling: worker capacity is taken from the WorkerPool's per-worker limits and the scheduler only places an actor on a worker whose capacity >= the actor's declared limits; a missing worker or actor dimension is treated as unconstrained so placement is never blocked by absent data. - Tests: unit tests for sizing and scheduling, plus an e2e suite (internal/e2e/suites/sizing) that resumes an actor and asserts, via the probe fixture's new /resources endpoint, that the running sandbox observes the declared CPU/memory from the inside. - Docs & demos: document the model in api-guide.md; the counter, sandbox, and micro-VM demos declare actor limits and their WorkerPool comments now describe the real model (worker limits size the worker pod + advertise capacity; the sandbox is sized by ActorTemplate.spec.resources).
6f01be6 to
58d0da9
Compare
Chenyi Wang (chw120)
changed the title
| vcpus = v | ||
| } | ||
| if sz.MemoryBytes > 0 { | ||
| if m := int(sz.MemoryBytes/(1024*1024)) - s.memReserveMiB; m > 0 { |
There was a problem hiding this comment.
🤖 should-fix 🟡 – When the reserve swallows the whole limit this silently falls back to the kata default, which is larger than what the actor declared. guestConfig defaults to 2048 MiB, so an actor declaring 256Mi — exactly the default reserve — takes the m > 0 branch as false and boots a 2 GiB VM, 8x its declared size. The scheduler already placed it against 256Mi, so a worker sized for that actor is overcommitted by the same factor and the pod OOMs rather than the actor being capped.
The band just above the reserve is the other half: 320Mi leaves 64 MiB of guest RAM, which will not boot a kata guest, and the failure surfaces as a boot hang rather than a sizing error.
Both cases are quiet. Failing the run with a message naming the declared limit and the reserve would make them obvious, and a minimum on spec.resources at admission would catch them before an actor is ever scheduled — there is no validation on that field today.
There was a problem hiding this comment.
Thanks Ben for reviewing the PR. I updated the code with the detailed error message and the validation check. Please review again.
Fail cold boot with a clear error (naming the declared limit, the reserve, and the guest minimum) instead of silently falling back to the larger kata default when the VMM reserve leaves too little guest RAM to boot. Add a matching admission floor on ActorTemplate.spec.resources.limits.memory (>= VMM reserve + guest minimum) for sandboxClass microvm so the case is caught at create time, before scheduling.
99bd032 to
dc1c37c
Compare
2 participants
Actors previously ran in sandboxes sized to the whole node; there was no way to declare how much CPU/memory a given actor should get. This adds an explicit, immutable sizing knob on the ActorTemplate and plumbs it into the sandbox's OCI spec for both the gVisor and micro-VM runtimes.
API —
ActorTemplate.spec.resources(*corev1.ResourceRequirements). Thelimitssize the sandbox and are baked into the immutable spec; the CRD and generated code are regenerated accordingly.internal/sizing (shared by both runtimes) — new
SandboxSizevalue (FromLimits/VCPUs/ApplyToOCISpec).ApplyToOCISpecwrites CPU quota+period and the memory limit onto the OCI spec, and is a no-op when neither dimension is set, so 0 means "unconstrained".VCPUsrounds milliCPU up to wholevCPUs.
Plumbing — ateapi reads the template limits (
actorResourceLimits→tmpl.Spec.Resources) and suppliesCpuMilli/MemoryBytesover the actor RPCs (ateapi → atelet → ateom); ateom applies them — gVisor via the cgroup leaf (runsc --cpu-num-from-quotaprovisions the sentry vCPU count), micro-VM via the guest VmConfig. The two fields are carried on the proto messages.Scheduling — worker capacity is taken from the WorkerPool's per-worker limits and advertised on the Worker; the scheduler only places an actor on a worker whose capacity >= the actor's declared limits. A missing worker or actor
dimension is treated as unconstrained, so placement is never blocked by absent data.
Docs & demos — document the model in
api-guide.md; the counter, sandbox, and micro-VM demos declare actor limits, and their WorkerPool comments now describe the real model (worker limits size the worker pod + advertise scheduling capacity; the sandbox itself is sized byActorTemplate.spec.resources).internal/e2e/suites/sizingresumes an actor and asserts, via the probe fixture's/resourcesendpoint, that the running sandbox observes the declared CPU/memory from the inside)