vm: per-VM cgroup v2 CPU scope (request / limit / burst) — vCPU count only bounds guest parallelism

[CMGS]

Problem

The only CPU control cocoon applies today is the vCPU count: CH gets --cpus boot=N,max=<host cores> (hypervisor/cloudhypervisor/args.go) and nothing constrains the VMM process on the host side. vCPU count bounds guest-code parallelism, not host consumption:

• VMM I/O threads run outside the vCPU threads: virtio-blk/net processing, io_uring io-wq kernel workers (accounted to the VMM process), vmm/api threads.
• KVM halt-polling (halt_poll_ns, default 200us) burns host CPU in vCPU threads even while the guest idles.
• There is no floor either: under host contention nothing guarantees a VM any share.

Neither backend can do this natively. CH --cpus offers boot/max/topology/affinity only — upstream expects the embedder to use cgroups (libvirt/Kata do exactly that). FC has cgroup support only via the jailer, which we do not run (#83), and FC is headed to maintenance tier (#161). The mechanism therefore belongs in cocoon, backend-agnostic, at the shared VMM spawn path.

Baseline evidence (measured)

Bare-metal 16-core testbed, cocoon master-07f1d69, CH v54.0.0. One VM, 1 vCPU / 2 GB / no NIC / COW disk direct=on, num_queues=1; whole-process %CPU via pidstat -t, 15–20 s averages:

guest workload	VMM process %CPU	breakdown
idle	0.1%	halt-polling negligible on this host
1× sha256sum (vCPU pegged)	100.0%	vcpu0 alone — the vCPU cap works exactly as expected
+ 1× dd O_DIRECT write loop	111.1%	vcpu0 99.6 + disk queue thread 11.2 + io-wq
+ 6× concurrent dd loops (4 write, 2 read)	113.5%	vcpu0 99.9 + disk queue 13.5 + io-wq

The >1-core claim is real and measured — and this is the floor case: one vCPU means one virtio queue per disk (num_queues=cpuCount) and a single guest thread issuing I/O. Queue threads scale with vCPU count and net-attached VMs add net queue processing on top, so the absolute overhead grows with exactly the workloads we deploy (multi-vCPU Windows agents doing real network I/O).

Proposal: per-VM cgroup v2 scope, raw knobs

At start, create a scope at a deterministic path under a configurable parent (default <cgroup2 root>/cocoon.slice/vm-<id>.scope) and spawn the VMM directly into it via clone3 CLONE_INTO_CGROUP (SysProcAttr.UseCgroupFD/CgroupFD, behind a Linux-only helper so the Darwin build stays valid) — no post-fork attach race, zero cost on the claim path. Every VMM thread inherits: vCPUs, virtio workers, io-wq workers, CH's pty child.

Knobs are the raw cgroup encodings — cocoon validates bounds and kernel constraints but does no unit translation; values are policy and stay with the caller:

knob	cgroup file	encoding and constraints
cpu_weight	cpu.weight	1..10000; proportional work-conserving share under contention, not an absolute floor
cpu_quota_us / cpu_period_us	cpu.max	hard ceiling; period defaults to 100000
cpu_burst_us	cpu.max.burst	accumulated runtime credit in us; kernel requires burst <= quota and a finite quota, so peak within one period is bounded by 2x quota

Defaults: --cpu N is the limit (K8s Guaranteed at N)

Every VM enters a scope unconditionally. For a VM created with --cpu N the scope defaults are:

knob	default	semantics	K8s analog
cpu_quota_us	N × period	hard cap at the vCPU count — --cpu finally means what it says	limit
cpu_weight	N	contention share proportional to size (today an N-vCPU VM's N busy threads earn it ~N shares, so this preserves current inter-VM proportions; a flat equal weight would silently change them). The parent contains only VM scopes, so only sibling ratios matter; weight = N stays within the kernel's 1..10000 for any real vCPU count, where 100 × N would overflow past 100 vCPUs	request (= limit → Guaranteed)
cpu_burst_us	0	no spikes past N unless explicitly granted	—

Callers override any of the three raw knobs: weight lowered below the quota share → Burstable overcommit; burst raised → spikes past N as an explicit policy choice; quota raised → headroom for VMM overhead (Kata's pod-overhead concept expressed through the existing knob — no fourth knob).

Documented cost of the default cap: the VMM's I/O service (virtio queues, io-wq) shares the guest's N-core budget. Two visible effects at defaults, quantified by the acceptance A/B below: a saturated VM doing I/O loses the virtio share out of its N cores (baseline floor case: ~13% of a 1-vCPU budget), and virtio threads throttled at period boundaries can add I/O completion latency up to ~one period worst case.

The bounded-average shape (low quota + burst; long-run average enforced at quota, spikes spend quota+burst per period with the credit cap bounding burst duration) remains available for strict metering / abuse containment — never a default.

Scope lifecycle (idempotent, converge-friendly)

Termination paths include stop/delete, hibernate, restore's kill/relaunch, failed launch/configure, and unexpected VMM exit. The contract:

• Deterministic path per VM; create-or-reconfigure before launch (create is idempotent). Start never waits for scope emptiness — a relaunch joins the same scope even if a dying predecessor still occupies it; only removal requires empty.
• Remove only after the VMM is known dead and the scope is empty; ENOENT counts as success.
• A populated or temporarily unremovable scope is left for the next start/stop/converge/GC pass — never kill remaining tasks; VMM liveness checks and the per-VM ops lock stay authoritative.
• GC scans vm-*.scope children of the configured parent and removes empty, unowned scopes (bounded residue: an empty dir). The parent is shared by both backends, so unowned means absent from the union of the CH and FC GC snapshots (the existing cross-module GC snapshot helpers provide this), and destructive cleanup of an owned-but-stale scope runs under the owning VM's ops lock. No new on-disk artifacts: knob values live in the existing VM meta record; the scope path derives from the VM ID.
• The force-stop path may use cgroup.kill (5.14+) to atomically kill everything in the owned scope — catches the pty child and any stray fork; GC of unowned scopes still never kills.
• The detached reseed child (perf(clone): hand post-resume reseed to a detached child #175) stays out of the scope: reseed is cocoon mechanism cost, not guest consumption.

Platform preflight and parent provisioning

Scopes are always on, so the preflight is unconditional — there is no reduced no-knob mode. At start, missing cgroup v2, a missing cpu controller, or a missing interface file required by the VM's effective knobs (cpu.max.burst needs kernel >= 5.14 + CONFIG_CFS_BANDWIDTH; the default shape needs only cpu.weight + cpu.max) is an actionable start error naming the missing file — never a silent fallback.

Parent provisioning is one idempotent contract: cocoon creates the configured parent (default cocoon.slice) if absent and enables +cpu in cgroup.subtree_control along the path it controls, then verifies the end state. The parent holds only VM scopes, so the no-internal-process rule is never violated. An admin who points the knob at a custom parent owns its placement; the same end-state verification applies.

Observability: expose cpu.stat (nr_throttled, throttled_usec) through inspect/debug. Durable metering is out of scope for this issue.

Explicit non-goals

• No cpuset/vCPU pinning — non-work-conserving, wastes cores; disk queue_affinity is orthogonal and unchanged.
• No memory.max / io.max here. The scope is the natural home for them later, but they are separate issues.
• No FC jailer integration; the scope covers whatever VMM process we spawn.
• No CPU-unit-to-weight translation in cocoon; callers that want K8s-style requests do the mapping themselves.

Requirements and caveats

• cgroup v2 unified hierarchy; verify the rt24 custom kernel has CONFIG_CFS_BANDWIDTH and >= 5.14 burst support.
• Windows guests: the default cap (quota = N) only bites under real saturation (idle vCPUs hlt properly), so the risk profile is much milder than bounded-average. Hardware validation of clock stability / DPC watchdog under bounded-average settings happens when a consumer actually ships that shape.
• cpu.max throttles at period boundaries (visible ~100ms stalls for interactive load at the default period); burst is the intended absorber, smaller periods the fallback.

Acceptance

• Baseline: 1-vCPU VM whole-process consumption > 1 host core — done, measured above (100.0% vCPU-only vs 111–113.5% with I/O).
• Default cap: the baseline workload on a --cpu 1 VM is capped at 100%; cpu.stat shows throttling.
• Tax A/B: same saturated CPU+I/O workload, default cap vs quota raised to N+headroom — record guest throughput delta (expect ~13% floor case) and guest disk latency p99 (expect period-boundary spikes when capped). These numbers are the caller's basis for raising quota.
• Weight discrimination: two VMs with weights 1:3 under a parent cpu.max below their combined demand settle at ~1:3 observed consumption; in a second uncontended phase each reaches its own quota.
• Burst: bounded-average VM shows spike-then-throttle at the credit boundary in cpu.stat.
• VM start latency A/B shows no material regression against the reference host's recorded numbers (scope creation, control-file writes, and clone3 are cheap but not zero); record the delta explicitly.
• Preflight: on a host without cgroup v2 (or where the parent cannot be provisioned), vm run fails with an actionable error naming the missing capability.
• kill -9 of the VMM leaves only an empty scope dir and converge removes it; stop/delete removes the scope inline; hibernate/restore relaunch reuses the deterministic path.
• Darwin build stays green (Linux-only spawn helper).

[CMGS]

The direction looks good, and the shared LaunchVMProcess path is the right backend-agnostic seam. Before implementation, I would tighten four contracts:

1. Define the knob encoding precisely. cpu_request=0.25 cannot be written directly to cpu.weight. If the caller owns policy, either expose the raw weight or define the exact CPU-unit-to-weight mapping, rounding, and bounds. Also describe cpu.weight as a proportional, work-conserving share rather than an absolute floor. cpu.max.burst is accumulated runtime credit in microseconds, not a simple peak-rate formula of "quota + burst per period"; a burst setting should require a finite limit.

2. Make the weighted-share acceptance test exercise weight. Two identical VMs with request 0.25 / limit 1 on a saturated two-core parent can each consume one core, so the test does not distinguish the request behavior. Use asymmetric requests, for example 1:3, under a parent quota below their combined demand; then remove parent contention in a separate phase and verify that each VM can reach its own limit.

3. Treat the scope as a shared, idempotent lifecycle resource. Production termination paths include stop/delete, hibernate, restore's kill/relaunch, failed launch/configure, and unexpected exit. The minimal convergence contract can stay simple: use a deterministic path; create or reconfigure it before launch; remove it only after the VMM is known dead and the scope is empty; treat ENOENT as success; and leave a populated or temporarily unremovable scope for the next start/stop/converge/GC pass. GC only needs to scan vm-*.scope children of the configured parent and remove empty, unowned scopes. There is no need to kill arbitrary remaining tasks—the existing VMM liveness checks and per-VM ops lock remain authoritative. The detached reseed child should remain outside the scope.

4. Define a small platform preflight. Probe cgroup v2, the delegated cpu controller, and the exact interface files requested by the VM, returning an actionable error when unavailable. Put the SysProcAttr.UseCgroupFD / CgroupFD assignment behind a Linux-specific helper so the repository's Darwin build remains valid.

Backward compatibility is not a constraint here. If every VM always enters its own scope, drop the "absent knobs mean today's behavior byte for byte" claim and document the default equal sibling weight as the new behavior. Live cpu.stat through inspect/debug is sufficient for this issue; durable metering can remain out of scope.

With those adjustments, the design looks implementation-ready.

[CMGS]

body updated per the review. knobs are now the raw cgroup encodings (weight 1..10000, quota/period us, burst us with the kernel constraint burst <= quota and finite quota, so peak per period is bounded by 2x quota, my original 0.25 to 1 core example was wrong). weight acceptance switched to asymmetric 1:3 under a parent quota plus an uncontended phase. scope lifecycle spelled out as the idempotent converge contract, reseed child stays out. platform preflight added, spawn helper is linux only so darwin build stays green. compat carve out dropped: every vm enters a scope, default weight scales with vcpu count so today's inter vm proportions hold, a flat default would silently change them.

baseline is measured and in the body: 1 vcpu vm, idle 0.1, cpu only exactly 100.0, one dd stream 111.1, six streams 113.5 percent of a core (vcpu 99.6 + disk queue + io wq), 16c bare metal, master-07f1d69, ch v54. and that is the floor case, queues scale with vcpu count and net adds on top.

[CMGS]

defaults settled after discussion: --cpu N is the limit. scope defaults for a vm created with --cpu N are quota = N x period, weight = 100 x N, burst = 0, which is exactly k8s guaranteed (request = limit). the caller overrides any of the three raw knobs freely: weight down for burstable overcommit, burst up to allow spikes past N as an explicit choice, quota up for vmm overhead headroom (kata's pod overhead expressed through the existing knob, no fourth knob needed).

the cap means the vmm's io service now shares the guest's N core budget, so two costs are documented instead of hidden: a saturated vm doing io loses the virtio share out of its N cores (baseline floor case ~13 percent), and virtio threads throttled at period boundaries can add io completion latency up to about one period. both became acceptance items, a capped vs raised-quota a/b recording guest throughput delta and disk latency p99, so the caller gets numbers to decide when to raise quota.

also folded in from the same discussion: no new on-disk artifacts (knobs live in the vm meta record, scope path derives from the vm id, the only crash residue is an empty scope dir which gc already covers), start never waits for scope emptiness so restore's kill relaunch is idempotent, and the owned force-stop path may use cgroup.kill to catch the pty child and stray forks while gc of unowned scopes still never kills.

body updated.

[CMGS]

The latest body closes the four points from my previous review, and the measured baseline makes the motivation concrete. Two contracts still need to be settled before implementation:

1. Always-on scopes make the no-knob preflight branch unreachable. Every VM now gets a default weight and finite quota, so missing cgroup v2 or the delegated CPU controller must be an unconditional, actionable start error. Please also state whether Cocoon idempotently creates the default cocoon.slice parent and enables +cpu in its cgroup.subtree_control, or whether the parent must be provisioned by the administrator.

2. cpu_weight = 100 * N exceeds the kernel's 10000 maximum when N > 100. The current CPU validation permits any value up to the host CPU count, so a large host can produce an invalid default. Pick a mapping that remains within 1..10000 for the supported vCPU range. If the parent contains only VM children, weight = N is the simplest proportional mapping; the common scale does not affect sibling ratios.

Two implementation watchpoints, not additional design scope:

• Scope GC uses one shared parent while both CH and FC GC modules are registered. Ownership must be checked against the union of both hypervisor snapshots and destructive cleanup must use the owning VM's ops lock; the existing cross-module GC snapshot helpers already support this.
• Change the start-latency acceptance from literal "unchanged / zero hot-path cost" to a small "no material regression" threshold. Scope creation, control-file writes, and clone3 are expected to be cheap, but not zero work.

With those adjustments, the plan is implementation-ready.

[CMGS]

both contracts settled, watchpoints folded in, body updated.

1. preflight is unconditional now, the reduced no knob mode is gone. parent provisioning is one idempotent contract, not two: cocoon creates the configured parent (default cocoon.slice) if absent, enables +cpu along the path it controls, verifies the end state, and fails with an error naming the missing piece. a custom parent's placement is the admin's responsibility but gets the same end state verification. i dropped the separate pre delegated acceptance scenario i had drafted, cocoon runs as root on our nodes so that branch was over defense.

2. default weight is N, not 100 x N. the parent holds only vm scopes so only sibling ratios matter, and N stays inside 1..10000 for any real vcpu count.

3. gc ownership: unowned means absent from the union of the ch and fc snapshots via the existing cross module helpers, and owned but stale cleanup runs under the owning vm's ops lock.

4. start latency acceptance is now no material regression with the delta recorded, not literal zero.

one more trim in the same spirit: the windows bounded average hardware run is out of acceptance. no consumer of that shape exists yet, the caveat stays and the validation happens when someone actually ships those settings.