Skip to content

vm: per-VM cgroup v2 CPU scope with Guaranteed-at-N defaults (#182) - #183

Merged
CMGS merged 5 commits into
masterfrom
feat/cgroup-cpu-scope
Aug 5, 2026
Merged

vm: per-VM cgroup v2 CPU scope with Guaranteed-at-N defaults (#182)#183
CMGS merged 5 commits into
masterfrom
feat/cgroup-cpu-scope

Conversation

@CMGS

@CMGS CMGS commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Implements #182 (frozen after three review rounds).

What

Every VMM now spawns directly into <cgroup2 root>/cocoon.slice/vm-<id>.scope via clone3 CLONE_INTO_CGROUP at the shared LaunchVMProcess seam — one integration point covers CH and FC across start, restore, and clone. Three raw knobs ride in types.Config and as vm run flags (--cpu-weight, --cpu-quota-us + --cpu-period-us, --cpu-burst-us); absent knobs resolve to the Guaranteed-at-N defaults: quota = N x period, weight = N, burst = 0. --cpu N finally means what it says.

Motivation is measured, not theoretical: on the 16-core testbed a 1-vCPU VM pegs 100.0% with pure CPU load, but 111–113.5% once it does I/O — virtio queue threads and io_uring io-wq workers live outside the vCPU count, and that overhead scales with vCPUs and NICs.

Lifecycle

  • Prepare is create-or-reconfigure and idempotent: restore's kill-relaunch rejoins the same scope even while a dying predecessor still occupies it; only removal requires empty.
  • Removal happens where the VMM is confirmed dead — stop, hibernate, delete teardown, and converge-dead — with cgroup.kill sweeping the CH pty child and stray forks before rmdir (EBUSY-aware poll).
  • GC: a new module removes empty scopes owned by no VM in the union of the CH and FC snapshots (existing cross-module helpers); it never kills. Crash residue is one empty directory.
  • Preflight is the operation itself: parent provisioning (mkdir + +cpu in subtree_control, end-state verified) fails with an error naming the exact missing file. subtree_control writes are read-gated — they take the kernel-wide cgroup_mutex, so steady-state launches must not pay a no-op write (clone-burst path).
  • The detached reseed child (perf(clone): hand post-resume reseed to a detached child #175) stays out of the scope; vm status gains a THROTTLED column from cpu.stat.

Not in this PR

Hardware acceptance on the testbed (default-cap enforcement, capped-vs-raised-quota tax A/B, 1:3 weight discrimination, start-latency delta) runs as a follow-up round per the issue's acceptance list.

Second commit is the whole-repo loc-justify cut application (net −41 prod): single-user map helpers inlined (utils/map.go deleted), OrDefault/cmp.Or/DeleteFunc reductions. applyFilters deliberately kept non-destructive — its test encodes that contract.

CMGS added 2 commits August 5, 2026 12:22
vCPU count only bounds guest parallelism: virtio queue threads and io-wq
workers burn host CPU outside it (measured 111-113% of a core for a
1-vCPU VM under I/O). Spawn the VMM via CLONE_INTO_CGROUP into
cocoon.slice/vm-<id>.scope with raw knobs (cpu.weight / cpu.max /
cpu.max.burst) defaulting to Guaranteed-at-N: quota = N x period,
weight = N, burst = 0.

Scope lifecycle is converge-friendly: create-or-reconfigure before
launch (relaunch reuses an occupied scope), owned removal after the VMM
is confirmed dead on stop/hibernate/teardown/converge-dead paths with
cgroup.kill sweeping stragglers, and a GC module that removes empty
scopes owned by no backend. subtree_control writes are read-gated so
steady-state launches stay off the kernel's hierarchy-wide cgroup_mutex.
…cmp.Or/DeleteFunc reductions

MergeSets folded into FilterUnreferenced's existing exclude variadic;
MapValues inlined at its only caller; applyFilters kept non-destructive
(its test encodes that contract) while filterRecords, with no observer
of the input, takes DeleteFunc.
@CMGS

CMGS commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Hardware acceptance — bare-metal 16-core testbed, binary pr183-1b07c48, CH v54.0.0

Environment: isolated root, non-root operator with a systemd user-delegated parent (cgroup_parent=user.slice/user-1000.slice/user@1000.service/cocoon-accept.slice) — which doubles as the custom-parent path from the issue.

item result
preflight (negative) non-root + default cocoon.sliceprepare cgroup scope: create cgroup parent: mkdir /sys/fs/cgroup/cocoon.slice: permission denied — exact file named, VM retryable
scope + defaults --cpu 1cpu.weight=1, cpu.max=100000 100000, VMM in scope (verified via /proc/pid/cgroup)
default cap the baseline CPU+I/O load that measured 111–113.5% uncapped now pegs exactly 100.00% (15s pidstat avg); cpu.stat: nr_throttled=203, throttled_usec=2.89s
THROTTLED column 456/7.742876s while throttling; - when down
kill -9 → converge empty scope dir remains; vm stop converges and removes it; state → stopped
tax A/B fixed hashing workload under background O_DIRECT I/O: capped 2014/2016/2366 ms vs quota-raised (200ms/period) 1800/1796/1820 ms → +12–13% capped, matching the predicted virtio-share floor case
weight 1:3 25 vs 75 under same-core runqueue contention: 7.50s vs 22.50s over 30s = 2.99:1; uncontended phase: both reach their full quota (10.0s/10s each)
start latency 5 runs each, old master-07f1d69 vs pr183: median 15ms vs 15ms (verified live run 29ms) — no material regression
GC two orphan scopes (records deleted) swept in one gc run, 5ms, reason=orphan-scope; owned stopped VMs untouched

Two findings worth recording:

  1. The issue's weight-acceptance scenario as written does not trigger weight. A parent cpu.max below combined demand causes bandwidth throttling, and children draw the parent quota first-come-first-served — both VMs settled at exactly 0.5 cores regardless of 25:75 weights. cpu.weight only arbitrates real runqueue contention (physical CPU saturation); pinning both vCPUs to one core produced the textbook 2.99:1 split. Callers planning burstable overcommit should size by physical saturation, not parent quotas.
  2. Non-root operation needs the CLI in the same delegation domain as the parent. clone3's CLONE_INTO_CGROUP enforces write access to the source/target common ancestor: launching from an ssh session scope into a user@.service-delegated parent fails with EACCES until the command runs inside that domain (e.g. systemd-run --user --scope). Root deployments — our production shape — are unaffected.

… the VM's

Codex P1: restore launched the VMM under the old record's cgroup config
while persisting the snapshot's afterwards, and Prepare could not lower
quota past a leftover burst.

Knobs now follow the Network precedent: restore keeps the VM's knobs
(the snapshot's describe its source VM), clone inherits from the
snapshot with per-knob flag overrides like --nics. Restore still
launches under the target config so a --force cross-config restore
derives the scope from the new vCPU count. Prepare zeroes a leftover
burst before writing cpu.max (kernel requires burst <= quota).
@CMGS

CMGS commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

cb4e9d3 — cgroup knobs are host policy (addresses the restore P1)

Direction settled in discussion: knobs follow the Network precedent — they are host-side policy that stays with the VM, not guest state that travels in snapshots.

  • Restore keeps the VM's knobs (RestoreVMConfigFromFlags now preserves them exactly like Network); the snapshot's knobs describe its source VM and no longer overwrite the target's. This dissolves most of the DB-vs-runtime mismatch by making the old record's values the correct ones.
  • The remaining real subset — a --force cross-config restore changes the vCPU count the default quota derives from — is fixed by launching under the target config (rec.Config = *vmCfg before launch, CH + FC).
  • Clone gains per-knob flag overrides (--cpu-weight/--cpu-quota-us/--cpu-period-us/--cpu-burst-us, inherit-unless-set like --nics), validated through the same ResolveKnobs().Validate() gate.
  • Prepare converges burst: zero a leftover cpu.max.burst before writing cpu.max (kernel rejects quota < burst), then write the target burst. ENOENT on the zeroing write tolerated for pre-5.14 kernels.

Coverage requested: unit tests for restore-keeps-knobs and clone-flag-overrides (incl. invalid-override rejection); hardware evidence on the 16-core testbed:

check result
kernel constraint scope with burst=150000: cpu.max write to 100000 rejected; zero burst first → same write succeeds
cross-config e2e, pre-fix binary (1b07c48) reproduced the P1 exactly: DB cpu=2, scope still 100000 100000 (old N=1), knobs zeroed in DB
cross-config e2e, fixed binary (cb4e9d3) cpu-1 VM with weight=25/burst=50000, restore --force from a cpu-2 snapshot → scope weight=25 max=200000 100000 burst=50000, DB cpu=2/weight=25/burst=50000 — runtime == DB, knobs kept, quota re-derived from the new N

CMGS added 2 commits August 5, 2026 13:50
… alone, restore preflights fit

A snapshot's cgroup knobs describe its source VM; applying them on clone
contradicted restore's keep-the-VM's-policy rule and made an explicit
zero flag inexpressible. Restore now rejects kept knobs that no longer
fit the snapshot's vCPU count before the destructive phase — the same
combination previously failed only at Prepare, after the VMM was gone,
and retried into the same failure.
…urst zeroing to reused scopes

LaunchSpec.SockPath/PIDPath became pure derivations of Rec.RunDir once
Rec landed; computing them inside LaunchVMProcess removes the only way
a caller could set them inconsistently. Fresh scopes get the kernel's
burst=0 default, so the zeroing write now runs only when Mkdir reports
the scope pre-existed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant