Skip to content

feat: slot-release hibernation for hibernatePolicy=release CocoonSets#23

Merged
CMGS merged 1 commit into
mainfrom
feat/hibernate-slot-policy
Jul 24, 2026
Merged

feat: slot-release hibernation for hibernatePolicy=release CocoonSets#23
CMGS merged 1 commit into
mainfrom
feat/hibernate-slot-policy

Conversation

@tonicmuroq

@tonicmuroq tonicmuroq commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Why

A hibernated VM's placeholder pod keeps its CPU/memory requests bound to the node, so the scheduler counts the seat as occupied even though vk-cocoon has pushed the VM to the registry and destroyed it locally. On a node with 60/80 VMs hibernated, 60 seats are dead weight: no new VM can take them and the live VMs run effectively un-oversold. hibernatePolicy=release turns hibernation into "give the seat back to the pool" — wake takes any free seat on any node, exactly like a GCE create can hit a zone stockout.

Design: https://github.com/cocoonstack/cocoon-specs/pull/5. Depends on cocoon-common: cocoonstack/cocoon-common#7 (merged; go.mod pinned to its merge commit 4363e4f).

How

All new logic in cocoonset/slotrelease.go, two insertions in Reconcile:

  • reconcileSuspendRelease (replaces reconcileSuspend for release sets): applies the hibernate annotation as before, then — only after the existing allOwnedPodsHibernated gate (lifecycle-state=hibernated ∧ observed-generation ≥ CR generation ∧ snapshot present in registry) — stashes the vm names (reusing the delete-time GC stash) and the main pod's node (wake hint) onto the CR, and deletes the owned pods. Terminal pods are kept for the existing post-unsuspend triage. Fails closed without a registry: the seat is never released unverified.
  • reconcileWake: owns the reconcile of a pod-less Suspended set coming back. Persists the new Waking phase before creating the restore pod (crash re-entry resumes the restore instead of falling through to createMainAgent, whose restore intent only covers CocoonHibernation CRs and would fresh-boot over the snapshot). The pod is created unpinned with restore-from-hibernate and a preferred (soft) affinity to the hibernated-on node — old seat still free ⇒ warm wake from the node-local snapshot, otherwise the scheduler places it anywhere in the pool and vk pulls from the registry. Unschedulable surfaces as WakeNoCapacity events + a metric while the pod keeps waiting. Suspended lands in the same pass as the deletes (the durable receipt); a fast unsuspend that still sees the old main in a stale cache is confirmed against the uncached API (stale view of the delete → wait, never-deleted pod → in-place wake), and a crash before the receipt leaves pod-less Suspending which the node-hint arm restores — no informer ordering can reach a fresh boot over the snapshot. The :hibernate tag is dropped only once the restored VM is live (same rule migration enforces). Gated on phase, never on the current policy or bare tag presence, so a policy edit made while suspended still restores correctly and a stale tag can never roll a running VM back.

Migration is unaffected and runs first: a pinned released set resumes through migration's aborted-migration branch, so a spec.nodeName pin always wins over wake-anywhere.

Metrics

slot_release_pods_deleted_total, slot_release_wake_total{placement=hint-node|pool} (soft-affinity hit rate — note every release wake restores via registry pull: vk drops local snapshots with the pod, so hint-node placement is about future warm-path potential, not a warm restore today), slot_release_wake_unschedulable_total (the out-of-stock signal for fleet sizing).

Tests

21 new cases in cocoonset/slotrelease_test.go (fake-client style, mirroring migrate_test.go): verify-before-delete ordering, stash-before-delete, terminal-pod retention, restore+preferred-affinity pod shape, pin-wins-over-hint, fresh-boot fallthrough when no snapshot exists, fail-closed probe errors, tag survival until live, Running-phase tag distrust, and normal-flow handoff for retain placeholders. Full go test ./... green.

@tonicmuroq
tonicmuroq force-pushed the feat/hibernate-slot-policy branch 2 times, most recently from ed5bbe7 to e47b4cc Compare July 23, 2026 13:03
@CMGS
CMGS force-pushed the feat/hibernate-slot-policy branch 7 times, most recently from b700d38 to 1c7e9bc Compare July 24, 2026 16:08
A hibernated VM's placeholder pod keeps its requests bound to the node,
so the scheduler counts the seat as occupied even though vk-cocoon has
pushed the VM to the registry and destroyed it locally. For
hibernatePolicy=release sets the operator now:

- suspend: after every managed VM's :hibernate snapshot is verified in
  the registry (the existing allOwnedPodsHibernated gate), stash the vm
  names (delete-time tag GC) and the main pod's node (wake hint), then
  delete the owned pods — the seat frees and Suspended lands in the
  same pass as the durable receipt that the deletes were issued.
  Terminal pods are kept for triage. Fails closed without a registry.
- wake: reconcileWake owns the reconcile of a set coming back from a
  pod-less Suspended state. It recreates the main pod unpinned with
  restore-from-hibernate and a preferred (soft) affinity to the
  hibernated-on node, persists the Waking phase before the create,
  surfaces Unschedulable as WakeNoCapacity events, and drops the
  :hibernate tag only once the restored VM is live. Gated on phase,
  not the current policy. Durable node-hint evidence closes the
  informer races a fast unsuspend can hit: a pod-less Suspending set
  (crash before the receipt) still restores, a visible main under
  Suspending/Suspended is confirmed against the uncached API — stale
  view of the delete waits it out, a never-deleted pod wakes in place —
  and a restore-marked, non-hibernating main re-engages completion
  across a stale phase read; completion clears the hint.

Migration is unaffected and runs first; a pinned released set resumes
through its aborted-migration branch, so the pin always wins over
wake-anywhere. New metrics: slot_release_pods_deleted_total,
slot_release_wake_total{placement=hint-node|pool},
slot_release_wake_unschedulable_total.

cocoon-common is pinned to the spec.hibernatePolicy merge commit.
@CMGS
CMGS force-pushed the feat/hibernate-slot-policy branch from 1c7e9bc to 6ffa5fa Compare July 24, 2026 16:19
@CMGS
CMGS merged commit 2c6ed5d into main Jul 24, 2026
2 checks passed
@CMGS
CMGS deleted the feat/hibernate-slot-policy branch July 24, 2026 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants