Skip to content

v4.4.0 — dynamic scheduling

Choose a tag to compare

@UnmetDecade UnmetDecade released this 13 Nov 03:42
· 28 commits to main since this release

Dynamic scheduling: a host resource pool enforced by cgroups, live capability scores, weighted fair-share grants, and a pressure state machine. Static deployments keep their existing behaviour.

Highlights

Dynamic scheduling (v0.3)

  • Host resource pool. Each host exposes osacp.slice/pool.slice, with one builder-<id>.scope per builder. Grants map to memory.low / cpu.weight, caps to memory.max / cpu.max, and pressure squeezes memory.high to 1.1× (PRESSURE) or 1.0× (CRITICAL) of the grant. Nothing is OOM-killed to reclaim memory.
  • Split capability score. W = W_static + H_res × W_res, where only the CPU/memory half degrades with real usage. Live, dispatch (25th percentile over 15 s) and guaranteed scores are tracked per instance; committed-but-unused grants lower admission budgets, not the score.
  • Weighted max-min water filling. Integer-only allocation with stable tie-breaks, independent CPU and memory fills, idle floors, and OVERCOMMITTED detection when a grant is squeezed below committed reservations.
  • Per-type concurrency k(i, τ). Replaces the fixed instance limit for dynamic builders, bounded by the remaining grant, the task request and the per-type cap.
  • Pressure state machine. NORMAL → BUSY → PRESSURE → CRITICAL → RECOVERING → NORMAL with separate enter and exit thresholds; new claims are refused under PRESSURE / CRITICAL instead of killing work.
  • Grant lifecycle. PENDING → APPLIED / FAILED / SUPERSEDED with monotonic generations. With require_applied_before_start, a dynamic builder may not start before its quota is applied.
  • Queue discipline. Claims scan up to 20 candidates, record block reasons durably, and retry with exponential backoff (1 s doubling to 60 s) instead of head-of-line blocking the queue.
  • Relay. Cross-mode eligibility uses the ceiling; same-mode ordering uses the live score. The smallest sufficient child is preferred for D > 0, the idlest for D = 0.

Removal reliability

  • A repeated REMOVE is now idempotent: it is re-dispatched instead of returning 409 DEPLOYMENT_REMOVED. A timed-out acknowledgement no longer strands a deployment in the UI.
  • Deployment acknowledgement updates are guarded by generation, so a late acknowledgement cannot roll back a newer command.

Documentation and localization

  • New bilingual scheduling-algorithm guide, wired into both READMEs, the scheduler specification and the docs index: English · 简体中文
  • The admin app UI and source comments are now fully English; .zh-CN documentation remains Chinese.

Upgrade notes

  • Run the schema migration scheduler/sql/008_dynamic_scheduling.sql before starting the scheduler. The deployment payload under deploy/scheduler/app/sql/ includes it.
  • Static mode is unchanged. The scheduling mode is selected once on the root scheduler and inherited by node schedulers and builders.
  • Dynamic mode requires cgroup v2 on the host. If the host runtime does not provide it, admission reports CGROUP_UNAVAILABLE and refuses to run rather than running without an enforced cap.

Verification

  • CI green for scheduler, pcs-issuer, host-runtime, secret-store and admin-app (typecheck, full test suites against PostgreSQL, lint and build).
  • Local checks: typecheck across backend packages, admin-app lint/build, DB-free suites (host-runtime 72, secret-store 6, scheduler resources 15).