Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 8 additions & 9 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -561,24 +561,23 @@ jobs:
run: make ci-smoke-mem512

scheduler-baseline:
name: Scheduler perf/stability baseline (ADR-0022 prereq 2, report-only)
name: Scheduler perf/stability baseline (ADR-0022 prereq 2)
runs-on: ubuntu-latest
needs: [build]
# ADR-0022 prerequisite 2: RECORD the scheduler baseline (preempt/switch per
# 1000 guest ticks, max reschedule gap, run-count spread) so a future scheduler-
# constant change is measured, not blind (ADR-0016). REPORT-ONLY in this PR: it
# asserts only a non-vacuous liveness floor (the scheduler is multiplexing a real
# roster), so it reddens on a dead/wedged scheduler but arms no perf band yet.
# A follow-up arms the calibration-selected scalar (preempt-per-1000-ticks) with a
# SCHED_QUANTUM_TICKS revert-confirm. Not a release blocker.
# ADR-0022 prerequisite 2: measure the scheduler baseline so a future scheduler-
# constant change is measured, not blind (ADR-0016). ASSERTS the calibration-
# selected scalar — preemptions per 1000 guest ticks (~200 at the default quantum)
# within [100, 600] — plus a non-vacuous liveness floor. Guest-tick-normalized, so
# host-CPU-invariant on a slow runner. Revert-confirmed: SCHED_QUANTUM_TICKS 5 -> 20
# drops the rate to ~50 and reddens the floor. Not a release blocker.
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y build-essential qemu-system-x86 nasm python3
- name: Boot and record the scheduler baseline (liveness floor asserted)
- name: Boot and assert the scheduler baseline (preemption-rate band + liveness)
run: make ci-smoke-sched

quiet-boot:
Expand Down
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,21 @@ floor of ~100 (= q5-median × 0.5), revert-confirmed by a `SCHED_QUANTUM_TICKS`
20 (q=80 grinds the harness itself, so 20 is the practical lever). Build clean under
`-Werror`; format/cppcheck/api-consistency green.

### ADR-0022 prereq-2 — arm the scheduler gate, **ADR Accepted**

`scripts/test_qos_sched.py` flips from report-only to a real gate: it asserts
**preemptions-per-1000-guest-ticks in `[100, 600]`** (the PR-1-calibrated scalar; ~200 at
the default quantum) on top of the liveness floor. Everything asserted is a delta over
guest ticks, so the band is host-CPU-invariant — a slow CI runner reads the same rate.
The floor catches a *coarsened* round-robin cadence (the scheduler getting more sluggish —
ADR-0022's regression direction); the loose ceiling catches pathological thrash without
reddening on the intended future fix (a *smaller* quantum raises the rate). **Revert-
confirmed** on WSL: at the default `SCHED_QUANTUM_TICKS=5` the gate is green (rate 199.9);
bumping it to 20 drops the rate to 49.9 and reddens the floor; restoring returns it to
green. This completes ADR-0022's second prerequisite, so the ADR flips to **Accepted** —
the deferral decision and both prerequisites (latency gate + scheduler baseline) are done;
the latency fix itself stays a deliberately deferred, now-unblocked future epic.

## [0.5.0] — 2026-07-11 — Agent-reachable QPU, hardening, dead-code payoff

Post-v0.4.0 increments landed autonomously through the panel → gate (revert-and-confirm)
Expand Down
13 changes: 7 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2211,12 +2211,13 @@ ci-smoke-latency: kernel
@echo "=== QuantumOS COM2 latency Test (ADR-0022) ==="
QOS_KERNEL=$(BUILD_DIR)/kernel.elf32 python3 scripts/test_qos_latency.py

# Scheduler perf/stability baseline recorder (ADR-0022 prereq 2). REPORT-ONLY:
# emits the tick-normalized SCHED baseline (preempt/switch per 1000 guest ticks,
# max reschedule gap, run-count spread) and asserts ONLY a non-vacuous liveness
# floor (the scheduler is multiplexing a real roster), so it reddens on a
# dead/wedged scheduler but arms no perf band yet — PR-2 arms the calibration-
# selected scalar. QOS_SCHED_LOAD=1 drives COM2 PING load during sampling.
# Scheduler perf/stability baseline gate (ADR-0022 prereq 2). Emits the tick-
# normalized SCHED baseline (preempt/switch per 1000 guest ticks, max reschedule
# gap, run-count spread) and ASSERTS the calibration-selected scalar — preemptions
# per 1000 guest ticks, ~200 at the default quantum — within [100, 600], plus a
# non-vacuous liveness floor. Guest-tick-normalized, so host-CPU-invariant on slow
# CI. Revert-confirmed: SCHED_QUANTUM_TICKS 5 -> 20 drops the rate to ~50 and reddens
# the floor. QOS_SCHED_LOAD=1 drives COM2 PING load during sampling.
ci-smoke-sched: kernel
@echo "=== QuantumOS scheduler baseline recorder (ADR-0022 prereq 2) ==="
QOS_KERNEL=$(BUILD_DIR)/kernel.elf32 python3 scripts/test_qos_sched.py
Expand Down
21 changes: 18 additions & 3 deletions docs/adr/0022-com2-latency-scheduler-bound.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,30 @@
# 22. COM2 Round-Trip Latency Is Scheduler-Cadence-Bound

Date: 2026-07-11
Status: Proposed (root cause measured; a fix is DEFERRED — **prerequisite 1, the latency gate, now shipped**)
Status: Accepted (2026-07-13; the deferral decision + both prerequisites are complete — the latency fix itself remains a deliberately deferred future epic)

> **Update (2026-07-11).** Prerequisite 1 below — a bounded latency assertion — **shipped** as
> `ci-smoke-latency` (`scripts/test_qos_latency.py`, via the new `QosVM.ping()` one-hop primitive).
> It asserts a bounded PING/PONG median (the pure transport + scheduler-cadence floor) and records the
> STATUS baseline; it is revert-confirmed against the real regression source — bumping
> `SCHED_QUANTUM_TICKS` reddens it. This gives any future latency-reduction work a proven guard and a
> recorded baseline (~0.45 s PING / ~0.90 s STATUS on the dev box). Prerequisite 2 (a scheduler perf
> baseline) and the fix itself remain open.
> recorded baseline (~0.45 s PING / ~0.90 s STATUS on the dev box).
>
> **Update (2026-07-13) — Accepted, both prerequisites complete.** Prerequisite 2 (the scheduler
> perf/stability baseline) shipped as `ci-smoke-latency`'s sibling `ci-smoke-sched`
> (`scripts/test_qos_sched.py`, CI job `scheduler-baseline`). A design panel refuted the naive metric —
> aggregate context switches are voluntary-yield-dominated, so `switch_count/tick` is both quantum-
> insensitive and host-throughput-dependent. The kernel instead grew a **dedicated `preempt_count`**
> incremented only at timer-quantum expiry (never on `SYS_YIELD`), exposed via a `SYSINFO_SCHED` sub-op
> / qsh `sched` command / `QosVM.sched()`, alongside a made-live `last_scheduled` and a per-PCB
> `sched_picks` for the fairness/tail snapshot. Calibration (WSL) confirmed **preemptions-per-1000-
> guest-ticks ≈ 1000/quantum, host-invariant** (~200 at q=5 idle+load with <0.5 % variance, ~50 at
> q=20); the gate asserts that scalar in `[100, 600]` plus a liveness floor, revert-confirmed by a
> `SCHED_QUANTUM_TICKS` 5→20 bump (rate drops to ~50, reddens the floor). Both prerequisites the fix was
> gated on now exist. **The latency fix itself (the I/O-priority-boost epic, or a measured quantum
> reduction) stays deferred** — it is now unblocked, with a proven latency gate AND a proven scheduler
> baseline to measure it against, but remains a future epic per this ADR's decision, not an autonomous
> increment.

## Context

Expand Down
66 changes: 45 additions & 21 deletions scripts/test_qos_sched.py
Original file line number Diff line number Diff line change
@@ -1,26 +1,24 @@
#!/usr/bin/env python3
"""
Scheduler perf/stability baseline recorder (ADR-0022, prerequisite 2) — REPORT-ONLY.
Scheduler perf/stability baseline gate (ADR-0022, prerequisite 2).

ADR-0022 root-caused the agent-tool latency floor to the scheduler cadence and DEFERS
the fix behind two prerequisites. Prereq 1 (the COM2 latency gate, test_qos_latency.py)
shipped. This is prereq 2: a perf/stability baseline so a future scheduler-constant
change (the deferred fix) is MEASURED, not blind (ADR-0016 anti-vacuous discipline).

WHY REPORT-ONLY (PR-1): the design panel's red-team refuted the naive metric. Aggregate
context switches (`switches`) are dominated by VOLUNTARY yields — every long-lived citizen
busy-yields, resetting the quantum counter thousands of times per 10 ms tick, so the timer-
preemption path is nearly dead at rest. switches/tick is therefore BOTH quantum-insensitive
(a SCHED_QUANTUM_TICKS change does not move it -> a gate on it is vacuous) AND host-CPU-
throughput dependent (yield-loop iterations per tick scale with guest speed -> flaky on slow
CI). The kernel now exposes a DEDICATED `preempt` counter incremented ONLY on timer-quantum
expiry (never on SYS_YIELD), which is 1/quantum-paced by construction. But whether `preempt`
is non-zero enough UNDER LOAD to separate q=5 from q=80 is an EMPIRICAL question. So PR-1
does NOT assert any perf band. It RECORDS every candidate (preempt-per-1000-ticks, switch-
per-1000-ticks, max reschedule gap, run-count spread) under idle and under COM2 PING load,
across the SCHED_QUANTUM_TICKS values, to CALIBRATE which scalar PR-2 will gate. The only
assertion here is a non-vacuous LIVENESS floor (the scheduler is actually multiplexing a
real roster) so this job still fails on a dead/wedged/frozen scheduler.
THE METRIC (design-panel-refined, calibration-selected): the design panel's red-team
refuted the naive metric. Aggregate context switches (`switches`) are dominated by
VOLUNTARY yields — every long-lived citizen busy-yields, resetting the quantum counter
thousands of times per 10 ms tick, so the timer-preemption path is nearly dead at rest.
switches/tick is therefore BOTH quantum-insensitive (a gate on it is vacuous) AND host-CPU
dependent (flaky on slow CI). The kernel instead exposes a DEDICATED `preempt` counter,
incremented ONLY on timer-quantum expiry (never on SYS_YIELD). PR-1's calibration confirmed
preempt-per-1000-guest-ticks = ~1000/quantum, host-invariant (~200 at q=5 idle+load with
<0.5% variance, ~50 at q=20). So this gate ASSERTS that scalar in [FLOOR, CEIL] (see the
constants below) — plus a non-vacuous LIVENESS floor so it also reddens on a dead/wedged/
frozen scheduler. Everything asserted is a delta over GUEST ticks, so a slow CI runner reads
the same value (it just takes longer wall-clock to fill the window).

Env:
QOS_KERNEL path to kernel.elf32 (else the bridge default)
Expand All @@ -46,6 +44,18 @@
LIVENESS_MIN_SWITCHES = 100 # context switches over the measured window
LIVENESS_MIN_RUNNABLE = 12 # settled runnable count (observed ~20 at rest)

# Armed perf band on the calibration-selected scalar: preemptions per 1000 guest ticks.
# WSL calibration measured preempt/1000t = ~200 at SCHED_QUANTUM_TICKS=5 (idle AND load,
# <0.5% variance) and ~50 at q=20 — i.e. it tracks 1000/quantum, host-invariant (a ratio
# of guest-side counters, so a slow CI runner reads the same value). FLOOR = q5-median x
# 0.5 = 100 is load-bearing: it reddens when the round-robin cadence COARSENS (quantum
# raised past ~10, the sluggish direction ADR-0022 warns about) — a SCHED_QUANTUM_TICKS
# bump to 20 drops P to ~50 and trips it (revert-confirmed). CEIL = q5-median x 3 = 600 is
# a loose thrash cap that does NOT redden on the intended future fix (a SMALLER quantum
# raises P; q=2 -> ~500 still passes), only on pathological churn.
PREEMPT_PER_1000T_FLOOR = 100.0
PREEMPT_PER_1000T_CEIL = 600.0

# State-gated warmup by GUEST TICKS: the transient boot self-tests (echo, watched-svc,
# quota-test, delegation-test, cpu-hog, qpu-test) run and exit inside the first few
# hundred ticks; cpu-hog in particular holds whole quanta and pollutes the preemption
Expand Down Expand Up @@ -136,19 +146,33 @@ def main():
f"switch_per_1000t={p_switch:.2f} maxgap={max_gap} spread={max_spread} "
f"runnable_min={min_runnable}")

# NON-VACUOUS LIVENESS FLOOR (the only assertion in PR-1): the scheduler must be
# multiplexing a real roster. Reddens on a dead scheduler (no switches), a single-
# runnable degenerate, or a frozen guest clock — but NOT on any healthy cadence, so
# it is not a perf band (that is PR-2, armed on the calibration-selected scalar).
# NON-VACUOUS LIVENESS FLOOR: the scheduler must be multiplexing a real roster.
# Reddens on a dead scheduler (no switches), a single-runnable degenerate, or a
# frozen guest clock. Checked before the perf band so a wedged scheduler is
# attributed correctly rather than as a band miss.
if d_switch < LIVENESS_MIN_SWITCHES:
_fail(f"scheduler not multiplexing: {d_switch} switches over {d_ticks} ticks "
f"(< {LIVENESS_MIN_SWITCHES}) — dead/wedged scheduler")
if min_runnable < LIVENESS_MIN_RUNNABLE:
_fail(f"runnable roster collapsed to {min_runnable} (< {LIVENESS_MIN_RUNNABLE}) "
"— citizens not staying schedulable")

# ARMED PERF BAND (the calibration-selected scalar): preemptions per 1000 guest
# ticks must sit in [FLOOR, CEIL]. The floor catches a coarsened round-robin cadence
# (the scheduler getting more sluggish — ADR-0022's regression direction); the loose
# ceiling catches pathological thrash without reddening on the intended smaller-
# quantum fix. Revert-confirmed: SCHED_QUANTUM_TICKS 5 -> 20 drops P to ~50 < FLOOR.
if p_preempt < PREEMPT_PER_1000T_FLOOR:
_fail(f"preemption rate {p_preempt:.1f}/1000t below floor {PREEMPT_PER_1000T_FLOOR:.0f} "
f"— round-robin cadence has COARSENED (quantum raised? scheduler more sluggish). "
f"dpreempt={d_preempt} over dticks={d_ticks}. See ADR-0022.")
if p_preempt > PREEMPT_PER_1000T_CEIL:
_fail(f"preemption rate {p_preempt:.1f}/1000t above ceiling {PREEMPT_PER_1000T_CEIL:.0f} "
f"— pathological reschedule thrash. dpreempt={d_preempt} over dticks={d_ticks}.")
print(f"OK: scheduler live ({d_switch} switches, {min_runnable} runnable over "
f"{d_ticks} ticks). Baseline recorded — no perf band asserted (PR-1 report-only).")
print("=== scheduler baseline recorder PASSED (liveness floor; calibration data emitted) ===")
f"{d_ticks} ticks); preemption rate {p_preempt:.1f}/1000t within "
f"[{PREEMPT_PER_1000T_FLOOR:.0f}, {PREEMPT_PER_1000T_CEIL:.0f}].")
print("=== scheduler baseline gate PASSED (liveness + preemption-rate band) ===")


if __name__ == "__main__":
Expand Down
Loading