Skip to content

OS-DEV v1916 — the intermittent-hang bug, found and fixed

Choose a tag to compare

@milescoviello milescoviello released this 31 Jul 06:46
· 681 commits to main since this release

This release is mostly one thing: a real kernel scheduler bug that had been
causing "intermittent" hangs for many milestones is finally found and fixed.

The bug (M1912)

schedule() charged CPU time as timer_ms() - last_in — millisecond
granularity. Every spin-then-yield lock in the kernel spins ~1000 pause
iterations (microseconds) and then yields, so the measured slice was 0 and the
waiter's vruntime never moved. CFS always picks the smallest vruntime, so the
waiters won every pick and the task holding the lock was never scheduled again.

Captured mid-deadlock:

min_vr=1420 | id7 vr=1420 (spinning on ata_lock)
            | id6 vr=1420 (spinning on ata_lock)
            | id0 vr=1430 READY, HOLDS ata_lock, never picked

Ten units behind was enough to starve forever. The fix floors the per-switch
charge at 1 so vruntime strictly increases.

One bug explained the boot TLS "stall", the HTTP server test's long-running
flakiness (a wedged holder means nothing can read the disk, so the server never
loads), and most likely the "residual" ATA flakiness blamed on the disk since
M1541 — that milestone added the task_yield() which converted a livelock into
this starvation.

Found by sampling the instruction pointer through the QEMU monitor after
printf-debugging kept moving the apparent hang location. Proven with a
deterministic self-test rather than statistics: 64 fast yields must advance
vruntime by ≥64 — 64 with the fix, 0 without.

Three more races, from auditing the class instead of stopping

  • CMOS clock (M1913) — 0x70/0x71 index/data pair with no lock and ~21
    concurrent callers (desktop clock, file timestamps, ring-3 clock_gettime). An
    interleave returns another register as the time; rtc_set wrote the time into
    the wrong register. Two unbounded loops also bounded.
  • PCI config space (M1914) — 0xCF8/0xCFC, no locking, 13 files, reachable
    from ring 3 via /pci/.... Sub-word writes were read-modify-write, so two
    drivers setting their own bit in one dword could lose an update.
  • kprintf (M1915) — no lock at all, so concurrent log lines spliced
    character-by-character and corrupted the serial log every headless test greps.

Each fix ships with a self-test that was mutation-tested by removing the lock —
including one first attempt that ran 3001 unlocked reads and detected nothing,
which is why the tests now deliberately widen the race window.

Also

  • CSS max-height (M1910), which a comment in the renderer wrongly claimed was
    architecturally impossible. It is the symmetric clamp to height/min-height,
    plus suppression of the block background under spilled content.
  • Boot self-tests are gated behind -append selftest (M1916) so normal
    boot-to-desktop stays at 1.21 s.

Full make check is green: 130 suites.

Try it

Download os-dev-tryit.zip, unzip, and see RUN_ME.md — one QEMU command.