SleepBench measures stage-fraction calibration and first-REM latency, not just kappa - #935
Conversation
… not just kappa
PR #348 fitted the stager to DREAMT. It raised kappa on all three benchmarks
with a held-out gap of -0.027 — clean by every ML criterion — and PR #437
reverted it 48 hours later because it re-scored a healthy night from 6% to 23%
awake. The revert's own words: "kappa doesn't guard stage-fraction
calibration."
SleepBench as it landed in #925 reproduces exactly that blind spot. Sections B
and C score agreement (accuracy, kappa, per-stage sensitivity/specificity) and
wake minutes, none of which constrain how much of the night a recipe spends at
each stage. A recipe can raise kappa while systematically reallocating stages,
because the epochs it newly gets right can outnumber the epochs it newly
mislabels.
Adds two sections. Nothing existing is replaced; the old section E is renamed G.
E. PER-STAGE FRACTION CALIBRATION. Predicted % of night vs reference % of
night for wake/light/deep/REM, as a signed bias in percentage points,
per night and aggregated, plus the unsigned MAE so a recipe that over- and
under-calls in equal measure cannot pass as well calibrated.
- E.0 prints accuracy and kappa on the SAME nights, so the number that is
not sufficient sits directly above the numbers that guard it.
- E.2 reports a HEALTHY stratum (in-bed >= 5 h) separately. The aggregate
hides the failure: #348's healthy-night blowout was ~17 pp of wake, which
pooling with short fragmented nights dilutes.
- E.3 reports the shipped population's stage fractions over every replayed
night with no reference at all, so a distribution shift is catchable on a
database with no human labels.
F. FIRST-REM LATENCY. Minutes from staged sleep onset to the first REM epoch,
per night and as a median/p10/p90/min/max, for each recipe and for the human
reference. Calibration pins how MUCH REM a recipe emits, not WHEN; REM in
the first minutes after onset is physiologically implausible in a healthy
adult. Nights that never reach REM are counted separately rather than folded
in as a zero.
Reference set. The calibration sections score the stage-locked rows, not
section B's set. B excludes an edited night whose stored hypnogram is a
byte-exact replay of the CURRENT V2 — right for B's question, but
version-dependent by construction: change the recipe and a night can enter or
leave the exclusion, silently swapping the denominator underneath a
before/after. The stagelock cursor set comes from `cursors` and does not move
when the recipe moves.
Read-only properties are unchanged: DB.swift is untouched, the open is still
SQLITE_OPEN_READONLY | immutable=1, there is no write surface, and the database
path is still a required argument.
Verification. 13 new unit tests over the pure label-array primitives, no
database needed (swift test in Tools/SleepBench). Measured on a real 36-session
database with 15 stage-locked human-authored references (9,163 epochs),
replaying the same harness against two builds of SleepStagerV2 — this branch's
base, and the base plus the REM-latency guard change proposed in #934:
V2 4-class kappa 0.691 -> 0.689, accuracy 78.2% -> 78.2%
stage bias, pp wake -7.26 -> -7.13
light -7.54 -> -6.10
deep +6.65 -> +6.65
rem +8.15 -> +6.57
first-REM latency median 72.2 -> 83.2 min, minimum 10.0 -> 43.0 min
healthy-stratum wake fraction 9.40% -> 9.43% (+0.03 pp)
Kappa moved -0.002 and would have called that change nothing. The latency
minimum moving off 10 minutes, and the healthy-stratum wake fraction holding to
+0.03 pp, are the evidence — and neither was measurable in this repository
before this commit.
|
#934 is merged, so the recipe change these metrics justified is on One thing before this lands, and it is the same ask from #925 rather than a new one. This PR adds 132 lines of tests that no CI job will ever run. Look at the checks: two, not eleven — That matters more here than it did on #925. You are already touching Happy to put that step up as its own PR instead if you would rather keep this one to the metrics — say The metrics themselves read well and the scoping is right: pure functions over label arrays, no database, |
|
Its own PR, please — and thank you for offering. The more important half of the answer first: you asked for exactly this on #925, and I let that one merge without it. Your review there was submitted So: yes, as its own PR. It is a CI change with its own blast radius, and it reviews better on its own than riding in under a metrics diff. Two details worth having when you write it, because I do not think it is five lines.
And the reason it should land soon rather than eventually, which is the part I under-weighted on #925: Nothing needed on this PR then — it stays the metrics plus the |
Section E's reference set keys on `stagelock` so the denominator cannot move between two builds — the right call, and the reason section B's recipe-dependent exclusion is not reused there. But a `stagelock` cursor proves only that the stages arrived through `edit_sleep_stages`; it does not prove they differ from what V2 emits. On the database this was developed against, 3 of the 15 stage-locked reference nights replay BYTE-EXACT from V2 and a 4th agrees at 99.09%. Those rows carry no information about V2: they hand the incumbent a guaranteed perfect night and charge every alternative recipe for the same nights. The effect is large enough to invert a verdict. Scoring PR #348's parameter set on this database gives 4-class kappa 0.640 against the incumbent's 0.689 on the contaminated 15-night set — #348 loses — and 0.705 against 0.604 on the same set with the three byte-exact nights held out — #348 wins by +0.101. Section E exists precisely to judge a #348-style tune, so a 0.085-kappa thumb on the scale in favour of "change nothing" is the one bias it must not have. Section E cannot simply drop the rows the way section B does without reintroducing the version-dependence it was built to avoid, so the harness names them instead: a new E.-1 audit lists every stage-locked night whose stored hypnogram replays from V2 at >= 95%, flags the byte-exact ones, reports what fraction of the reference set they are, and prints the exact `--exclude` line to pin the same held-out set across both builds of a comparison. Excluding by explicit timestamp keeps the set frozen; excluding by "matches this build's V2" would not. The audit asserts no mechanism for the match, only that a row cannot be an independent reference for the recipe it replays. Verification: `swift test` in Tools/SleepBench = 13 tests, 0 failures; release build clean; run against a 37-session device database copy, E.-1 names the four nights above and every downstream section is unchanged when `--exclude` is not passed. Tools-only — no analytics, no stager, no Android twin, no goldens. Refs #348, #437.
Follow-up commit: the calibration reference set was scoring V2 against V2Pushed one commit on top of this PR. It came out of actually using section E for the job it was built for — re-testing #348 under a label-shift prior correction — and finding that the section's own reference set had a thumb on the scale. The defect. Section E keys its reference set on It inverts the verdict. Scoring PR #348's parameter set on today's recipe, 4-class kappa against the human-authored hypnograms:
The more self-comparison you remove, the more the sign flips and the wider the gap. Section E exists precisely to judge a #348-style tune, so a 0.085-kappa bias toward "change nothing" is the one bias it must not have. The fix. Section E can't drop the rows the way section B does without reintroducing the version-dependence it was built to avoid, so it names them instead. A new Excluding by explicit timestamp keeps the set frozen; excluding by "matches this build's V2" would not. The audit asserts no mechanism for the match — only that a row cannot be an independent reference for the recipe it replays. Default output is unchanged when Tools-only. No analytics, no stager, no Android twin, no goldens. What the fixed instrument then said about #348 and #437 — CONFIRMED, n = 12 nights, 1 subjectSince this is the first time #437's stated precondition for #348's return ("validated against healthy-sleeper ground truth") has actually been testable, here is the measurement. One healthy adult (resting HR 41–59, HRV 72–120 ms, efficiency 0.90–0.94), 12 hand-authored hypnograms, 37 replayed nights. Read every number as n=1 subject. I am not proposing any stager change on this evidence. Each row is #348's parameter values replayed on today's recipe, decomposed. "healthy wake bias" is section E.2, in-bed ≥ 5 h, signed percentage points against the human labels.
Three things fall out, and I think the second is the one worth acting on: 1. Both halves of the #348/#437 story reproduce on an instrument neither PR had. #348 verbatim lifts κ by +0.101 on a wearer and a dataset it never saw — so it was measuring something real — and swings healthy-stratum wake bias by +22.7 pp, taking realised healthy wake fraction from 9.65% to 33.10%. That is #431's field signature, reproduced from raw. #437 was right to revert, and the tune was not noise. 2. The blowout was the prior; the kappa was the physiology — and #437 had to throw out both. #348 raised wake through two independent channels: the motion/jerk gating ( 3. The prior correction works as the theory says, and is worth less than advertised. Correcting Measured priors, for anyone repeating this. The incumbent
Light, deep and REM are close to the shipped values. Wake is not: three independent references put this wearer at 13.6–18.2%, against a shipped prior of 10%. DREAMT's 34% is far too high, but 10% appears to be too low, and the two bracket the truth. The Oura figure is the one to weigh — 362 nights, a different device, a different era and a different scorer, so it is not circular the way the hand-label figure is (that target was estimated on the same nights it is scored against). One more note on the doc comment's own arithmetic: A per-night prior is not testable on this data. A static prior can only move the mean, not the spread, and the between-night spread is itself off: realised wake sd is 4.39 pp (2.18 on the healthy stratum) against 6.7–7.6 pp in both references. The only label-free per-night driver available on this device is the strap's own band wake fraction — but band capture starts 2026-07-11 and 10 of the 12 hand-labelled nights predate it, so the correlation that would justify it rests on n=2. Not a negative result; no result. Worth revisiting once band coverage and hand labels overlap. Limits, stated plainly. One subject, 12 nights, one device. The hand labels were authored against an older, wake-heavier recipe, so a wearer correcting an over-call may not land on their own true value. The |
ryanbr
left a comment
ryanbr
left a comment
There was a problem hiding this comment.
Good catch on your own work, and the fix is the right shape: naming the contaminated nights rather than
dropping them keeps the set frozen, --exclude takes explicit timestamps so it can't drift between
builds, and the audit prints either way — including "clean" when there's nothing to report, so you can
tell it ran from it finding nothing. Default output is unchanged.
Worth saying out loud since it's already merged: #934's tables came from the biased instrument. The
contaminated nights hand the unchanged recipe a perfect score and penalise anything else, so #934's
changed config was marked down, not up. Its claim was "no accuracy win" — that survives, and if anything
was understated. Nothing to redo, but the numbers in that PR body should be read as measured before this
fix. Happy to note that on #934 so the record is straight.
Your #348 re-analysis is the most interesting thing in the thread, and you've scoped it correctly as n=1
and not proposed a change on it.
Approving. I'll put the CI step up separately as you asked — it'll cover tools/linux-capture too, which
has the same problem after #942.
* Run the tool tests that nothing was running Three places in this repo had tests, or code the app's conclusions depend on, that no CI job ever executed. tools/linux-capture carries 212 tests across 8 files — the frame decoder, the HCI extractor, the activity and sync decoders, spot HRV, and the SpO2 promote gate #103 is waiting on. They ran only when someone remembered to. All 212 pass today, so this starts green; the point is that a regression would previously have gone unnoticed by a suite that exists to catch it. Tools/SleepBench is the instrument the sleep work is argued with — #934's stage-fraction tables came out of it — and #935 added 13 tests nothing ran. It path-depends on Packages/StrandAnalytics and Packages/WhoopProtocol, so it also bitrots the first time either changes shape. #935 is the case in point: that tool was scoring V2 against V2's own output for a week and no check could have told anyone. Tools/Backfill has no tests yet, so it is build-only, guarded by a grep for a testTarget rather than assuming one. The Swift side joins swift-packages.yml, whose paths now include Tools/** so a package change can fail its dependents. The Python side is its own ubuntu job: stdlib unittest, no dependencies, and `discover` rather than a hand-listed set so a new test file cannot be silently left unrun — which is the exact failure this closes. app-build.yml stays disabled; app-target Swift is a separate problem. * Fix the Backfill build the new job just caught The tool did not compile: line 57's stage dictionary mixes Double? sources with untyped 0 fallbacks, so Swift could only infer [String: Any] and the emit-module step failed. Nothing built Tools/Backfill, so nobody found out. Annotated [String: Double] — all four sources are Double? on WearableSleepSession, and that is the minute-dict shape SleepStageTotals already decodes. No behaviour change is possible here: the tool could not run at all before this. This is the job earning its place on its first run. * Fix three things the re-review found in this PR pipefail. GitHub's default shell is `bash -e` WITHOUT pipefail, so piping the test run into tee returned tee's status and a failing suite would have exited 0. The count assertion I added would therefore have MASKED every real failure — strictly worse than not adding it. Proven both ways locally: without pipefail a red suite reports exit 0; with it, exit 1. Tools/** was too wide. It matched appchangelog-gen.py, doc_comment_lint.py, the anonymize shell scripts and __pycache__, so editing a Python lint script would have queued ten macos-15 jobs. Narrowed to Tools/SleepBench/** and Tools/Backfill/**, which is what the Swift job actually builds. Tools/test_i18n_audit.py was still unrun — 36 tests covering the audit that gates every PR's translations. i18n-coverage.yml runs `i18n_audit.py --ci` but never its test file. Now covered, from inside Tools/ since the suite imports the module by bare name. Both steps now assert a floor on the collected count, because `discover` exits 0 having found NOTHING: a wrong working-directory would otherwise turn this job green while running no tests, which is the exact failure the PR exists to end. * Write the scratch capture outside the checkout tee wrote out.txt into tools/linux-capture and Tools, neither of which is gitignored — so rehearsing these commands locally leaves a stray file that can be committed by accident. $RUNNER_TEMP keeps the streamed log and leaves the working tree clean.
2 participants
Follows #925 (which added SleepBench) and #930 / #934.
Why
PR #348 fitted the stager to DREAMT. It improved κ on all three benchmarks with a held-out gap of −0.027 — clean by every ML criterion — and PR #437 reverted it 48 hours later because it re-scored a healthy night from 6% to 23% awake. The revert's own words:
SleepBench as it landed in #925 reproduces exactly that blind spot. Sections B and C score agreement (accuracy, κ, per-stage sensitivity/specificity) and wake minutes. None of those constrain how much of the night a recipe spends at each stage: a recipe can raise κ while systematically reallocating stages, because the epochs it newly gets right can outnumber the epochs it newly mislabels. That is what #348 did, and SleepBench in its current form would have scored it as an improvement too.
A comment on #930 stated that per-stage fraction calibration and first-REM latency would be added to SleepBench. #934 instead computed both from a throwaway scratchpad harness to keep that PR to a single concern, and said so in the issue rather than letting it pass. This PR closes that gap: the two metrics now live in the repository, where the next person to tune the stager will actually run them.
What
Nothing existing is replaced. κ, accuracy, sensitivity/specificity, wake minutes and the band-state comparison are all untouched. Two sections are added; the old section E (per-night detail) is renamed G.
E. Per-stage fraction calibration. Predicted % of night vs reference % of night for wake / light / deep / REM, as a signed bias in percentage points — positive meaning the recipe spends more of the night at that stage than the human did — per night and aggregated, plus the unsigned MAE so a recipe that over- and under-calls in equal measure cannot pass as well calibrated.
F. First-REM latency. Minutes from staged sleep onset to the first REM epoch — per night, and as median / p10 / p90 / min / max for each recipe and for the human reference. Calibration pins how much REM a recipe emits, not when; REM in the first minutes after onset is physiologically implausible in a healthy adult, and a stager that emits it is mislabelling early light sleep whatever its κ says. Nights that never reach REM are counted separately rather than folded in as a zero — a recipe that stops emitting REM must not read as one with an excellent short latency.
The
--csvoutput gains the same per-night columns (stageLocked,healthy, per-stage percentages for V2 and truth, and both first-REM latencies), with blanks rather than sentinels where a night has no value.Reference set
The calibration sections score the stage-locked rows, not section B's set. B excludes an edited night whose stored hypnogram is a byte-exact replay of the current V2 — the right call for B's question, but version-dependent by construction: change the recipe and a night can enter or leave the exclusion, silently swapping the denominator underneath a before/after. The
stagelockcursor set comes fromcursorsand does not move when the recipe moves. Both counts are printed, so the difference is visible rather than implied.Safety
Read-only properties are unchanged.
DB.swiftis not touched by this PR; the open is stillSQLITE_OPEN_READONLY | immutable=1, there is no write surface, and the database path is still a required argument. No health data and no database path is committed — the only path in the source is the pre-existing/path/to/whoop.sqliteplaceholder in the usage line.Verification
13 new unit tests over the pure label-array primitives, run with
swift testinTools/SleepBench— no database, no--dbargument, no health data. They pin the sign convention (positive = over-call), that an unused stage reads as 0% rather than a missing key, that latency is measured from onset and not from session start, that a night with no REM isnilrather than zero, and that a mean bias of zero does not imply good calibration when the MAE is 12.Measured on a real 36-session database with 15 stage-locked human-authored references (9,163 epochs), replaying the harness against two builds of
SleepStagerV2— this branch's base, and the base plus the REM-latency guard change proposed in #934:κ moved −0.002 and would have called that change nothing. The latency minimum moving off 10 minutes, the REM fraction bias dropping 1.6 pp, and the healthy-stratum wake fraction holding to +0.03 pp are the evidence — and none of it was measurable in this repository before this commit. For contrast, #348's healthy-stratum wake bias blew out ~17 pp, which is exactly the shape this section is built to catch.
Not in this PR
Tools/SleepBenchis not covered byswift-packages.yml, whose matrix isPackages/**, so these tests do not yet run in CI. Adding it means changing theworking-directoryshape for all eight existing package jobs, which is a separate concern from the metrics — happy to follow up if you want it.