Sleep: correct which stager ships (V2, not V1), and add SleepBench so the claim is measurable - #925
Conversation
Several comments on both platforms described `SleepStagerV2` as an "opt-in experimental" stager and V1 as "the default". That is backwards, and one Swift comment in `Repository.restageFromRaw` stated it outright: "V1 stays the default and is untouched". Both preferences are default ON: Strand/BLE/PuffinExperiment.swift experimentalSleepV2Enabled — `object(forKey:) == nil ? true : …` android/.../ble/PuffinExperiment.kt experimentalSleepV2 — `prefs.getBoolean(KEY, true)` Strand/Screens/SettingsView.swift @AppStorage(...) = true V2 was promoted over V1 in #277 and extended to every strap family in #351, so V2 is what stages a normal user's nights. The confusion has a single source: `useSleepStagerV2: Bool = false` in the `detectSleep` / `analyzeDay` signatures (SleepStager.swift:851, AnalyticsEngine.swift:328). That is the LIBRARY default, there so pure-function callers and the frozen-golden tests stay byte-identical. The app never takes it; the live call site threads the preference. Reading the signature alone gets the product's behaviour exactly wrong, so every one of these comments now names both defaults and says which is which. Verified empirically before changing anything, by replaying both stagers over one user's 36 recorded sessions and comparing epoch-for-epoch against the stored hypnogram. On the 21 UNEDITED nights (where the stored stages are whatever the live stager emitted), V2 reproduces the stored hypnogram byte-exactly on 21 of 21; V1 on 0 of 21, agreeing only 30.6–65.8%. The harness is `Tools/SleepBench`, added in the following commit, so anyone can re-run this against their own database. Comments only — no behaviour change on either platform. swift test (Packages/StrandAnalytics): 1167 tests, 0 failures ./gradlew assembleFullDebug testFullDebugUnitTest: 3201 tests, 0 failures, 5 skipped (JDK 17; counts read from app/build/test-results/testFullDebugUnitTest/TEST-*.xml, not from the wrapper's exit code)
Eleven open PRs claim sleep-accuracy improvements and none of them is scored
against a shared reference, so nobody can say which ones help or by how much.
This adds the missing yardstick.
`sleepbench --db <copy-of-a-noop-sqlite>` replays `SleepStager` (V1) and
`SleepStagerV2` over every recorded night in a database and scores each
hypnogram against whichever independent references that database carries:
(a) hand-authored hypnograms — `sleepSession.userEdited` rows that ALSO carry a
`stagelock:<device>:<startTs>` cursor;
(b) the strap's own band sleep_state (the v18 @81 high nibble), which is an
independent hardware verdict NOOP does not compute — and needs no human
labelling, so it scales to a whole history for free;
(c) the stored hypnogram itself, which identifies the recipe that produced it.
Reported per stager: wake-minute error, sleep-onset and final-wake offsets,
per-stage sensitivity/specificity, Cohen's kappa on both the 4-class and the
sleep/wake problem, and per-night variance. PR #738's band veto is mirrored so
its effect can be scored on the isolated staging path too.
Two correctness points the harness makes rather than assumes:
- `userEdited` alone does NOT mean human stage labels. The local editor
corrects bed/wake BOUNDS and re-derives the hypnogram from raw, so its stages
are machine output over a human-chosen window. Only a `stagelock` row had its
stages authored directly. Section 0 reports the split so a caller cannot
silently score a stager against its own output.
- Session rows and stream rows can live under DIFFERENT deviceIds in a real
database, so `--device` and `--stream-device` are separate arguments.
Read-only by construction: the database is opened `SQLITE_OPEN_READONLY` with
`immutable=1`, its path is always an argument, and nothing is ever written back.
No health data lives in this repository.
Built and run against a real 36-session database; `swift build -c release` clean.
ryanbr
left a comment
ryanbr
left a comment
There was a problem hiding this comment.
Verified the load-bearing claim independently rather than from the table in the description, since a
"documentation correction" is worth exactly whether the new text is true.
It is. experimentalSleepV2Enabled is object(forKey:) == nil ? true : …;
experimentalSleepV2 is prefs.getBoolean(KEY_EXPERIMENTAL_SLEEP_V2, true). There is one live
staging call site per platform and both thread the preference — IntelligenceEngine.swift:672, and
Repository.restageFromRaw, which gates on the flag directly via stageSession rather than the
useSleepStagerV2 parameter (so a census on the parameter name alone misses it — worth knowing, since
that is the function whose comment said "V1 stays the default and is untouched"). The library
= false really is only the library's contract.
Comments-only holds. Every changed line outside Tools/ is a comment or blank — checked
mechanically, not by eye, across all 13 files. Parity is kept: each Swift edit has its Kotlin twin.
#277 and #351 resolve correctly — "promote Sleep V2 to the default staging engine" and "V2 default
for every strap". Good citations to add.
Two things.
1. Nothing builds Tools/, and the tool path-depends on two packages that change constantly.
Package.swift points at ../../Packages/StrandAnalytics and ../../Packages/WhoopProtocol, and no
workflow compiles anything under Tools/. So the first API change in either package breaks
sleepbench and nothing reports it. A harness meant to be the shared yardstick for the sleep PRs is
worth little if it stops compiling in three weeks and we find out when someone tries to use it. One
swift build step in swift-packages.yml fixes it; worth doing here rather than as a follow-up.
2. #600 and #690 point at unrelated work in this repo — a Target Strain notification and a
body-location probe. They are pre-existing and not yours, but this PR repeats them in the comments it
is making trustworthy, so they are worth correcting in the same pass. The V2 lineage that resolves is
#347 ("remove the V2 toggle") → #351 ("reworks #347") → #277. If #600/#690 are upstream
numbering, saying so inline would stop the next reader chasing them.
On the tool: immutable=1 + SQLITE_OPEN_READONLY is the right call and the reasoning about
suppressing WAL recovery writes on a pulled device file is correct. Cohen's kappa is implemented
correctly, including returning .nan at p_e >= 1 instead of dividing by zero. Excluding the
4 userEdited-without-stagelock nights as self-comparison is the distinction that makes the
21-of-21 number mean anything, and scoping it to "which code path runs, not a physiological
validation" is the right claim to make from one user's 36 sessions.
Approving on the substance. Please add the CI build step before merge.
|
Correcting my own note above, since it would have sent you the wrong way. I said The real problem is narrower: the tree mixes two trackers without saying which is which, so |
… not just kappa (#935) * SleepBench measures stage-fraction calibration and first-REM latency, not just kappa PR #348 fitted the stager to DREAMT. It raised kappa on all three benchmarks with a held-out gap of -0.027 — clean by every ML criterion — and PR #437 reverted it 48 hours later because it re-scored a healthy night from 6% to 23% awake. The revert's own words: "kappa doesn't guard stage-fraction calibration." SleepBench as it landed in #925 reproduces exactly that blind spot. Sections B and C score agreement (accuracy, kappa, per-stage sensitivity/specificity) and wake minutes, none of which constrain how much of the night a recipe spends at each stage. A recipe can raise kappa while systematically reallocating stages, because the epochs it newly gets right can outnumber the epochs it newly mislabels. Adds two sections. Nothing existing is replaced; the old section E is renamed G. E. PER-STAGE FRACTION CALIBRATION. Predicted % of night vs reference % of night for wake/light/deep/REM, as a signed bias in percentage points, per night and aggregated, plus the unsigned MAE so a recipe that over- and under-calls in equal measure cannot pass as well calibrated. - E.0 prints accuracy and kappa on the SAME nights, so the number that is not sufficient sits directly above the numbers that guard it. - E.2 reports a HEALTHY stratum (in-bed >= 5 h) separately. The aggregate hides the failure: #348's healthy-night blowout was ~17 pp of wake, which pooling with short fragmented nights dilutes. - E.3 reports the shipped population's stage fractions over every replayed night with no reference at all, so a distribution shift is catchable on a database with no human labels. F. FIRST-REM LATENCY. Minutes from staged sleep onset to the first REM epoch, per night and as a median/p10/p90/min/max, for each recipe and for the human reference. Calibration pins how MUCH REM a recipe emits, not WHEN; REM in the first minutes after onset is physiologically implausible in a healthy adult. Nights that never reach REM are counted separately rather than folded in as a zero. Reference set. The calibration sections score the stage-locked rows, not section B's set. B excludes an edited night whose stored hypnogram is a byte-exact replay of the CURRENT V2 — right for B's question, but version-dependent by construction: change the recipe and a night can enter or leave the exclusion, silently swapping the denominator underneath a before/after. The stagelock cursor set comes from `cursors` and does not move when the recipe moves. Read-only properties are unchanged: DB.swift is untouched, the open is still SQLITE_OPEN_READONLY | immutable=1, there is no write surface, and the database path is still a required argument. Verification. 13 new unit tests over the pure label-array primitives, no database needed (swift test in Tools/SleepBench). Measured on a real 36-session database with 15 stage-locked human-authored references (9,163 epochs), replaying the same harness against two builds of SleepStagerV2 — this branch's base, and the base plus the REM-latency guard change proposed in #934: V2 4-class kappa 0.691 -> 0.689, accuracy 78.2% -> 78.2% stage bias, pp wake -7.26 -> -7.13 light -7.54 -> -6.10 deep +6.65 -> +6.65 rem +8.15 -> +6.57 first-REM latency median 72.2 -> 83.2 min, minimum 10.0 -> 43.0 min healthy-stratum wake fraction 9.40% -> 9.43% (+0.03 pp) Kappa moved -0.002 and would have called that change nothing. The latency minimum moving off 10 minutes, and the healthy-stratum wake fraction holding to +0.03 pp, are the evidence — and neither was measurable in this repository before this commit. * Stop SleepBench's calibration set scoring V2 against V2's own output Section E's reference set keys on `stagelock` so the denominator cannot move between two builds — the right call, and the reason section B's recipe-dependent exclusion is not reused there. But a `stagelock` cursor proves only that the stages arrived through `edit_sleep_stages`; it does not prove they differ from what V2 emits. On the database this was developed against, 3 of the 15 stage-locked reference nights replay BYTE-EXACT from V2 and a 4th agrees at 99.09%. Those rows carry no information about V2: they hand the incumbent a guaranteed perfect night and charge every alternative recipe for the same nights. The effect is large enough to invert a verdict. Scoring PR #348's parameter set on this database gives 4-class kappa 0.640 against the incumbent's 0.689 on the contaminated 15-night set — #348 loses — and 0.705 against 0.604 on the same set with the three byte-exact nights held out — #348 wins by +0.101. Section E exists precisely to judge a #348-style tune, so a 0.085-kappa thumb on the scale in favour of "change nothing" is the one bias it must not have. Section E cannot simply drop the rows the way section B does without reintroducing the version-dependence it was built to avoid, so the harness names them instead: a new E.-1 audit lists every stage-locked night whose stored hypnogram replays from V2 at >= 95%, flags the byte-exact ones, reports what fraction of the reference set they are, and prints the exact `--exclude` line to pin the same held-out set across both builds of a comparison. Excluding by explicit timestamp keeps the set frozen; excluding by "matches this build's V2" would not. The audit asserts no mechanism for the match, only that a row cannot be an independent reference for the recipe it replays. Verification: `swift test` in Tools/SleepBench = 13 tests, 0 failures; release build clean; run against a 37-session device database copy, E.-1 names the four nights above and every downstream section is unchanged when `--exclude` is not passed. Tools-only — no analytics, no stager, no Android twin, no goldens. Refs #348, #437.
2 participants
What this PR does
Two commits on the sleep-staging subsystem: a documentation correction, and the measuring tool that made the correction provable.
1.
docs(sleep)— say which stager actually shipsComments on both platforms described
SleepStagerV2as an "opt-in experimental" stager and V1 as "the default". That is backwards. One comment inRepository.restageFromRawsaid it outright: "V1 stays the default and is untouched".Both preferences are default ON:
Strand/BLE/PuffinExperiment.swiftexperimentalSleepV2Enabled—object(forKey:) == nil ? true : …android/.../ble/PuffinExperiment.ktexperimentalSleepV2—prefs.getBoolean(KEY, true)Strand/Screens/SettingsView.swift@AppStorage(...) private var experimentalSleepV2Enabled = trueV2 was promoted over V1 in #277 and extended to every strap family in #351, so V2 is what stages a normal user's nights.
The confusion has a single source:
useSleepStagerV2: Bool = falsein thedetectSleep/analyzeDaysignatures (SleepStager.swift:851,AnalyticsEngine.swift:328). That is the library default, there so pure-function callers and the frozen-golden tests stay byte-identical. The app never takes it — the live call site threads the preference. Reading the signature alone gets the product's behaviour exactly wrong, so each of these comments now names both defaults and says which is which.Comments only. No behaviour change on either platform.
2.
tools(sleep)— addTools/SleepBenchAn offline harness that replays
SleepStager(V1) andSleepStagerV2over every recorded night in a NOOP database and scores each hypnogram against whichever independent references that database carries:sleepSession.userEditedrows that also carry astagelock:<device>:<startTs>cursor;sleep_state(the v18@81high nibble) — an independent hardware verdict NOOP does not compute, and one that needs no human labelling, so it scales to a whole history for free;Reported per stager: wake-minute error, sleep-onset and final-wake offsets, per-stage sensitivity/specificity, Cohen's kappa on both the 4-class and the sleep/wake problem, and per-night variance. #738's band veto is mirrored so its effect can be scored on the isolated staging path too.
Two correctness points the harness makes rather than assumes:
userEditedalone does not mean human stage labels. The local editor corrects bed/wake bounds and re-derives the hypnogram from raw, so its stages are machine output over a human-chosen window. Only astagelockrow had its stages authored directly. Section 0 reports the split so a caller cannot silently score a stager against its own output. On the database I ran it against, this excluded 4 of 15 edited nights as byte-exact replays of the current V2 — i.e. self-comparison.deviceIds in a real database, so--deviceand--stream-deviceare separate arguments.Why
The subsystem has been changing without a shared yardstick. #348 (@tanarchytan) re-tuned the V2 deep boundary on DREAMT (n=100 gold) and merged 2026-07-13; #437 restored the pre-#348 defaults 36 hours later, because the tune "over-calls awake in the field" (#431). Both were reasonable calls on the evidence available. Neither side could put a number on the disagreement, because there was no way to score a candidate against a wearer's own recorded nights.
Right now 12 of the 60 open PRs touch a
Sleep/Stager-named file, from 6 contributors (@Newbbsss, @DX23876, @tigercraft4, @digitalerdude, @pipiche38, and me). None of them shares a metric. This is the missing piece — not a tuning change, just the ability to measure one.The measurement that motivated commit 1
Replaying both stagers over one user's 36 recorded sessions, epoch-for-epoch against the stored hypnogram. On the 21 unedited nights — where the stored stages are whatever the live stager emitted — V2 reproduces the stored hypnogram byte-exactly on 21 of 21; V1 on 0 of 21, agreeing only 30.6–65.8%.
That is what settles which stager ships, and it is reproducible by anyone with a NOOP database.
Scope of that evidence, stated plainly: one user, one strap family, 36 sessions. It establishes which code path runs — which is a fact about this repository, not about sleep physiology — and nothing more. It is not a validation of V2's accuracy.
Safety of the tool
SQLITE_OPEN_READONLYwithimmutable=1(DB.swift:15-16), so it cannot write, and cannot trigger a WAL/journal recovery write on a pulled device file.--dbexits 2.--csv <path>report you explicitly ask for.Type of change
How it was tested
No BLE path is touched by either commit — commit 1 is comments only, commit 2 is a standalone offline tool that reads a SQLite file.
Android counts were read from
app/build/test-results/testFullDebugUnitTest/TEST-*.xmlrather than the wrapper's exit status, since the Gradle wrapper can report success on a failed build. Built with JDK 17, matching the project'sjvmTarget.sleepbenchwas additionally run end-to-end against a real 36-session database to produce the numbers above.Checklist
swift testinPackages/StrandAnalytics)android/(./gradlew testFullDebugUnitTest)StrandDesigntokens — no hardcoded colors, fonts, or spacing (n/a — no UI change)docs/CONTRIBUTING.mdStrand.xcodeproj/) or any secrets/keystoresRelated issues
Context for the documentation fix: #277, #351, #690.
Context for why a shared harness is worth having: #348, #431, #437.