Sleep: forbid wake → deep/REM transitions — the one part of #348 that survives a de-contaminated benchmark - #987
Conversation
…8 that survives measurement
PR #348 re-tuned seven things about SleepStagerV2 on DREAMT; PR #437 reverted all of them 48 h later
because one healthy night went from 6 % to 23 % awake ("kappa doesn't guard stage-fraction calibration").
Re-litigating that revert component-by-component, on a reference set de-contaminated with the
`--exclude` flag added in af44563, six of the seven components measure neutral-to-negative and stay
reverted. One does not: the AWAKE transition row.
Restored from #348, on both platforms:
"awake": deep 0.01 -> 0.0, rem 0.02 -> 0.0, light 0.27 -> 0.10, awake 0.70 -> 0.90
This is a physiological claim, not a fitted constant: sleep onset descends through N1/N2, so wake never
transitions straight into N3 or REM, and the freed mass makes a WASO episode span several epochs rather
than flicker back to sleep after one. The `ln(max(v, 1e-9))` viterbi floor that makes a zeroed entry
safe already landed with #348 and survived #437, so no other change is needed to support it.
MEASUREMENT (Tools/SleepBench, one wearer's 36 recorded nights).
The only reference here that V2 cannot contaminate is the strap's own band `sleep_state` — the v18 @81
nibble, WHOOP's verdict rather than a re-derivation of NOOP's. Over 21 banded nights / 15 554 epochs:
sleep/wake kappa 0.105 -> 0.118 wake sensitivity 16.0 % -> 17.6 %
accuracy 79.1 % -> 79.2 %
The #437 guard holds — this is the number that must not blow out, and does not:
healthy-stratum wake fraction (n = 20) 9.43 % -> 9.96 % (#348 entire: 32.66 %)
healthy deep 22.09 % -> 22.11 %
first-REM latency MAE 53.9 -> 41.6 min
Why the other six stay reverted, each measured alone against the same band reference / wake guard:
base priors (deep .15, awake .34) healthy wake 9.43 % -> 17.76 % the #437 blow-out, confirmed
motion gate (jerk 75/35, boost 4) healthy wake 9.43 % -> 15.92 % a SECOND wake channel, not free
emission coefficients band kappa 0.105 -> 0.094
deep gate .25 -> .40 healthy deep 22.09 % -> 25.47 % worsens an existing over-call
awake dead-zone 0.30 band kappa 0.105 -> 0.099
deep/rem/light transition rows band kappa 0.105 -> 0.101, healthy REM 29.58 % -> 33.11 %
LIMITS, stated plainly. n = 1 wearer. #348's out-of-cohort evidence was +0.028 (AAUWSS) and +0.006
(Walch); its headline +0.17 is DREAMT in-sample, on the cohort its constants were tuned to. 13 of this
database's 15 stage-locked nights are byte-exact or >= 95 % V2 replays and cannot serve as an independent
reference for V2 at all, leaving 2 usable hand-labelled nights (kappa 0.857 -> 0.863 there, a tie); the
band comparison above is the load-bearing one precisely because it does not depend on those labels.
Verification: StrandAnalytics `swift test` 1248 tests, 0 failures. Android
`:app:testFullDebugUnitTest --rerun-tasks` 3239 tests, 0 failures, 0 errors (counted from JUnit XML),
SleepStagerV2Test 12/12. Both frozen goldens pass UNCHANGED — they are insensitive to this row, which
is why this adds a direct pin for it on both platforms (twin of the existing deep-row pin), asserting
the zeros and that a zeroed entry still reaches the lattice as a finite log-weight.
|
Reviewed. The measurement work here is the most careful this repo has seen on the stager — ablating all seven components separately, and reporting that six of your own candidates measured neutral-to-negative, is the opposite of cherry-picking. Correcting the "+0.17" figure to the in-sample number it actually is deserves calling out too. Checked and confirmed:
One thing to fix, in this PRThe comment above the floor is now false, on both platforms: This PR is what introduces the zero entries, so the floor stops being inert and becomes load-bearing. Worse, your new docstring sends readers straight there — "See the header note on Two smaller notes, no action needed"Forbid" overstates what the code does, in a good way. With the floor, wake→deep/rem costs ≈ −20.7 against wake→light ≈ −2.3 — a ~18.4 log-unit penalty, not a prohibition. A strong enough emission can still cross it. That is the better behaviour: a genuine sleep-onset REM period stays representable rather than being structurally impossible, it just has to earn it. Might be worth a clause in the docstring, since "ZERO" reads as absolute. The new tests cannot actually detect the floor being removed. They recompute VerdictSound, and I would land it after the comment fix. The reason I am comfortable at n = 1 is that this component is not a fitted constant — "sleep onset descends through N1/N2" holds independently of your 36 nights, which is exactly what separates it from the six you reverted. The #437 guard is explicitly checked and does not reproduce (9.43 % → 9.96 %, against #348's 32.66 %). Worth stating plainly for the record: this changes staging for every user going forward, stored hypnograms are untouched, so timelines will mix recipes across the change — same as #348/#437 before it. Nothing to do, but it should be in the release notes rather than discovered. On the request for a second wearer: agreed that it is the binding constraint. Also happy to open your closing finding — REM rising as a fraction of sleep on longer nights (+0.579) — as its own issue, since it will get lost at the bottom of a merged PR. |
|
Correction: the first-REM latency row in the table above is wrong, and I am withdrawing it. That row claims first-REM latency MAE improves 53.9 → 41.6 min. Measured against human-scored PSG hypnograms instead of the strap's band state, the same change goes 45.1 → 49.9 min — it makes first-REM latency slightly worse. The row is unsupported and should not count toward landing this PR. Where the original figure came from is the reference set described above; why it disagrees with truth this sharply I have not established, and I am not going to guess. The instrument is Everything else here survives that instrument, and the case gets stronger rather than weaker. Same cohort, same run:
κ, REM F1 and wake sensitivity all move the direction the band comparison predicted, and the #437 stage-fraction guard holds against truth as well: wake tightens 4.34 → 4.15 % and deep does not move at all. So the physiological claim this PR actually rests on — sleep onset descends through N1/N2, so wake does not transition straight into N3 or REM — is now confirmed against a human-scored reference rather than only against another vendor's band. Net: one supporting row withdrawn, the main result independently confirmed on a better instrument than the one it shipped with. Happy to strike the row from the description before this lands — I have left the original table intact so this comment has something to point at. Your review notes are a separate matter and I will answer them there; this number seemed worth correcting before the merge rather than after. Two limits that apply to every figure above: sleep-accel carries no R-R, so |
… withdrawn latency figure Review fix for #987, both platforms. The comment above the ln floor read "Inert for the current matrix (no zero entries)" — a statement this PR is precisely what falsifies. The awake row now carries wake->deep = wake->rem = 0.0, so the floor is the only thing standing between those entries and ln(0) = -Inf, while the transition docstring sends readers to that note to learn "why a zero is safe" and they arrived at an assertion that zeros do not exist. Someone tidying dead code later had written permission to delete the one line holding this up. The comment now says LOAD-BEARING, names the two entries that depend on it, and gives the magnitude (-20.7 floored vs wake->light's -2.3). Same reason, in the transition docstring: ZERO reads as absolute and is not. Floored, wake->deep costs ~18.4 log-units more than wake->light — a penalty a strong enough emission can still cross, so a genuine sleep-onset REM period stays representable rather than structurally impossible. That is the intended behaviour and it was not written down. Also drops the "first-REM latency MAE 53.9 -> 41.6 min" clause from the same docstring. That figure is withdrawn: measured against human-scored PSG hypnograms rather than the strap's band state, the same change moves first-REM MAE 45.1 -> 49.9 min, i.e. slightly worse. Replaced with the figures that do reproduce on that instrument (PhysioNet sleep-accel, 31 subjects / 26 773 epochs): 4-class kappa 0.356 -> 0.363, REM F1 0.569 -> 0.575, wake sensitivity 30.42 % -> 30.84 %, #437 stage-fraction guard intact. Comments only; the matrix, the recipe and every test are untouched. Verification: StrandAnalytics `swift test` 1175 tests, 0 failures (the "1248" in the PR description does not reproduce and is corrected there). Android `:app:testFullDebugUnitTest --rerun-tasks` 3232 tests, 0 failures, 0 errors, 5 skipped, counted from the JUnit XML rather than the wrapper exit code — which returned 0 on a run that never found the SDK. SleepStagerV2Test 12/12, SleepStagerV2Tests 14/14 incl. testFrozenGoldenHypnogram, SleepStageTotalsTests 63/63. Both frozen goldens pass unchanged and their sources are byte-identical: the diff touches two files and only comment lines in them.
|
Comment fix pushed in The line now reads load-bearing rather than inert: it names Two things found while doing it, both worth flagging rather than quietly fixing. The withdrawn latency figure was in the shipped source, not just the PR body. The body's "1248 tests" does not reproduce. CI runs the identical command — |
…991) * Tools/SleepPSG: score the shipped sleep stager against PSG truth, reproducibly Every reference NOOP's sleep work can currently reach is either the recipe's own output handed back to it or another vendor's black box. Tools/SleepBench says so in its own header. A harness that scored SleepStagerV2 against real polysomnography existed once, produced the single most decisive result this project has had — that REM detection measures physiology and not the clock — and then vanished, because it lived in a session scratchpad. The next sleep investigation had to run on one instrument instead of two. This is that harness, in the repository, with a CI job. WHAT IT IS. `sleeppsg --dataset <path>` replays SleepStagerV2 over PhysioNet sleep-accel (Walch, Huang, Forger & Goldstein, SLEEP 42(12) zsz180, 2019) — 31 subjects of wrist accelerometer and heart rate recorded alongside human-scored PSG hypnograms — and scores it epoch-for-epoch. Read-only, dataset path always an argument, dataset never committed. README.md carries the download step and the attribution both the Open Data Commons licence and the MIT companion code require. THE PORT, AND WHY IT CANNOT DRIFT. Baseline numbers come from StrandAnalytics.SleepStagerV2.stageSession itself — the shipped file, no reimplementation. Variant numbers cannot: the recipe holds its constants as `static let`s and keeps Epoch/features() internal, so asking "what would this score with one transition row changed" needs the recipe restated with knobs. RecipePort.swift does that, and PortValidation measures the equivalence rather than asserting it: 48 randomised nights (off-grid starts, channel dropout, arousals, motion bursts, unsorted input, R-R present so the RSA term is exercised — the dataset cannot exercise it) plus 7 degenerate cases (all-HR-missing, zero-variance HR, 1- and 2-epoch nights, saturated motion gate, no coverage at all, motion absent). Every epoch label from both paths must agree. It runs in `swift test`, needs no dataset, and names the night and epoch when it fails. A failure there is never flakiness — it means RecipeConfig.shipped and SleepStagerV2.swift have diverged. WHAT IT REPORTS, AND WHY IN THAT SHAPE. - Stage fractions are a first-class result, not an appendix. Kappa does not constrain them: #348 improved kappa on all three of its benchmarks and #437 reverted it 48 h later for re-scoring a healthy night from 6 % to 23 % awake. A harness reporting only kappa could not have caught that, and ScoringTests pins a case where kappa rises while the wake fraction blows out, so the point survives as an assertion. - Splits are leave-one-SUBJECT-out, never epoch-level. Consecutive 30 s epochs from one night are nearly identical; a random epoch split lets any model memorise the subject. AblationTests fails if a held-out subject's rows can reach its own training fold. - Night length is a stratification variable, not a footnote. A REM-fraction-vs-duration coupling measured +0.579 on one wearer's nights; the report gives terciles and the same correlation for PSG TRUTH beside the prediction, because a coupling truth also carries is physiology and one only the prediction carries is an artefact. Wired into the `tools:` matrix that #943 added, alongside SleepBench and Backfill, so the equivalence check actually runs. Limits are stated in the README rather than buried: sleep-accel carries no beat-to-beat intervals, so the RSA respiration term is inert here (nil on every epoch, contributing exactly 0.0) and results are a lower bound on the same recipe with R-R present; the accelerometer is an Apple Watch's, not a WHOOP gravity decode, which tests rather than violates the recipe's night-relative motion thresholds; only staging is exercised, not session detection; and n = 31. * SleepPSG: print a self-check against the old harness's figures, and a variant that falsifies the easy answer The rebuilt harness has to be able to say whether it is the SAME instrument as the one it replaces, so section 3 now prints the previous harness's reported figures beside its own. Nothing is tuned to them; the point is to expose disagreement, not to close it. Every truth-side figure reproduces exactly — 31 subjects, 26 773 PSG-scored epochs, deep 14.76 % of night, truth median first-REM latency 88.5 min. Predicted deep lands at 19.24 % against a reported 19.25 %, and four-class kappa at 0.356 against 0.349. Three independent numbers from PR #348's OWN Walch column also reproduce, which is the stronger check because they were produced by a different (Python) implementation of the same recipe: that PR reported the pre-#348 incumbent at kappa 0.356 (measured here: 0.356), mean deep 19.2 % (19.24 %) and wake recall 33 % (30.4 %), and the full #348 build at deep 9.9 % (10.63 %). The REM prediction does not reproduce: 27.0 % of night against a reported 20.8 %, REM F1 0.569 against 0.515, median first-REM 91.5 min against 142.0. There was an obvious explanation and it is WRONG. #930 replaced a `c < 0.12 ? 3.0 : 0` step in the session -fraction domain with a graded penalty in minutes from sleep onset; the old figures were reported while #930 was the candidate, so the incumbent they describe predates it, and the graded guard is the weaker one averaged over the epochs it covers — which would produce exactly this "more REM, arriving earlier" pattern. `Variants.preNine30Guard` runs that guard. It moves nothing: kappa 0.356 -> 0.356, REM F1 0.569 -> 0.569, REM 27.02 % -> 26.97 %. On 8-hour lab nights the guard barely binds in either form. The hypothesis is falsified and is kept in the tree because the variant is what falsified it. Two constraints on whatever the residual turns out to be, both readable off the same table: it is not the session window (`clock` drives the deep prior and the REM ramp alike, and predicted deep reproduces to 0.01 pp), and it is not the latency guard. One coincidence recorded as an observation, not a conclusion: the reported "REM F1 0.515" is exactly this harness's measured REM PRECISION. Also here: - wake SENSITIVITY joins the variant table, because that is the quantity #987 was landed on against the strap's band state, and a claim you cannot read off the table is a claim you cannot check; - the variant CSV carries precision/recall/F1/fraction/bias for all four stages instead of three F1s; - the ablation fits IRLS over one flat design matrix per model instead of refiltering row structs and reallocating a feature array per row per iteration — same arithmetic, ~93x fewer allocations, which is the difference between a minute and an afternoon in an unoptimised build; - the dataset reader clips to the recipe's own read window during the read. Each subject's files span DAYS around the lab night (the earliest rows sit four days before lights-out) and stageSession clips to [start-330, end+390) before doing anything, so this drops only rows the recipe would discard itself; - second bucketing is a floor, not a round, so a sample at t = 0.9 s stays in second 0 rather than shifting the motion series against the heart-rate series by up to a second. 33 tests, 0 failures. Full report runs in 28 s on the 31-subject cohort. * SleepPSG: make the #987 comparison work from either side of the PR `RecipeConfig.shipped` has to describe `SleepStagerV2` as compiled on the branch it sits on — that is the whole basis of the port-validation check. The fork and upstream currently disagree about one row of the transition matrix: this repository's main carries PR #987 (wake→deep and wake→rem zeroed), upstream's does not until the PR merges. Hard-coding the variant as "apply #987" is therefore wrong on one of the two branches, and would have shipped a benchmark whose headline comparison is a no-op against itself. `Variants.pr987` now reads the shipped awake row and offers the OTHER one, naming itself accordingly ("#987 awake-transition row" upstream, "#987 REVERTED (pre-#987 row)" here). Same comparison, same pair of rows; only the sign of the delta and the label flip. The test asserts both directions rather than assuming one, so it passes wherever it is compiled, and it fails if the shipped row is neither side of #987 — which would mean `RecipeConfig.shipped` had gone stale for a different reason. `#348 entire` pins the awake row to the literal instead of to `pr987`, since #348's own row is the one #987 later restored and that is a fixed historical fact, not a branch-relative one. 33 tests, 0 failures. * SleepPSG: say which denominator every stage fraction uses The tool has always printed both conventions, labelled, one table above the other. The README quoted one of them in the self-check and the other in the variant section without ever saying so, which read as two different answers to the same question: section 3, pooled over all scored epochs deep truth 13.76 % predicted 18.94 % bias +5.18 pp section 3, mean of per-subject percentages deep truth 14.76 % predicted 19.24 % bias +4.48 pp Both are right and both are measured on the same 26 773 epochs. The denominator WITHIN a night is identical either way — all scored epochs, wake included. What differs is the weighting ACROSS the 31 subjects: pooled gives a 490-minute night more weight than a 208-minute one, the per-subject mean gives every subject one vote. For deep that is a ~1 pp gap, which is large enough to look like a contradiction and small enough to go unnoticed. The self-check is stated in the per-subject mean because that is the convention the previous harness reported in, and reproducing it on any other convention would not be reproducing it. Section 6's variant table and every `bias pp` column in it are pooled, which is why the incumbent's deep bias reads +5.18 pp there. Neither changes; both are now named at the point of use. Adds a convention table to the README, labels every percentage in the self-check section, and records the same thing on `Variants.preNine30Guard`, whose docstring quotes the per-subject figures. Also un-rounds "REM 27.0 %" to 26.96 % — 27.0 was ambiguous between the two conventions (26.96 per-subject, 27.02 pooled) precisely where the distinction matters. Documentation only: no measurement, no threshold and no reported number changes. Verified against the full 31-subject cohort locally; `swift test` in Tools/SleepPSG 33 tests, 0 failures.
ryanbr
left a comment
ryanbr
left a comment
There was a problem hiding this comment.
Re-reviewed. Everything I raised is addressed, and you found more than I asked for. Verified rather than taken on trust:
- The withdrawn latency figure is gone from the shipped source, not just struck in the body — zero occurrences of 53.9 / 41.6 anywhere in the diff. That it was baked into the docstring at all is the part worth noting: a PR body is read once, a docstring is read for years, so catching it there mattered more than the body edit.
- The floor comment now reads load-bearing. Naming
wake→deepandwake→remas the two entries that depend on it, with the magnitude, means the next person tidying "dead defensive code" hits a sentence telling them it is the only thing between those entries and-Inf. - The ZERO/prohibition wording is reconciled. Two comments in the same file disagreeing about whether a zero is absolute would have been worse than either wording alone.
- The whole-PR behavioural diff is still exactly four lines: two transition rows and two tests. Everything else is comments.
- 13/13 green,
MERGEABLE CLEAN.
On the withdrawal
Withdrawing your own headline improvement, unprompted, after building the instrument that contradicted it, is the strongest thing in this PR. And it does not sink the case: against PSG truth the change still moves κ 0.356 → 0.363, REM F1 0.569 → 0.575, wake sensitivity 30.42 → 30.84 %, with the #437 stage-fraction guard holding. Three metrics up, one down, the guard intact — and the physiological argument (sleep onset descends through N1/N2) never depended on any dataset, which is what separates this from the six components you reverted.
Correcting "1248 tests" to a measured 1175 is the same instinct applied to something nobody would ever have checked.
One small thing, and it is your own standard
The docstring lists the four things that improved and does not mention the one that got worse. First-REM latency MAE 45.1 → 49.9 against PSG is struck in the PR body, but the body lives on GitHub and the docstring lives in the file. Someone reading SleepStagerV2.swift next year sees an unbroken row of wins.
Your own validation protocol (#988, rule 8) is "a negative result is a result". One clause — that latency moves the wrong way against PSG and this landed anyway because the other four held — would make the record match the standard you wrote. Not blocking; approving either way, and happy for it to ride a later commit.
Approving. Land it whenever suits.
…d) (#997) #987 changed SleepStagerV2's awake transition row; #991's port-equivalence guard landed before it and correctly failed on main: RecipeConfig.shipped no longer reproduces SleepStagerV2 — 195 of 12377 epoch labels differ. This is the guard working, not a defect in it. #991 merged AFTER #987 was opened, so #987's own green CI never ran the SleepPSG job — its branch predates the tool. MERGEABLE CLEAN only ever meant no textual conflict, and there was none: the two files never touch. The coupling is semantic, which is exactly the kind a test has to catch because review will not. RecipePort's own comment already called this out — the awake row "is the one line that legitimately differs between branches" and RecipeConfig.shipped "must always describe SleepStagerV2 AS COMPILED on this branch". So the fix is the single line it predicted, and Variants.pr987 needs no edit: it reads the shipped row and offers whichever one is not there, so the #987 ablation keeps working from the other side.
….2.2/304 (#998) main has been on 9.2.1 for 42 commits. That was invisible until the testing build cut from main stamped 9.2.1 / versionCode 303, and Android refused to install it over the 29 July staging build (9.2.2 / versionCode 304) — a downgrade, surfaced as the unhelpful "App not installed". versionCode goes to 305 rather than 304 because 304 is already published on the rolling testing-latest tag; reusing it would leave two different builds sharing a code. MARKETING_VERSION and versionName move together per CLAUDE.md. No release notes here — this is the version bump only, and 9.2.2's notes still owe users the user-visible changes from the last 42 commits (Effort shifts from #963 and #992, the new Apple Health permission prompt, the Oura resting-HR heal that needs a re-sync, and the sleep-staging change from #987).
…#979) Stored hypnograms carry both 'wake' and 'awake'. Five segment comparisons on EACH platform recognised only one, and they are the same five sites — the reimplementation reproduced the bug faithfully: SleepStager 1075/2193 <-> SleepStager.kt 1157/2378 WakeMotionRefinement 188 <-> WakeMotionRefinement.kt 223 WearableImporter 134 <-> WearableExportImporter.kt 788 XiaomiImporter 136 <-> XiaomiBandImporter.kt 335 The damaging shape is `stage != "wake"` used to mean asleep: an imported 'awake' segment falls through it and is counted as SLEEP, inflating the sleep efficiency figure. The mirror shape `stage == "wake"` under-counts wake time, and made #987's wake refinement skip those segments entirely. Six other sites already defended with `case "wake", "awake"`, which is what makes this a missing shared rule rather than a missing idea. Two vocabularies exist by design and neither moves here: segment strings canonicalise to 'wake', minutes-dictionary keys to 'awake'. The bug is the dictionary vocabulary reaching a segment comparison, which happens because imports do not pass through SleepStagerV2 — Oura's phase table is [deep, light, rem, awake]. So this is a PREDICATE, not a canonicaliser: it fixes the comparisons without rewriting any stored string, and no persisted hypnogram changes meaning. SleepStagerV2's own "awake" tests are deliberately untouched; that is its internal model vocabulary, renamed to 'wake' on output. Android's UI canonicalStage() now folds through the shared predicate so the alias rule has one definition. It still returns 'awake', because that is the key its colour table and the minutes dictionaries use. 6 twin tests per platform, same cases in the same order.
…#979) (#1011) * Sleep stages: recognise both spellings of wake in segment comparisons (#979) Stored hypnograms carry both 'wake' and 'awake'. Five segment comparisons on EACH platform recognised only one, and they are the same five sites — the reimplementation reproduced the bug faithfully: SleepStager 1075/2193 <-> SleepStager.kt 1157/2378 WakeMotionRefinement 188 <-> WakeMotionRefinement.kt 223 WearableImporter 134 <-> WearableExportImporter.kt 788 XiaomiImporter 136 <-> XiaomiBandImporter.kt 335 The damaging shape is `stage != "wake"` used to mean asleep: an imported 'awake' segment falls through it and is counted as SLEEP, inflating the sleep efficiency figure. The mirror shape `stage == "wake"` under-counts wake time, and made #987's wake refinement skip those segments entirely. Six other sites already defended with `case "wake", "awake"`, which is what makes this a missing shared rule rather than a missing idea. Two vocabularies exist by design and neither moves here: segment strings canonicalise to 'wake', minutes-dictionary keys to 'awake'. The bug is the dictionary vocabulary reaching a segment comparison, which happens because imports do not pass through SleepStagerV2 — Oura's phase table is [deep, light, rem, awake]. So this is a PREDICATE, not a canonicaliser: it fixes the comparisons without rewriting any stored string, and no persisted hypnogram changes meaning. SleepStagerV2's own "awake" tests are deliberately untouched; that is its internal model vocabulary, renamed to 'wake' on output. Android's UI canonicalStage() now folds through the shared predicate so the alias rule has one definition. It still returns 'awake', because that is the key its colour table and the minutes dictionaries use. 6 twin tests per platform, same cases in the same order. * Add an integration test per platform, so a reverted call site fails The six predicate tests pass whether or not the five call sites were actually changed - they exercise the rule, not its users. Nothing in the suite noticed a reverted site, which for a latent bug is the only way a regression would ever surface. Each platform now also drives a real caller: SleepStager.hypnogramMetrics over the same night as the existing AASM test, with the WASO segment spelled 'awake'. tst comes from a positive list (light/deep/rem) so it is immune either way at 1080 s; WASO and the disturbance count are not, and read 0/0 before the fix. Verified the two implementations agree before asserting: onset, sptEnd and the clip arithmetic are identical in Swift and Kotlin, so both expect waso 60 and disturbances 1.
2 participants
Re-litigates the #437 revert of #348 component-by-component, and lands the one component of seven that survives measurement on a de-contaminated reference set.
#348 re-tuned seven things about
SleepStagerV2on DREAMT. #437 reverted all seven 48 h later because one healthy night went from 6 % to 23 % awake — "kappa doesn't guard stage-fraction calibration." That revert was correct about the outcome. This PR asks the narrower question it did not have the tooling to ask at the time: was every one of the seven components guilty?Six were. One was not.
This row is a physiological claim rather than a fitted constant: sleep onset descends through N1/N2, so wake does not transition straight into N3 or REM, and the freed mass makes a WASO episode span several epochs instead of flickering back to sleep after one. The
ln(max(v, 1e-9))viterbi floor that makes a zeroed entry safe already landed with #348 and survived #437, so nothing else is needed to support it.How this was measured
Tools/SleepBenchover one wearer's 36 recorded nights, using the--excludeflag and theE.-1self-comparison audit from #935.The reference-set contamination is worse than assumed. On this database the audit reports 10 of 15 stage-locked nights are byte-exact V2 replays and 3 more sit at ≥ 95 % — 13 of 15 cannot serve as an independent reference for the recipe that produced them. They hand the incumbent free perfect scores and charge every alternative. Any before/after on the full 15 is biased toward "the incumbent is fine": the incumbent scores 4-class κ 0.978 there, which is a measurement of V2 against itself.
So the load-bearing instrument here is not the hand labels. It is the strap's own band
sleep_state(the v18 @81 nibble) — WHOOP's verdict, not a re-derivation of NOOP's, and something the recipe cannot contaminate. 21 banded nights, 15 554 epochs, and the night set is fixed across every build compared.first-REM latency MAE53.9 min41.6 min— withdrawn, see belowThe first-REM latency row is struck and does not count toward landing this PR. Measured against human-scored PSG hypnograms instead of the strap's band state, the same change goes 45.1 → 49.9 min — slightly worse, a reversal, not an improvement. It is struck rather than deleted so the correction comment still has something to point at. Treat every band-derived first-REM latency figure in this description as unsupported, including the 53.9 → 32.1 min quoted for the emission coefficients below; that component is rejected on other grounds and none of the reasoning here depends on a latency number.
Everything else survives the PSG instrument, and the case gets stronger.
Tools/SleepPSG(#991) replays the shippedSleepStagerV2.stageSessionover PhysioNet sleep-accel — 31 subjects, 26 773 human-scored epochs. Same cohort, same run: 4-class κ 0.356 → 0.363, REM F1 0.569 → 0.575, wake sensitivity 30.42 % → 30.84 %, and the #437 stage-fraction guard holds against truth as well — wake tightens 4.34 % → 4.15 % and deep does not move (18.94 % → 18.94 %, pooled over all 26 773 scored epochs). So the physiological claim this PR actually rests on is now confirmed against a human-scored reference and not only against another vendor's band. That instrument was built after this PR was opened, which is why the struck row shipped unchallenged.For scale on the guard column: #348 in its entirety puts healthy-stratum wake at 32.66 %. This change puts it at 9.96 %.
On the 2 genuinely independent hand-labelled nights that remain after de-contamination, 4-class κ goes 0.857 → 0.863 — a tie at n = 2, reported for completeness rather than as evidence.
Why the other six stay reverted
Each measured alone, against the same band reference and the same wake guard:
The motion-gate row is the one worth dwelling on. It is tempting to read #348 as "the prior did the damage, the physiology was fine" — that is not what the data says. The prior and the motion gate are two independent wake channels, each roughly doubling the wake fraction on its own; removing only the prior still leaves healthy wake at 16.03 %. #437 was right to take both.
The emission coefficients are the closest call: they cut healthy deep 22.09 % → 15.30 % and first-REM latency MAE 53.9 → 32.1 min. But they cost band κ (0.105 → 0.094), cost the 2 clean hand-labelled nights (κ 0.857 → 0.658), and worsen the REM/night-length coupling below. Their deep correction is only an improvement if a population PSG deep fraction is trusted over this wearer's own labels, which disagree by ~6 pp. Not landed.
What this PR is not entitled to claim
Request: a second labelled wearer. The honest ceiling on this line of work is one person's nights. If anyone else has hand-corrected hypnograms — especially on a 4.0, or from a sleeper whose architecture differs — a second subject would do more for
SleepStagerV2than any further re-tune of these constants. Happy to share the exact SleepBench invocation.Separately found, not fixed here
SleepStagerV2emits more REM as a fraction of sleep on longer nights: corr(total sleep, REM %) = +0.579 Pearson / +0.501 Spearman over 36 nights, with nights ≥ 9 h running 37.3 % REM against only 37.7 % light, which is not physiological.The mechanism was tested, not guessed. Two diagnostic builds (neither shipped):
remLatencyGuard→ corr collapses +0.579 → +0.241, long-vs-short spread +5.90 pp → +0.55 pp1.0 * cREM ramp in the minute domain → corr worsens to +0.679, spread +14.28 ppSo the
1.0 * cramp is exonerated; the coupling comes from the guard. It is a fixed 60-minute, 3.0-log-odds penalty, so it covers ~36 % of a 166-minute night and ~9 % of a 654-minute one, and proportional REM suppression therefore scales inversely with night length. This is a side effect of #930's (correct) move of the guard from the fraction domain to the minute domain — that PR's premise about latency is well-evidenced and should stand; what was not measured was the effect on REM fraction. Worth its own change, deliberately, rather than folding into this one. This PR is neutral-to-slightly-positive on it (+0.579 → +0.563).Review fix, second commit
bb50f85takes the one change asked for, on both platforms. The floor comment read "Inert for the current matrix (no zero entries)" — the statement this PR falsifies — while the transition docstring sent readers there to learn "why a zero is safe". It now says LOAD-BEARING, nameswake→deepandwake→remas the two entries that depend on it, and gives the magnitude (−20.7 floored againstwake→light's −2.3), so deleting it reads as breaking the stager rather than as tidying dead code.Also taken, since "ZERO" does read as absolute: the docstring now states that the floored zero is a ~18.4 log-unit penalty a strong enough emission can still cross, so a genuine sleep-onset REM period stays representable rather than structurally impossible. And the withdrawn latency figure is gone from that docstring too — it was quoted there as well, replaced by the PSG figures that do reproduce.
Comments only. Two files, no non-comment line touched.
Verification
swift test(StrandAnalytics): 1175 tests, 0 failures — the earlier "1248" in this description does not reproduce and is corrected here;SleepStagerV2Tests14/14 incl.testFrozenGoldenHypnogram,SleepStageTotalsTests63/63./gradlew :app:testFullDebugUnitTest --rerun-tasks: 3232 tests, 0 failures, 0 errors (5 skipped; counted from the JUnit XML, not the wrapper exit code — which returned 0 on a run that never located the SDK),SleepStagerV2Test12/12Refs #348, #437, #935, #930, #991.