sleep: score stages instead of AND-ing gates, drop RMSSD - #34
Merged
Conversation
Scored the stager against DREAMT (99 PSG-labelled wrist nights) via a new harness and the old rules came out at kappa 0.036 — deep PPV 5.7% against a 4.5% base rate, REM PPV 12.1% against 14.0%, so basically chance on both. Per-subject effect sizes say why: axis Deep vs Light REM vs Light what we did with it Rk d -0.53 d +0.43 throttled to z>4, ~never fired hrSd d -0.43 d +0.27 deep only sdnn d -0.41 d +0.32 didn't compute it lfhf d -0.33 d +0.12 throttled to z>4 hr d +0.31 d +0.18 deep gate, wrong direction rmssd d -0.13 d -0.02 REM PRIMARY So the deep rule was AND-ing one good gate, one null gate and one inverted gate. Those only line up by luck, which is why deep came out as 30-second specks that the 3-min bout rule then deleted. Now: weighted sums of robust-z axes, weights = the measured d values. Dropped rmssd and mean HR from both scores. Added sdnn. Atonia + HR floor stay as gates on REM since they're preconditions, not evidence. kappa 0.036 -> 0.121 (0.101 on 49 held-out subjects). Deep 10.3/5.7 -> 55.0/12.5 sens/PPV, REM 30.6/12.1 -> 59.2/19.7. Cutoffs are NOT the kappa optimum. kappa keeps climbing as you suppress deep (deepCut 1.0 -> 0.2% deep, kappa 0.158) because DREAMT is a clinic cohort with only 4.5% deep. Shipping that gives users ~0 deep sleep again. Picked for normative proportions instead. Also had to calibrate cutoffs on our own captures, not DREAMT — the DREAMT-optimal remCut 0.8 gives 23% REM there but 10.3% on WHOOP, and failed the real-night regression test (62 min REM vs ~162 expected). Weights travel, cutoffs don't. Split cardioStager into feature extraction + classifyCardioEpochs so the rules can be scored on their own. No behaviour change from the split itself. Caveats: one cohort, one sensor, DREAMT's feature extraction not ours, deep rests on 19-29 subjects, no PSG on our own device. Proportion matching says the hypnogram shape is believable, not that any epoch is right. 386 tests green.
📝 WalkthroughWalkthroughThe cardio stager now uses weighted robust-z scores over SDNN, HR dispersion, LF/HF, and R(k). It exposes feature extraction and classification APIs. Two command-line tools validate cutoffs and calculate sleep-stage proportions from fixture and night data. ChangesCardio staging and validation
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant cardioStager
participant CardioEpochFeatures
participant classifyCardioEpochs
Caller->>cardioStager: Provide HR, accelerometer, RR data, and score cutoffs
cardioStager->>CardioEpochFeatures: Extract epoch features
cardioStager->>classifyCardioEpochs: Pass feature arrays and cutoffs
classifyCardioEpochs->>Caller: Return sleep-stage classifications
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 14
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/src/onehz/sleep/cardio_stager.dart`:
- Around line 604-611: The hrSd baseline and scoring paths need the same
validity gates as the other axes. In lib/src/onehz/sleep/cardio_stager.dart
lines 604-611, update the hrSdScale sample to include only still(e) && hrSd[e] >
0 values; in lines 665-670, make hrSdZ return null when the gated sample has
fewer than four entries or hrSd[e] is NaN or zero. Re-run the cutoff sweep in
tool/whoop_proportions.dart after both changes.
- Around line 604-611: The hrSd baseline currently includes non-sleep epochs
unlike the other stage-score axes. Update the hrSdScale sample construction in
the surrounding staging logic to include only values where still(e) is true and
hrSd[e] is positive, while leaving the existing robust scaling and downstream
hrSdZ usage unchanged.
- Around line 429-464: Update classifyCardioEpochs to validate that motion, hr,
hrSd, rmssd, lfhf, rk, and sdnn all have the same length before deriving nEpoch
or indexing any list. Report the contract violation explicitly in a release-safe
check rather than relying on assertions, while preserving normal classification
when all lengths match.
In `@tool/stager_harness.dart`:
- Around line 320-337: Extract the shared subject label mapping into a helper
such as _labelSubject, returning aligned truth and prediction index lists using
the existing SleepStage/deepFlag mapping. Replace the duplicated builders in
_scoreSet, _deepMetrics, and the main loop with calls to this helper, preserving
the main loop’s per-subject 30-epoch limit where required.
- Around line 227-238: After the grid search in the sweep logic, check whether
best remains below zero and report the failed sweep to stderr instead of
printing the seeded bestR, bestD, and kappa values. Preserve the existing
optimization behavior when at least one grid point produces a valid score.
- Around line 202-210: The _featuresOf function must validate that every feature
key, including sdnn, exists, contains a list, and has the same length as motion
before constructing CardioEpochFeatures; preserve clear fixture errors instead
of allowing _nums to throw. In tool/stager_harness.dart lines 27-32, update the
documented fixture schema to include "sdnn":[..] and correct the flag list on
line 28.
- Around line 129-142: Update the stage-processing loop around kClasses.indexOf
so unrecognised labels are counted rather than silently skipped, while
preserving the existing handling of recognised stages. After processing the run,
report the accumulated unknown-label counts to stderr once, including each label
and its count, before the tool exits or prints the final scoring summary.
- Around line 88-89: Expose the library’s _remScoreCut and _deepScoreCut
defaults from classifyCardioEpochs as public constants, then update the harness
argOf defaults and shipped-cutoff reporting to reference those constants instead
of hardcoded 0.3 values. Preserve the existing override behavior for --rem-cut
and --deep-cut.
- Around line 166-185: Guard the summary block when perSubjectKappa is empty:
print the subject-count line indicating that zero subjects were scored, then
return before sorting, percentile calculation, reduce, or percentage division.
Also update _report to avoid computing correct / t.length for an empty report
and emit the established no-scores outcome instead of NaN.
- Around line 27-32: Update the usage block to list the implemented
flags—--sleep-window, --sweep, --deep-curve, --dev, --holdout, --rem-cut <v>,
and --deep-cut <v>—and remove the unsupported --profile flag. Expand the fixture
schema description to include the sdnn field, preserving the existing
null-means-not-measurable guidance.
- Around line 90-101: Apply the existing split filter before the --deep-curve
branch so _deepMetrics receives only the development subjects during the deepCut
sweep. Remove the duplicated split-filter block that follows the branch, while
preserving the curve output and early return behavior.
- Around line 194-199: Update _splitOf to select the split using a high bit of
the completed FNV-1a hash instead of h.isEven, preserving the existing
_Split.dev and _Split.holdout outcomes while avoiding correlation with
sequential subject identifiers. Recalculate any documented dev/holdout counts
affected by the repartition.
In `@tool/whoop_proportions.dart`:
- Around line 28-32: Update the file selection flow in the `files` declaration
to remove the hard-coded `2026-` path condition, allowing all JSON files in the
configured directory to be included. Keep the existing `File`-type and `.json`
extension filters unchanged.
- Around line 41-66: Update the accelerometer initialization in the
stream-building loop around `AccelSample` so every missing sample starts with
its own timestamp (`ts * 1000.0`) and `valid: false`. Preserve observed samples
as valid and only mark gap-filled samples valid when they are within the
existing 120-second carry-forward window, ensuring pre-observation and long-gap
slots remain invalid.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 4e29e913-d80b-4a92-b6b1-84bb01c79bfc
📒 Files selected for processing (3)
lib/src/onehz/sleep/cardio_stager.darttool/stager_harness.darttool/whoop_proportions.dart
Comment on lines
+227
to
+238
| var best = -1.0, bestR = 0.3, bestD = 0.3; | ||
| final cuts = [0.0, 0.2, 0.4, 0.6, 0.8, 1.0, 1.3, 1.6, 2.0]; | ||
| for (final r in cuts) { | ||
| for (final d in cuts) { | ||
| final k = _scoreSet(dev, epochSec, r, d); | ||
| if (k > best) { | ||
| best = k; | ||
| bestR = r; | ||
| bestD = d; | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Report a failed sweep instead of printing kappa=-1.000.
_scoreSet returns _kappa, which returns NaN when the epoch list is empty or the agreement is degenerate. NaN > best is always false, so a NaN grid point never updates best. If every grid point returns NaN, best stays at -1.0 and bestR/bestD stay at the seed values 0.3. Lines 250-253 then print dev optimum remCut=0.3 deepCut=0.3 DEV kappa=-1.000, which reads as a real but very poor optimum rather than as a sweep that scored nothing.
Detect best < 0 after the grid and write a diagnostic to stderr.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tool/stager_harness.dart` around lines 227 - 238, After the grid search in
the sweep logic, check whether best remains below zero and report the failed
sweep to stderr instead of printing the seeded bestR, bestD, and kappa values.
Preserve the existing optimization behavior when at least one grid point
produces a valid score.
Critical one was real. The new hrSd axis skipped the sample-selection every other axis gets: - hrSdScale pooled ALL epochs, not just still(e), so wake epochs raised the median and widened the MAD. hrSd carries the second-largest deep weight (0.43) so that alone shifts stage proportions. - no >=4 sample floor, and no rejection of hrSd == 0. That's the "fewer than 2 valid HR samples" sentinel, not a real zero, and it z-scores very negative, i.e. scores as evidence FOR deep sleep. One real capture had 226 of 1120 epochs (20%) with no HR, every one of them scoring toward deep. Gappy-wear users, which is exactly who we're trying to help. Fixed both. Proportions actually improved: 0.5/0.3 now gives REM 23.6% / Deep 14.9%, both centred in the normative bands. Cutoffs unchanged. Also the harness was hardcoding 0.3/0.3 as its defaults while the library shipped 0.5/0.3, so every number I reported was measured at the wrong cutoffs. Now reads kDefaultRemScoreCut/kDefaultDeepScoreCut from the library and prints which it used. Corrected numbers: kappa 0.128 all / 0.132 holdout (was reporting 0.121 / 0.101). Other tool fixes: - --deep-curve was sweeping over holdout subjects, now dev-only - crash when no subject scored - split hashed on the low bit, which for S001/S002/... degenerates to alternating by last character. Uses a middle bit now. - fixture schema validated up front instead of producing a confident wrong kappa - extracted the truth/prediction builder, it existed three times - whoop_proportions matched files by a hardcoded '2026-', now matches by record shape - whoop_proportions filled gaps with a fake (0,0,1) sample at a stale timestamp; now tracks usable runs and stages each separately like production Length invariant on CardioEpochFeatures asserted in dev, and length() returns the shortest list so a bad caller truncates instead of RangeError. 386 tests green.
Contributor
Author
|
Went through all 14. Verified each against the code before touching anything — most were right, and one was a genuinely important catch. the critical one — real, and worse than describedThe Measured it: one of our real captures has 226 of 1120 epochs (20%) with no HR, every one of them scoring toward deep. The other ten nights have ~0%. So this specifically hit users with gappy wear — precisely the bad-tail population this whole PR is about. Good catch. Both parts fixed. Proportions got better: at 0.5/0.3 we now get REM 23.6% / Deep 14.9%, both centred in the normative bands rather than at the edge. Cutoffs unchanged. the harness default drift — this invalidated my reported numbers
Corrected numbers:
Better than I claimed, but I claimed it wrong. Description and the source doc comment are both corrected. restAll valid, all fixed:
one I did differentlyFor the 386 tests green, analyze clean. 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Built a harness that runs the real stager against DREAMT (99 PSG-labelled wrist nights). Our rules scored kappa 0.036. Deep PPV 5.7% against a 4.5% base rate, REM PPV 12.1% against 14.0% — chance, basically.
Checked each axis per-subject to see why:
So the deep rule AND-ed one good gate, one null gate and one inverted gate. Those only line up by luck — which is why deep sleep came out as 30-second specks that the 3-min bout rule then deleted. The "needs smoothing" theory was treating the symptom.
what changed
Weighted sums of robust-z axes instead of boolean conjunctions. Weights are the measured d values, not tuned. Dropped rmssd and mean HR from both scores, added sdnn. Atonia + HR floor stay as gates on REM (preconditions, not evidence).
Also split
cardioStagerinto feature extraction +classifyCardioEpochsso the rules can be scored on their own. Split itself is behaviour-neutral.numbers
Base rates are 4.5% Deep and 14% REM, so both minority classes now land well above chance where they used to sit at or below it.
Still miles off the 0.60-0.66 literature ceiling. Most of what's left is the wake rule (11% sens), which I didn't touch.
two things worth arguing about
Cutoffs are not the kappa optimum, on purpose. kappa keeps climbing as you suppress deep — deepCut 1.0 gives 0.3% deep and kappa 0.151 on the dev split. That's arithmetic on a clinic cohort where deep is only 4.5% of epochs, not physiology. Shipping it means users see ~0 deep sleep again, which is the bug we're fixing. Picked for normative proportions (REM 20-25%, Deep 13-23% of TST) and reported the kappa that falls out.
Cutoffs are calibrated on our own captures, not DREAMT. The DREAMT-optimal remCut 0.8 gives ~23% REM there and ~10% on WHOOP, and blew up the real-night regression test (62 min REM vs ~162 expected). Weights transfer, cutoffs don't — different sensor, different feature spread. Swept on 11 real nights instead (
tool/whoop_proportions.dart): 0.5/0.3 gives REM 23.6% / Deep 14.9%.caveats
One cohort, one sensor, DREAMT's feature extraction rather than ours, deep resting on 19-29 subjects, and no PSG on our own device at all. Proportion matching says the hypnogram has a believable shape, not that any epoch is right. All written into the source comments too.
Needs a kAlgoVersion bump in edge when it repins.
386 tests green, analyze clean.
🤖 Generated with Claude Code