diag(hrv): shadow-log a deduped R-R stream to validate the over-count fix (#1331/#1008/#1118) by ryanbr · Pull Request #1352 · ryanbr/noop

ryanbr · 2026-08-15T13:37:17Z

Stage 1 of the WHOOP 4.0 over-count fix — instrumentation only, no behaviour change. Shadow-logs a candidate de-dup so it can be validated against ground truth before it ever becomes the read path.

Why

The WHOOP 4.0 R-R over-count (#1008 — each beat emitted 2–3×: an exact duplicate plus a two-optical-channel twin ~34 ms off) is behind three symptoms: HRV ≠ WHOOP (#1008), the HRV "unverified" caveat (#1118), and — via #1127's RSA beat-accuracy gate — the respiratory blank on 4.0 since V10 (#1331). The real fix is a de-dup, but it changes a physiological signal (HRV + respiration), so per the "validate against the artifact, not one match" rule it can't be shipped on a guess.

What (this PR)

HRVAnalyzer.collapseOverCount / HrvAnalyzer.collapseOverCount — a pure, byte-identical de-dup: a same-second beat within ~40 ms of one already kept (the exact dups + the ~34 ms channel twins) is dropped, keeping one representative. Returns the deduped (tsSec, rrMs).
On an over-count night only, the always-on hrv diag appends a hrv dedup … line with the deduped stream's rmssd / sdnn / meanNN / coverage / beatAccurate, beside the raw values already logged. The shipped HRV/resp path is unchanged — this only logs a candidate.

What it buys us

A strap log now shows, per 4.0 over-count night, whether de-dup:

drives coverage → ~1.0 and beatAccurate high — i.e. it would pass fix(oura): gate RSA respiration to no-data when the R-R stream is banked #1127's RSA gate, so respiratory rate returns (the [master] Bug report #1331 fix);
brings rmssd/sdnn toward physiological and matching WHOOP.

That's the exact data to validate against WHOOP's own numbers and @artemc's Polar H10 dataset (#1118) — across his over-count and clean nights — before Stage 2 flips the read path to the deduped stream (which closes #1008 + #1118 + #1331 together).

Verification

New HrvCollapseOverCount{Test,Tests} (both platforms): a synthetic 3× over-count collapses 180→60 beats, coverage ~3× → ~1.0, and a clean stream passes through untouched. Kotlin green locally.
StrandAnalytics via swift-packages; the IntelligenceEngine shadow line is app-target → app-build gate.

Refs #1331, #1008, #1118, #1127. Instrumentation-first, mirroring #1117/#1332/#1344.

@artemc

… fix (#1331/#1008/#1118) The WHOOP 4.0 R-R over-count (#1008 — each beat emitted 2-3x, an exact dup plus a two-optical-channel twin ~34ms off) corrupts HRV (#1008/#1118) AND, via #1127's RSA beat-accuracy gate, blanks respiratory rate on 4.0 since V10 (#1331). The fix is a de-dup, but it changes a physiological signal, so it must be validated against ground truth before it becomes the read path. Stage 1 (this PR) — instrument, do not change behaviour: - HRVAnalyzer/HrvAnalyzer.collapseOverCount: pure, byte-identical de-dup that collapses same-second beats within ~40ms of one already kept (the exact dups + the ~34ms channel twins). Returns the deduped (tsSec, rrMs). - On an over-count night the always-on hrv diag appends a `hrv dedup` line with the DEDUPED stream's rmssd/sdnn/meanNN/coverage/beatAccurate beside the raw. Shipped HRV/resp is UNCHANGED — this only logs a candidate. So a strap log now shows, per 4.0 over-count night, whether de-dup drives coverage→~1.0 and beatAccurate high (which would pass #1127's RSA gate → resp returns, the #1331 fix) and rmssd/sdnn toward physiological — validated against WHOOP's own numbers and @artemc's Polar H10 (#1118) before Stage 2 flips it live. Tests: HrvCollapseOverCount{Test,Tests} pin a synthetic 3x over-count collapsing 180→60 beats, coverage ~3x→~1.0, clean streams untouched. compileFullDebugKotlin + the Kotlin test green; StrandAnalytics via swift-packages; IntelligenceEngine shadow is app-target → app-build gate. Refs #1331, #1008, #1118, #1127.

…te (#1331) Re-review caught that the ~40ms same-second collapse can over-merge two REAL neighbouring beats whose R-R values sit within 40ms (normal at sleeping HR), so its coverage undershoots 1.0 even when de-dup is working — confounding the validation signal. Add the EXACT-duplicate collapse (rrTolMs=0, same ts AND value, provably no real-beat loss) as a safe reference line, so the shadow logs the honest spectrum: covExact (safe floor, the unambiguous exact-dup magnitude) alongside cov40/beatAcc40/rmssd40 (aggressive upper bound). The real de-dup sits between them. Test pins the exact-dup floor (180->120, coverage ~2x, twins kept).

… between doc + fn) doc_comment_lint flagged HrvAnalyzer.kt:503 — collapseOverCount landed between collapsedCoverage's #550 KDoc and its declaration, detaching the doc. Move collapseOverCount above the #550 doc so it reattaches. No code change.

@artemc

… fix (ryanbr#1331/ryanbr#1008/ryanbr#1118) (ryanbr#1352) * diag(hrv): shadow-log a deduped R-R stream to validate the over-count fix (ryanbr#1331/ryanbr#1008/ryanbr#1118) The WHOOP 4.0 R-R over-count (ryanbr#1008 — each beat emitted 2-3x, an exact dup plus a two-optical-channel twin ~34ms off) corrupts HRV (ryanbr#1008/ryanbr#1118) AND, via ryanbr#1127's RSA beat-accuracy gate, blanks respiratory rate on 4.0 since V10 (ryanbr#1331). The fix is a de-dup, but it changes a physiological signal, so it must be validated against ground truth before it becomes the read path. Stage 1 (this PR) — instrument, do not change behaviour: - HRVAnalyzer/HrvAnalyzer.collapseOverCount: pure, byte-identical de-dup that collapses same-second beats within ~40ms of one already kept (the exact dups + the ~34ms channel twins). Returns the deduped (tsSec, rrMs). - On an over-count night the always-on hrv diag appends a `hrv dedup` line with the DEDUPED stream's rmssd/sdnn/meanNN/coverage/beatAccurate beside the raw. Shipped HRV/resp is UNCHANGED — this only logs a candidate. So a strap log now shows, per 4.0 over-count night, whether de-dup drives coverage→~1.0 and beatAccurate high (which would pass ryanbr#1127's RSA gate → resp returns, the ryanbr#1331 fix) and rmssd/sdnn toward physiological — validated against WHOOP's own numbers and @artemc's Polar H10 (ryanbr#1118) before Stage 2 flips it live. Tests: HrvCollapseOverCount{Test,Tests} pin a synthetic 3x over-count collapsing 180→60 beats, coverage ~3x→~1.0, clean streams untouched. compileFullDebugKotlin + the Kotlin test green; StrandAnalytics via swift-packages; IntelligenceEngine shadow is app-target → app-build gate. Refs ryanbr#1331, ryanbr#1008, ryanbr#1118, ryanbr#1127. * diag(hrv): log an exact-dup reference beside the ~40ms de-dup candidate (ryanbr#1331) Re-review caught that the ~40ms same-second collapse can over-merge two REAL neighbouring beats whose R-R values sit within 40ms (normal at sleeping HR), so its coverage undershoots 1.0 even when de-dup is working — confounding the validation signal. Add the EXACT-duplicate collapse (rrTolMs=0, same ts AND value, provably no real-beat loss) as a safe reference line, so the shadow logs the honest spectrum: covExact (safe floor, the unambiguous exact-dup magnitude) alongside cov40/beatAcc40/rmssd40 (aggressive upper bound). The real de-dup sits between them. Test pins the exact-dup floor (180->120, coverage ~2x, twins kept). * fixup: reattach collapsedCoverage doc (collapseOverCount was inserted between doc + fn) doc_comment_lint flagged HrvAnalyzer.kt:503 — collapseOverCount landed between collapsedCoverage's ryanbr#550 KDoc and its declaration, detaching the doc. Move collapseOverCount above the ryanbr#550 doc so it reattaches. No code change.

@artemc

…w candidate (#1331/#1118) The #1331 4.0 logs read `crossSecondOverCount` every night: the same-second shadow collapses (#1352's exact + 40 ms candidates) can't reach the duplicate twins because they straddle the second boundary, so `cov40` stays ~1.7-2.0 on the heavy nights and RSA-respiration stays blanked by #1127's beat-accuracy gate. Add a THIRD shadow candidate, `xsec` — the 40 ms collapse widened to a 1-second window — logged beside the others so we can SIZE how much of a night is cross-second (does coverage fall to ~1.0, does beat-accuracy clear the 0.5 gate?). `collapseOverCount` gains a `windowSec` param (default 0 = same-second, byte- identical for every existing caller). Instrumentation ONLY — the shipped HRV/resp path is unchanged. The window is a strict UPPER BOUND, deliberately NOT a shippable de-dup: a steady real HR has near-identical intervals one second apart, so a cross-second collapse over-merges real beats (the test pins this: a 10-beat steady stream drops to 5 at window=1). The real fix is density/timeline-based and must be validated against ground truth (@artemc's H10) before it becomes the read path — this just quantifies the target. Both platforms (byte-identical `hrv dedup` line + twin tests).

@artemc

…w candidate (#1331/#1118) (#1399) The #1331 4.0 logs read `crossSecondOverCount` every night: the same-second shadow collapses (#1352's exact + 40 ms candidates) can't reach the duplicate twins because they straddle the second boundary, so `cov40` stays ~1.7-2.0 on the heavy nights and RSA-respiration stays blanked by #1127's beat-accuracy gate. Add a THIRD shadow candidate, `xsec` — the 40 ms collapse widened to a 1-second window — logged beside the others so we can SIZE how much of a night is cross-second (does coverage fall to ~1.0, does beat-accuracy clear the 0.5 gate?). `collapseOverCount` gains a `windowSec` param (default 0 = same-second, byte- identical for every existing caller). Instrumentation ONLY — the shipped HRV/resp path is unchanged. The window is a strict UPPER BOUND, deliberately NOT a shippable de-dup: a steady real HR has near-identical intervals one second apart, so a cross-second collapse over-merges real beats (the test pins this: a 10-beat steady stream drops to 5 at window=1). The real fix is density/timeline-based and must be validated against ground truth (@artemc's H10) before it becomes the read path — this just quantifies the target. Both platforms (byte-identical `hrv dedup` line + twin tests).

ryanbr added 3 commits August 16, 2026 01:36

ryanbr merged commit 6418618 into main Aug 15, 2026
14 checks passed

ryanbr deleted the 1331-overcount-dedup-shadow branch August 15, 2026 14:08

This was referenced Aug 16, 2026

fix(today): bound the classic dashboard's vital carry, one skin-temp format (#1331 display facet, both platforms) #1362

Merged

WHOOP 4.0: measured nights still crossSecondOverCount post-#1071/#1072 — R-R over-count contaminates HRV #1118

Open

ryanbr mentioned this pull request Aug 17, 2026

diag(hrv): size the cross-second R-R over-count with a windowed shadow candidate (#1331/#1118) #1399

Merged

ryanbr mentioned this pull request Aug 17, 2026

[master] Bug report #1331

Open

Navigation Menu

diag(hrv): shadow-log a deduped R-R stream to validate the over-count fix (#1331/#1008/#1118) - #1352

diag(hrv): shadow-log a deduped R-R stream to validate the over-count fix (#1331/#1008/#1118)#1352
ryanbr merged 3 commits into
mainfrom
1331-overcount-dedup-shadow

ryanbr commented Aug 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ryanbr commented Aug 15, 2026

Why

What (this PR)

What it buys us

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant