Respiratory rate: prefer PPG-derived estimate over RSA when available (#103)

[digitalerdude]

Summary

Follows up on the #103 investigation. That thread ruled out SpO2 and landed
a proof-of-concept: respiratory rate can be recovered from the WHOOP5 v26
single-wavelength 24Hz optical PPG buffer via burst-level spectral analysis
(RIAV/RIIV band search, 0.15–0.40Hz).

NOOP already ships a respiratory-rate estimator for WHOOP5 —
SleepStager.respRateFromRR (RSA, from R-R intervals) — feeding
DailyMetric.respRateBpm and an illness-detection signal in
ReadinessEngine. So this is a second, more accurate estimator, added
as a default-off, diagnostic-only Experimental toggle (never the shipped
default, per the project's derived-biosignal validation standard):

• PpgResp (Swift Packages/WhoopProtocol, Kotlin
  com.noop.protocol): a burst-level estimator on the same v26 buffer
  PpgHr already reads, one estimate per ~40s burst. Validated against
  two real overnight BLE captures vs. the WHOOP app's own reading:
  within ~2–6%, vs. the RSA estimator's 6–11% low bias on the same
  nights. A band-edge guard (bandEdgeGuardHz) rejects peaks pinned at
  the band floor by residual low-frequency drift — confirmed on real data
  (see the review-response below) and it improved accuracy (top-burst
  median 14.04 → 14.52 vs. 14.6 truth).
• SleepStager.respRateFromPpg: filters bursts to a sleep session,
  requires ≥9 qualifying bursts (else NaN), median of the top-third by
  confidence.
• Illness gate stays on a single estimator. DailyMetric.respRateBpm
  (and hence the ReadinessEngine z-gate) always comes from RSA. The
  PPG estimator never overrides it — a value that silently switched
  estimator night to night could inject a ~1–1.5 bpm step and brush the
  illness WATCH threshold on its own (a false signal from the switch, not
  physiology). When the toggle is on, PPG is only computed alongside RSA
  and logged (rsa=… ppg=… used=rsa) through the existing diagnostic sink —
  the "keep the instrumentation" ask.
• PuffinExperiment.ppgRespRate (default false), mirroring the
  existing ppgHrSubLagInterp / hrvReadiness pattern, with a Test Centre
  Experimental-algorithms row on both platforms. Default path is
  byte-identical to what's shipped today.
• New migration for the ppgRespSample table, wired into device-scoped
  delete, timestamp healing, and the raw-sensor CSV export on both
  platforms. Decoded + persisted unconditionally as instrumentation
  regardless of the toggle. Migration numbers: GRDB
  v27-ppg-resp-sample / Room MIGRATION_19_20 (schemaVersion 20),
  renumbered from v26/18→19 after v26-efficiency-heal landed in that slot
  first (the coordination note in the review — resolved).

Test plan

• swift test — WhoopProtocol (288), WhoopStore (263), StrandAnalytics
  (1048), all green
• ./gradlew compileFullDebugKotlin + testFullDebugUnitTest — green
• Local xcodebuild of the macOS Strand app target — BUILD
  SUCCEEDED (touches app-target files app-build.yml CI doesn't cover
  by default)
• End-to-end sanity check against the two real captured nights via the
  actual production code path
• No BLE/hardware surface touched — pure historical-offload + analytics
  path.

[sunny-noop]

Nice result, and the honesty about the evidence base is appreciated.

We've been working the same v26 buffer from the Linux side (the spot-HRV pair, NoopApp/noop#537 for Android and #541 for the Linux reference tool) and can independently confirm two of your observations. First, v26 stretches really are a sparse minority of any night, with v18 taking the rest and never both at once. Second, the aggregation shape you landed on (one estimate per burst, filter on a per-burst quality score, median of the top slice) is the same shape that turned out to be necessary to get a stable RMSSD out of those same bursts. Two independent metrics converging on the same aggregation suggests it's the burst structure forcing it rather than a coincidence.

Offer: we have v26 bursts from two WHOOP 5 straps across a decent number of nights, each with a per-night reference reading to score against. Happy to run PpgResp.deriveRespRate + SleepStager.respRateFromPpg over those and report back the spread against the reference. That would turn the two-night check into something with an actual n, and it would cover a second wearer. Say the word and I'll do it.

Two things I'd want to check before trusting the estimator broadly:

RIAV envelope window. amplitudeEnvelope takes max-min over 24 samples (~1 s). At a sleeping HR around 45 bpm the beat period is ~1.33 s, longer than the window, so the envelope stops measuring beat amplitude and starts measuring intra-beat shape. That injects a beat-phase ripple rather than a breathing one, and it would be worst exactly in deep sleep where HR is lowest. PpgHr already recovers beat locations on the same buffer, so a per-beat peak-to-trough amplitude resampled uniformly would sidestep it. Concrete check against your 57 bursts: does conf (or the estimate itself) correlate with the burst's own mean HR?

Low-band leakage. The detrend is a 12.5 s trailing mean, so the high-pass corner sits around 0.08 Hz, not far below the 0.15 Hz band floor. Residual perfusion or posture drift can leak into the lowest bins and pin the peak at the band edge, and bandPeak has no boundary check. Cheap diagnostic: how many of the 57 bursts peaked in the bottom bin or two? If it's more than a couple, a boundary reject or a longer detrend window is probably worth it.

Neither is a blocker for prefer-PPG-with-RSA-fallback, which reads as the right call given the evidence you have.

[ryanbr]

Really strong, honest work — clean decode → storage → estimator → fusion on both platforms, well gated (≥9 bursts else NaN, median of top-third confidence), and refreshingly candid about the evidence. Two things I want to credit up front, because they matter given the history here:

• This isn't the Widget (iOS): throttle the live-HR republish to 60s (#114 follow-up) #194 autocorr artifact. Widget (iOS): throttle the live-HR republish to 60s (#114 follow-up) #194 (PPG→HR) got withdrawn because autocorrelating fixed-N-per-record buffers manufactured a lag=N peak that "matched ground truth" on a night or two anyway. This clears that two ways: the record-boundary periodicity sits near ~1 Hz, far above the 0.15–0.40 Hz resp band, so record structure can't contaminate it; and the synthetic tests vary the input (breathBpm: 10 → 10, 15 → 15), so it demonstrably tracks frequency rather than emitting a constant. That's the right test.

The ask: gate the fusion behind a default-off Experimental toggle

The remaining gap is the real-data validation — 2 nights, one person "whose respiratory rate barely moves," weak correlation (all your words). The synthetic tests prove the method tracks synthetic modulation; they don't prove the peak it locks onto on real PPG is respiration vs. a coincidental band peak near this person's stable ~14/min. Given #194, that's thin ground to make PPG override the proven RSA estimator for everyone by default.

A default-off Experimental toggle ("Prefer PPG respiratory estimate") solves this cleanly and lets the feature ship now:

• Default OFF ⇒ zero change to everyone's current (RSA) behavior — no regression on thin evidence.
• Opt-in users self-select as the validation cohort. Combined with the logging below, N opt-in nights across subjects/varying resp are exactly the data needed to later make it the default.
• Fits the existing pattern (PuffinExperiment / experimentalSleepV2Enabled / ExperimentalBrand). The code change is small: wrap the AnalyticsEngine prefer-PPG branch in if respPpgEnabled. Both platforms (byte-parity gate + a Settings row).

One thing the toggle alone doesn't fix — keep the illness gate off the mixed baseline

DailyMetric.respRateBpm now switches per-night between PPG (~2–6% off) and RSA (6–11% low bias) by coverage, and that same series feeds ReadinessEngine's illness z-gate. At ~14/min the two estimators differ ~1–1.5 bpm, so for a stable sleeper (resp SD ~1 bpm) a night that switches estimator injects a ~1–1.5 bpm step → z ≈ 1–1.5, right at WATCH (1.5) — a false "illness watch" from the estimator flipping, not physiology. You noted the thresholds were sized for RSA's noise floor, but not that a mixed baseline manufactures z-spikes on switches. So even for opt-in users: decouple it — the toggle controls the displayed/stored value, but the illness gate reads a consistent single estimator (RSA-only baseline, or de-bias RSA→PPG before mixing). "Show me the PPG number" shouldn't corrupt "is my resp trending up."

Keep the instrumentation

Retain the PPG-vs-RSA logging (or add it) so opt-in nights emit PPG vs RSA vs the value actually used — that's what turns the toggle into a real experiment and produces the evidence to promote it to default later.

Nits

• Synthetic tests cover 10 & 15; add an upper-band case (18–20/min) so the whole 0.15–0.40 Hz band is exercised.
• "0.24 breaths/min resolution" is oversold — a 40 s / ~960-sample burst has ~0.025 Hz (~1.5 breaths/min) native resolution; the 0.004 Hz DTFT step interpolates but doesn't add real precision. Worth softening the comment so it doesn't read as sub-breath accuracy.

Coordination note (not this PR's fault)

This uses Room MIGRATION_18_19 / GRDB v26 / schemaVersion 19 for ppgRespSample. There's a parked change targeting the same version numbers for a different column, so whichever lands first, the other rebases to v20/v27 — flagging so the migration numbers don't collide.

Net: great method, honestly scoped. Land it behind a default-off Experimental toggle with the illness gate kept on a single estimator and the logging retained, and flip the preference to default once the opt-in field data shows PPG actually tracks real variation.

[digitalerdude]

Thanks @ryanbr — this is exactly the right call, and I've addressed all of it.

Toggle: a default-off PuffinExperiment.ppgRespRate already went in on the previous push (it was the "fix the CI / follow the derived-biosignal standard" round). But your illness-gate point exposed that gating alone wasn't enough, so I went further:

Illness gate now reads a single estimator, always. DailyMetric.respRateBpm (and hence the ReadinessEngine z-gate) comes from RSA unconditionally — the PPG estimator can never override it. You're right that a mixed baseline manufactures z-spikes on switches, independent of either estimator's own accuracy; even for opt-in users, "show me the PPG number" must not corrupt "is my resp trending up." So when the toggle is on, PPG is computed only alongside RSA and emitted as a diagnostic log line (rsa=… ppg=… used=rsa) through the existing trace sink — which doubles as the "keep the instrumentation" ask: opt-in nights now generate the RSA-vs-PPG-vs-used evidence needed to eventually promote it. Toggle copy on both platforms reworded from "prefer" to "diagnostic."

Nits:

• Added an upper-band synthetic case (19 breaths/min) so the whole 0.15–0.40 Hz band is exercised.
• Softened the "0.24 breaths/min resolution" comment — it now states the ~40s/~960-sample burst's native ~1.5 breaths/min bin spacing and clarifies the 0.004 Hz step interpolates the DTFT between bins without adding real resolution.

Coordination note: confirmed — v26-efficiency-heal landed in my slot during the rebase, so this renumbered to GRDB v27-ppg-resp-sample / Room MIGRATION_19_20 (schemaVersion 20). Migration tests updated to match.

(Also confirmed & fixed a real band-edge leakage issue sunny-noop flagged — details in the reply to that comment.)

[digitalerdude]

@sunny-noop Really appreciate this — the convergent-aggregation observation (independent RMSSD work landing on the same one-estimate-per-burst / quality-filter / top-slice-median shape) is a genuinely reassuring signal that it's the burst structure forcing the aggregation, not a coincidence. And yes please to the offer: v26 bursts from two straps across a decent number of nights, each with a per-night reference, is exactly what turns this two-night check into a real n across a second wearer. That's the evidence that would let the diagnostic toggle become the default. PpgResp.deriveRespRate → SleepStager.respRateFromPpg is the path to run; happy to coordinate on capture format.

I ran both diagnostics you suggested against the real capture data:

RIAV envelope window (conf/estimate vs. burst mean HR): weak and inconsistent across the two nights — corr(HR, conf) was −0.14 / +0.02 and corr(HR, bpm) was −0.10 / −0.39. Not a clean smoking gun for the beat-period-vs-window concern, but the sign flip on n=2 means I can't rule it out either. I left the RIAV path as-is rather than do a speculative rewrite (your per-beat-amplitude-resample idea via the existing PpgHr beat locations is the right fix if more nights confirm it) and documented it as an open question. This is squarely the kind of thing your multi-night/second-strap data would settle.

Low-band leakage: confirmed, and fixed. This one was real — without a boundary check, 32% (night 1) / 15% (night 2) of bursts peaked in the bottom one or two bins (≤9.5 breaths/min), far more than plausible slow breathing. Added PpgResp.bandEdgeGuardHz (0.02 Hz): a peak within that margin of either band edge is rejected (nil for that channel), on the reasoning you gave — a real breathing peak is a local maximum, while residual drift monotonically increases toward 0 Hz and so always "wins" at a bounded edge. It also improved accuracy against ground truth (top-confidence-burst median 14.04 → 14.52 vs. the app's 14.6). I tested a longer detrend window too, but the boundary reject was both more effective and more targeted. Ported to both platforms.