Oura: stamp banked IBIs at their real ring-time, not the drain-arrival time#677
Conversation
…rrival time A banked Oura IBI (a 0x60/0x80/0x6E history record) was enqueued with `ts: now` - the wall-clock moment the history drain arrived - instead of its real ring-time, unlike every sibling banked stream (.hrv/.temp/.spo2/.sleepPhase, which anchor via `unixSeconds(forRingTimestamp:)`). The anchor was wired for those streams when it was introduced (52b6e88) but `.ibi` was left on `now`. Effect: since a night's history is drained the next day, every overnight beat is written to `rrInterval` at the DAYTIME sync moment - the deep-night hours (00-06 local) come out empty while the sync hour piles up an implausible density of beats. Any consumer of the Oura R-R series (HR charts, HRV / daytime-stress analytics) therefore reads the wrong time axis. The `oura-ibihr` diagnostic corpus was always correct (it uses the anchored ring-time), so the data itself is fine - only the datastore timestamp was wrong. Fix (Swift + Android twin): anchor `.ibi` exactly like the sibling streams, with the hold-until-anchor `pendingAnchorEvents` fallback for a beat that arrives before the session's UTC anchor lands. Live HR is untouched (that is the `.hr` live push); the live R-R strip (setRRIntervals / liveSink) is unchanged - only the PERSIST timestamp moves. NOTE: main has no Oura sleep-session persist yet, so the visible payoff here is a correctly time-stamped `rrInterval`; the downstream benefit (a persisted Oura sleep session carrying restingHr/avgHrv) lands once that path merges. Verified: xcodebuild Strand macOS BUILD SUCCEEDED; Android compileFullDebugKotlin OK. Hardware validation (next drain): overnight `rrInterval` populates at real times and matches the `oura-ibihr` corpus (00-06 local no longer empty). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016DN6cU2ERWZUS9tWYCjPvi
ryanbr
left a comment
There was a problem hiding this comment.
Thanks for this — the core fix is correct and worth landing. .ibi really was the one banked stream still stamped at now instead of anchored via unixSeconds(forRingTimestamp:), so a night drained the next day wrote every beat to the daytime sync moment (deep-night rrInterval empty, sync hour impossibly dense). Anchoring it like the siblings is right; I confirmed a live IBI still anchors to ~now, the live R-R strip is untouched, and the park/flush fallback handles the pre-anchor beat. The Kotlin side is good as-is.
One request before merge, on the Swift side only.
The noteStoredHistoryRingTime(ibi.ringTimestamp) line can skip banked history
The Swift .ibi case adds noteStoredHistoryRingTime(ibi.ringTimestamp), which feeds the IBI ring-time into the history resume cursor (drain.maxStoredRingTime, committed by commitResumeCursor). The sibling streams do this safely because they are history-only — their ring-times always fall inside the drained sleep window. IBI is the only stream that arrives both live and banked, and the two are indistinguishable at that call site except by ring-time.
Failure path:
- A large backlog drain force-stops early — the 300 s deadline or the stall guard fires with
bytes_left > 0, so an un-drained banked remainder is still pending. (Those guards exist precisely for big backlogs.) - A live IBI push interleaves during that drain (ring-time ~now, resolves under the anchor) →
maxStoredRingTimejumps to ~now. commitResumeCursor(drainCompleted: false)commitsmaxStoredRingTimeanyway → the cursor leaps to ~now.- The next fetch seeks from ~now and skips the un-drained middle backlog — permanently, never re-fetched.
The pre-fix code didn't have this, because .ibi never touched the cursor: a force-stop resumed from the last banked sample. The new line introduces it.
Suggested change
Drop just that one line — keep the anchoring:
case .ibi(let ibi):
if feedsLive { live.setRRIntervals([ibi.ibiMs]) }
if let ts = driver.unixSeconds(forRingTimestamp: ibi.ringTimestamp) {
enqueue([e], ts: ts)
// NOTE: unlike the history-only siblings, do NOT noteStoredHistoryRingTime here —
// IBI also arrives live (ring-time ~now), and letting a live beat advance the resume
// cursor can skip an un-drained banked remainder on a force-stopped drain.
} else {
pendingAnchorEvents.append((e, ibi.ringTimestamp))
}This costs nothing: the resume cursor is still driven correctly by the history-only sleep streams (hrv/temp/spo2/sleepphase) that share the same night window, and it makes the Swift path match Kotlin, which never notes any stream's ring-time (so the Kotlin side of this PR is already safe). The safe failure direction — re-drain + dedup — is exactly what Kotlin already does.
Minor
The ingest doc comment (both platforms) still lists IBI under "Live-push events … stamped at wall-clock arrival time" — after this change IBI is anchored, so that line is now stale. A one-line update would keep it honest.
Happy to push this myself if you'd rather — just say the word.
…tense, §9 full records) - Roadmap: correct "SyncTime (0x12/0x13) UTC anchoring" — 0x12/0x13 is the app→ring time-SET handshake; the actual UTC anchor is the ring-emitted 0x42 time-sync event (§6.11). Fixes the self-contradiction the reviewer flagged. - §6.4: reword the rrInterval caveat to present tense + "PR ryanbr#677 (pending merge)"; the docs no longer describe an unmerged fix as already landed on main. - §9: reproduce full type/len/rt records + 2–3 samples for the high-rate tags 0x61/0x72/0x6d (like 0x76) so future RE sees framing and cross-sample variation directly; note 0x61's payload length is not fixed (len byte moves 0x10/0x11/0x12). Docs-only. Full records extracted verbatim from the NOOP Gen-3 raw capture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016DN6cU2ERWZUS9tWYCjPvi
…yanbr#677 review) ryanbr's review: IBI is the only stream that arrives both live (ring-time ~now) and banked, indistinguishable at the ingest call site except by ring-time. The prior noteStoredHistoryRingTime(ibi.ringTimestamp) let a live beat that interleaves during a force-stopped drain (300s deadline / stall guard, bytes_left > 0) leap maxStoredRingTime to ~now, so commitResumeCursor(drainCompleted:false) would seek the next fetch from ~now and permanently skip the un-drained backlog. - Swift .ibi case: keep the ring-time anchoring (the actual fix), drop the resume-cursor note. The cursor is still driven by the history-only siblings (hrv/temp/spo2/sleepPhase) sharing the same night window. - Swift drainPendingAnchorEvents: a parked IBI can also be a live beat that arrived before the anchor, so skip the cursor note for .ibi there too (closes the same hole via the park/flush path). - Doc comments (both platforms): IBI is now anchored, not wall-clock — reword the stale "live-push events (HR/IBI/battery) stamped at arrival time" line. This matches the Kotlin side, which already notes no stream's ring-time; the safe failure direction (re-drain + dedup) is what Kotlin already does. Built: Strand macOS BUILD SUCCEEDED; Android compileFullDebugKotlin clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016DN6cU2ERWZUS9tWYCjPvi
|
Done in 71d391d. Dropped the cursor note in .ibi (kept the anchoring) — and while there, closed the same hole in the park/flush path: a parked live IBI (arrived before the anchor) would've advanced the cursor at flush for the identical reason, so drainPendingAnchorEvents now skips the note for .ibi too. Reworded the stale wall-clock doc comment on both platforms. Kotlin cursor logic untouched (already safe). Built: macOS Strand BUILD SUCCEEDED, Android compileFullDebugKotlin clean. |
…yanbr#677 review) ryanbr's review: IBI is the only stream that arrives both live (ring-time ~now) and banked, indistinguishable at the ingest call site except by ring-time. The prior noteStoredHistoryRingTime(ibi.ringTimestamp) let a live beat that interleaves during a force-stopped drain (300s deadline / stall guard, bytes_left > 0) leap maxStoredRingTime to ~now, so commitResumeCursor(drainCompleted:false) would seek the next fetch from ~now and permanently skip the un-drained backlog. - Swift .ibi case: keep the ring-time anchoring (the actual fix), drop the resume-cursor note. The cursor is still driven by the history-only siblings (hrv/temp/spo2/sleepPhase) sharing the same night window. - Swift drainPendingAnchorEvents: a parked IBI can also be a live beat that arrived before the anchor, so skip the cursor note for .ibi there too (closes the same hole via the park/flush path). - Doc comments (both platforms): IBI is now anchored, not wall-clock — reword the stale "live-push events (HR/IBI/battery) stamped at arrival time" line. This matches the Kotlin side, which already notes no stream's ring-time; the safe failure direction (re-drain + dedup) is what Kotlin already does. Built: Strand macOS BUILD SUCCEEDED; Android compileFullDebugKotlin clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016DN6cU2ERWZUS9tWYCjPvi
…tense, §9 full records) - Roadmap: correct "SyncTime (0x12/0x13) UTC anchoring" — 0x12/0x13 is the app→ring time-SET handshake; the actual UTC anchor is the ring-emitted 0x42 time-sync event (§6.11). Fixes the self-contradiction the reviewer flagged. - §6.4: reword the rrInterval caveat to present tense + "PR ryanbr#677 (pending merge)"; the docs no longer describe an unmerged fix as already landed on main. - §9: reproduce full type/len/rt records + 2–3 samples for the high-rate tags 0x61/0x72/0x6d (like 0x76) so future RE sees framing and cross-sample variation directly; note 0x61's payload length is not fixed (len byte moves 0x10/0x11/0x12). Docs-only. Full records extracted verbatim from the NOOP Gen-3 raw capture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016DN6cU2ERWZUS9tWYCjPvi
…pipeline scores an Oura night (ryanbr#728) The Oura ring banks overnight IBI (0x60/0x80/0x6E) but NO heart-rate stream, so an Oura night lands only in `rrInterval`. The nightly analytics pipeline gates on `hrSample` — `resolveDayOwner`'s presence probe and the `hr.count >= 200` guard — so with zero overnight HR the day-owner falls back off the ring and the whole night scores NULL: no restingHr / avgHrv, and skin temp (gated behind the same probe) never surfaces. (Confirmed on device: overnight `rrInterval` ~25k/night but `hrSample` 0.) Fix: materialise one median HR per IBI-record as `hrSample`. - `OuraIbiHr.perRecordMedianHR([OuraIBI]) -> [OuraHR]` (pure, both platforms): every IBI decoded from a record shares that record's ring-time, so grouping by ringTimestamp == grouping by record; HR = round(60000 / median(IBI)) over the record's physiological beats (IBI 300..2000 ms), gated to 30..220 bpm. Grouping also collapses a record's 6-or-7 same-ts beats into ONE row, matching hrSample's (deviceId, ts) key. - Wired ONLY into the banked path: in `ingest`/`emit`, a batch with NO live-HR push (`.hr`) is history data (the live push always emits [.hr, .ibi] together), so live HR is never double-counted. The derived HR is enqueued as `.hr` → the mapping writes `hrSample`, but it does NOT pass through the `.hr` switch case, so it triggers no wear/live-badge side-effects. Per-record ring-time anchored (ryanbr#677); an unanchored record is skipped and re-derived when it re-serves after the 0x42 anchor. hrSample's (deviceId, ts) primary key makes re-drains idempotent. Byte-identical Swift/Kotlin twins; `OuraIbiHr` fixture-tested on both (grouping, median, IBI gate, 6-beats-collapse, ordering, empties). `OuraHR`'s existing physiological gate + the live path are untouched. Verification: swift test OuraProtocol green (incl. OuraIbiHrTests); macOS Strand BUILD SUCCEEDED; Android compileFullDebugKotlin OK. Materialisation on real banked IBI is validated on the next capture (nightly scoring lighting up skin temp / RHR / HRV); the 0x5D hr_bpm gives a ground-truth cross-check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…pipeline scores an Oura night (ryanbr#728) The Oura ring banks overnight IBI (0x60/0x80/0x6E) but NO heart-rate stream, so an Oura night lands only in `rrInterval`. The nightly analytics pipeline gates on `hrSample` — `resolveDayOwner`'s presence probe and the `hr.count >= 200` guard — so with zero overnight HR the day-owner falls back off the ring and the whole night scores NULL: no restingHr / avgHrv, and skin temp (gated behind the same probe) never surfaces. (Confirmed on device: overnight `rrInterval` ~25k/night but `hrSample` 0.) Fix: materialise one median HR per IBI-record as `hrSample`. - `OuraIbiHr.perRecordMedianHR([OuraIBI]) -> [OuraHR]` (pure, both platforms): every IBI decoded from a record shares that record's ring-time, so grouping by ringTimestamp == grouping by record; HR = round(60000 / median(IBI)) over the record's physiological beats (IBI 300..2000 ms), gated to 30..220 bpm. Grouping also collapses a record's 6-or-7 same-ts beats into ONE row, matching hrSample's (deviceId, ts) key. - Wired ONLY into the banked path: in `ingest`/`emit`, a batch with NO live-HR push (`.hr`) is history data (the live push always emits [.hr, .ibi] together), so live HR is never double-counted. The derived HR is enqueued as `.hr` → the mapping writes `hrSample`, but it does NOT pass through the `.hr` switch case, so it triggers no wear/live-badge side-effects. Per-record ring-time anchored (ryanbr#677); an unanchored record is skipped and re-derived when it re-serves after the 0x42 anchor. hrSample's (deviceId, ts) primary key makes re-drains idempotent. Byte-identical Swift/Kotlin twins; `OuraIbiHr` fixture-tested on both (grouping, median, IBI gate, 6-beats-collapse, ordering, empties). `OuraHR`'s existing physiological gate + the live path are untouched. Verification: swift test OuraProtocol green (incl. OuraIbiHrTests); macOS Strand BUILD SUCCEEDED; Android compileFullDebugKotlin OK. Materialisation on real banked IBI is validated on the next capture (nightly scoring lighting up skin temp / RHR / HRV); the 0x5D hr_bpm gives a ground-truth cross-check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…pipeline scores an Oura night (#728) (#774) The Oura ring banks overnight IBI (0x60/0x80/0x6E) but NO heart-rate stream, so an Oura night lands only in `rrInterval`. The nightly analytics pipeline gates on `hrSample` — `resolveDayOwner`'s presence probe and the `hr.count >= 200` guard — so with zero overnight HR the day-owner falls back off the ring and the whole night scores NULL: no restingHr / avgHrv, and skin temp (gated behind the same probe) never surfaces. (Confirmed on device: overnight `rrInterval` ~25k/night but `hrSample` 0.) Fix: materialise one median HR per IBI-record as `hrSample`. - `OuraIbiHr.perRecordMedianHR([OuraIBI]) -> [OuraHR]` (pure, both platforms): every IBI decoded from a record shares that record's ring-time, so grouping by ringTimestamp == grouping by record; HR = round(60000 / median(IBI)) over the record's physiological beats (IBI 300..2000 ms), gated to 30..220 bpm. Grouping also collapses a record's 6-or-7 same-ts beats into ONE row, matching hrSample's (deviceId, ts) key. - Wired ONLY into the banked path: in `ingest`/`emit`, a batch with NO live-HR push (`.hr`) is history data (the live push always emits [.hr, .ibi] together), so live HR is never double-counted. The derived HR is enqueued as `.hr` → the mapping writes `hrSample`, but it does NOT pass through the `.hr` switch case, so it triggers no wear/live-badge side-effects. Per-record ring-time anchored (#677); an unanchored record is skipped and re-derived when it re-serves after the 0x42 anchor. hrSample's (deviceId, ts) primary key makes re-drains idempotent. Byte-identical Swift/Kotlin twins; `OuraIbiHr` fixture-tested on both (grouping, median, IBI gate, 6-beats-collapse, ordering, empties). `OuraHR`'s existing physiological gate + the live path are untouched. Verification: swift test OuraProtocol green (incl. OuraIbiHrTests); macOS Strand BUILD SUCCEEDED; Android compileFullDebugKotlin OK. Materialisation on real banked IBI is validated on the next capture (nightly scoring lighting up skin temp / RHR / HRV); the 0x5D hr_bpm gives a ground-truth cross-check. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Summary
A banked Oura IBI (a
0x60/0x80/0x6Ehistory record) is enqueued withts: now— the wall-clock moment the history drain arrived — instead of its real ring-time, unlike every sibling bankedstream (
.hrv/.temp/.spo2/.sleepPhase, which anchor viaunixSeconds(forRingTimestamp:)). The anchor was wired for those streams when it landed (52b6e88d) but.ibiwas left onnow.The effect
A night's history is drained the next day, so every overnight beat is written to
rrIntervalat the daytime sync moment: the deep-night hours (00–06 local) come out empty while the sync hourpiles up an implausible density of beats. Any consumer of the Oura R‑R series (HR charts, HRV / daytime-stress analytics) reads the wrong time axis. The
oura-ibihrdiagnostic corpus was alwayscorrect (it uses the anchored ring-time), so the data is fine — only the datastore timestamp was wrong.
Fix
Anchor
.ibiexactly like the sibling banked streams, with the hold-until-anchorpendingAnchorEventsfallback for a beat that arrives before the session's UTC anchor lands. Swift + Android twin(
OuraLiveSource.swift/OuraLiveSource.kt)..hrlive push.setRRIntervals/liveSink) is unchanged — only the persist timestamp moves to real time.Scope note
mainhas no Oura sleep-session persist yet, so the visible payoff here is a correctly time-stampedrrInterval. The downstream benefit — a persisted Oura sleep session carryingrestingHr/avgHrv— lands once that path merges; this fix is the correct prerequisite either way.Cross-platform
Both platforms, same change; the bug and the sibling-stream anchor both exist on each. No stored-data shape change — only the timestamp of an already-persisted row — so no migration.
Verification
xcodebuild … -scheme Strand -destination 'platform=macOS'— BUILD SUCCEEDED../gradlew compileFullDebugKotlin— OK.unixSeconds(forRingTimestamp:)it reuses already is). Hardware validation, next drain: overnightrrIntervalpopulates at real times andmatches the
oura-ibihrcorpus (00–06 local no longer empty).