Skip to content

Oura: stamp banked IBIs at their real ring-time, not the drain-arrival time#677

Merged
ryanbr merged 2 commits into
ryanbr:mainfrom
pipiche38:oura-ibi-anchor-fix
Jul 22, 2026
Merged

Oura: stamp banked IBIs at their real ring-time, not the drain-arrival time#677
ryanbr merged 2 commits into
ryanbr:mainfrom
pipiche38:oura-ibi-anchor-fix

Conversation

@pipiche38

Copy link
Copy Markdown

Summary

A banked Oura IBI (a 0x60/0x80/0x6E history record) is enqueued with ts: now — the wall-clock moment the history drain arrived — instead of its real ring-time, unlike every sibling banked
stream (.hrv/.temp/.spo2/.sleepPhase, which anchor via unixSeconds(forRingTimestamp:)). The anchor was wired for those streams when it landed (52b6e88d) but .ibi was left on now.

The effect

A night's history is drained the next day, so every overnight beat is written to rrInterval at the daytime sync moment: the deep-night hours (00–06 local) come out empty while the sync hour
piles up an implausible density of beats. Any consumer of the Oura R‑R series (HR charts, HRV / daytime-stress analytics) reads the wrong time axis. The oura-ibihr diagnostic corpus was always
correct (it uses the anchored ring-time), so the data is fine — only the datastore timestamp was wrong.

Fix

Anchor .ibi exactly like the sibling banked streams, with the hold-until-anchor pendingAnchorEvents fallback for a beat that arrives before the session's UTC anchor lands. Swift + Android twin
(OuraLiveSource.swift / OuraLiveSource.kt).

  • Live HR is untouched — that's the .hr live push.
  • The live R‑R strip (setRRIntervals / liveSink) is unchanged — only the persist timestamp moves to real time.

Scope note

main has no Oura sleep-session persist yet, so the visible payoff here is a correctly time-stamped rrInterval. The downstream benefit — a persisted Oura sleep session carrying
restingHr/avgHrv — lands once that path merges; this fix is the correct prerequisite either way.

Cross-platform

Both platforms, same change; the bug and the sibling-stream anchor both exist on each. No stored-data shape change — only the timestamp of an already-persisted row — so no migration.

Verification

  • xcodebuild … -scheme Strand -destination 'platform=macOS'BUILD SUCCEEDED.
  • Android ./gradlew compileFullDebugKotlinOK.
  • BLE dispatch isn't unit-testable (the anchor primitive unixSeconds(forRingTimestamp:) it reuses already is). Hardware validation, next drain: overnight rrInterval populates at real times and
    matches the oura-ibihr corpus (00–06 local no longer empty).

…rrival time

A banked Oura IBI (a 0x60/0x80/0x6E history record) was enqueued with `ts: now` - the
wall-clock moment the history drain arrived - instead of its real ring-time, unlike every
sibling banked stream (.hrv/.temp/.spo2/.sleepPhase, which anchor via
`unixSeconds(forRingTimestamp:)`). The anchor was wired for those streams when it was
introduced (52b6e88) but `.ibi` was left on `now`.

Effect: since a night's history is drained the next day, every overnight beat is written
to `rrInterval` at the DAYTIME sync moment - the deep-night hours (00-06 local) come out
empty while the sync hour piles up an implausible density of beats. Any consumer of the
Oura R-R series (HR charts, HRV / daytime-stress analytics) therefore reads the wrong time
axis. The `oura-ibihr` diagnostic corpus was always correct (it uses the anchored
ring-time), so the data itself is fine - only the datastore timestamp was wrong.

Fix (Swift + Android twin): anchor `.ibi` exactly like the sibling streams, with the
hold-until-anchor `pendingAnchorEvents` fallback for a beat that arrives before the
session's UTC anchor lands. Live HR is untouched (that is the `.hr` live push); the live
R-R strip (setRRIntervals / liveSink) is unchanged - only the PERSIST timestamp moves.

NOTE: main has no Oura sleep-session persist yet, so the visible payoff here is a correctly
time-stamped `rrInterval`; the downstream benefit (a persisted Oura sleep session carrying
restingHr/avgHrv) lands once that path merges.

Verified: xcodebuild Strand macOS BUILD SUCCEEDED; Android compileFullDebugKotlin OK.
Hardware validation (next drain): overnight `rrInterval` populates at real times and
matches the `oura-ibihr` corpus (00-06 local no longer empty).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016DN6cU2ERWZUS9tWYCjPvi

@ryanbr ryanbr left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this — the core fix is correct and worth landing. .ibi really was the one banked stream still stamped at now instead of anchored via unixSeconds(forRingTimestamp:), so a night drained the next day wrote every beat to the daytime sync moment (deep-night rrInterval empty, sync hour impossibly dense). Anchoring it like the siblings is right; I confirmed a live IBI still anchors to ~now, the live R-R strip is untouched, and the park/flush fallback handles the pre-anchor beat. The Kotlin side is good as-is.

One request before merge, on the Swift side only.

The noteStoredHistoryRingTime(ibi.ringTimestamp) line can skip banked history

The Swift .ibi case adds noteStoredHistoryRingTime(ibi.ringTimestamp), which feeds the IBI ring-time into the history resume cursor (drain.maxStoredRingTime, committed by commitResumeCursor). The sibling streams do this safely because they are history-only — their ring-times always fall inside the drained sleep window. IBI is the only stream that arrives both live and banked, and the two are indistinguishable at that call site except by ring-time.

Failure path:

  1. A large backlog drain force-stops early — the 300 s deadline or the stall guard fires with bytes_left > 0, so an un-drained banked remainder is still pending. (Those guards exist precisely for big backlogs.)
  2. A live IBI push interleaves during that drain (ring-time ~now, resolves under the anchor) → maxStoredRingTime jumps to ~now.
  3. commitResumeCursor(drainCompleted: false) commits maxStoredRingTime anyway → the cursor leaps to ~now.
  4. The next fetch seeks from ~now and skips the un-drained middle backlog — permanently, never re-fetched.

The pre-fix code didn't have this, because .ibi never touched the cursor: a force-stop resumed from the last banked sample. The new line introduces it.

Suggested change

Drop just that one line — keep the anchoring:

case .ibi(let ibi):
    if feedsLive { live.setRRIntervals([ibi.ibiMs]) }
    if let ts = driver.unixSeconds(forRingTimestamp: ibi.ringTimestamp) {
        enqueue([e], ts: ts)
        // NOTE: unlike the history-only siblings, do NOT noteStoredHistoryRingTime here —
        // IBI also arrives live (ring-time ~now), and letting a live beat advance the resume
        // cursor can skip an un-drained banked remainder on a force-stopped drain.
    } else {
        pendingAnchorEvents.append((e, ibi.ringTimestamp))
    }

This costs nothing: the resume cursor is still driven correctly by the history-only sleep streams (hrv/temp/spo2/sleepphase) that share the same night window, and it makes the Swift path match Kotlin, which never notes any stream's ring-time (so the Kotlin side of this PR is already safe). The safe failure direction — re-drain + dedup — is exactly what Kotlin already does.

Minor

The ingest doc comment (both platforms) still lists IBI under "Live-push events … stamped at wall-clock arrival time" — after this change IBI is anchored, so that line is now stale. A one-line update would keep it honest.

Happy to push this myself if you'd rather — just say the word.

pipiche38 added a commit to pipiche38/noop that referenced this pull request Jul 20, 2026
…tense, §9 full records)

- Roadmap: correct "SyncTime (0x12/0x13) UTC anchoring" — 0x12/0x13 is the
  app→ring time-SET handshake; the actual UTC anchor is the ring-emitted 0x42
  time-sync event (§6.11). Fixes the self-contradiction the reviewer flagged.
- §6.4: reword the rrInterval caveat to present tense + "PR ryanbr#677 (pending merge)";
  the docs no longer describe an unmerged fix as already landed on main.
- §9: reproduce full type/len/rt records + 2–3 samples for the high-rate tags
  0x61/0x72/0x6d (like 0x76) so future RE sees framing and cross-sample variation
  directly; note 0x61's payload length is not fixed (len byte moves 0x10/0x11/0x12).

Docs-only. Full records extracted verbatim from the NOOP Gen-3 raw capture.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016DN6cU2ERWZUS9tWYCjPvi
…yanbr#677 review)

ryanbr's review: IBI is the only stream that arrives both live (ring-time ~now)
and banked, indistinguishable at the ingest call site except by ring-time. The
prior noteStoredHistoryRingTime(ibi.ringTimestamp) let a live beat that interleaves
during a force-stopped drain (300s deadline / stall guard, bytes_left > 0) leap
maxStoredRingTime to ~now, so commitResumeCursor(drainCompleted:false) would seek
the next fetch from ~now and permanently skip the un-drained backlog.

- Swift .ibi case: keep the ring-time anchoring (the actual fix), drop the
  resume-cursor note. The cursor is still driven by the history-only siblings
  (hrv/temp/spo2/sleepPhase) sharing the same night window.
- Swift drainPendingAnchorEvents: a parked IBI can also be a live beat that
  arrived before the anchor, so skip the cursor note for .ibi there too (closes
  the same hole via the park/flush path).
- Doc comments (both platforms): IBI is now anchored, not wall-clock — reword the
  stale "live-push events (HR/IBI/battery) stamped at arrival time" line.

This matches the Kotlin side, which already notes no stream's ring-time; the safe
failure direction (re-drain + dedup) is what Kotlin already does. Built: Strand
macOS BUILD SUCCEEDED; Android compileFullDebugKotlin clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016DN6cU2ERWZUS9tWYCjPvi
@pipiche38

Copy link
Copy Markdown
Author

Done in 71d391d. Dropped the cursor note in .ibi (kept the anchoring) — and while there, closed the same hole in the park/flush path: a parked live IBI (arrived before the anchor) would've advanced the cursor at flush for the identical reason, so drainPendingAnchorEvents now skips the note for .ibi too. Reworded the stale wall-clock doc comment on both platforms. Kotlin cursor logic untouched (already safe). Built: macOS Strand BUILD SUCCEEDED, Android compileFullDebugKotlin clean.

@pipiche38

Copy link
Copy Markdown
Author

Kindly merge with the order is still #677#678

pipiche38 added a commit to pipiche38/noop that referenced this pull request Jul 20, 2026
…yanbr#677 review)

ryanbr's review: IBI is the only stream that arrives both live (ring-time ~now)
and banked, indistinguishable at the ingest call site except by ring-time. The
prior noteStoredHistoryRingTime(ibi.ringTimestamp) let a live beat that interleaves
during a force-stopped drain (300s deadline / stall guard, bytes_left > 0) leap
maxStoredRingTime to ~now, so commitResumeCursor(drainCompleted:false) would seek
the next fetch from ~now and permanently skip the un-drained backlog.

- Swift .ibi case: keep the ring-time anchoring (the actual fix), drop the
  resume-cursor note. The cursor is still driven by the history-only siblings
  (hrv/temp/spo2/sleepPhase) sharing the same night window.
- Swift drainPendingAnchorEvents: a parked IBI can also be a live beat that
  arrived before the anchor, so skip the cursor note for .ibi there too (closes
  the same hole via the park/flush path).
- Doc comments (both platforms): IBI is now anchored, not wall-clock — reword the
  stale "live-push events (HR/IBI/battery) stamped at arrival time" line.

This matches the Kotlin side, which already notes no stream's ring-time; the safe
failure direction (re-drain + dedup) is what Kotlin already does. Built: Strand
macOS BUILD SUCCEEDED; Android compileFullDebugKotlin clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016DN6cU2ERWZUS9tWYCjPvi
@pipiche38
pipiche38 requested a review from ryanbr July 21, 2026 12:26
pipiche38 added a commit to pipiche38/noop that referenced this pull request Jul 21, 2026
…tense, §9 full records)

- Roadmap: correct "SyncTime (0x12/0x13) UTC anchoring" — 0x12/0x13 is the
  app→ring time-SET handshake; the actual UTC anchor is the ring-emitted 0x42
  time-sync event (§6.11). Fixes the self-contradiction the reviewer flagged.
- §6.4: reword the rrInterval caveat to present tense + "PR ryanbr#677 (pending merge)";
  the docs no longer describe an unmerged fix as already landed on main.
- §9: reproduce full type/len/rt records + 2–3 samples for the high-rate tags
  0x61/0x72/0x6d (like 0x76) so future RE sees framing and cross-sample variation
  directly; note 0x61's payload length is not fixed (len byte moves 0x10/0x11/0x12).

Docs-only. Full records extracted verbatim from the NOOP Gen-3 raw capture.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016DN6cU2ERWZUS9tWYCjPvi
@ryanbr
ryanbr merged commit 2742f9f into ryanbr:main Jul 22, 2026
1 check passed
@ryanbr ryanbr mentioned this pull request Jul 22, 2026
@pipiche38
pipiche38 deleted the oura-ibi-anchor-fix branch July 22, 2026 18:02
pipiche38 added a commit to pipiche38/noop that referenced this pull request Jul 24, 2026
…pipeline scores an Oura night (ryanbr#728)

The Oura ring banks overnight IBI (0x60/0x80/0x6E) but NO heart-rate stream, so an Oura night lands
only in `rrInterval`. The nightly analytics pipeline gates on `hrSample` — `resolveDayOwner`'s presence
probe and the `hr.count >= 200` guard — so with zero overnight HR the day-owner falls back off the ring
and the whole night scores NULL: no restingHr / avgHrv, and skin temp (gated behind the same probe)
never surfaces. (Confirmed on device: overnight `rrInterval` ~25k/night but `hrSample` 0.)

Fix: materialise one median HR per IBI-record as `hrSample`.
- `OuraIbiHr.perRecordMedianHR([OuraIBI]) -> [OuraHR]` (pure, both platforms): every IBI decoded from a
  record shares that record's ring-time, so grouping by ringTimestamp == grouping by record; HR =
  round(60000 / median(IBI)) over the record's physiological beats (IBI 300..2000 ms), gated to 30..220
  bpm. Grouping also collapses a record's 6-or-7 same-ts beats into ONE row, matching hrSample's
  (deviceId, ts) key.
- Wired ONLY into the banked path: in `ingest`/`emit`, a batch with NO live-HR push (`.hr`) is history
  data (the live push always emits [.hr, .ibi] together), so live HR is never double-counted. The
  derived HR is enqueued as `.hr` → the mapping writes `hrSample`, but it does NOT pass through the
  `.hr` switch case, so it triggers no wear/live-badge side-effects. Per-record ring-time anchored (ryanbr#677);
  an unanchored record is skipped and re-derived when it re-serves after the 0x42 anchor. hrSample's
  (deviceId, ts) primary key makes re-drains idempotent.

Byte-identical Swift/Kotlin twins; `OuraIbiHr` fixture-tested on both (grouping, median, IBI gate,
6-beats-collapse, ordering, empties). `OuraHR`'s existing physiological gate + the live path are
untouched.

Verification: swift test OuraProtocol green (incl. OuraIbiHrTests); macOS Strand BUILD SUCCEEDED;
Android compileFullDebugKotlin OK. Materialisation on real banked IBI is validated on the next capture
(nightly scoring lighting up skin temp / RHR / HRV); the 0x5D hr_bpm gives a ground-truth cross-check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
pipiche38 added a commit to pipiche38/noop that referenced this pull request Jul 24, 2026
…pipeline scores an Oura night (ryanbr#728)

The Oura ring banks overnight IBI (0x60/0x80/0x6E) but NO heart-rate stream, so an Oura night lands
only in `rrInterval`. The nightly analytics pipeline gates on `hrSample` — `resolveDayOwner`'s presence
probe and the `hr.count >= 200` guard — so with zero overnight HR the day-owner falls back off the ring
and the whole night scores NULL: no restingHr / avgHrv, and skin temp (gated behind the same probe)
never surfaces. (Confirmed on device: overnight `rrInterval` ~25k/night but `hrSample` 0.)

Fix: materialise one median HR per IBI-record as `hrSample`.
- `OuraIbiHr.perRecordMedianHR([OuraIBI]) -> [OuraHR]` (pure, both platforms): every IBI decoded from a
  record shares that record's ring-time, so grouping by ringTimestamp == grouping by record; HR =
  round(60000 / median(IBI)) over the record's physiological beats (IBI 300..2000 ms), gated to 30..220
  bpm. Grouping also collapses a record's 6-or-7 same-ts beats into ONE row, matching hrSample's
  (deviceId, ts) key.
- Wired ONLY into the banked path: in `ingest`/`emit`, a batch with NO live-HR push (`.hr`) is history
  data (the live push always emits [.hr, .ibi] together), so live HR is never double-counted. The
  derived HR is enqueued as `.hr` → the mapping writes `hrSample`, but it does NOT pass through the
  `.hr` switch case, so it triggers no wear/live-badge side-effects. Per-record ring-time anchored (ryanbr#677);
  an unanchored record is skipped and re-derived when it re-serves after the 0x42 anchor. hrSample's
  (deviceId, ts) primary key makes re-drains idempotent.

Byte-identical Swift/Kotlin twins; `OuraIbiHr` fixture-tested on both (grouping, median, IBI gate,
6-beats-collapse, ordering, empties). `OuraHR`'s existing physiological gate + the live path are
untouched.

Verification: swift test OuraProtocol green (incl. OuraIbiHrTests); macOS Strand BUILD SUCCEEDED;
Android compileFullDebugKotlin OK. Materialisation on real banked IBI is validated on the next capture
(nightly scoring lighting up skin temp / RHR / HRV); the 0x5D hr_bpm gives a ground-truth cross-check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ryanbr pushed a commit that referenced this pull request Jul 24, 2026
…pipeline scores an Oura night (#728) (#774)

The Oura ring banks overnight IBI (0x60/0x80/0x6E) but NO heart-rate stream, so an Oura night lands
only in `rrInterval`. The nightly analytics pipeline gates on `hrSample` — `resolveDayOwner`'s presence
probe and the `hr.count >= 200` guard — so with zero overnight HR the day-owner falls back off the ring
and the whole night scores NULL: no restingHr / avgHrv, and skin temp (gated behind the same probe)
never surfaces. (Confirmed on device: overnight `rrInterval` ~25k/night but `hrSample` 0.)

Fix: materialise one median HR per IBI-record as `hrSample`.
- `OuraIbiHr.perRecordMedianHR([OuraIBI]) -> [OuraHR]` (pure, both platforms): every IBI decoded from a
  record shares that record's ring-time, so grouping by ringTimestamp == grouping by record; HR =
  round(60000 / median(IBI)) over the record's physiological beats (IBI 300..2000 ms), gated to 30..220
  bpm. Grouping also collapses a record's 6-or-7 same-ts beats into ONE row, matching hrSample's
  (deviceId, ts) key.
- Wired ONLY into the banked path: in `ingest`/`emit`, a batch with NO live-HR push (`.hr`) is history
  data (the live push always emits [.hr, .ibi] together), so live HR is never double-counted. The
  derived HR is enqueued as `.hr` → the mapping writes `hrSample`, but it does NOT pass through the
  `.hr` switch case, so it triggers no wear/live-badge side-effects. Per-record ring-time anchored (#677);
  an unanchored record is skipped and re-derived when it re-serves after the 0x42 anchor. hrSample's
  (deviceId, ts) primary key makes re-drains idempotent.

Byte-identical Swift/Kotlin twins; `OuraIbiHr` fixture-tested on both (grouping, median, IBI gate,
6-beats-collapse, ordering, empties). `OuraHR`'s existing physiological gate + the live path are
untouched.

Verification: swift test OuraProtocol green (incl. OuraIbiHrTests); macOS Strand BUILD SUCCEEDED;
Android compileFullDebugKotlin OK. Materialisation on real banked IBI is validated on the next capture
(nightly scoring lighting up skin temp / RHR / HRV); the 0x5D hr_bpm gives a ground-truth cross-check.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants