feat(oura): reconstruct BLE sleep-stage hypnogram - #325
Closed
pipiche38 wants to merge 20 commits into
Closed
Conversation
…, drain to bytes_left==0 The 0x11 GetEvents summary was misread on main (the old ryanbr#91 fix never merged): body[0] is events_received (a batch COUNT, not a status - the 2026-07-11 18:53 drain stopped on events==0 with 400,873 bytes STILL LEFT, abandoning the newest data incl. the night's hypnogram), and bytes[2..5] is bytes_left (a remaining-BYTE count, not a cursor - persisting it and comparing across sessions as clocks minted a phantom "ring-time regression" -> reset-to-0 -> full history re-dump on every connect; both morning/evening logs show it byte-for-byte). Package: - parseGetEventsResponse -> (eventsReceived, bytesLeft, moreData) per open_oura's EventBatchSummary; moreData = bytesLeft > 0. - 3 golden tests from the real captured summaries, incl. the exact events==0/bytes_left==400,873 case that must continue. App (OuraLiveSource): - The drain now runs until bytes_left == 0. The response byte-count is NEVER persisted. - The durable resume point (open_oura nextEventToSync) is the newest STORED history sample's ring-time (maxStoredRingTime), advanced only by samples that resolved a REAL anchored time (incl. drained pending-anchor parks; never wall-clock fallbacks), bounded by a plausibility ceiling, and committed only at drain end when it resolves under the current anchor. - Genuine ring reboot detection: a stored sample OLDER than the seek cursor (sawPreResumeData) -> full pull next connect. - Backstop guards only (a healthy drain ends at 0): bytes_left stall (3 flat summaries) and a 300s wall deadline; a forced stop keeps banked forward progress, never resets to 0. - A persisted cursor above the ceiling (pre-fix garbage) resets to a full pull at load. - Log marker [cursor-fix v1] so the running build is identifiable. swift test: 91/91. macOS app builds (BUILD SUCCEEDED). To validate on-device: connect twice - 1st drain should end "caught up (bytes_left 0)" + "resume cursor advanced", 2nd should fetch only the delta (no phantom regression, no full re-dump), and the previous night's sleep data should finally arrive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bj38eZ4JMejA813XhYvMk6
…al log Three fixes so the ring's hypnogram can be validated against WHOOP: (2) Stage mapping: OuraSleepStage now follows open_oura's VALIDATED decode_sleep_phases order (0=deep, 1=light, 2=rem, 3=awake). NOOP had 0=awake/2=deep/3=rem from the same unverified doc as the rest of s6.12; two live captures contradict it - phase records decoded AT WAKE (wearer demonstrably awake) carry code 3, which is awake under open_oura's mapping and "REM" under the old one. Raw codes persist unchanged (stage.rawValue is the wire code); only the labels move. Golden tests updated (0x4E + 0x4B: codes 1,2,3,0 -> light, rem, awake, deep). (3) PK-collapse fix (OuraStreamMapping): the event store's PK is (deviceId, ts, kind) and every code in a phase record shared the record's ts, so a whole hypnogram record collapsed to ONE surviving row (the DB held ~1 row/record; nights lost). Each code's ts is now offset by its index in seconds - an ORDER-preserving synthetic spacing, documented as provisional until the real per-code epoch is pinned; the raw code + index persist unchanged. (4) Arrival visibility (OuraLiveSource, investigation log): every 0x4B/0x4E/0x5A record now logs its anchored time, code count, and deep/light/rem/awake histogram inline, plus a per-code cadence self-check (record gap / previous record's code count, median at drain-end) - the same technique that pinned the 60s activity MET epoch. This settles when/whether the ring transmits the night's phase timeline (the drain now provably reaches the newest data, yet the night's hypnogram has not appeared in the phase rows - the log will show what actually arrives). Tests: OuraProtocol 91/91, WhoopStore 243/243, StrandAnalytics 994/994. macOS app BUILD SUCCEEDED. No Android Oura twin on main. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bj38eZ4JMejA813XhYvMk6
…rt drain The 2026-07-12 morning captures delivered the ring's FULL night hypnogram (23 records x 52 codes; deep 88 / light 331 / REM 79 / awake 100 min at 30s/code - structure and totals consistent with WHOOP's staging of the same night), but every record read "[no anchor yet]": a short resume drain carries no organic 0x42 time-sync, so the session never anchored. Consequences: the hypnogram wall-clock-stamped at connect time, and the resume cursor could not commit (4 connects re-dumped the same window). Root cause: this clean stack never asks the ring for its clock - the SyncTime send (and the corrected 0x12 layout) were old-branch fixes that never merged to main. Ported both, from the on-device-proven versions: - Commands.syncTime is now the authoritative open_oura layout `12 09 <unix_secs u64 LE> <tz i8 half-hours>` (supersedes the reverse-engineered unix/256 + 0xF6-trailer guess, which did not match the native client). Tests updated + a tz-byte test added. - OuraLiveSource writes syncTime(now) once per session at .streaming, BEFORE the first history fetch, so the ring emits a usable 0x42 and every fetched record (incl. the night's hypnogram) gets its real anchored time; the resume cursor can then commit and the repeated re-dump stops. swift test: 92/92. macOS app BUILD SUCCEEDED. To validate on-device: "-> sync_time" then "UTC time anchor acquired" early in the connect, sleep-phase records with REAL night timestamps + the per-code cadence self-check line, and "resume cursor advanced" at drain end. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bj38eZ4JMejA813XhYvMk6
Upstream absorbed the ryanbr#91 cursor fix and refactored it into the pure, unit-tested OuraHistoryDrain core (ryanbr#291) — all 8 conflicts in OuraLiveSource resolved in favor of that refactor (functionally identical to this branch's inline version, better factored). Kept from this branch: the sleep-phase arrival log + per-code cadence self-check (re-added its teardown resets, which sat inside a conflict hunk), the SyncTime send, and the stage-mapping/PK-collapse/estimator work (auto-merged cleanly). Verified after merge: OuraProtocol 106/106 (incl. upstream's OuraHistoryDrainTests), WhoopStore 256/256, macOS app BUILD SUCCEEDED. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bj38eZ4JMejA813XhYvMk6
…st end
The ring's SleepNet finalizes a night AFTER wake and writes the whole
hypnogram to its event log in ONE burst (observed 2026-07-12: 23 records
x 52 codes, envelope ring-times all ~09:30 = the WRITE moment). The
envelope timestamp marks when the analysis was saved, not when the sleep
happened - anchoring per-record collapsed an entire night onto a few
seconds of PK slots (the "vanished hypnogram").
Reconstruction: a burst's codes are one contiguous sequence at 30 s/code
ENDING at the burst's envelope time. The epoch is triple-confirmed:
open_oura sleepnet.md ("classification at 30-second intervals"), the
observed window math (1,196 codes over a ~23:35-09:00 night), and the
0x49 summary's 600-minute window over those same codes.
- OuraProtocol: new pure OuraHypnogramAssembler - groups consecutive
phase records into bursts by envelope ring-time gap (60 s), lays codes
backward from the anchored burst end (codesWithTimes). 6 tests incl.
the real 23x52 night shape and a two-night split.
- OuraLiveSource: phase records feed the assembler instead of enqueueing
at envelope time; bursts persist on close/drain-end/teardown with
reconstructed per-code times (wall-clock end fallback when unanchored,
honest + logged); banked bursts advance the resume cursor; a
"hypnogram reconstructed [start -> end] ... deep/light/rem/awake min"
log makes each capture self-evident.
- OuraStreamMapping: store the given ts verbatim (the provisional
ts+index offset would double-shift reconstructed codes); 30 s spacing
now guarantees distinct (deviceId, ts, kind) rows.
Tests: OuraProtocol 112/112, WhoopStore 256/256. macOS app BUILD
SUCCEEDED. On-device validation: reconstructed night rows should appear
across the real 23:3x-09:xx window in the event table, matching the
manual reconstruction (deep 88 / light 331 / REM 79 / awake 100 min).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bj38eZ4JMejA813XhYvMk6
Owner
|
Thanks for this — the reconstruction work is well done and the decode side checks out locally ( It can't land as-is, though — the "no Android twin exists" note isn't accurate, so as written it would break the cross-platform parity contract (the #1 rule in CLAUDE.md):
The time-axis reconstruction (assembler + We'll fold the corrected Android |
tigercraft4
reviewed
Jul 12, 2026
tigercraft4
left a comment
tigercraft4
left a comment
There was a problem hiding this comment.
Review (external, non-blocking). The protocol fix + reconstruction read as sound and the assembler is well unit-tested. One data-integrity point worth confirming before this drives any rendered value, plus a couple of consistency nits. Note: Decoders.swift:334 carries the same stale 0=awake…3=REM doc comment but isn't in this diff, so it can't be flagged inline — worth updating in the same pass.
…wall-clock axis Resolves the review remarks on upstream ryanbr#325: PARITY BLOCKER (ryanbr): the Kotlin twin decodes the same tags with the OLD stage mapping - merging Swift-only would make byte-identical stored rows decode as deep on iOS and awake on Android. Fixed: - android OuraEvents.kt: OuraSleepStage flipped to the open_oura VALIDATED order (DEEP=0, LIGHT=1, REM=2, AWAKE=3), doc-commented as the byte-identical twin of the Swift enum. - DecoderGoldenTest: 0x6C golden updated (codes 1,2,3,0 -> light, rem, awake, deep; byte-identical to the Swift golden) + a raw-value pin test so a regression to the old mapping breaks loudly. - OuraStreamMappingTest: DEEP.raw==0 / REM.raw==2 expectations. - docs/OURA_PROTOCOL.md s6.12: the line both platforms inherited the bug from now teaches 0=deep,1=light,2=rem,3=awake (open_oura decode_sleep_phases), with a CORRECTION note, and includes 0x4B. REVIEW (tigercraft4): - Stale old-mapping doc lines removed/corrected (OuraEvents.swift leftover line above the corrected block; Decoders.swift twin). - Assembler order robustness: bursts now surface hasNonMonotonicRingTimes (arrival order is deliberately NOT re-sorted - envelope rts are near-identical write moments, so an unstable sort could scramble the true sequence); OuraLiveSource logs the flag. New test pins flagged-but-not-resorted behavior. - Wall-clock fallback removed: an unanchored burst is now HELD (pendingUnanchoredBursts, same discipline as pendingAnchorEvents) and reconstructed when the 0x42 anchor lands; if the session ends unanchored it is DROPPED with an honest log - safe, the resume cursor only advances on an anchored persist, so the ring re-serves the same records next drain. A night's time axis is never guessed. - Stored-rows note: no consumer renders OURA_SLEEP_PHASE rows yet (no sleep-session builder exists); rows written under the old labels are investigation-only. Called out in the PR thread. Tests: OuraProtocol 116/116 (swift test); macOS app BUILD SUCCEEDED. Kotlin: mirror edits verified by inspection - no local Java runtime; android.yml/upstream CI covers testFullDebugUnitTest.
Author
|
I have merged against the 8.7.0 and also addressed the review points in the last commit cebde9d
@ryanbr I would be pleased if you could give me some hints on how you would like me to move forward, when I'll be a stage where I can push into persistent storage, see #323 From the latest commit of today (based on 8.7.0 )What worked (this is the real validation with Gen3)
|
…ame-window re-serve loop On-device 2026-07-12: one session re-served the same ~955 KB window five times and never reached the 409 KB tail (last night's hypnogram). Root cause is ours, not the ring's: the continuation request re-sent the SESSION-START cursor (historyCursor only commits at drain end) as an open_ring-style max=0 "ack", and it was fired the moment the 0x11 summary arrived - which the ring emits BEFORE its batch finishes streaming. Re-requesting mid-stream at a non-advancing cursor makes the ring restart its serve from that cursor. Aligned with open_oura's drain_events (crates/oura-link/src/client.rs): - Continuation = getEvents(maxSeenRingTime + 1, max=255, flags=-1) - same shape as the initial request; the max=0 ack-fetch is gone (open_oura has no such request). - The next request waits for the batch to go QUIET (1.5 s with no records, the Swift equivalent of open_oura transact()'s quiet window) instead of firing on summary receipt. - A batch that yields no records, or no cursor advance, STOPS the drain (open_oura !progressed) instead of looping until the 3-flat-summaries stall guard trips. - New OuraHistoryDrain.maxSeenRingTime tracks EVERY history record's envelope ring-time (anchored or not, plausibility-capped) for the in-session continuation; the DURABLE cursor still commits only from anchored stored samples (unchanged). - OuraEvent.envelopeRingTimestamp exposes each record's envelope ring-time; history TLV ingest routes through ingestHistory() which feeds the drain + keeps the quiet window open. - Live-HR re-engage keepalive is skipped while a drain is in flight (the Oura app never runs live mode during a sync); it resumes on the next 15 s tick. - OURA_PROTOCOL.md s5.2/s5.3 corrected: 0x11 = events_received/progress/bytes_left (no cursor), summary is NOT an end-of-batch marker, and the open_ring ack-fetch is marked SUPERSEDED with the observed re-serve evidence. Verified: OuraProtocol swift test 121/121 (5 new continuation-cursor tests + driver continuation shape repinned); macOS app xcodebuild BUILD SUCCEEDED. On-device validation next: reconnect from cursor 3595428 should stream the tail in advancing batches and deliver the 2026-07-11 night's burst. Android's Oura driver still carries the pre-ryanbr#91 summary model - upstream's tracked Android follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bj38eZ4JMejA813XhYvMk6
…t / deadline backlog) 0x71 green_ibi_amp (ryanbr#287/ryanbr#333): every occurrence now logs with anchored time, envelope rt, payload length, and full raw hex (was once-per-kind with no timestamp) - a verified decoder needs several real payloads, and the anchored time is what aligns each record with the concurrent live-HR R-R ground truth. Capped at 50/session against a night-long green-IBI wall; a drain-end roll-up reports the session count + distinct payload lengths regardless. Drain self-resume: a drain that ends with KNOWN remaining work now schedules its own next pass 5 s later instead of waiting for a manual reconnect or the 15 min periodic fetch: - ring reboot detected (sawPreResumeData -> cursor honestly reset to 0): the full re-pull starts immediately on the same connection; - deadline-guard stop with backlog: progress banked, next pass continues from the new cursor. A STALL/no-progress stop never chains (that is the ring looping). Capped at 6 passes per session; a healthy bytes_left==0 completion re-arms the cap. Timer torn down on stop/ disconnect/connect like the other drain timers. Verified: OuraProtocol swift test all suites 0 failures; macOS app BUILD SUCCEEDED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bj38eZ4JMejA813XhYvMk6
… solved, 0x60 flagged suspect Ingested github.com/ringverse/protocol/oura (the [ringverse] source our doc already cited piecemeal). Three findings actioned: 1. 0x71 green_ibi_and_amp: ringverse carries the COMPLETE firmware-addressed decoder (p_green_ibi_and_amp @0x503960) - exact scrambled bit map for 5x 11-bit IBIs + 5 amplitude mantissas, shift=(s==7?0:s+1) from payload[13] bits [2:0], bit [3] reserved (set = firmware mismatch), first emitted entry amplitude-only, timestamps walk backward by each IBI. Ported verbatim as OuraDecoders.decodeGreenIBIAmpCandidate (Tier-B ryanbr#287, strict len-14 gate, never stored) + round-trip/gate golden tests via an inverse packer. The 0x71 fixture-capture log now prints the candidate decode (shift/ibis_ms/amps) side by side with the raw hex, so captures are immediately comparable to concurrent live R-R. OURA_PROTOCOL.md s6.2 rewritten with the full layout + correction of the old "5 deltas + 6 amplitudes" summary. 2. 0x49 sleep_summary_1 SOLVED: both uint16 LE fields are minutes-before-event offsets - the tracked sleep window is [event-start_off, event-end_off]. VALIDATED against our own samples: 600/10 on the 7/11->12 night = 23:30->09:20, matching the reconstructed hypnogram and the reported sleep. Every 0x49 now logs the derived window as an independent cross-check of the reconstruction axis (Tier-B, log-only). s6.12 updated; future refinement noted (anchor burst end at event-end_off instead of the write moment). 3. 0x60 decodeIBIAmplitude flagged SUSPECT (s6.1 WARNING): ringverse shows the 11-bit IBI pieces are scrambled across non-adjacent bytes; our sequential BitReader read disagrees. Cross-check vs live R-R scheduled with the parked history-IBI work - no behavior change tonight. Verified: OuraProtocol swift test 123/123 (2 new); macOS app BUILD SUCCEEDED. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bj38eZ4JMejA813XhYvMk6
Author
|
@tigercraft4 it would be interesting if you could you use that branch, for the 0x71 decoding issue. I have added :
|
tigercraft4
reviewed
Jul 12, 2026
tigercraft4
left a comment
tigercraft4
left a comment
There was a problem hiding this comment.
(Re-anchoring two earlier inline notes that landed as position-only.)
|
Thanks @pipiche38 — pulled your branch ( Honest constraint up front: I don't have an Oura ring, so I can't produce a real 0x71 capture myself. What I can do is (a) review the candidate decoder and (b) hand-decode any raw payload you paste from the fixture log and cross-check the numbers against the concurrent live-HR R-R. Two things I'd check once a capture lands:
If you paste one or two logged 0x71 records here (raw hex + the anchored time + the decoder's output + the concurrent live-HR R-R for that window), I'll hand-decode them against both the candidate layout and the ringverse |
…anchorless-session gap On-device 2026-07-13 morning: a mechanically perfect drain (45 advancing batches, bytes_left 253839 -> 0) ended with NOTHING anchored - no 0x42 arrived all session, so the night's 1040-code hypnogram burst parked honestly and the resume cursor could not advance. Root cause: the 0x42 time_sync_ind record is only LOGGED when the ring actually adjusts its clock; on an already-synced ring (we SyncTime every connect now) a session may never see one. The anchor was luck-based. Fix per ringverse BLE.md: the SyncTime RESPONSE (0x13) carries the ring's CURRENT device timestamp - `13 05 <device_ts:4 LE> <status:1>`. Paired with the host wall-clock at receipt this is a deterministic ring-time->UTC anchor available at EVERY connect: - OuraFraming.parseSyncTimeResponse + syncTimeResponseOp (0x13 peeked non-destructively like 0x11/0x0D). - OuraDriver.syncTimeAnchorCandidate (pure): ringverse labels the field "seconds" but the record clock runs 100 ms ticks - try raw-ticks and seconds x10 against [historyCursor, cursor + 7 days]; exactly ONE must fit, else nil (ambiguous or fresh cursor -> log-only, never a guessed anchor). - OuraDriver.adoptSyncTimeAnchor: same 2020-2035 plausibility gate as 0x42/0x85. - OuraLiveSource.handleSyncTimeResponse: on success logs the adopted rt + unit and drains everything parked while unanchored; on ambiguity logs the raw value for investigation. Since the 0x13 lands right after connect (before the drain serves data), history now anchors inline and the drain-end cursor commit resolves. - OURA_PROTOCOL.md s5.4 rewritten: correct u64+tz request layout (the old open_ring token/counter/0xf6 guess was still documented), response layout, anchor policy. Verified: OuraProtocol swift test 126/126 (3 new); macOS app BUILD SUCCEEDED. On-device next connect should log "UTC anchor from SyncTime response (0x13)" and immediately reconstruct the re-served night. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bj38eZ4JMejA813XhYvMk6
…e moment trailed by 43 min)
On-device 2026-07-13 (first run with the 0x13 anchor - which worked exactly as designed:
deterministic anchor at connect, unit resolved as ticks, cursor finally advanced): the
night reconstructed at 00:11->08:51 while the wearer slept 23:34->08:03. The burst end was
anchored at the SleepNet WRITE moment, which trailed the true sleep end by 43 minutes and
shifted the whole night. The same drain's 0x49 window (event-559/-43 min = 23:32->08:08)
matched the wearer's report within minutes - so the parked refinement is now implemented:
- The 0x49 sleep_summary_1 offsets are stashed (lastSleepWindow049) and paired with the
hypnogram burst of the SAME finalization (envelope ring-times within 10 min - observed
seconds apart, the 0x49 arrives right before the phase records).
- The burst end then anchors at event - end_offset (the true sleep end) instead of the
write envelope; sanity-gated (true end must precede the write by < 6 h) with the write
moment as fallback when no 0x49 matched. Refinement is logged
("burst end refined by 0x49 - SleepNet write ... -> true sleep end ...").
- OURA_PROTOCOL.md s6.12 updated (refinement implemented, +43 min evidence).
Expected axis for last night after a re-pull: ~23:28->08:08 vs reported 23:34->08:03.
Note: rows persisted under the old write-anchored axis are investigation-only (no consumer
renders OURA_SLEEP_PHASE yet); a bulk clean is already on the TODO with the rrInterval one.
Verified: OuraProtocol swift test all green; macOS app BUILD SUCCEEDED.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bj38eZ4JMejA813XhYvMk6
Author
Today's session, summed upEvery layer of the stack proved itself in one run:
|
Author
|
@ryanbr working on the Android twin ... will keep you posted |
…t, PR ryanbr#325 review) ryanbr is right that the Android twin exists and must not diverge - it carried every bug Swift fixed this cycle. This ports the complete chain, file by file, with the same fixes, the same comments, and JVM twin tests mirroring the Swift suites (same fixtures/values): - Framing.kt: 0x11 summary corrected to open_oura EventBatchSummary (events_received/progress/bytes_left - the old cursor misread WAS the ryanbr#91 full re-dump loop, still live here); 0x13 SyncTime response parse (deterministic anchor pair); lenient parseRecord + ONE-packet-per-notification reassembler (twin of dae3d7a, the phantom-storm fix - the old buffering loop was still live here too). - Commands.kt: syncTime -> u64 seconds LE + tz half-hours (on-device-proven layout; the open_ring token/counter/0xF6 guess never anchored real hardware). - OuraDriver.kt: continuation fetch at the ADVANCED cursor (max=255, open_oura drain_events; the max=0 "ack" caused the same-window re-serve loop); syncTimeAnchorCandidate (raw-ticks vs seconds x10 disambiguation against the persisted cursor) + adoptSyncTimeAnchor (same 2020-2035 plausibility gate); 0x4B routed to the validated phase decode. - EventTags.kt: SLEEP_SUMMARY_B(0x4B) -> SLEEP_PHASE_B, Tier A (twin of dddbe95 - the divergence flagged in the PR ryanbr#325 review). - OuraEvents.kt: envelopeRingTimestamp (feeds the in-session continuation cursor). - Decoders.kt: decodeGreenIBIAmpCandidate (ringverse p_green_ibi_and_amp @0x503960, Tier-B ryanbr#287, log-only). - NEW OuraHistoryDrain.kt: seen/stored two-cursor drain core (ryanbr#91/ryanbr#291) - guards, progress test, reboot detection, cursor commit rules. - NEW HypnogramAssembler.kt: burst grouping + 30s/code backward time-axis reconstruction. - ble/OuraLiveSource.kt: sync_time now sent at streaming (was never sent!); 0x13 anchor handler; quiet-window batch continuation + no-progress stop; drain-end cursor commit (stored-anchored only) + self-chained passes (reboot/deadline backlog); hypnogram burst persistence with park-until-anchor + honest teardown drop + the 0x49 true-sleep- end refinement; 0x71 fixture-capture log with candidate decode; per-session resets. STORED-DATA PARITY: OURA_SLEEP_PHASE rows now get the same reconstructed time axis on both platforms (per-code envelope-time rows were the old Android behavior - the whole night collapsed onto the write moment). Verified: compileFullDebugKotlin BUILD SUCCESSFUL; testFullDebugUnitTest 2647 tests - all com.noop.oura + OuraStreamMapping suites green incl. 30 new/updated twin tests (OuraHistoryDrainTest, HypnogramAssemblerTest, 0x71 round-trip/gates, 0x13/candidate, one-packet reassembler, continuation-shape pins). The 3 remaining failures (AiCoachContextTest, StandardHrSensorFormatTest x2) fail identically on the untouched tree - pre-existing locale/env issues on this machine, not this change. BLE session choreography is inspection-only (no Android device on hand) - same validation ask as the Swift side had before its on-device passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bj38eZ4JMejA813XhYvMk6
Author
|
@ryanbr Android twin ported in cae2847 — the full stack (0x11/#91 fix, one-packet framing, u64 SyncTime + 0x13 anchor, drain core, hypnogram reconstruction incl. the 0x49 end refinement, 0x4B |
Author
|
This is the cleanest night-sync yet — every mechanism we built this week fired correctly on the first connect, and the reconstructed night matches your ground truth almost exactly. Against your 23:31 → 7:38 night:
Pipeline health, all green:
|
Author
|
I'll be starting working on moving it to Tier-A based on thsi approach as we don't have any HR/RR Reuse the stage-timeline rows path from #988 / the HC import (#240): the Health Connect import already ingests third-party-staged sleep with stage timelines. Could a BLE source feed the same structures (as source: oura-ring / provided rather than computed)? This seems closest to "richer record wins" semantics that already arbitrate WHOOP vs imported nights. Waiting for this PR #325 been approved to move forward on :
|
pipiche38
added a commit
to pipiche38/noop
that referenced
this pull request
Jul 14, 2026
…session
Bank the anchored Oura hypnogram as a CachedSleepSession with a
[{start,end,stage}] breakdown, upserted under the ring's own deviceId so
SleepMerge's imported-over-computed rule surfaces Oura's SleepNet staging over
NOOP's sparse-motion computed night ("richer record wins" — the same
arbitration that already picks a WHOOP/HC import over a computed night). Reuses
the existing ryanbr#988/HC stage-timeline path; no new merge inputs, read-id plumbing,
or schema migration. Sidesteps the ryanbr#960 Tier-B durable-rows pin: sleep phases
are Tier-A provided data, not a Tier-B derivation.
- Pure byte-identical mappers (Swift OuraSleepSessionMapping + Kotlin twin);
stagesJSON hand-built in fixed key order; 6+6 tests assert the JSON verbatim.
- Wired via a new persistSleepSession closure in OuraLiveSource ->
store.upsertSleepSessions, keyed by the ring deviceId. A confirmation strap-log
line ("sleep session persisted ...") makes the persist observable on-device.
Follow-on to ryanbr#325 (hypnogram reconstruction). App-target Swift compiled (Strand
macOS + NOOPiOS); WhoopStore swift test + Android compile green. On-hardware
validation pending.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bj38eZ4JMejA813XhYvMk6
This was referenced Jul 14, 2026
pipiche38
changed the title
Author
|
@ryanbr let me know if this PR can be merged, or not. All my work is based on that. |
Author
|
Heads-up — the 0x60/0x80 IBI decoders on this branch are now known-wrong; a dedicated fix is up #511. While building HR-history instrumentation on a real Gen 3 overnight capture, the banked IBIs decoded to an 82% beat-to-beat >200 ms jump rate — not a heartbeat train. Re-decoding the same bytes against open_oura's arse_api_ibi_and_amplitude_event / parse_api_green_ibi_quality_event pinned it to our bit layout:
Fix is isolated in #511 (Swift + Kotlin twins, real-capture golden vectors, docs §6.1/§6.4, no public-API change). Impact on this PR: none functionally — the hypnogram reconstruction rides on the 0x4E/0x5A sleep-phase tags (and #447 on 0x50 MET), neither of which touches 0x60/0x80. But this branch still carries the old decoder + its golden-test ectors, so when the fix lands on main this will need a rebase that picks up the corrected decodeIBIAmplitude/decodeGreenIBIQuality and the updated DecoderGoldenTests vectors (a likely small conflict there). Suggested order: land the decode fix first, then rebase this. |
Author
|
@ryanbr I just merge this branch with the recent update on origin/main |
Author
|
#325 is included in #447 and ready for approval ( #447 (comment)) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR makes the ring's own sleep analysis (SleepNet, computed on-device) fully recoverable over BLE: the night hypnogram now decodes with correct stage labels, gets a real time axis, and persists as a 30-second stage timeline
— validated against a WHOOP reference for the same night. It also fixes the missing time anchor that left short drains unstamped.
Changes
SyncTime (0x12) on connect — corrected layout (Commands.swift, OuraLiveSource)
The app never asked the ring for its clock, so a short resume drain carried no 0x42 anchor: every fetched record stayed [no anchor yet] and the resume cursor could never commit (observed: 4 connects re-dumping the same window).
Now sends SyncTime once per session before the first fetch, using the authoritative open_oura layout (12 09 ), superseding the earlier reverse-engineered guess (unix/256 + 0xF6 trailer).
On-device: anchor lands within ~1 s of connect.
Sleep-stage mapping corrected (OuraEvents.swift)
OuraSleepStage now follows open_oura's validated order (0=deep, 1=light, 2=rem, 3=awake); the old mapping (0=awake/2=deep/3=rem) came from an unverified doc and was contradicted by live captures (records decoded at wake carry
code 3). Raw wire codes persist unchanged — labels only.
Hypnogram time-axis reconstruction (new OuraHypnogramAssembler + OuraLiveSource wiring)
Key protocol finding: the ring finalizes a night after wake and writes the whole hypnogram in one burst — every record's envelope ring-time marks the write moment (~09:30), not the sleep. Persisting at envelope time collapsed
an entire night onto seconds. The new pure assembler groups consecutive phase records into bursts (ring-time gap) and lays the codes backward at 30 s/code from the anchored burst end. The epoch is triple-confirmed: open_oura's
sleepnet.md ("classification at 30-second intervals"), window math (1,196 codes over a ~23:35–09:00 night), and the 0x49 summary's 600-minute window. Bursts flush on gap/drain-end/teardown, advance the resume cursor, and log a
self-evident summary (hypnogram reconstructed [start → end] deep/light/rem/awake min).
Phase persistence (OuraStreamMapping)
Each code now persists at its reconstructed 30 s-spaced timestamp (distinct (deviceId, ts, kind) rows); the interim ts + index collision workaround is removed.
Validation
disagreement). Reconstruction re-verified via a forced full re-pull.
Notes
reusing the docs: add a validation protocol for scoring changes #988 stage-timeline path.