Skip to content

v0.21.4

Choose a tag to compare

View all tags
@TheArchitectit TheArchitectit released this 13 Aug 16:58
· 36 commits to master since this release
v0.21.4
c2903a4

v0.21.4 — C2 fixes: session-resume duplicate-turn fix (Sprint R) + health observability (Sprint H)

This release closes the two mega-compact findings from the bridge C2 end-to-end investigation (see docs/blog/2026-08-13-bridge-c2-the-bug-that-wasnt.md). Both are COMPLETE scope — no deferred telemetry debt.

Install: pi update --extensions.

Sprint R — session-resume duplicate-turn fix (Finding 2)

The bug: 557 turn_write_failed events (DuplicateTurnError) on a live device. When pi resumes a session, it reuses the same sessionId → mega reuses the same conversationId (persistent session_conversations mapping) → but turnIndex restarts at 0 → the UNIQUE(conversation_id, turn_index) constraint collides on every resumed turn. Not just log noise: the resumed session's new turns all failed to record, so fork (which reads the turn store) silently lost post-resume work. 502 of the 557 came from one 18h conversation resumed ~7 times.

The fix (Option B + sessionTurnIndex carry): recordTurnRow now stores turn_index = MAX(existing for the conversation) + 1 (the conversation's monotonic turn number) instead of event.turnIndex (pi's per-session counter, which restarts at 0). Fresh conversations are byte-identical to before (0,1,2…); resumed conversations continue (11,12,13…) — no collision.

Why it's safe (the load-bearing insight, audit-verified): forkFromConversation resolves a turn via getTurnByIndex(conversationId, turnIndex) — it already treats turnIndex as conversation-relative ("the Nth turn in this conversation"), NOT as pi's session counter. Option B makes the stored data match what fork already expects. The full dependents audit (fork, stampTurnsEpoch, turn_recall/HyDE, dashboard Turns tab, bridge child path) confirmed all SAFE except one metrics join.

The sessionTurnIndex carry (no corners): src/metrics/turns.ts joins turns.turn_index to raw_transcript.turn_index (which is seeded from the per-session counter). Under Option B that join would silently zero rawMessageCount for resumed turns. So TurnEntry now carries sessionTurnIndex (pi's per-session counter) alongside the monotonic turn_index; the metrics join re-keys on sessionTurnIndex (with ?? turnIndex coalesce so pre-migration rows still join). The bridge child path (recordTurn) gets the same treatment. Documented residual: raw_transcript stays session-scoped, so resumed-segment metrics can blend (over-count, never a false zero) — full elimination needs dbMirrorAppend to carry the monotonic index; tracked as a follow-up.

Files: recordTurnRow.ts, mega-turn-store.ts, bridge/factory.ts + bridge/types.ts, metrics/turns.ts, store/turns/types.ts (sessionTurnIndex on TurnEntry + nextTurnIndexFor on TurnReader), store/turns/schema.ts (session_turn_index column), store/turns/sqlite-store/{read,write,rows}.ts, store/turns/memory-store.ts (split under the soft limit — see below). Tests: helper unit, recordTurnRow resume, integration (real turns.db 0..10 → 11), metrics resume-divergence, bridge child collision/resume, store resend-after-gap.

Sprint H — health observability + drift clarity (Findings 3 + 4)

Finding 3: errorRate read 1.0 (healthy) the entire time 557 turn_write_failed events accumulated. The dashboard showed green while the log showed red. Root cause: errorRate is 1 - (errors/total) over recentErrorCategories, fed by lastErrorCategory — which is only set in the API-error retry path. Internal store-write errors never reached it.

The fix (B2 — a distinct 6th health axis): a new storeErrorRate sub-score, separate from the API-retry errorRate, so internal store-write failures surface in the composite. Weights rebalanced: errorRate 0.18 → 0.09 + storeErrorRate 0.09 (equally loud — a silent store-write failure is as much a health problem as a retryable API error). New store_error_score column in context_health (idempotent; pre-upgrade DBs degrade to NULL → 1.0). Step-change note: historical rows keep their stored composite (point-in-time snapshots, no backfill); the trend line shows a boundary step at this deploy — not a real health change.

Option A ingestion (no central filter): the recentInternalErrors ring is populated by an explicit runtime.recordInternalError(category) call at each failure emit site — not by a central events.log filter (which would recreate the blind spot as a drift-prone filter list). The audit found only turn_write_failed existed; the sibling *_failed emits were instrumentation work (~14 sites, including several silent catches that now gain an emit — completing the ledger rule's "a write failure logs locally" requirement). Categories: store_write, fork, vector_index, embedder, migration.

The INTERNAL-ERR scanner (no-corners backstop): a new guardrails rule (PREVENT-INTERNAL-ERR-001, scripts/internal-err-scan.mjs) — every *_failed emit must pair with a recordInternalError call OR carry a // guardrails-allow INTERNAL-ERR: <reason> annotation. Wired into npm run lint. A future emit site that forgets the pairing fails the gate — the blind spot can't silently reappear. (It already caught a real gap: db-auto-maintain-fail in session-handlers.ts was unpaired — instrumented in this release.)

Finding 4 (clarity): the status-line drift chip relabeled to comp lag. It was the cross-repo compaction_lag signal (repo active but not compacted), not an error-rate drift — the old label conflated them during triage.

Cortex seam: the vector-cortex fire() sink uses a standalone Logger (the cortex contract forbids subscription hooks — §6 ledger protocol). On the failure branch, the host-owned adapter calls recordInternalError("vector_index") — contract test untouched.

Class-separation invariant (verified): the API-error retry path (errorRetry.ts → lastErrorCategory) does NOT push to the internal-errors ring. errorRate and storeErrorRate are independent.

Dashboard surface: the Health tab now shows a separate "Internal Errors" gauge next to "Error Rate" — the two classes are visually distinct, not folded.

Files: runtime.ts + internal-errors.ts (ring + recordInternalError), runtime-instrumentation.ts, 14 emit sites, health-handler.ts (6th axis feed), contextHealth.ts (storeErrorRate + storeErrorScore), store/sqlite/schema.ts + context-health.ts (column + reader; context-health.ts split — cache-poison extracted to context-health-cache-poison.ts), internal-err-scan.mjs + .test.mjs, pattern-rules.json (rule registered), HealthTab.tsx + api/health.ts + widget.ts (dashboard surface). Tests: contextHealth.test.ts (weights sum 1.0; axis independence; equal 0.09 weight), internal-err-scan.test.mjs (8 cases).

Refactors (soft-limit splits, no behavior change)

  • src/store/turns/memory-store.ts (492 → 194): split into delegate-shell + memory-store/{ctx,read,write,admin}.ts (mirrors sqlite-store). Arrays mutated in place so ctx references stay valid.
  • src/store/sqlite/context-health.ts (345 → 270): cache-poison accessors extracted to context-health-cache-poison.ts; re-exported.

Verification

  • Gate: build + lint (guardrails + semantic + stub + mock + INTERNAL-ERR) + npm test (4322 pass / 0 fail) + regression_check --all --soft-as-hard --pre-commit (exit 0) + dashboard build + bundle-presence.
  • Device verify (post pi update --extensions): resume a session → no new turn_write_failed; resumed turns land at the high-water mark in turns.db; the Health tab "Internal Errors" gauge reflects store errors (not flat 1.0); the status-line chip reads comp lag.

What's next

  • C2-cont: continue bridge end-to-end validation (parent recall injection, child dispatch with the second -e, cross-dispatch recall, fork after recordTurns, ITHACUS_MEGA_BRIDGE=false byte-identity).
  • Documented residual (Sprint R): dbMirrorAppend carrying the monotonic index to fully eliminate the resumed-segment metrics blend.
  • Merge fix/c2-findings + feat/ithacus-bridge to master (docs: the C2 spec + blog post).
Assets 2
Loading