v0.21.12 — invisible-overhead calibration (400-loop fix, attempt #9)

Fix

Invisible request overhead (2026-08-20 incident, attempt #9 / TRUE root cause of the surviving 400 loop). pi adds a fixed per-request overhead H at request time — system prompt + tool definitions + extension systemPrompt prepends — that never appears in the stored transcript. Every estimator counted messages only: pi's banner, pi's `estimateContextTokens`, and our `estimateSessionTokens`/`applyTailCap`. On a 32k-window model (GLM-4.7, qwen3.8-27b via plexus) H is ~13–16k tokens — roughly HALF the window. After the first 400, provider usage goes null, the estimate path undercounted by ~50%, the anchor floor's `criticalOver` hatch stayed closed at an estimated ~80%, `cut=null` shipped the raw view, and the loop repeated forever — pi's one-shot overflow recovery only resets on user `message_start`, so auto-retries burn it permanently.

Mechanisms:

• Wire-truth parse — the provider's 400 text (`request (39048 tokens) exceeds the available context size (32768 tokens)`) is parsed into ground-truth request/available counts that override the blind estimate for THAT event's gate. Mid-loop, usage is null — but the 400 text carries the real numbers, so the loop un-blinds itself.
• Per-model overhead EMA — every usage-bearing context event samples `H = wire − messageEstimate` and persists it in SQLite meta as a fraction of the window. Calibrated H is added back on the estimate path so the headroom gate sees the real request size, and subtracted from the tail-cap budget so the cap bounds the REAL wire prompt, not just messages.
• Cap-the-skip-path invariant — when `computeLiveTrimCut` returns null (anchor floor refuses), the view is STILL tail-capped instead of shipping raw. The trim path can never return a view the budget wouldn't allow; this alone breaks the 400 loop even if calibration is cold.

Flags (dashboard Settings > Compaction): `MEGACOMPACT_WIRE_OVERHEAD` (default ON; OFF = byte-identical v0.21.11), `MEGACOMPACT_WIRE_OVERHEAD_DEFAULT_PCT` (0.15 fallback fraction when no EMA sample exists, clamped 0–0.85).

Tests + docs

+52 tests: wire-truth regex fixtures (incl. the exact incident string), EMA init/update/clamp, percent-based budget ratios at 32k/200k/1M windows, the incident regression (est 26k + H 16k on 32768 window now caps correctly), liveTrim skip-cap with flag-OFF byte-identity. `docs/LIVE_TRIM_AND_OVERFLOW_RECOVERY.md` §5 documents the mechanism end-to-end.

Verification

• 4464 tests / 0 fail, lint clean, regression `--all` + `--soft-as-hard --pre-commit` both green, guardrails clean
• Live: `pi update --extensions` → resume the idolaigame01 GLM/qwen session; expect `wire_truth_parse` + `skip_cap_applied` events and the 400 loop to break. Watch dashboard.json diag (`diagCtxWireTruth`, `diagCtxSkipCapped`, `headroomTrips`), not the banner (untrimmed transcript).

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

v0.21.11 — tail-cap undercount fix (GLM-4.7 32k truncate loop)

Fix

Tail-cap token undercount (2026-08-20 incident, attempt #8). applyTailCap estimated preserved-tail messages with text-only extraction while pi's convertToLlm ships assistant/toolResult messages verbatim (thinking blocks + toolCall arguments JSON included). A ~30k-token tail registered as ~5k tokens, passed the ~10.3k budget on a 32k-window model with a 20k output reserve, and overflowed the provider on both the original turn and pi's one-shot overflow retry → "Context overflow recovery failed after one compact-and-retry attempt."

New estimateAgentMessageBudgetTokens (extensions/mega-events/context-handler/headroom.ts) counts every block the provider receives, wired into both the fire path (applyTailCap) and replay path (recapReplayedTail). Failing-first regression tests reproduce the incident shape.

Docs

docs/LIVE_TRIM_AND_OVERFLOW_RECOVERY.md — end-to-end mechanism reference: pi emitContext chain, gate (fast/headroom/Phase H), replay cache + re-cap, thrash guard + debounce with headroom exemptions, tail-cap budget math, the verbatim-accounting invariant, how to validate future fixes, and known sharp edges. Registered in INDEX_MAP.md / HEADER_MAP.md.

Verification

• 4505 tests / 0 fail, lint clean, regression 0 blocking, guardrails clean
• Live: pi update --extensions → dashboard 0.21.11; GLM-4.7 32k session completes turns without output truncation

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

v0.21.10

What changed

• test: split degenerate-guard cascade tests into sibling file
• fix: degenerate-summary cascade — content healing, replay keying, debounce bypass (v0.21.10)

Incident fixes (2026-08-19, GLM-4.7 32k loop):

• A1/A2 content healing — extractive.ts no longer uses an rs/ts/js-only allowlist (Go/C/Python/etc. projects produced 34-token skeleton summaries). Universal blocklist-based path detection + salvage digest for skeleton sessions + placeholder-filtered user requests.
• B degenerate-dedup guard — a fuzzy-tier (L1/L2) match onto a degenerate skeleton is declined when the incoming region is strictly richer, so the skeleton stops absorbing every future compaction. Flag-off = byte-identical. (DEDUP_DEGENERATE_GUARD, dashboard-tunable)
• C1 replay keying — rt.lastCheckpointId now set on dedup-only epochs too, so D.2 live-trim replay matches after a restart (fixes liveTrimReplays:0 / comp lag warn).
• C2 debounce headroom exemption — pi's 400→compact→retry recovery no longer hits the 2s debounce (headroomExceeded bypasses it).

Install: `pi update --extensions`

v0.21.9

What changed

Fix: the output-headroom gate — compaction now fires BEFORE the request overflows the model window.

The bug (attempt #6, GLM-4.7 at 32k)

The compaction gate judged only INPUT utilization (fires at ~80% of the context window). But vLLM-style providers reject a request when input_tokens + declared_max_tokens > context_window. For a model with a 32000-token window and 20000 maxTokens, requests overflow at ~37% INPUT — so the gate never tripped (compact never) and the session 400-looped:

Error: 400: ... request (33263 tokens) exceeds the available context size (32768 tokens)
Context overflow recovery failed after one compact-and-retry attempt
Compaction failed: Nothing to compact (session too small)

The fix

A percent-based output-headroom pre-fire check, shared between the gate and the live-trim tail cap (single source of truth, context-handler/headroom.ts):

• Reserve = the model's plausible declared maxTokens (≤95% of the window), else clamp(MEGACOMPACT_OUTPUT_RESERVE_PCT, 0.1–0.95) × window (default 30%).
• Trip when input + reserve + safetyMargin >= contextWindow.
• The 0.95 plausibility bound matters: vLLM reserves the FULL declared maxTokens, so the user's real 32000/20000 (62.5%) config must be trusted, not treated as junk.
• Replay paths (D.2/D.3) re-cap cached trim views against the CURRENT window (a mid-epoch model switch can shrink it).
• ThrashGuard exempts headroom fires — an overflowed session is unrecoverable, so a headroom-triggered compact is never refused.
• Tail-cap hardenings (budget floor, PREVENT-PI-002 pair-safe front-drop) are unconditional guardrail fixes.

Percent-based by construction: identical math at 32k / 64k / 128k / 200k / 256k / 512k / 1M / 5M. No hardcoded token constants in the firing path.

New flags (both dashboard-toggleable, Compaction group)

• MEGACOMPACT_OVERFLOW_HEADROOM — the pre-fire gate (default ON).
• MEGACOMPACT_OUTPUT_RESERVE_PCT — fallback reserve fraction (default 0.3, clamped 0.1–0.95).

New diag counter: headroomTrips.

Verification

• 24 new tests (19 headroom + 5 settings round-trip); full suite green (4290 pass / 0 fail).
• Full QA review via three parallel agent lenses: overflow-scenario matrix (13 window-size cases), PREVENT-PI invariants + thrash/replay interactions, settings/scanner exposure — all PASS after resolving the one flagged MAJOR (flag-scope documentation).

Install: pi update --extensions

v0.21.8 — plexus contextWindow fallback (live-trim tail-cap fix)

Fix

plexus (OpenAI-compatible) providers do not report contextWindow in getContextUsage(). The context handler set lastCtxWindow = usage.contextWindow ?? 0, so it was 0 for those providers. This silently disabled:

• The live-trim tail-cap (guarded on ctxWindow > 0) — which reserves output headroom
• The token-gate window math

Without the tail-cap, a 32k-context model's own ~12k output overflowed the window every turn → truncation loop (stopReason: "length"), even though mega-compact was running.

Root cause

context-handler.ts:119 had no fallback when usage.contextWindow was 0.

Fix

Fall back to runtime.currentModel?.contextWindow (populated from models.json via captureModel on model_select) when the provider-reported window is 0. Mirrors the gate's existing pct fallback (gateCheck.ts S27).

const reportedWindow = usage?.contextWindow ?? 0;
runtime.lastCtxWindow =
    reportedWindow > 0
        ? reportedWindow
        : (runtime.currentModel?.contextWindow ?? 0);

Test

New test auto-trigger: resolves lastCtxWindow from captured model when provider omits contextWindow (S29-b plexus fallback) — fires model_select (32k/20k model) then context with contextWindow: 0, asserts lastCtxWindow === 32000 and currentModel.maxTokens === 20000.

Note on the test

The test accesses lastRuntime via the module namespace (reg.lastRuntime) rather than destructuring (const { lastRuntime } = ...), because destructuring breaks the ES module live binding and snapshots a prior test's runtime.

---

LTS patch. No new features. No config changes required.

v0.21.7 — Revert + corrected output-error catch

v0.21.7 — Revert + corrected output-error catch

Hotfix — reverts the Phase H.1 regression introduced in v0.21.5/v0.21.6

v0.21.5 and v0.21.6 are DEPRECATED. Upgrade to v0.21.7.

Bug

Phase H.1 (v0.21.6) added a guard in context-handler.ts that called piCompactWouldNoop(ctx) — the predicate for pi's durable ctx.compact() path — before evaluateGate consumed the forceCompactNextGate flag. When the session was small (the exact scenario Phase H targeted: a 32k-context model truncating below the input threshold), piCompactWouldNoop returned true, so the guard cleared the flag and returned early, suppressing the live-trim pipeline entirely. The default compaction path (buildLiveTrimView in liveTrim.ts) never calls ctx.compact(), so the durable-path predicate was irrelevant — it guarded nothing it needed to and broke the one thing Phase H was built to fix.

Result: small-context models entered an infinite truncation loop ("compact never").

Fix

• Full revert of the v0.21.5/v0.21.6 sprint to the v0.21.4 baseline (39584739).
• Cherry-picked the audit-verified correct phases (A–G + H) back, without the H.1 guard.
• The Phase H output-error catch trips compaction on stopReason: "length" via a one-shot forceCompactNextGate flag consumed by gateCheck.ts evaluateGate — the correct and sufficient mechanism.
• Safety valves already in place (live-trim null-cut return, thrashGuard, 2s debounce, fallbackCompaction) handle every edge the H.1 guard tried (and failed) to cover.

Threshold normalization (Phases A–G, kept)

• A: COMPACT_TIERS folded to a fraction map (low 0.5 / medium 0.6 / high 0.7 / ultra 0.7 / mega 0.75)
• B: DEFAULT_CONTEXT_WINDOW named constant (env MEGACOMPACT_DEFAULT_CONTEXT_WINDOW, 200_000, display-seed only)
• C: umbrella-OFF DEFER now also returns +Infinity when the window is unknown (LTS correctness)
• D: autoCompactCheck takes a required threshold param (footgun closure)
• E: DEFAULT_SAVED_GOAL seeds the saved goal
• F: DEFAULT_FAST_GATE_PCT_CUSTOM = 70 (pure refactor)
• G: test rewrites for the new tier map

Dashboard rule compliance

• MEGACOMPACT_OUTPUT_ERROR_COMPACT added to the Settings → Compaction group (default ON)
• MEGACOMPACT_DEFAULT_CONTEXT_WINDOW added to EXCLUDED_SETTINGS (display-seed only, never a firing gate)

Cross-system impact

git diff 39584739..HEAD -- src/vector-cortex/ src/memory.ts src/recall.ts src/engine.ts src/store/ → 0 lines. No vector-cortex, memory, recall, engine, or store code was touched.

Device-side verify

pi update --extensions
# open a 32k-context model session, drive to ~77% context, trigger output truncation
# compaction should fire on the NEXT context event; no "Nothing to compact" error

v0.21.4

v0.21.4 — C2 fixes: session-resume duplicate-turn fix (Sprint R) + health observability (Sprint H)

This release closes the two mega-compact findings from the bridge C2 end-to-end investigation (see docs/blog/2026-08-13-bridge-c2-the-bug-that-wasnt.md). Both are COMPLETE scope — no deferred telemetry debt.

Install: pi update --extensions.

---

Sprint R — session-resume duplicate-turn fix (Finding 2)

The bug: 557 turn_write_failed events (DuplicateTurnError) on a live device. When pi resumes a session, it reuses the same sessionId → mega reuses the same conversationId (persistent session_conversations mapping) → but turnIndex restarts at 0 → the UNIQUE(conversation_id, turn_index) constraint collides on every resumed turn. Not just log noise: the resumed session's new turns all failed to record, so fork (which reads the turn store) silently lost post-resume work. 502 of the 557 came from one 18h conversation resumed ~7 times.

The fix (Option B + sessionTurnIndex carry): recordTurnRow now stores turn_index = MAX(existing for the conversation) + 1 (the conversation's monotonic turn number) instead of event.turnIndex (pi's per-session counter, which restarts at 0). Fresh conversations are byte-identical to before (0,1,2…); resumed conversations continue (11,12,13…) — no collision.

Why it's safe (the load-bearing insight, audit-verified): forkFromConversation resolves a turn via getTurnByIndex(conversationId, turnIndex) — it already treats turnIndex as conversation-relative ("the Nth turn in this conversation"), NOT as pi's session counter. Option B makes the stored data match what fork already expects. The full dependents audit (fork, stampTurnsEpoch, turn_recall/HyDE, dashboard Turns tab, bridge child path) confirmed all SAFE except one metrics join.

The sessionTurnIndex carry (no corners): src/metrics/turns.ts joins turns.turn_index to raw_transcript.turn_index (which is seeded from the per-session counter). Under Option B that join would silently zero rawMessageCount for resumed turns. So TurnEntry now carries sessionTurnIndex (pi's per-session counter) alongside the monotonic turn_index; the metrics join re-keys on sessionTurnIndex (with ?? turnIndex coalesce so pre-migration rows still join). The bridge child path (recordTurn) gets the same treatment. Documented residual: raw_transcript stays session-scoped, so resumed-segment metrics can blend (over-count, never a false zero) — full elimination needs dbMirrorAppend to carry the monotonic index; tracked as a follow-up.

Files: recordTurnRow.ts, mega-turn-store.ts, bridge/factory.ts + bridge/types.ts, metrics/turns.ts, store/turns/types.ts (sessionTurnIndex on TurnEntry + nextTurnIndexFor on TurnReader), store/turns/schema.ts (session_turn_index column), store/turns/sqlite-store/{read,write,rows}.ts, store/turns/memory-store.ts (split under the soft limit — see below). Tests: helper unit, recordTurnRow resume, integration (real turns.db 0..10 → 11), metrics resume-divergence, bridge child collision/resume, store resend-after-gap.

---

Sprint H — health observability + drift clarity (Findings 3 + 4)

Finding 3: errorRate read 1.0 (healthy) the entire time 557 turn_write_failed events accumulated. The dashboard showed green while the log showed red. Root cause: errorRate is 1 - (errors/total) over recentErrorCategories, fed by lastErrorCategory — which is only set in the API-error retry path. Internal store-write errors never reached it.

The fix (B2 — a distinct 6th health axis): a new storeErrorRate sub-score, separate from the API-retry errorRate, so internal store-write failures surface in the composite. Weights rebalanced: errorRate 0.18 → 0.09 + storeErrorRate 0.09 (equally loud — a silent store-write failure is as much a health problem as a retryable API error). New store_error_score column in context_health (idempotent; pre-upgrade DBs degrade to NULL → 1.0). Step-change note: historical rows keep their stored composite (point-in-time snapshots, no backfill); the trend line shows a boundary step at this deploy — not a real health change.

Option A ingestion (no central filter): the recentInternalErrors ring is populated by an explicit runtime.recordInternalError(category) call at each failure emit site — not by a central events.log filter (which would recreate the blind spot as a drift-prone filter list). The audit found only turn_write_failed existed; the sibling *_failed emits were instrumentation work (~14 sites, including several silent catches that now gain an emit — completing the ledger rule's "a write failure logs locally" requirement). Categories: store_write, fork, vector_index, embedder, migration.

The INTERNAL-ERR scanner (no-corners backstop): a new guardrails rule (PREVENT-INTERNAL-ERR-001, scripts/internal-err-scan.mjs) — every *_failed emit must pair with a recordInternalError call OR carry a // guardrails-allow INTERNAL-ERR: <reason> annotation. Wired into npm run lint. A future emit site that forgets the pairing fails the gate — the blind spot can't silently reappear. (It already caught a real gap: db-auto-maintain-fail in session-handlers.ts was unpaired — instrumented in this release.)

Finding 4 (clarity): the status-line drift chip relabeled to comp lag. It was the cross-repo compaction_lag signal (repo active but not compacted), not an error-rate drift — the old label conflated them during triage.

Cortex seam: the vector-cortex fire() sink uses a standalone Logger (the cortex contract forbids subscription hooks — §6 ledger protocol). On the failure branch, the host-owned adapter calls recordInternalError("vector_index") — contract test untouched.

Class-separation invariant (verified): the API-error retry path (errorRetry.ts → lastErrorCategory) does NOT push to the internal-errors ring. errorRate and storeErrorRate are independent.

Dashboard surface: the Health tab now shows a separate "Internal Errors" gauge next to "Error Rate" — the two classes are visually distinct, not folded.

Files: runtime.ts + internal-errors.ts (ring + recordInternalError), runtime-instrumentation.ts, 14 emit sites, health-handler.ts (6th axis feed), contextHealth.ts (storeErrorRate + storeErrorScore), store/sqlite/schema.ts + context-health.ts (column + reader; context-health.ts split — cache-poison extracted to context-health-cache-poison.ts), internal-err-scan.mjs + .test.mjs, pattern-rules.json (rule registered), HealthTab.tsx + api/health.ts + widget.ts (dashboard surface). Tests: contextHealth.test.ts (weights sum 1.0; axis independence; equal 0.09 weight), internal-err-scan.test.mjs (8 cases).

---

Refactors (soft-limit splits, no behavior change)

• src/store/turns/memory-store.ts (492 → 194): split into delegate-shell + memory-store/{ctx,read,write,admin}.ts (mirrors sqlite-store). Arrays mutated in place so ctx references stay valid.
• src/store/sqlite/context-health.ts (345 → 270): cache-poison accessors extracted to context-health-cache-poison.ts; re-exported.

Verification

• Gate: build + lint (guardrails + semantic + stub + mock + INTERNAL-ERR) + npm test (4322 pass / 0 fail) + regression_check --all --soft-as-hard --pre-commit (exit 0) + dashboard build + bundle-presence.
• Device verify (post pi update --extensions): resume a session → no new turn_write_failed; resumed turns land at the high-water mark in turns.db; the Health tab "Internal Errors" gauge reflects store errors (not flat 1.0); the status-line chip reads comp lag.

What's next

• C2-cont: continue bridge end-to-end validation (parent recall injection, child dispatch with the second -e, cross-dispatch recall, fork after recordTurns, ITHACUS_MEGA_BRIDGE=false byte-identity).
• Documented residual (Sprint R): dbMirrorAppend carrying the monotonic index to fully eliminate the resumed-segment metrics blend.
• Merge fix/c2-findings + feat/ithacus-bridge to master (docs: the C2 spec + blog post).

v0.21.3

What changed

• feat(ithacus-bridge-A): pi-agnostic bridge API + child extension + flag

Install: `pi update --extensions`

v0.21.2

What changed

• Merge pull request #21 from TheArchitectit/fix/pr18-review-followups
• fix(recall): close PR #18 review follow-ups — fts5 removed-filter, NaN quorum guard
• Merge pull request #18 from drwhofan2k18-pixel/audit-fixes-v21
• fix: H1 dedup-resurrection + M4 MinHash precision + E1 NaN recall guard

Install: `pi update --extensions`

v0.21.1

What changed

• Merge pull request #20 from TheArchitectit/fix/thrashguard-session-start-bind-ordering
• fix(thrashguard): bind repo before session-start disarm
• Merge pull request #19 from drwhofan2k18-pixel/fix-thrashguard
• fix(thrashguard): clear blocked_until on effective compaction + session start

Install: `pi update --extensions`