perf(daemon): drain raw-materialization backlog in bursts, unblock startup#3102
Merged
Merged
Conversation
…artup Problem: the raw source->index conveyor materialized at most 1 row per 30s tick (2 rows/min -> a 73k-row backlog needs ~25 days), every tick paid fixed recovery-scan overhead amortized over that single row, startup Drive catch-up (serial network I/O) was awaited before every periodic loop and the live watcher, and the conveyor additionally waited for full watcher catch-up despite operating purely on durable local source.db rows. Together these made the archive unable to self-heal any index gap bigger than a trickle (2026-07-18 incident; bead polylogue-5jak). What changed: - Raw-materialization loop is ungated and backlog-aware: bounded 16-row passes run back-to-back while candidates remain and progress is made, yielding the writer for 1s between passes; a no-progress pass (blocked candidates) or drained backlog returns the loop to the 30s interval, and pending browser-capture spool files still preempt. - Interrupted-frontier recovery runs on the first pass of a cycle only; burst continuations skip the scan. - RawMaterializationCounts carries remaining_candidates (from the existing remaining-candidate repair metric) so the loop can see the backlog. - Startup Drive catch-up no longer blocks the world: the periodic Drive loop now runs its first pass immediately in the background and the dedicated awaited startup pass is deleted. Ref polylogue-5jak Co-Authored-By: Claude <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
Next review available in: 46 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Jul 18, 2026
Sinity
added a commit
that referenced
this pull request
Jul 18, 2026
…-up (#3105) ## Summary Second daemon-drain fix from the 2026-07-18 restore telemetry (follows #3102): during the watcher's initial source catch-up, the embed convergence stage ran inline in every ingest chunk — serial Voyage network calls dominating chunk wall time. Embeddings now defer into convergence debt until catch-up completes. ## Problem Live chunk telemetry during the archive restore: `catch-up chunk 1/726 complete: … convergence_s=41.425 stages=embed:29.636,…` — ~30s of embedding inside a ~60s chunk, projected across 726 chunks ≈ 6 hours of serial network embedding inside the bulk drain path, for a rebuildable read-model that already has a dedicated post-catch-up backlog loop (`periodic_embedding_backlog_check`, gated on `catch_up_complete`). ## Solution - `make_embed_stage(db_path, *, defer=None)`: while the predicate holds, every execute lane (`execute`/`execute_many`/`execute_sessions`) returns `False`, which the stage's existing `false_means_pending=True` contract converts to convergence debt — no new mechanism, no scope reduction, just ordering: sessions first, vectors when quiet. - `make_default_convergence_stages` passes the predicate through; the daemon wires it to the watcher catch-up gate (`not catch_up_complete_gate.is_set()`). All other call sites default to no deferral. - Embed has no barrier hooks, so deferral cannot block downstream stages (workflow/insights). The debt retry loop and embedding backlog loop are both already gated on catch-up completion, so a deferral cannot ping-pong during catch-up. ## Verification - `devtools test tests/unit/daemon/test_convergence_stages.py tests/unit/daemon/test_daemon_cli.py` — 130 passed; new test proves all three execute lanes defer without touching the embedding implementation while the predicate holds, then fall through when it releases, and pins `false_means_pending=True`. - `mypy` clean via `devtools verify --quick` (pre-push gate green). - Not run: full `devtools verify --all`. Ref polylogue-5jak 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QUsH3Rhq6oAZpYPWcsZqnZ --------- Co-authored-by: Claude <noreply@anthropic.com>
Sinity
added a commit
that referenced
this pull request
Jul 18, 2026
…3108) ## Summary Third daemon-drain fix from the 2026-07-18 restore (follows #3102, #3105): the browser-capture spool yield treated *excluded* and *repeatedly-failed* spool files as "pending live ingest", parking the raw-materialization conveyor on every tick — indefinitely when such files never succeed. ## Problem `_browser_capture_spool_has_pending_files` (added in #2975 to keep raw maintenance from monopolizing the writer ahead of fresh browser evidence) returned True for `cursor.excluded` and `cursor.failure_count > 0`. Observed live during the restore: "raw materialization: yielding to pending browser-capture spool files" every tick with 607 spooled captures queued — and after catch-up, any capture left in a failure/excluded disposition would park the conveyor forever. That inverts the priority the yield was built to protect: terminal files are not waiting on the writer. ## Solution Only files genuinely awaiting the live route count as pending: no cursor yet, or bytes changed since the cursor. Excluded files are terminally disposed (exclusion policy owns them); failed files are owned by retry/backoff. Both now `continue` instead of yielding. ## Verification `devtools test tests/unit/daemon/test_daemon_cli.py` — 89 passed; new test `test_spool_pending_check_ignores_terminal_cursor_states` pins: no-cursor ⇒ yield, excluded ⇒ no yield, failure_count ⇒ no yield. Pre-push quick gate green. Ref polylogue-5jak 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QUsH3Rhq6oAZpYPWcsZqnZ Co-authored-by: Claude <noreply@anthropic.com>
Sinity
added a commit
that referenced
this pull request
Jul 18, 2026
…lds (#3109) ## Summary Fourth daemon-drain fix from the 2026-07-18 restore (follows #3102, #3105, #3108): the hook-event spool drain re-opened and re-initialized the whole 5-tier archive **per record** inside the daemon writer hold, and every live hook-spool file event triggers a full drain — with 2,226 pending events and five active agent sessions continuously producing hook events, the single writer was monopolized for tens of minutes and watcher catch-up starved behind it. ## Problem py-spy against the live daemon during a 22-minute stalled catch-up chunk showed the writer thread `polylogue-writer:watcher.hook_spool.drain` pinned inside `initialize_archive_tier` ← `ArchiveStore.open_existing` ← `_persist_record`, called from `drain_hook_event_spool`'s per-record loop. Index session count was frozen the whole time. ## Solution - `drain_hook_event_spool` opens the archive once per pass; each record still commits individually (explicit `archive.commit()`) before its file is acknowledged — the ack-after-commit receipt contract is unchanged, and a failed record rolls back and stays pending, isolated from its neighbors. - Archive-open failure marks the whole selected batch failed and leaves every file pending (same fail-closed behavior the blocked-root test pins). - New `limit` parameter bounds one writer hold; `HookSpoolDrainResult` gains `remaining` so callers know to continue. - The watcher drains in 250-event batches in a loop, releasing the writer between batches, and logs the total once. ## Verification - `devtools test tests/unit/sources/test_hook_spool.py` — 24 passed, including new `test_drain_opens_archive_once_per_pass_and_honors_limit` (one open per pass counted via patched `open_existing`, limit honored, `remaining` drives the second pass, second pass drains the rest). - `devtools test tests/unit/sources/test_live_watcher.py -k spool` — 1 passed. - `mypy` clean; pre-push quick gate green. Ref polylogue-5jak 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QUsH3Rhq6oAZpYPWcsZqnZ --------- Co-authored-by: Claude <noreply@anthropic.com>
Sinity
added a commit
that referenced
this pull request
Jul 18, 2026
#3123) ## Summary Fifth daemon-drain fix from the 2026-07-18 restore: the catch-up plan's round-robin interleave (#1616) spreads browser-capture spool files across the entire plan, and the raw-materialization conveyor yields the writer while *any* spool file lacks a cursor — so the conveyor (the archive's only self-heal path for quarantined cohorts, i.e. most of the restore population) stays parked for the whole multi-hour catch-up. ## Problem Observed live: ~600 spooled captures interleaved across ~700 chunks; `raw materialization: yielding to pending browser-capture spool files` every tick for hours across two daemon restarts, with the burst-capable conveyor (#3102) never executing a single pass. ## Solution `_interleave_by_source` emits the browser-capture bucket first, then round-robins the remaining families exactly as before (the #1616 progress-visibility rationale is preserved for them). Spool files gain cursors in the first chunks; the conveyor unparks minutes into catch-up. ## Verification `devtools test tests/unit/sources/test_live_watcher.py -k "interleave or catch_up"` — 13 passed, including the new test pinning spool-first ordering with round-robin preserved for the rest. `mypy` clean; pre-push quick gate green. Ref polylogue-5jak 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QUsH3Rhq6oAZpYPWcsZqnZ Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the P0 daemon-convergence starvation from the 2026-07-18 poisoned-index incident investigation (bead
polylogue-5jak): the raw source→index conveyor could only self-heal at 1 row per 30s tick (~25 days for the current 73,311-row backlog), and daemon startup serialized all convergence and the live watcher behind serial Drive network I/O.Problem
Live evidence from the incident restore (see
.agent/scratch/warroom-2026-07-17/perf-investigation-2026-07-18.md):_RAW_MATERIALIZATION_CONVERGENCE_BATCH_LIMIT = 1every 30s → 2 rows/min; a rebuilt-but-empty index inherits the entire corpus as backlog and the daemon "heals" it homeopathically. Each tick also paid fixed overhead (blob-ref restore scan, interrupted-frontier recovery, FTS close) amortized over one row.startup.drive_catchupwas awaited before every periodic loop and before the LiveWatcher was constructed — measured live at ~4 serial attachment fetches/min; every restart on a Drive-heavy archive blocks all local convergence for tens of minutes to hours.catch_up_completegate) despite operating purely on durable localsource.dbrows, which have no acquisition precondition.Solution
_periodic_raw_materialization_convergenceis now ungated and backlog-aware: bounded 16-row passes run back-to-back whileremaining_candidates > 0and the pass made progress, sleeping 1s between passes so the write coordinator stays responsive to live appends (per the refactor(storage): remove archive body duplication #1498-cascade retro discipline: bounded per-cycle work, no busy loops — a no-progress pass with blocked candidates ends the burst). Pending browser-capture spool files still preempt the loop, including mid-burst.RawMaterializationCountsgainsremaining_candidates, populated from the repair pass's existingraw_materialization_remaining_candidate_countmetric.recover=Falseon burst continuations) — it only has work after a crash/restart._periodic_drive_source_catchupnow runs its first pass immediately as a background task. Nothing gates on it.Insight/embedding/derived-debt loops keep their catch-up gates — they read from the index the catch-up populates, so waiting is correct there.
Verification
devtools test tests/unit/daemon/test_daemon_cli.py— 87 passed, including new tests: burst-through-backlog (multi-pass single cycle, recovery gated to first pass, sleep schedule pinned), burst-stops-without-progress (blocked candidates can't hot-loop), starts-without-catch-up, recover=False skips the frontier recovery scan, and the lifecycle-ordering test now asserts Drive catch-up does NOT precede the watcher.mypy polylogue/daemon/cli.py polylogue/product/raw_authority.py— clean;ruffclean; pre-push quick gate green.devtools verify --all(testmon-affected scope only).Ref polylogue-5jak
🤖 Generated with Claude Code
https://claude.ai/code/session_01QUsH3Rhq6oAZpYPWcsZqnZ