fix(live): avoid reclassifying established append cohorts#2849
Conversation
Problem: Each small live append reran historical full-snapshot classification despite already having a byte-proven predecessor and baseline. That repeatedly loaded all retained full blobs and caused the v35 catch-up memory and I/O spike. What changed: Use the durable raw revision replay metadata on the established append path. Keep historical classification as a recovery fallback only when that metadata has no accepted chain. The production-route evidence harness now proves both branches and bounds normal-path blob reads to the selected baseline and tail. Compatibility/migration: No archive schema or cursor format changes. Existing incomplete cohorts retain the conservative classifier path. Ref polylogue-cnaj Co-Authored-By: Claude <noreply@anthropic.com>
📝 WalkthroughWalkthroughAppend ingestion now prefers durable replay metadata and falls back to cohort classification only when no raw IDs are accepted. Instrumentation and integration tests cover the primary route, established cohorts, and incomplete-cohort recovery. ChangesDurable replay metadata routing
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant WatcherAppend
participant AppendIngest
participant ArchiveStore
participant HistoricalFullBlob
WatcherAppend->>AppendIngest: process appended raw payload
AppendIngest->>ArchiveStore: raw_revision_replay_plan(logical_source_key)
ArchiveStore-->>AppendIngest: replay plan
alt no accepted raw IDs
AppendIngest->>ArchiveStore: classify_raw_revision_cohort(logical_source_key)
ArchiveStore->>HistoricalFullBlob: read_all historical full blob
HistoricalFullBlob-->>ArchiveStore: historical snapshot
ArchiveStore-->>AppendIngest: recovery replay plan
end
AppendIngest-->>WatcherAppend: execute bounded replay
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eaa8537b46
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A non-empty metadata replay plan can describe an older accepted chain while a newer full snapshot remains asserted. Reclassify whenever the current append is absent and defer it if proof still cannot include it, so the watcher never advances a cursor over unindexed bytes. The append cohort harness covers both the empty and non-empty omitted-current recovery states. Ref polylogue-cnaj Co-Authored-By: Claude <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 01f4c44142
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if raw_id not in replay_plan.accepted_raw_ids: | ||
| replay_plan = archive.classify_raw_revision_cohort(logical_source_key) |
There was a problem hiding this comment.
When a previous append for this same predecessor was deferred/quarantined, raw_revision_replay_plan() ignores it because only byte-proven children participate. With this new guard, a later append that has the same byte-contiguous parent can be accepted as soon as its own raw_id appears in the metadata plan, so the classifier is skipped; but classify_raw_revision_cohort() is the path that promotes quarantined contiguous appends via _promote_contiguous_append_evidence, after which sibling tails would be marked ambiguous instead of selecting one by arrival order. In a recovery/legacy cohort with a quarantined append tail, this can advance the cursor/index to the wrong branch.
Useful? React with 👍 / 👎.
1 participant
Summary
Replace repeated historical full-snapshot classification on the established
live-append path with the existing durable revision replay plan. Incomplete or
legacy/crash-interrupted cohorts still take the conservative classifier
fallback; if the current append remains outside the accepted chain, it is
deferred without advancing the cursor.
Problem
On 2026-07-13, a 46 MiB actively appended Codex JSONL selected by catch-up
every ~16 seconds took 37–38 seconds per 0.1–0.3 MiB append. During one
cycle, anonymous memory rose from roughly 0.4 GiB to 4.2 GiB; the daemon
recorded 6,655
memory.highevents before it was intentionally stopped. Theproduction evidence harness identified repeated
classify_raw_revision_cohortreads of every retained historical fullsnapshot as the dominant avoidable work.
Solution
polylogue/sources/live/append_ingest.pynow first derives the accepted chainfrom durable raw-revision metadata immediately after recording the
byte-contiguous append witness. It invokes historical full-snapshot
classification whenever that metadata omits the current append. If the
classifier still cannot prove the current raw, the append remains deferred, so
the watcher cannot advance its cursor over unindexed bytes.
The production-route harness distinguishes the normal metadata route from the
fallback. It proves that an established cohort makes zero classifier and
historical-full reads, while incomplete cohorts still invoke classification and
are deferred until their authority proof includes the current append.
Ref polylogue-cnaj
Verification
devtools test tests/integration/test_append_cohort_memory.py— 4 passed.devtools verify --quick— all 15 static/generated checks passed.append without any predecessor baseline deferred before either old or new
replay path and therefore did not exercise their injected index/crash
failures. The production-route integration coverage above supplies the
established and recovery cohort checks for this change.