fix(daemon): bound ordinary revision replay

[Sinity]

Summary

Correct the unsafe ordinary-daemon replay amplification merged in #2688. Ordinary replay now censuses only the selected authority cohort; explicit offline rebuild remains the only archive-wide census route. Payload size is enforced after full durable cohort expansion and before opening any retained blob.

Problem

Independent adversarial review of #2688 found two critical defects:

1. backfill_historical_revision_evidence(selected_raw_ids=...) still parsed and rewrote membership census for every retained raw before applying the selection. A 120-row daemon batch would therefore rescan roughly 18k raws / 45 GiB and mutate unrelated source evidence.
2. The pre-expansion size guard could select a small raw whose authority sibling exceeded 1 GiB. Stream-capable formats were also incorrectly treated as executable even though raw_revision_material materializes the full retained payload before stream parsing.

#2688 remains explicit evidence of the superseded unsafe implementation; this PR corrects it forward without rewriting history.

Solution

• Add selected-row support to the source-tier census reader.
• Expand selected IDs through durable source-path and membership authority, census only that fixed-point cohort, and leave unrelated raw census rows untouched.
• Keep selected_raw_ids=None as the explicit archive-wide/offline behavior.
• Enforce max_payload_bytes over the fully expanded cohort before _parse_retained_raw opens any blob.
• Treat all oversized retained raws as blocked; stream capability remains diagnostic until the retained reader actually streams.
• Return a typed resource-blocked result to ordinary repair with stable metrics and detail.
• Add anti-amplification, oversized-stream, and small-selected/oversized-sibling regressions that fail if unrelated or oversized blobs are opened.

Verification

• devtools test tests/unit/storage/test_archive_readiness.py tests/unit/storage/test_repair.py tests/unit/sources/test_revision_backfill.py — 56 passed in 81.81s.
• devtools verify --quick — all 13 steps passed in 22.37s.

Ref polylogue-yla8.

Summary by CodeRabbit

• New Features

  • Added configurable limits for replaying historical revision data.
  • Added clear resource-blocking errors when individual or combined payloads exceed configured limits.
  • Expanded repair diagnostics to include related payload cohorts and authority components.
  • Added support for targeted census selection and payload-size reporting.

• Bug Fixes

  • Prevented oversized or blocked payloads from being opened or replayed during repair.
  • Improved handling of parsing failures, append fragments, and related session data.

• Tests

  • Added coverage for oversized cohorts, independent processing, and targeted rebuild isolation.

[coderabbitai]

📝 Walkthrough

Walkthrough

Raw revision backfill now enforces payload limits across authority components. Archive selection exposes cohort and size metadata, while materialization repair uses that metadata to block oversized components and process executable components across bounded passes.

Changes

Bounded raw replay

Layer / File(s)	Summary
Authority component selection polylogue/storage/sqlite/archive_tiers/archive.py	ArchiveStore supports filtered census queries, payload-size lookup, transitive authority components, and connection-aware membership expansion.
Bounded revision replay polylogue/sources/revision_backfill.py, tests/unit/sources/test_revision_backfill.py	Historical replay enforces per-raw and aggregate payload limits, raises a new exported resource-blocked exception, and tests targeted cohort isolation.
Materialization repair gating polylogue/storage/repair.py, tests/unit/storage/test_repair.py	Repair expands candidates to authority cohorts, reports blocking metrics, filters executable components, handles replay blocking, and tests oversized and multi-pass scenarios.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant repair_raw_materialization
  participant ArchiveStore
  participant backfill_historical_revision_evidence
  repair_raw_materialization->>ArchiveStore: expand raw ids into authority components
  ArchiveStore-->>repair_raw_materialization: expanded ids, payload sizes, components
  repair_raw_materialization->>backfill_historical_revision_evidence: replay executable raw ids with max_payload_bytes
  backfill_historical_revision_evidence->>ArchiveStore: census selected component raws
  ArchiveStore-->>backfill_historical_revision_evidence: census rows and payload sizes
  backfill_historical_revision_evidence-->>repair_raw_materialization: replay result or RawRevisionReplayResourceBlockedError

Loading

Possibly related PRs

• Sinity/polylogue#2387: Overlaps the raw-materialization replay and repair coordination flow.
• Sinity/polylogue#2511: Also modifies raw-materialization candidate selection and repair handling.
• Sinity/polylogue#2670: Also changes replay blocking logic in repair_raw_materialization.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and clearly summarizes the main change: bounding ordinary daemon revision replay.
Description check	✅ Passed	The description includes the required Summary, Problem, Solution, and Verification sections and is mostly complete.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/fix/bound-ordinary-authority-replay

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4fc38aaadc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Include raw revision siblings in the bounded cohort

When a selected raw has another raw_sessions.logical_source_key sibling under a different source_path, this expansion does not include that sibling unless it already has a raw_session_memberships row. Normal single-session revision ingest binds raw_sessions.logical_source_key and applies revision replay without writing membership rows, but later this function still calls raw_revision_rebuild_selection() and classify_raw_revision_cohort(), which read every full raw for the logical key. In that scenario an oversized sibling excluded from executable_raw_ids can still be opened by classification without the new max_payload_bytes guard, and accepted siblings were never spilled, so selected repair can still bypass the intended 1 GiB bound or fail during replay.

Useful? React with 👍 / 👎.

[Sinity]

Automated review classification: CodeRabbit is rate-limited and posted no code finding. An independent adversarial review ran four correction iterations over the combined #2688/#2689 surface; the final head 3b928e3 has no remaining P0 blocker. The review specifically validated scoped ordinary census, transitive cohort expansion, per-component resource admission, oversized-before-open behavior, offline-policy preservation, and non-starvation. All substantive CI gates are green; the skipped test job is the documented per-PR policy, with 59 focused managed tests run locally.

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

polylogue/storage/repair.py (2)

1693-1702: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

elif can under-report the blocked scope in the detail message when both conditions hold.

When oversized_raw_ids (per-raw check against candidates.raw_blob_bytes, scoped to this pass's raw_ids) is non-empty, the elif blocked_component_raw_ids: branch is skipped entirely — even if the authority-component expansion pulled in additional raw ids beyond oversized_raw_ids (e.g. materialized siblings not in this pass's candidate list) that are also blocked. The raw_materialization_resource_blocked_count metric still reports the full, correct len(blocked_component_raw_ids), but the human-readable detail string would only mention len(oversized_raw_ids), which can be smaller — misleading for someone diagnosing why replay stalled. This combination isn't currently covered by a test (the existing tests only exercise cases where exactly one of the two is non-empty).

Consider reporting both counts when both are non-empty, e.g.:

-    if oversized_raw_ids:
+    if oversized_raw_ids and not blocked_component_raw_ids - set(oversized_raw_ids):
         detail += (
             f"; {len(oversized_raw_ids):,} non-stream-safe raw row(s) exceed execution limit "
             f"{_format_bytes(RAW_MATERIALIZATION_EXECUTE_BLOB_LIMIT_BYTES)}"
         )
-    elif blocked_component_raw_ids:
+    if blocked_component_raw_ids - set(oversized_raw_ids):
         detail += (
-            f"; {len(blocked_component_raw_ids):,} raw row(s) belong to authority components whose aggregate "
+            f"; {len(blocked_component_raw_ids - set(oversized_raw_ids)):,} additional raw row(s) belong to "
+            "authority components whose aggregate "
             f"payload exceeds {_format_bytes(RAW_MATERIALIZATION_EXECUTE_BLOB_LIMIT_BYTES)}"
         )

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@polylogue/storage/repair.py` around lines 1693 - 1702, The detail message
conditionally reports only one blocked scope, under-reporting when both
oversized_raw_ids and blocked_component_raw_ids are non-empty. Update the detail
construction in the repair logic to report both counts when both collections
contain IDs, while preserving the existing messages for either-only case; add or
update tests covering the combined condition.

---

362-407: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

New backlog keys aren't present in the "unavailable" early-return dict.

expanded_candidate_count, expanded_total_blob_bytes, expanded_aggregate_blocked, authority_component_count, blocked_authority_component_count, executable_authority_component_count are only added to the dict returned on the "available" path (lines 402-407). The early return above (when source.db/index.db are missing) still returns the older, smaller key set. A caller that unconditionally indexes these new keys (rather than checking available first) will get a KeyError when tiers are missing, whereas previously-existing keys like oversized_count/oversized_stream_safe_count already had this gap — this PR just widens it with more keys that a caller might now start depending on.

Consider adding these keys with zero/False defaults to the early-return dict for shape consistency.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@polylogue/storage/repair.py` around lines 362 - 407, Add the new expansion
and authority-component fields—expanded_candidate_count,
expanded_total_blob_bytes, expanded_aggregate_blocked,
authority_component_count, blocked_authority_component_count, and
executable_authority_component_count—to the unavailable early-return dictionary
in the relevant storage repair function, using zero or False defaults so its
result shape matches the available path.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@polylogue/sources/revision_backfill.py`:
- Around line 70-128: Prevent archive-wide census from being combined with a
payload limit: at the start of the relevant backfill function, before
constructing census_selections, validate that max_payload_bytes is None when
selected_raw_ids is None and raise a clear ValueError otherwise. Document this
contract near the function signature or validation, using the existing
selected_raw_ids and max_payload_bytes symbols.

In `@polylogue/storage/sqlite/archive_tiers/archive.py`:
- Around line 1717-1732: raw_membership_selection_components_sync redundantly
expands raw IDs already belonging to discovered components. Before calling
expand_raw_membership_selection_sync for each raw_id, skip it when any existing
component already contains that ID; retain the existing merge logic for
uncovered IDs and preserve deterministic sorted output.

---

Outside diff comments:
In `@polylogue/storage/repair.py`:
- Around line 1693-1702: The detail message conditionally reports only one
blocked scope, under-reporting when both oversized_raw_ids and
blocked_component_raw_ids are non-empty. Update the detail construction in the
repair logic to report both counts when both collections contain IDs, while
preserving the existing messages for either-only case; add or update tests
covering the combined condition.
- Around line 362-407: Add the new expansion and authority-component
fields—expanded_candidate_count, expanded_total_blob_bytes,
expanded_aggregate_blocked, authority_component_count,
blocked_authority_component_count, and executable_authority_component_count—to
the unavailable early-return dictionary in the relevant storage repair function,
using zero or False defaults so its result shape matches the available path.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 1cae9e82-239b-44ad-ac59-990b12690ad2

📥 Commits

Reviewing files that changed from the base of the PR and between df559d8 and 3b928e3.

📒 Files selected for processing (5)

• polylogue/sources/revision_backfill.py
• polylogue/storage/repair.py
• polylogue/storage/sqlite/archive_tiers/archive.py
• tests/unit/sources/test_revision_backfill.py
• tests/unit/storage/test_repair.py

[coderabbitai]

🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Payload-limit check operates on the whole census_selection regardless of censused state — correct, but combining selected_raw_ids=None with max_payload_bytes would block the entire archive-wide census in one shot.

When selected_raw_ids is None, census_selections = (None,), so census_selection never gets narrowed to a component — the size check (if max_payload_bytes were set) would sum all retained raws' blob sizes at once and raise for the whole archive, rather than gating per-cohort. Today no caller passes both selected_raw_ids=None and max_payload_bytes together (repair.py always passes a concrete selected_raw_ids list when it sets max_payload_bytes), so this isn't currently reachable, but it's a sharp edge for future callers given the docstring/PR intent that "explicit offline rebuilds retain archive-wide census behavior" (i.e. size limits are meant to be inapplicable to the None path).

Consider asserting/documenting that max_payload_bytes is only meaningful together with an explicit selected_raw_ids, e.g. raising a ValueError up front if both are combined, to prevent a future caller from unintentionally blocking the whole archive census.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@polylogue/sources/revision_backfill.py` around lines 70 - 128, Prevent
archive-wide census from being combined with a payload limit: at the start of
the relevant backfill function, before constructing census_selections, validate
that max_payload_bytes is None when selected_raw_ids is None and raise a clear
ValueError otherwise. Document this contract near the function signature or
validation, using the existing selected_raw_ids and max_payload_bytes symbols.

[coderabbitai]

🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Redundant per-raw expansion inside raw_membership_selection_components_sync.

Each raw_id in the input (after dedup) triggers its own call to expand_raw_membership_selection_sync, which internally re-runs the full fixpoint expansion query loop from scratch. Since expand_raw_membership_selection_sync([raw_id]) already computes the raw's entire connected component (it's a full closure over shared source_path/logical_source_key edges), any later raw_id that lands inside an already-computed component still triggers a brand-new expansion pass instead of being skipped. For a tightly-connected component with k members present in the input list, this repeats the same multi-query fixpoint computation k times.

This method is called from _raw_materialization_candidate_ids (twice per repair_raw_materialization pass: once up front, once for the "remaining" recheck) and again inside backfill_historical_revision_evidence's own census loop — so the redundant cost compounds across layers. Given the PR's goal is bounding replay work, it's worth avoiding this amplification in the component-discovery step itself.

♻️ Proposed fix: skip raw_ids already covered by a merged component

     `@staticmethod`
     def raw_membership_selection_components_sync(
         conn: sqlite3.Connection,
         raw_ids: list[str],
     ) -> tuple[tuple[str, ...], ...]:
         """Partition scheduling hints into transitive authority components."""
         components: list[set[str]] = []
+        covered: set[str] = set()
         for raw_id in dict.fromkeys(raw_ids):
+            if raw_id in covered:
+                continue
             expanded, _keys = ArchiveStore.expand_raw_membership_selection_sync(conn, [raw_id])
             component = set(expanded)
             overlapping = [existing for existing in components if existing & component]
             for existing in overlapping:
                 component.update(existing)
                 components.remove(existing)
             components.append(component)
+            covered.update(component)
         return tuple(tuple(sorted(component)) for component in sorted(components, key=lambda item: min(item)))

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	@staticmethod
	def raw_membership_selection_components_sync(
	conn: sqlite3.Connection,
	raw_ids: list[str],
	) -> tuple[tuple[str, ...], ...]:
	"""Partition scheduling hints into transitive authority components."""
	components: list[set[str]] = []
	for raw_id in dict.fromkeys(raw_ids):
	expanded, _keys = ArchiveStore.expand_raw_membership_selection_sync(conn, [raw_id])
	component = set(expanded)
	overlapping = [existing for existing in components if existing & component]
	for existing in overlapping:
	component.update(existing)
	components.remove(existing)
	components.append(component)
	return tuple(tuple(sorted(component)) for component in sorted(components, key=lambda item: min(item)))
	`@staticmethod`
	def raw_membership_selection_components_sync(
	conn: sqlite3.Connection,
	raw_ids: list[str],
	) -> tuple[tuple[str, ...], ...]:
	"""Partition scheduling hints into transitive authority components."""
	components: list[set[str]] = []
	covered: set[str] = set()
	for raw_id in dict.fromkeys(raw_ids):
	if raw_id in covered:
	continue
	expanded, _keys = ArchiveStore.expand_raw_membership_selection_sync(conn, [raw_id])
	component = set(expanded)
	overlapping = [existing for existing in components if existing & component]
	for existing in overlapping:
	component.update(existing)
	components.remove(existing)
	components.append(component)
	covered.update(component)
	return tuple(tuple(sorted(component)) for component in sorted(components, key=lambda item: min(item)))

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@polylogue/storage/sqlite/archive_tiers/archive.py` around lines 1717 - 1732,
raw_membership_selection_components_sync redundantly expands raw IDs already
belonging to discovered components. Before calling
expand_raw_membership_selection_sync for each raw_id, skip it when any existing
component already contains that ID; retain the existing merge logic for
uncovered IDs and preserve deterministic sorted output.