Skip to content

Gap healing rescans the full comparison window on every pass #26

Description

@ewhauser

Problem

C-R4 gap healing runs every GAP_HEAL_PERIOD (default 5m) over the full GAP_COMPARISON_WINDOW (default 6h), reading GAP_PAGE_SIZE×GAP_MAX_PAGES (default 1,000) deliveries per run and immediately scheduling a continuation at the page cap. The same delivery history is re-read on every pass: in one production hour the ten-page cap was hit 587 times — at least 5,870 delivery-list requests plus 1,651 redelivery requests, all against the App-JWT-only deliveries API.

PR #24's lease serialized overlapping scans but did not reduce volume.

Proposed behavior

  • Durable high-watermark cursor on delivery ID: steady-state passes scan only deliveries since the last completed pass, plus a bounded overlap for late arrivals. Cost becomes proportional to delivery rate, not window size.
  • Pacing between capped continuations instead of immediate rescheduling.
  • Optionally retain a full-window deep scan at a much lower cadence (hourly/daily) if C-R4's bound requires it; document the guarantee either way.

Acceptance criteria

  • Steady-state request volume for gap healing is O(new deliveries), verified by test.
  • A delivery missed by ingestion is still detected and redelivered within the documented C-R4 bound (regression test with the fake's delivery feed).
  • Cursor survives restarts; a stale/lost cursor falls back safely to a bounded window, never an unbounded rescan.
  • Page-cap continuations are paced and observable without error-level noise.

Observed on v0.3.3.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions