Skip to content

ADR-021 D3's visibility half is still open: pending events expire by deleteMany, not by a status flip #1007

Description

@lilyshen0722

Split out of #993 / #1001 so it is not marked discharged by mistake.

What #1001 did and did not do

ADR-021 D3 (Accepted, ratified 2026-08-13) has two requirements. #1001 (13f68032) satisfies one:

D3 requirement status after #1001
hosted events exempt from the 30-minute deleteMany, TTL above the tolerated-outage window moot — the 30-minute default is gone for everyone; pending now ages out on the same 168h clock as delivered/acked
"expired hosted events retire to failed (visible), never silent deletion" open

D3 is explicit that the second is the point, not a detail:

:691 is deleteMany on pending, not a status flip, so once it fires which users and pods lost turns is unrecoverable (only the aggregate pending=N reaches stdout) … That asymmetry is the whole reason hosted events retire to failed rather than inheriting the default: the fault must survive its own sweep.

agentEventService.ts:715 on main is still deleteMany({ status: 'pending', createdAt: { $lt: deliveredThreshold } }). Later, but still a delete.

What is and isn't covered now

  • Events that reach the cap: covered, and newly so. fix(agents): the collector destroyed the events the requeue had just rescued #1001 gives the lifecycle room to run, so three deliveries at 10–20 min spacing reach attempts >= 3 and the retire pass sets status: 'failed' with a reason. That path had never fired before (0 rows at failed, 0 at attempts >= 3).
  • Events that never reach the cap: not covered. An event no seat ever fetched — uninstalled agent, dead wrapper, a seat down longer than the window — still vanishes at 168h with no row, no status, no error. Silent deletion, later.

The shape this wants

The delete/update split proposed and discarded during #993's thread, on the grounds that "the horizon change handles it for free." It does handle destruction for free. It does not handle evidence, and D3 asks for evidence:

// nobody ever took it, and it has aged out — no consumer, nothing to explain
deleteMany({ status: 'pending', createdAt: { $lt: threshold }, attempts: 0 })

// somebody had it and it aged out — say so, and leave a row for the 168h failed sweep
updateMany(
  { status: 'pending', createdAt: { $lt: threshold }, attempts: { $gte: 1 } },
  { $set: { status: 'failed', error: 'aged out in pending after delivery without an ack' } },
)

Note this is not the attempts-based predicate that was refuted in #993 — that one used attempts to decide whether to destroy, and a merely-stopped seat produces the same row as a dead one. Here attempts only decides how to record an expiry that is happening either way, at a horizon long enough that the distinction no longer gates data loss.

Why file rather than fold into #1001

#1001 was already six commits and five files past its original diff. And the D3 clause is a decision someone ratified for a reason — it deserves its own thread rather than a paragraph in a PR body that a squash-merge flattens.

Related: #993 (root cause), #1001 (the fix that landed), ADR-021 D3.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions