Skip to content

feat(selfhost): add bounded retention for webhook_events, audit_events, ai_usage_events, product_usage_events, agent_context_snapshots #3896

Description

@JSONbored

Context

signal_snapshots was already fixed to a bounded, deduped latest-per-key retention (#3810/#3857). A follow-up sweep found five more tables that grow forever with no prune/delete path anywhere in src/:

  • webhookEvents (src/db/schema.ts) — one row per inbound webhook delivery, never deduped or aged out.
  • auditEvents (src/db/schema.ts) — one row per contributor-affecting decision, written from 170+ call sites via recordAuditEvent.
  • aiUsageEvents (src/db/schema.ts) — one row per AI review/summary call; only needed for the rolling daily-neuron-budget window (sumAiEstimatedNeuronsSince).
  • productUsageEvents (src/db/schema.ts) — rolled up daily into productUsageDailyRollups, but the raw per-event rows are never deleted even after being folded into that rollup.
  • agentContextSnapshots (src/db/schema.ts) — one payload blob per agent run, capped only by a per-runId read .limit(50), not a delete.

None of the backup/litestream services in docker-compose.yml touch app-level row retention — they only snapshot/replicate the whole database file, so this is a genuine unbounded growth vector distinct from #3810 (which is scoped to signal_snapshots + central-cloud D1 size probing/alerting specifically).

Requirements

  • Add a periodic maintenance pass (alongside the existing maybeRunAgentMaintenance in src/queue/processors.ts) that prunes rows older than a configurable retention window for webhook_events, audit_events, and agent_context_snapshots.
  • For ai_usage_events, prune while always preserving at least the current UTC day (needed by sumAiEstimatedNeuronsSince).
  • For product_usage_events, prune rows once they are older than the oldest rollup window they still need to contribute to (PRODUCT_USAGE_RETENTION_MAX_WINDOW_DAYS, already computed in src/db/repositories.ts).
  • Make each retention window configurable with sane defaults; fail-safe (a pruning failure must never affect the review/gate path).

Acceptance criteria

  • All five tables have a bounded, tested retention mechanism.
  • Existing rollup/budget-window consumers (sumAiEstimatedNeuronsSince, productUsageDailyRollups) are unaffected by pruning.
  • 100% branch coverage on the new pruning logic; a regression test proves pruning doesn't remove rows still needed by a live consumer.

Parent: #1667

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Status
In Progress

Relationships

None yet

Development

No branches or pull requests

Issue actions