[P0][relay] Investigate and contain relay OOM restarts

[psam21]

Migrated from docs/pending-work-plan.md §4.

Problem

The deployment journal showed repeated relay termination by the Linux OOM killer, with the restart counter reaching 61. This is independent of Blossom's HTTP availability and must be treated as a production incident. Repeated systemd restarts can also make dashboard totals, NIP tests, and upload-adjacent relay authentication appear intermittently broken.

Concrete recent evidence (2026-09-02):

• 09:49:49 — OOM kill, 23.1M memory peak (likely killed by another process's OOM, not the relay itself).
• 10:31:34 — OOM kill, 499.0M memory peak, 234.0M swap.
• 12:16:34 — OOM kill, 512.2M memory peak, 38.0M swap.

Required investigation

Capture the following over a representative window:

• Memory usage over time (RSS, swap).
• Service limits (MemoryMax, MemoryHigh in the systemd drop-in).
• Database pool size and connection counts.
• WebSocket connection counts.
• Cache sizes (events cache, event-kind stats cache, grouped breakdown cache).
• Event processing queue depths.
• Timing of grouped telemetry refreshes.

Determine whether the aggregate query, event cache, NIP test traffic, connection pool, or a combination is responsible.

Containment plan

Immediate containment (independent of root cause):

1. Keep systemd restart behavior but prevent an unbounded memory-heavy dashboard refresh from competing with relay traffic.
2. The grouped cache should have a bounded context, a single-flight refresh, a result-size limit, and a controlled backoff after failure.
3. Memory limits should be used as a guardrail rather than as the primary fix.
4. Any change to MemoryMax, pool sizing, or cache allocation should be tested against authenticated Nostr traffic and the NIP harness.

Acceptance criteria

• Root cause identified and recorded in the issue comment with evidence.
• Containment shipped: bounded grouped cache + single-flight refresh + result-size limit + backoff.
• OOM kills reduced to zero over a 7-day observation window.
• Memory profile documented in the issue.

Dependencies

• Depends on no other issues.
• Blocks [P0][relay] Investigate and contain relay OOM restarts #107 (deploy safeguards) — the deployment script must not declare success while the relay is in an OOM-restart loop.

Source

Migrated from docs/pending-work-plan.md §4.

[psam21]

Dependency map:

• Independent (no blockers).
• Blocks [P0][deploy] Tighten deployment safeguards (clean checkout, atomic rollback, separate status reporting) #108 (deployment script must not declare success while the relay is in an OOM-restart loop).