Idle log eviction
Topics that were used once and abandoned used to hold their runtime cost forever: every opened partition log kept two goroutines (a 100ms-tick flusher and a retention reaper), open segment file descriptors, and buffer/index memory until the topic was deleted or the process restarted.
Narad now closes any partition log untouched for storage.idle_log_eviction_ms (default 30 minutes, 0 disables, floor 60s — file-configurable) and reopens it lazily, invisibly, on the next produce, consume, or replay (#103).
What makes it safe:
- Observation never keeps a log warm. Metrics polls peek without opening, and a fan-out child that is attached but silent no longer pins its parent: caught-up cursors are answered from the durable high-watermark file (force-synced on every clean close) without reopening the log.
- Committed backlog always opens the log — correctness over eviction, always. A record produced while a cursor waits against a closed log is picked up within 250ms.
- Retention is never stranded: eviction defers until the reaper has finished deleting aged segments.
- No races with produce: the close runs under the same locking discipline as topic deletion, re-verified under lock; a produce-vs-evict hammer test (eviction forced on every sweep) loses zero records under the race detector.
New metrics: narad_open_partition_logs, narad_idle_logs_evicted_total.
Also: the config docs no longer show locked storage internals in the example file — the strict loader rejects them, and the example now matches (data_dir, codec, compression_level, idle_log_eviction_ms are the four accepted keys).
Create short-lived topics and forget to delete them? Now rude but free.
🤖 Generated with Claude Code