Reduce peak RSS: smaller read buffers + per-role SQLite cache budgets

[martinus]

Two memory optimizations for large trees, both measured perf-neutral. Investigation started from massif: peak heap was 88% the per-thread read buffer, and the 64 MiB SQLite cache (a lazily-filled ceiling) multiplies across the walker and search-pool connections at scale.

Changes

1. Read/hash buffer 8 MiB → 1 MiB (file_scan.c). One static __thread buffer per csum thread, so at --io-threads=8 the old size cost 64 MiB of resident buffers on large-file trees. Files larger than the buffer are already read in successive passes, and 1 MiB saturates sequential read throughput (scan is I/O/metadata-bound).
2. Per-connection SQLite cache budgets (dbfile). Every connection defaulted to 64 MiB. Connections that don't run the heavy dedupe joins now get less: the find_dupes search pool (up to cpu_threads connections) → 32 MiB, and the walkers (which only read the fs-uuid config, if that) → 2 MiB. The loader/reader/writer keep 64 MiB. Hot pages stay warm in the shared OS page cache, so the joins keep their speed. (sqlite3_soft_heap_limit64 was ruled out — the distro SQLite isn't built with SQLITE_ENABLE_MEMORY_MANAGEMENT.)

Benchmarks (btrfs, io/cpu-threads=8, interleaved A/B via scripts/bench-ram.sh)

Workload	wall base → new	peak RSS base → new
Large files (2 GB, read-buffer path)	2.00 s → 2.00 s	70 → 15 MiB (−54)
250k files, dup-heavy (66 MiB hashfile)	10.92 s → 10.91 s	200 → 167 MiB (−35)

The #2 saving grows with hashfile size (this was only a 66 MiB hashfile; the 870 MiB/1.7M-file reports are where the ≤8 search connections × 64 MiB really bite).

Also

• New reproducible harness scripts/bench-ram.sh (non-destructive oans -r, peak RSS + wall).
• Correctness: scripts/verify.sh green (88 tests + valgrind).

Follow-ups not in this PR (measured separately): shrinking seen_inodes (~50 B/file) and trying PRAGMA mmap_size.

🤖 Generated with Claude Code