v1.5.0 — retention runs (downsampling cascade), index crash-recovery hardening

Highlights

Retention now actually runs, and downsampling is a cascade (#8)

Per-measurement retention had an API, a persisted policy and a working fold, but none of it ran: TSMCompactor::setRetentionContext() had no production caller. Downsampling never happened. TTL was only applied by deleting files that had fully expired, so a file with a single point still inside retention kept all of its expired points forever.

• Retention is wired through an Engine-injected RetentionContextProvider. If the provider fails, the compaction fails instead of silently compacting without retention.
• Downsampling cascade: downsampleTiers accepts 1–4 ordered stages, e.g. 1 Hz → 1m after 7d → 15m after 90d. Each tier's interval must evenly divide the next tier's, so no bucket straddles a coarser boundary. The legacy downsample object mirrors the finest tier.
• Age-driven sweep: series that have stopped receiving writes are still folded, with no persisted watermark.
• Per-field methods: fieldMethods routes, for example, totalizers to sum/max and status words to latest. Booleans and strings never fold by default and are never coerced to 1.0/0.0.
• Rollup state is preserved across stages (TSM V4, see upgrade notes). Point counts and original latest timestamps survive cascading, so min/max/sum/latest compose exactly. avg across stages is documented as an unweighted mean of bucket means.
• Cascade compaction is 13–21% faster from run-batched folding. Single-fold output is bit-identical to the per-point implementation.

See docs/api-retention.md and docs/downsampling-cascade-plan.md.

Index crash recovery and transactional retries

• Index metadata and exact day membership are synced to the index WAL before the data WAL, so an acknowledged write is always discoverable after a crash.
• Field/tag schema updates are serialised read-modify-write transactions, so local series creation and schema broadcasts cannot overwrite each other's unions.
• Failed WAL appends, SST flushes and rotations keep their batches and memtables for retry instead of dropping or resurrecting state.
• An upgrade performs a one-time full postings repair. An unclean start repairs day membership from TSM bounds, and outside the verified range discovery falls back to unpruned filtering rather than answering with a wrong empty. See docs/index-recovery.md.

Cached bitmap lifetime

Postings/day bitmaps are now read and updated through synchronous visitors rather than pointers into the cache. A ready-future preemption can no longer expose an entry after a rehash or eviction.

Derived queries, forecasting and anomaly detection

• /derived supports per-series fan-out (by {tag}) and opt-in multi-series arithmetic (multi_series), with explicit pairing and broadcast rules. A mismatch is an error, never a silent intersection.
• Forecast and anomaly results carry per-group series and declined-group counts. See docs/api-derived.md, docs/forecasting.md and docs/anomaly-detection.md (including known limitations).

Upgrade notes

• On-disk format: compaction now writes TSM V4. Raw flushes still write V3. 1.5.0 reads V2–V4. Older binaries cannot read V4 files, so once 1.5.0 has compacted anything, rolling back to 1.4.x loses access to that data. Take a volume snapshot before upgrading if you need a rollback path.
• First boot is slower: the index clean-shutdown marker moves to recorder generation 3 and a one-time full postings repair runs. The service is not ready until it finishes.
• Writes pay an index durability barrier once per batch.
• Retention policies that were stored but inert will start to apply. Check GET /retention before upgrading. A policy set months ago will begin deleting and downsampling data on the first compaction.
• Retention records are read leniently from this version on. Downgrading to a build that predates the cascade drops policies whose record contains downsampleTiers/fieldMethods (logged as "Failed to parse retention policy JSON"). Nothing stored is lost.

CI on the release commit: 3,909 C++ unit tests — release build 3,909 passed; coverage (Seastar debug) build 3,906 passed, 3 skipped, 0 failed; lib/ line coverage 84.7%.

🤖 Generated with Claude Code