v1.5.0 — retention runs (downsampling cascade), index crash-recovery hardening

Highlights

Retention now actually runs, and downsampling is a cascade (#8)

Per-measurement retention had an API, a persisted policy and a working fold, but none of it ran: TSMCompactor::setRetentionContext() had no production caller. Downsampling never happened. TTL was only applied by deleting files that had fully expired, so a file with a single point still inside retention kept all of its expired points forever.

• Retention is wired through an Engine-injected RetentionContextProvider. If the provider fails, the compaction fails instead of silently compacting without retention.
• Downsampling cascade: downsampleTiers accepts 1–4 ordered stages, e.g. 1 Hz → 1m after 7d → 15m after 90d. Each tier's interval must evenly divide the next tier's, so no bucket straddles a coarser boundary. The legacy downsample object mirrors the finest tier.
• Age-driven sweep: series that have stopped receiving writes are still folded, with no persisted watermark.
• Per-field methods: fieldMethods routes, for example, totalizers to sum/max and status words to latest. Booleans and strings never fold by default and are never coerced to 1.0/0.0.
• Rollup state is preserved across stages (TSM V4, see upgrade notes). Point counts and original latest timestamps survive cascading, so min/max/sum/latest compose exactly. avg across stages is documented as an unweighted mean of bucket means.
• Cascade compaction is 13–21% faster from run-batched folding. Single-fold output is bit-identical to the per-point implementation.

See docs/api-retention.md and docs/downsampling-cascade-plan.md.

Index crash recovery and transactional retries

• Index metadata and exact day membership are synced to the index WAL before the data WAL, so an acknowledged write is always discoverable after a crash.
• Field/tag schema updates are serialised read-modify-write transactions, so local series creation and schema broadcasts cannot overwrite each other's unions.
• Failed WAL appends, SST flushes and rotations keep their batches and memtables for retry instead of dropping or resurrecting state.
• An upgrade performs a one-time full postings repair. An unclean start repairs day membership from TSM bounds, and outside the verified range discovery falls back to unpruned filtering rather than answering with a wrong empty. See docs/index-recovery.md.

Cached bitmap lifetime

Postings/day bitmaps are now read and updated through synchronous visitors rather than pointers into the cache. A ready-future preemption can no longer expose an entry after a rehash or eviction.

Derived queries, forecasting and anomaly detection

• /derived supports per-series fan-out (by {tag}) and opt-in multi-series arithmetic (multi_series), with explicit pairing and broadcast rules. A mismatch is an error, never a silent intersection.
• Forecast and anomaly results carry per-group series and declined-group counts. See docs/api-derived.md, docs/forecasting.md and docs/anomaly-detection.md (including known limitations).

Upgrade notes

• On-disk format: compaction now writes TSM V4. Raw flushes still write V3. 1.5.0 reads V2–V4. Older binaries cannot read V4 files, so once 1.5.0 has compacted anything, rolling back to 1.4.x loses access to that data. Take a volume snapshot before upgrading if you need a rollback path.
• First boot is slower: the index clean-shutdown marker moves to recorder generation 3 and a one-time full postings repair runs. The service is not ready until it finishes.
• Writes pay an index durability barrier once per batch.
• Retention policies that were stored but inert will start to apply. Check GET /retention before upgrading. A policy set months ago will begin deleting and downsampling data on the first compaction.
• Retention records are read leniently from this version on. Downgrading to a build that predates the cascade drops policies whose record contains downsampleTiers/fieldMethods (logged as "Failed to parse retention policy JSON"). Nothing stored is lost.

CI on the release commit: 3,909 C++ unit tests — release build 3,909 passed; coverage (Seastar debug) build 3,906 passed, 3 skipped, 0 failed; lib/ line coverage 84.7%.

🤖 Generated with Claude Code

v1.4.4

Fixes

Batch writes never recorded day-bitmap membership (#5). Engine::insertBatch recorded membership from request timestamps the memory store had already moved out, so the loop was a no-op on every batch write since 1.4.1. Established series only had membership for the day their first-batch MetadataOp announced (plus whatever the startup repair restored), and time-scoped queries whose range started after that returned nothing with the data present. In production the discoverable builtIn.node.stats series in one project decayed 194 → 161 → 111 → 2 → 1 → 0 over the six days after the 1.4.3 restart. Membership is now recorded before the data write. Regression test goes through the real batch path.

One repair on upgrade, even after a clean stop (#5). The clean-shutdown marker now carries a recorder generation ("2"); open() treats any other value as unclean and runs the day-bitmap repair once. Expect Repaired day bitmaps: … on every shard on the first boot of 1.4.4; later restarts skip it as before.

Decode failures name the block (#6). A block decode error now carries [block offset=… size=… count=… time=a..b hdr.count=… hdr.tsBytes=… skip=… n=… valueBytes=…] ahead of [tsm <file>], so descriptor-vs-bytes can be told apart from the log line alone. readSingleBlockImpl also copies the block descriptor before its DMA suspension instead of re-reading it through the caller's reference afterwards.

Upgrade notes

• First boot repairs day bitmaps for the last index.day_bitmap_rebuild_window_days (32) days from TSM bounds — ~4 s per shard at ~75k series. The service is not ready until it finishes.
• No on-disk format change. 1.4.3 remains a valid rollback target (it will run the repair on its own first boot because the marker generation differs).

v1.4.3 — day bitmaps survive an unclean shutdown

Fixes time-scoped queries silently returning nothing for series whose data is present and readable.

The bug: day bitmaps gate time-scoped series discovery, but membership was persisted only when the index memtable crossed write_buffer_size — a threshold fed by new series metadata, so a shard writing to established series can go days between flushes — and open() never rebuilt it. Any unclean exit erased every day recorded since the last flush, permanently. A query starting inside that window returned zero series; a wider range returned all of them.

Fixes:

• Day bitmaps are rebuilt at startup from TSM per-series time bounds after an unclean shutdown (INDEX_CLEAN_SHUTDOWN marks a clean one so ordinary restarts skip the work), and flushed on a 30s timer.
• recordDaySpan's 366-day clamp is recorded, so a query reaching into history the clamp refused falls back instead of reporting nothing — this is the cause of the "near-epoch points invisible unless endTime >= 1e17" behaviour.
• Postings: getOrLoadBitmapForInsert now re-marks its entry dirty after the KV load, closing a path that could make a series permanently invisible to every tag-filtered query with no crash involved.
• Neither loader's in-flight empty placeholder can now be read as "no such series", nor flushed over the full persisted bitmap.
• Failed WAL appends roll back the dirty flags they cleared, instead of leaving new membership in RAM marked clean.
• A zero-key or zero-length bloom filter no longer rejects everything; an SSTable with an unreadable metadata region fails loudly instead of answering "absent" forever; a truncated HLL is not flushed over the persisted sketch; single-tag queries report their series limit instead of silently truncating.

Operational note: the repair runs at startup, so upgrading requires a restart to recover already-lost membership. day_bitmap_memberships_repaired_total reports what it restored. New config: index.day_bitmap_flush_interval_seconds (30), index.day_bitmap_rebuild_window_days (32).

v1.4.2 — rebuild measurement blooms at startup

Bug fix

Stale measurement blooms are now repaired at startup, before serving (#3, follow-up to #2 / v1.4.1).

v1.4.1 only scheduled the rebuild "for the next index flush" — but that flush is triggered solely by 16 MB of new-series index entries, close() skipped it when the memtable was empty, and nothing external can force one. On a steady fleet the ≤1.4.0 blooms therefore stayed on disk and scoped queries stayed blind except where a by {} enumeration had happened to warm the bitmap cache.

• open() rebuilds and persists every measurement's bloom from the persisted postings keys before the HTTP server serves a request (one prefix scan per measurement; seconds for a few thousand).
• close() always flushes dirty caches (bitmaps, day bitmaps, HLLs, blooms), not only when the memtable has entries.
• Only the small MEASUREMENT_BLOOM index records are rewritten; TSM/WAL data and postings bitmaps are untouched.

Regression test: PostingsBitmapTest.OpenRepairsStalePersistedBloomImmediately plants a bloom missing a series and asserts a bare open() makes scoped lookups correct with no other activity.

v1.4.1 — fix scoped queries silently dropping series (measurement bloom false negatives)

Bug fix

Scoped queries could silently return no series for tag values that exist (#2).

flushDirtyMeasurementBlooms rebuilt a measurement's bloom filter — consulted before every scoped postings lookup — from the in-memory bitmap cache alone after the first full scan. The cache never holds every bitmap, so any tag value not queried since the last full scan dropped out of the bloom at the next rebuild and was rejected before reaching the KV store from then on: a permanent false negative. by {tag} enumeration was unaffected, which is how it was caught (193 of 369 tag values invisible in one production measurement on 1.4.0).

• Blooms are now always rebuilt from the persisted postings keys.
• open() schedules a one-time rebuild of every measurement's bloom at the first flush, so upgrading repairs blooms written by ≤1.4.0 with no operator action.
• Read-only defect: no data or index entries were lost; queries simply under-reported.

Regression test: PostingsBitmapTest.BloomRebuildKeepsSeriesNotResidentInCache.

v1.4.0 — durable acks, bounded non-numeric query plans, coalesced reads

Highlights

This release addresses the multi-second bucketed queries and reactor stalls reported on bool-heavy IoT measurements, and closes a write-durability gap.

Query performance — many-series and boolean/string workloads

• Boolean series with an aggregationInterval no longer materialise their full raw range. They ride the bucketed-LATEST pushdown (sparse-stat resolution / early-terminating reverse scan for coarse grids, the batched general pushdown for dense grids), answering from the few blocks that decide buckets. Incident shape (8 bool series × 3M points): whole-range bucket 1717ms → 0.5ms, 1h buckets 1677ms → 49ms, 5m buckets 1690ms → 54ms. Results are bit-identical to the canonical LATEST-per-bucket semantics — pinned by a new plan-equivalence test across memory/TSM/overlapping-file placements.
• String series (and LWW-gated booleans) use the bounded chunked reader as their normal path: peak memory of one chunk instead of the whole range, with reactor yields throughout. The non-numeric reduction loops are fully yield-chunked — the 80–150ms reactor stalls seen in production are gone.
• Cross-series DMA read elevator: concurrent series' block reads against the same TSM file are sorted, gap-merged (≤128KB) and issued as a few large reads instead of one small random read per (series, file). Measured on 960-series queries: 4,951 → 1,161 ops (raw, multi-file), 998 → 84 ops (bucketed LATEST). On IOPS-throttled volumes this translates directly to latency.
• Dense bucket grids (>256 buckets) for LATEST/FIRST now use the batched read plan instead of per-block reverse scans (same answers, ~100× fewer read ops on fine grids).

Durability — acknowledged writes survive crashes

• New [storage] wal_sync_mode, default "always" (group commit): a write is acknowledged only after its bytes are flushed and fdatasync'd. Concurrent writers share one flush round, so the cost amortises: insert throughput is unchanged (11.9M pts/s warm on the reference bench) and p99 batch latency improved (38.5ms → 21.9ms). Previously, a SIGKILL/OOM-kill silently lost the buffered tail of acked writes — up to 256KiB per shard-WAL with no time bound, hitting compressed boolean series 10–100× harder than float. The kill-test that lost 60k/12k/1k acked points now loses zero.
• "interval" mode (ack-immediate, periodic flush every wal_sync_interval_ms) is available for fsync-hostile storage; "rollover" preserves the legacy behaviour.
• WAL segment preallocation now follows wal_size_threshold (was a stale 16 MiB).

Diagnostics

• TSM read/decode failures now name the file: ... CompressedSlice - attempted to read beyond buffer bounds [tsm shard_1/tsm/000123.tsm] — in the dropped-series log, QUERY_INCOMPLETE responses, and compaction errors. Corrupt blocks can finally be traced to the file that holds them.

Upgrade notes

• No format changes; TSM/WAL files from v1.3.0 read unchanged.
• The durability default changed from the legacy volatile-until-rotation behaviour to group commit. On fsync-slow volumes, if ingest latency regresses, set wal_sync_mode = "interval".

All suites green: 3,591 C++ unit / 46 perf / 184 jest / 8 standalone.

🤖 Generated with Claude Code

v1.3.0 — rollup.js migration compat + reactor stall fixes

Query: rollup.js migration compatibility (opt-in, defaults unchanged)

Two per-request options for readers migrating from rollup.js-style aggregation, available on both the JSON and protobuf query interfaces:

• bucketAlignment: "start" (bucket_alignment): anchors the bucket grid at startTime instead of the epoch — bucket = startTime + floor((ts−startTime)/interval)·interval. Labels remain bucket starts, empty buckets remain omitted, endTime remains inclusive. Unknown values are rejected with INVALID_QUERY. Anchored queries take a single query plan on every data placement, so results cannot depend on where the data lives.
• booleansAsNumeric: true (booleans_as_numeric): boolean fields aggregate arithmetically as 1.0/0.0 (avg of [t,t,f,t,f] = 0.6) and are returned as numbers, raw reads included. Strings remain non-numeric. A boolean series whose read fails under this flag reports QUERY_INCOMPLETE rather than silently substituting a latest-per-bucket value.

The canonical defaults — epoch-aligned buckets, booleans as non-numeric latest-per-bucket — are unchanged.

Reactor stall elimination on the query path

A production incident (2 vCPU container) showed reactor stalls up to 980 ms during large queries. Root cause was an accidentally quadratic per-series merge in the fallback aggregation path, compounded by zero yield points across aggregation and response serialization.

• The shard-side map phase now emits one compact raw partial per series (O(N), was O(K²·N)): the incident shape (195 series / 3.5M points) dropped from ~5,100 ms to well under 1 ms of map-phase work.
• The full aggregation chain, protobuf response encoder, JSON response formatter, and series discovery are now coroutines that yield to the reactor on bounded chunks. End-to-end reproduction of the incident workload logs zero reactor stall warnings (was: stalls up to 980 ms), with server-side execution around 200 ms.
• NaN handling in the raw merge paths now matches the documented policy on every path (NaN never folds, never counts); pinned by new merge-correctness tests.
• New perf gate: AggregationStallBench measures both wall time and the longest reactor monopolization via a concurrent probe.

CI

• The SinglePointEncodeCost wall-clock pins skip under unoptimized/instrumented builds; the Coverage job is green again.

Suite: 3,586 C++ unit tests, all perf gates, 184 jest + 8 standalone API tests.

v1.2.5

Full Changelog: v1.2.4...v1.2.5

v1.2.4

test(e2e): update LatestAggregationQuery to the current no-interval s…

V1.2.3

Type enforcement on ingest