Skip to content

v1.4.3 — day bitmaps survive an unclean shutdown

Choose a tag to compare

@funkBuild funkBuild released this 01 Sep 06:20
· 21 commits to main since this release
1b8ce24

Fixes time-scoped queries silently returning nothing for series whose data is present and readable.

The bug: day bitmaps gate time-scoped series discovery, but membership was persisted only when the index memtable crossed write_buffer_size — a threshold fed by new series metadata, so a shard writing to established series can go days between flushes — and open() never rebuilt it. Any unclean exit erased every day recorded since the last flush, permanently. A query starting inside that window returned zero series; a wider range returned all of them.

Fixes:

  • Day bitmaps are rebuilt at startup from TSM per-series time bounds after an unclean shutdown (INDEX_CLEAN_SHUTDOWN marks a clean one so ordinary restarts skip the work), and flushed on a 30s timer.
  • recordDaySpan's 366-day clamp is recorded, so a query reaching into history the clamp refused falls back instead of reporting nothing — this is the cause of the "near-epoch points invisible unless endTime >= 1e17" behaviour.
  • Postings: getOrLoadBitmapForInsert now re-marks its entry dirty after the KV load, closing a path that could make a series permanently invisible to every tag-filtered query with no crash involved.
  • Neither loader's in-flight empty placeholder can now be read as "no such series", nor flushed over the full persisted bitmap.
  • Failed WAL appends roll back the dirty flags they cleared, instead of leaving new membership in RAM marked clean.
  • A zero-key or zero-length bloom filter no longer rejects everything; an SSTable with an unreadable metadata region fails loudly instead of answering "absent" forever; a truncated HLL is not flushed over the persisted sketch; single-tag queries report their series limit instead of silently truncating.

Operational note: the repair runs at startup, so upgrading requires a restart to recover already-lost membership. day_bitmap_memberships_repaired_total reports what it restored. New config: index.day_bitmap_flush_interval_seconds (30), index.day_bitmap_rebuild_window_days (32).