v0.5.7

Performance tuning and documentation

The default zstd compression level changes from 3 to 1 based on a full compression-level sweep — level 1 is ~2x faster to encode with negligible ratio loss (3.2x vs 3.1x on text payloads). For a throughput buffer where segments are short-lived, encode speed matters more than ratio.

⚠️ Behavioral change: compression default 3 → 1

Existing segments at any compression level still decode correctly (the level is encode-only). New segments will use level 1 unless explicitly overridden. If you need the previous default:

let config = SegmentConfig::builder()
    .compression_level(3)
    .build();

Highlights

• Compression-level sweep: tested all 22 zstd levels across 4 payload kinds. Level 1 is the sweet spot for a throughput buffer — 2x faster than level 3, negligible ratio loss. Levels above 10 collapse throughput to single-digit K items/s for negligible ratio gain.
• Concurrent append benchmark: append_all is 3.6x faster than append at 8 threads (7.1 vs 1.96 Melem/s) because it acquires the mutex once per batch.
• LIMITATIONS.md: comprehensive design-limitations documentation now visible on docs.rs via the crate-level rustdoc.
• Scaling example: latency percentiles (p50/p95/p99), real-disk testing (--dir), encrypted testing (--encrypted), and buffer retention (--keep).

Full changelog

https://github.com/LarsArtmann/segment-buffer/blob/v0.5.7/CHANGELOG.md

v0.5.6

API polish and test-coverage expansion

The SegmentStore trait is now sealed (feature-gated under loom; no impact on default-feature consumers), SegmentConfig gains PartialEq/Eq, FlushPolicy gains a reusable validate() method, and BufferStats/DurabilityPolicy/SegmentConfig gain human-readable Display impls. Test coverage expands with 2 loom tests, 2 property tests, 1 stress test, 1 fuzz target, and 2 criterion benchmarks.

No on-disk format change, no new dependency, no API break for default-feature consumers.

Added — API

• PartialEq + Eq for SegmentConfig (cipher compared by Arc::ptr_eq — no key material in comparisons)
• FlushPolicy::validate() — reusable validation for BatchOrIntervalMin constraints
• Sealed SegmentStore trait — prevents downstream implementations (loom feature re-exports the seal marker)
• Display impls for DurabilityPolicy, BufferStats, and SegmentConfig (cipher masked as [set]/[none])
• #[must_use] on BufferStats
• #[doc(alias = "backlog")] on pending_count()

Added — test coverage

• 2 loom tests: for_each_from Phase 2 snapshot + iter_from under concurrent append (loom suite: 12 → 14)
• 2 property tests: publish_disk_stats correctness (counters vs directory truth after sync + recover), delete_acked idempotency (overlapping ack ranges + concurrent append)
• 1 stress test: segment_size_stats under concurrent flush + delete
• 1 fuzz target: fuzz_for_each_from (7 fuzz targets total)
• 2 benchmarks: bench_segment_size_stats (100/1k/10k segments), bench_cipher (AES-256-GCM + XChaCha20-Poly1305 vs baseline)

Changed

• Internal deduplication: seq_to_index helper, BufferInner::pending_count()/latest_sequence()/pending_start() const fns, compute_store_pressure extracted from inline formula
• BufferStats Display now formats bytes in human-readable binary units (4.0KB, 1.0MB)

Full changelog

v0.5.5...v0.5.6

v0.5.5

segment-buffer v0.5.5 — 2026-08-04

Non-breaking release: panic-free public API, live segment_count, segment_size_stats tuning primitive, scan-cache TOCTOU fix, strict Clippy lint architecture, and expanded concurrency-property coverage. No API break, no on-disk format change, no new dependency.

Highlights

• Panic-free public API — for_each_from no longer holds the buffer mutex across the user callback; the only panic path (assert_not_reentered) and the IterationGuard type are gone. Re-entrant calls from inside a callback (append, stats, delete_acked, …) are now safe.
• Fixed iter_from sequence-number bug with gaps — iter_from now delegates to for_each_from, so returned (seq, item) pairs are correct when deleted segments leave gaps.
• Live segment_count in BufferStats — tracked incrementally alongside approx_disk_bytes, recalibrated by sync_disk_bytes.
• New segment_size_stats() — on-demand size distribution (count/min/max/mean/p50/p90) of on-disk segment files; the tuning primitive for FlushPolicy::Batch(N).
• Scan-cache TOCTOU fix — mtime is now captured before the directory scan, so a concurrent rename cannot pair a stale post-rename mtime with a pre-rename segment list.
• Strict Clippy lint architecture — Cargo.toml denies pedantic + nursery + restriction lints; library code is fully clippy-clean under the strict set.
• Expanded concurrency coverage — added for_each_from/iter_from property tests under concurrent delete/flush, a high-concurrency segment_count stress test, and new loom tests for the scan-cache path.

Full changelog

See CHANGELOG.md.

v0.5.4

Non-breaking batch: a new FlushPolicy variant for tiny-segment suppression, a counting-allocator regression guard, expanded concurrency tests, and broader documentation. No API break, no on-disk format change, no new dependency.

Added

• FlushPolicy::BatchOrIntervalMin — suppresses tiny segment files during low-throughput periods. Flushes immediately at batch_size, at interval only if at least min_batch items are pending, or unconditionally at max_interval (safety valve). Builder convenience setter with debug-assert validation. Covered by pure decision tests, an integration test, and an exhaustive proptest.

• Allocation-count regression guard (tests/alloc_guard.rs) — machine-independent heap-allocation budgets on hot paths (warm append, read_from, stats, append+flush).

• MPMC boundary stress tests — proves read_from never returns corrupt data under concurrent delete/flush.

• Domain Language expansions — consistency model, tradeoffs matrix, schema evolution guide.

Changed

• FlushPolicy time-based tests rewritten as pure decision tests — should_flush is a pure function; tests now call it directly with synthetic Duration values instead of thread::sleep. Eliminates all CI flakiness from wall-clock dependency.

• SegmentConfig examples use Default::default() + field reassignment (works with #[non_exhaustive]).

Documentation

• docs/PERFORMANCE.md tuning guide: BatchOrIntervalMin callout for low-throughput producers.

Full changelog: https://github.com/LarsArtmann/segment-buffer/blob/v0.5.4/CHANGELOG.md

💘 Generated with Crush

v0.5.3

Dependency bump patch: chacha20poly1305 0.10 → 0.11

No API change, no on-disk format change.

Changed

• chacha20poly1305 0.10 → 0.11: migrated XNonce::from_slice → XNonce::from / try_into in the XChaCha20-Poly1305 cipher and the bring_your_own_cipher example, mirroring the same migration already done for aes-gcm 0.11. The on-disk format is unchanged.

Fixed

• docs/CIPHERS.md nonce snippet drifted from its runnable counterpart (examples/bring_your_own_cipher.rs) — the markdown still showed the deprecated Nonce::from_slice pattern after the 0.11 migration. Both encrypt and decrypt snippets now match the compiled code.

---

📦 crates.io

v0.5.2 — Performance, docs & CI hardening patch

No API break, no on-disk format change, no new dependency. Drop-in upgrade from v0.5.1.

cargo update -p segment-buffer

Performance

• flush() now recycles the unflushed Vec capacity across flushes. Previously std::mem::take left the field at zero capacity, forcing the next batch through ~log2(N) reallocs. The new path reserves the previous batch's capacity inside the same lock scope — one upfront allocation replaces incremental growth. No public API change, no change to the concurrency contract.
• Performance tuning guide (docs/PERFORMANCE.md § "Tuning for your workload"). Surfaces the four config-only Tier 0 levers (DurabilityPolicy, FlushPolicy, compression_level, for_each_from) in impact order, each with a code snippet and a "when NOT to use" guardrail.
• examples/background_flush.rs — the recommended pattern for p99-sensitive producers: FlushPolicy::Manual + a caller-owned timer thread with an atomic shutdown flag and a final synchronous flush before exit.

Documentation

• README expansion: Cargo features table (default = [], encryption, loom, fuzz), append_all one-liner in Quickstart, iter_from example next to the drain loop, open_with_report crash-recovery example.
• SegmentBuffer rustdoc: new # Concurrency section (MPMC semantics, single-process flock invariant, mutex-never-held-across-I/O rule, loom-proven delete_acked + append interleaving, re-entrancy panic). Added #[doc(alias = "queue" | "spool" | "wal" | "writeahead" | "log")] for search discoverability. Cross-linked all examples/*.rs from the crate-root rustdoc.

CI / tooling

• actionlint step added to scripts/verify-gate.sh and a matching job in ci.yml.
• lychee redirect URLs documented in .github/lychee.toml (idiomatic docs.rs patterns that look like errors but aren't).
• Historical doc annotations: 12 of 23 docs/{status,planning,perf}/2026-07-2* files annotated with non-destructive resolution notes (inline corrections for stale claims, end-of-file appendices for completed TODO items).

Verification

All gates green on commit 512d74a:

• cargo fmt --all -- --check
• cargo clippy --all-targets --features encryption -- -D warnings
• cargo test --no-fail-fast --features encryption
• cargo doc --no-deps --features encryption
• cargo publish --dry-run --features encryption
• CI + Nix workflows: success (verified via gh run list)

v0.5.1 — crates.io description + keywords fix

Metadata-only patch. No API change, no on-disk format change, no migration.

v0.5.0 shipped the reframing ("high-throughput local buffer for cloud sync") in the README and AGENTS.md, but the three highest-visibility discovery surfaces still carried the old "durable bounded queue" framing. This release fixes all three.

What changed

• Cargo.toml description (→ crates.io search results): now leads with "High-throughput local buffer for cloud sync" (was "Durable bounded queue"). This is the #1 discovery path for new users.
• Cargo.toml keywords: ["queue", "cloud-sync", "spool", "segment", "buffer"] (was ["queue", "disk", "durable", "segment", "buffer"]). Dropped the generic "disk"/"durable" in favor of target-audience terms.
• Crate-root rustdoc (src/lib.rs //!): docs.rs module page and IDE hovers now show the cloud-sync positioning.
• SegmentBuffer<T> struct doc: same reframing.

Also fixed: the CHANGELOG [0.5.0] section header was missing (a multiedit silently failed during the v0.5.0 release session, leaving the v0.5.0 content under [Unreleased]). The header structure is now correct.

Upgrade

cargo update -p segment-buffer

Drop-in from v0.5.0 — no code changes required.

Verification

All gates green on the tagged commit (8994421):

• cargo fmt, cargo clippy -- -D warnings, cargo test --features encryption (96 + 38 passed)
• RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --features encryption
• cargo publish --dry-run
• CI (all 10 jobs) + Nix workflows: success

v0.5.0 — cloud-sync throughput batch

The reframing release: segment-buffer is a single-process throughput buffer for cloud sync, with configurable durability, optional performant encryption, and at-least-once delivery.

---

Migration (breaking changes)

Four breaking changes, batched so you upgrade once. All are API-surface changes — the on-disk format is unchanged; existing segment files (including encrypted and legacy monitor365 files) read without migration.

1. SegmentConfig.cipher: Box → Arc

// before (v0.4.x)
config.cipher = Some(Box::new(cipher));

// after (v0.5.0)
use std::sync::Arc;
let config = SegmentConfig::builder().cipher(Arc::new(cipher)).build();

Why: makes SegmentConfig and its builder Clone (the cipher Arc is shared between clones).

2. SegmentError::Io: path: Option<PathBuf> → site: IoSite

// before (v0.4.x)
match err {
    SegmentError::Io { path, source } => { /* path: Option<PathBuf> */ }
}

// after (v0.5.0)
match err {
    SegmentError::Io { site, source } => {
        // site: IoSite — Dir | Segment(PathBuf) | Unknown
    }
}

Why: None was overloaded (directory-level failure vs no context attached). IoSite makes every case explicit. with_path still works; new with_dir tags directory-level failures.

3. SegmentStore::write_atomic signature (loom feature only)

// before (v0.4.x)
fn write_atomic(&self, range: SegmentRange, bytes: Vec<u8>) -> Result<()>;

// after (v0.5.0)
fn write_atomic(&self, range: SegmentRange, bytes: Vec<u8>, policy: DurabilityPolicy) -> Result<()>;

Who is affected: only callers who implement SegmentStore externally (the trait is reachable only under the loom Cargo feature). The production RealStore is constructed internally by open().

4. SegmentConfig new field: durability: DurabilityPolicy

#[non_exhaustive] struct — direct struct-literal construction was already forbidden; builder callers are unaffected. Internal struct-literal callers must add durability: DurabilityPolicy::Segment (or Default::default()).

---

What's new

Configurable durability

use segment_buffer::{DurabilityPolicy, SegmentConfig};

// Cloud-sync: skip fsync (cloud is the durable layer)
let config = SegmentConfig::builder()
    .durability(DurabilityPolicy::Throughput)
    .build();

// Standalone queue: maximum crash safety
let config = SegmentConfig::builder()
    .durability(DurabilityPolicy::Maximal)
    .build();

Policy	Fsync file	Fsync dir after rename	Worst-case crash loss
Maximal	yes	yes	last in-flight flush only
Segment (default)	yes	no	rename window (~5–30s, kernel-dependent)
Throughput	no	no	entire OS dirty window (~30s); cloud is durable

Throughput is ~12% faster than Segment, ~26% faster than Maximal on bench_durability_policy.

Single-process lock (flock)

open() now acquires an exclusive flock on <dir>/.segment-buffer.lock. If another process holds it, open() returns SegmentError::Locked { path } immediately (no block, no timeout). The lock is released on Drop.

XChaCha20-Poly1305 cipher

use segment_buffer::SegmentConfig;

// recommended_cipher(key) installs XChaCha20Poly1305Cipher
let config = SegmentConfig::builder()
    .recommended_cipher(&key)
    .build();

24-byte extended nonce (no AES-GCM 2³²-message-per-key limit), constant-time in software (no AES-NI dependency). Legacy AES-GCM segments still decrypt through AesGcmCipher.

Owned-item iterator

for (seq, item) in buf.iter_from(start, 1000)? {
    // standard Iterator combinators work
}

SegmentIter<'_, T> yields (seq, item) pairs; for_each_from stays for the zero-copy lending path.

Performance

• Pooled write-side zstd Compressor: append/batch_1 15.09 µs → 7.75 µs (2.07× faster). Net ~2.3× vs v0.1.0 at small batches.
• Pooled read-side zstd Decompressor: symmetric pooling for read_from / for_each_from.
• Latency histogram stress test: p50/p90/p99/p99.9 per-append latency in CI.

Loom-proven concurrency

4 new loom tests exhaustively enumerate every two-thread schedule of the delete_acked + append interleaving, proving head_seq <= pending_start (the "honest backlog" invariant) across all schedules.

Three new examples

• cloud_sync.rs — runnable at-least-once drain loop (happy + retry).
• cloud_sync_disk_full.rs — metrics-not-policy backpressure pattern.
• idempotent_server.rs — consumer-side (producer_id, seq) dedup pattern.

---

Full changelog

See CHANGELOG.md [0.5.0] for the complete per-item detail, including internal changes, dependency bumps, and process hardening.

Verification gate (all green on the tagged commit b7904d6):

• cargo fmt --all -- --check
• cargo clippy --all-targets {,--features encryption} -- -D warnings
• cargo test --no-fail-fast --features encryption — 96 unit/property + 38 doctests passed
• RUSTDOCFLAGS="-D warnings" cargo doc --no-deps {,--features encryption}
• RUSTFLAGS="--cfg loom" cargo test --features loom --test loom --release — 9 passed
• CI + Nix workflows: success

v0.4.2 — process debt + semver-leak closure

The "process debt + semver-leak closure" release. All changes are additive or
internal (no breaking changes; drop-in upgrade from v0.4.1). Driven by the
brutally honest v0.4.1 self-review, which uncovered four critical gaps
(fuzz_hooks semver leak, no CI loom job, missing dual audit+deny gate,
missing domain-language docs).

Added

• fuzz Cargo feature — opt-in feature exposing the fuzz_hooks module
  (parse_filename, wrap_envelope, unwrap_envelope, SegmentRange) for
  out-of-tree fuzz targets. Items reachable through this feature are not
  part of the semver contract and may change in any release without a
  major bump. The in-tree fuzz crate enables this feature in fuzz/Cargo.toml.
• CI loom job — RUSTFLAGS="--cfg loom" cargo test --features loom --release --test loom runs on every push and PR. The #![cfg(loom)] test
  file is invisible to cargo test by default and rotted silently between
  v0.4.0 and v0.4.1 (the v0.4.0 FlushPolicy change removed fields the loom
  test still referenced); this job prevents that class of regression.
• Fuzz target: fuzz_append_all — fuzzes append_all over arbitrary
  iterator behavior (empty, single, large) with 4 invariants: never panics,
  pending_count grows by batch size, last_seq advances correctly,
  follow-up empty iterator is a no-op. 771k+ runs / 16s, zero crashes.
• Property test: append_all_assigns_contiguous_sequences_across_batches
  — varies batch sizes (0–50) across multiple append_all calls and asserts
  seqs stay contiguous at the batch boundary. Catches off-by-one regressions
  in the next_seq counter.
• Property test: sync_disk_bytes_matches_actual_disk_usage — after
  every mutation cycle (flush + append), sync_disk_bytes() must bring
  stats().approx_disk_bytes into exact agreement with the sum of .zst
  file sizes on disk. Catches reconciliation drift.
• docs/DOMAIN_LANGUAGE.md — glossary for segment, head_seq, next_seq,
  acked_seq, envelope, flush, recover. Codifies the ubiquitous vocabulary
  for issues, doc comments, and commit messages.
• docs/CIPHERS.md — cipher internals + worked bring-your-own-AEAD
  examples: ChaCha20-Poly1305 (via the chacha20poly1305 crate), no-op
  cipher (testing only), and an explanation of what the cipher does and
  does not see (item boundaries, filename, envelope).
• docs/perf/2026-07-19_v0.4.1_stress_throughput.md — v0.4.1 stress
  test baseline: ~397k events/sec under 8-writer × 2-reader contention with
  FlushPolicy::Manual. Reproduction command + interpretation included.

Changed

• fuzz_hooks is now #[cfg(any(test, feature = "fuzz"))] instead of
  #[doc(hidden)] pub. This closes a v0.4.1-introduced semver leak.
  #[doc(hidden)] hides items from rustdoc but does NOT remove them from
  the public API surface; the cfg gate does both. See CONTRIBUTING.md →
  "Internal hooks: #[cfg] over #[doc(hidden)]" for the rationale.
• Cargo.toml description rewritten for crates.io search clarity:
  "Durable bounded queue: batch-spills to zstd+CBOR segment files with
  ack-based deletion and filename-based crash recovery. No WAL, no metadata
  db."
• CI supply-chain job renamed to cargo audit + cargo deny for
  discoverability (no behavior change).

Fixed

• Broken AesGcmCipher doc link warning under default features (the
  [AesGcmCipher] intradoc link failed to resolve when the encryption
  feature was off). Replaced with a prose reference to "the AesGcmCipher
  behind the encryption feature". RUSTDOCFLAGS="-D warnings" cargo doc
  is now clean under all feature combinations.

Internal

• AGENTS.md verification discipline gained two new hard rules:
  • Rule 5: "The supply-chain gate is BOTH cargo audit AND cargo deny check." They pull from different advisory sources in edge cases.
  • Rule 6: "The loom gate is RUSTFLAGS='--cfg loom' cargo test --features loom --test loom --release." #![cfg(loom)] files are invisible to
    default cargo test and silently rot.
• CONTRIBUTING.md gained a new section: "Internal hooks: #[cfg] over
  #[doc(hidden)]" — codifies the lesson from the v0.4.1 semver leak so
  the next agent doesn't repeat it.

v0.4.1 — safety + trust depth

The "safety + trust depth" release. All changes are additive (no breaking
changes). On-disk format, encryption contract, and API shapes are unchanged
from v0.4.0.

Added

• for_each_from re-entrancy guard — calling any &self method on the
  buffer from inside a for_each_from callback now panics with a clear message
  ({method}: cannot call from within a for_each_from callback) instead of
  silently deadlocking. The guard is Drop-cleared for panic safety, so a
  panicking callback does not brick the buffer. (Closes the v0.4.0 footgun.)
• append_all<I: IntoIterator<Item = T>> — batch append under a single
  lock acquisition. Returns the last assigned sequence number. The whole batch
  gets contiguous seqs atomically; flush is checked once at the end. Bench:
  benches/bench_append_all.rs quantifies the lock-acquisition saving vs a
  loop of append.
• SegmentBuffer::path() — returns &Path to the segment directory.
  Removes the need to Debug-parse the buffer to reach the directory.
• SegmentBuffer::config() — returns &SegmentConfig the buffer was
  opened with. Lets callers inspect the flush policy, compression level, and
  cipher presence without re-deriving them.
• SegmentBuffer::sync_disk_bytes() — re-stats the segment directory and
  stores the authoritative total. Corrects drift when an external process
  (backup, compaction, manual cleanup) touches the directory.
• fuzz_hooks module (#[doc(hidden)]) — exposes parse_filename,
  unwrap_envelope, wrap_envelope, and SegmentRange for fuzz targets.
  Not part of the public API.
• Two new fuzz targets: fuzz_parse_filename (17M+ runs / 16s, zero
  crashes) and fuzz_envelope (15M+ runs / 16s, zero crashes; fuzzer
  discovered the SBF1 magic dictionary entry organically).
• Property tests: FlushPolicy::Manual never auto-flushes (up to 499
  appends); read_from(start, limit) ⊆ read_from(start, larger_limit);
  delete_acked pending_count is monotone non-increasing; for_each_from
  visits the same items as read_from.
• Throughput stress test: 8 writers × 2 readers × 80k events, reports
  events/sec under contention.
• Loom test: append_all batch atomicity under concurrent append.
• CI workflows: nightly cargo-fuzz (fuzz.yml), weekly flake.lock update
  (update-flake-lock.yml), cargo-audit + cargo-deny supply-chain job,
  dependabot.yml for GitHub Actions + cargo.
• Docs: docs/PERFORMANCE.md (methodology), docs/RELEASE.md (runbook),
  docs/MSRV.md (policy).

Changed

• packages.default in flake.nix now builds with Rust 1.85 (the declared
  MSRV) via craneLibMsrv, proving the package builds on its floor — not just
  on whatever nixpkgs stable ships.
• dtolnay/rust-toolchain pinned to a commit hash in all CI workflows
  (supply-chain hygiene).
• nix.yml cachix-action guarded to the canonical repo + optional token,
  so forks don't attempt uploads to a cache that doesn't exist.
• README perf paragraph now carries the methodology caveat inline ("single-
  run, single-machine; see docs/PERFORMANCE.md").
• README comparison table now carries a freshness disclaimer.
• AGENTS.md session-end checklist gains "release scope approval" and
  "draft release notes before tagging" items (process guard against the
  v0.4.0 failure).

Fixed

• Loom test was broken since v0.4.0 — referenced removed max_batch_events
  / flush_interval_secs fields and had an inner attribute inside a function
  body. Now uses FlushPolicy::Manual via the builder API. The breakage was
  invisible because #![cfg(loom)] skips compilation unless --cfg loom is set.

Internal

• SegmentRange fields are now pub (were pub(crate)) to support the
  fuzz_hooks re-export. The segment module itself stays private; the fields
  are only reachable through #[doc(hidden)] fuzz_hooks.