Skip to content

v0.5.0 — cloud-sync throughput batch

Choose a tag to compare

@LarsArtmann LarsArtmann released this 20 Jul 05:35
· 197 commits to master since this release

The reframing release: segment-buffer is a single-process throughput buffer for cloud sync, with configurable durability, optional performant encryption, and at-least-once delivery.


Migration (breaking changes)

Four breaking changes, batched so you upgrade once. All are API-surface changes — the on-disk format is unchanged; existing segment files (including encrypted and legacy monitor365 files) read without migration.

1. SegmentConfig.cipher: Box → Arc

// before (v0.4.x)
config.cipher = Some(Box::new(cipher));

// after (v0.5.0)
use std::sync::Arc;
let config = SegmentConfig::builder().cipher(Arc::new(cipher)).build();

Why: makes SegmentConfig and its builder Clone (the cipher Arc is shared between clones).

2. SegmentError::Io: path: Option<PathBuf> → site: IoSite

// before (v0.4.x)
match err {
    SegmentError::Io { path, source } => { /* path: Option<PathBuf> */ }
}

// after (v0.5.0)
match err {
    SegmentError::Io { site, source } => {
        // site: IoSite — Dir | Segment(PathBuf) | Unknown
    }
}

Why: None was overloaded (directory-level failure vs no context attached). IoSite makes every case explicit. with_path still works; new with_dir tags directory-level failures.

3. SegmentStore::write_atomic signature (loom feature only)

// before (v0.4.x)
fn write_atomic(&self, range: SegmentRange, bytes: Vec<u8>) -> Result<()>;

// after (v0.5.0)
fn write_atomic(&self, range: SegmentRange, bytes: Vec<u8>, policy: DurabilityPolicy) -> Result<()>;

Who is affected: only callers who implement SegmentStore externally (the trait is reachable only under the loom Cargo feature). The production RealStore is constructed internally by open().

4. SegmentConfig new field: durability: DurabilityPolicy

#[non_exhaustive] struct — direct struct-literal construction was already forbidden; builder callers are unaffected. Internal struct-literal callers must add durability: DurabilityPolicy::Segment (or Default::default()).


What's new

Configurable durability

use segment_buffer::{DurabilityPolicy, SegmentConfig};

// Cloud-sync: skip fsync (cloud is the durable layer)
let config = SegmentConfig::builder()
    .durability(DurabilityPolicy::Throughput)
    .build();

// Standalone queue: maximum crash safety
let config = SegmentConfig::builder()
    .durability(DurabilityPolicy::Maximal)
    .build();
Policy Fsync file Fsync dir after rename Worst-case crash loss
Maximal yes yes last in-flight flush only
Segment (default) yes no rename window (~5–30s, kernel-dependent)
Throughput no no entire OS dirty window (~30s); cloud is durable

Throughput is ~12% faster than Segment, ~26% faster than Maximal on bench_durability_policy.

Single-process lock (flock)

open() now acquires an exclusive flock on <dir>/.segment-buffer.lock. If another process holds it, open() returns SegmentError::Locked { path } immediately (no block, no timeout). The lock is released on Drop.

XChaCha20-Poly1305 cipher

use segment_buffer::SegmentConfig;

// recommended_cipher(key) installs XChaCha20Poly1305Cipher
let config = SegmentConfig::builder()
    .recommended_cipher(&key)
    .build();

24-byte extended nonce (no AES-GCM 2³²-message-per-key limit), constant-time in software (no AES-NI dependency). Legacy AES-GCM segments still decrypt through AesGcmCipher.

Owned-item iterator

for (seq, item) in buf.iter_from(start, 1000)? {
    // standard Iterator combinators work
}

SegmentIter<'_, T> yields (seq, item) pairs; for_each_from stays for the zero-copy lending path.

Performance

  • Pooled write-side zstd Compressor: append/batch_1 15.09 µs → 7.75 µs (2.07× faster). Net ~2.3× vs v0.1.0 at small batches.
  • Pooled read-side zstd Decompressor: symmetric pooling for read_from / for_each_from.
  • Latency histogram stress test: p50/p90/p99/p99.9 per-append latency in CI.

Loom-proven concurrency

4 new loom tests exhaustively enumerate every two-thread schedule of the delete_acked + append interleaving, proving head_seq <= pending_start (the "honest backlog" invariant) across all schedules.

Three new examples

  • cloud_sync.rs — runnable at-least-once drain loop (happy + retry).
  • cloud_sync_disk_full.rs — metrics-not-policy backpressure pattern.
  • idempotent_server.rs — consumer-side (producer_id, seq) dedup pattern.

Full changelog

See CHANGELOG.md [0.5.0] for the complete per-item detail, including internal changes, dependency bumps, and process hardening.

Verification gate (all green on the tagged commit b7904d6):

  • cargo fmt --all -- --check
  • cargo clippy --all-targets {,--features encryption} -- -D warnings
  • cargo test --no-fail-fast --features encryption — 96 unit/property + 38 doctests passed
  • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps {,--features encryption}
  • RUSTFLAGS="--cfg loom" cargo test --features loom --test loom --release — 9 passed
  • CI + Nix workflows: success