Skip to content

v0.1.1 — send_batch + LZ4 OOM fix

Choose a tag to compare

@Vadimus1983 Vadimus1983 released this 25 May 17:15
· 12 commits to main since this release

Security

  • Fixed: LZ4 decompress OOM (DoS). Records with Compression::Lz4 whose
    payload had an adversarial 4-byte LE size prefix could coerce
    lz4_flex::decompress_size_prepended into a ~4 GiB Vec::with_capacity
    call before the compressed bytes were validated, OOM-crashing the host
    process. Triggerable by any read of a corrupt or adversarial segment file.
    Fixed by bounding the claimed decompressed size at 16 MiB before delegating
    to lz4_flex (matching the existing Zstd cap). Discovered via the new
    cargo-fuzz decode_record_structured target. v0.1.0 is yanked.

Added

  • Producer::send_batch(Vec<(Option<Bytes>, Bytes)>) and
    Producer::send_batch_records(Vec<Record>) — atomic, single-mpsc-round-trip
    batched appends. ~10× throughput vs a loop of send() at N = 1024
    (2.53 M rec/s vs 252 K rec/s, criterion-measured, 256 B records, in-process).
  • Partition::append_batch(Vec<Record>) -> Result<Vec<u64>> — the actor
    primitive underlying Producer::send_batch.
  • Two new criterion benches: send_batch (batch-size × codec sweep),
    config_sweep (segment_size_threshold / index_interval / preset configs).
  • Cargo-fuzz scaffold under fuzz/ with five targets: decode_record,
    decode_record_structured, recovery_torn_tail, sparse_index_parse,
    log_op_sequence. WSL2 setup documented in the README.
  • Proptest coverage expanded to all three compression codecs, the
    SparseIndex::lookup invariant, Log append-then-read invariant, and
    Log rotation under random LogConfig.
  • "Performance recipes — pick once, ship it" section in both READMEs:
    decision table + four code recipes + "what NOT to tune" guidance.

Fixed

  • Segment::would_overflow no longer reports overflow on an empty segment.
    Previously the first append (or batch) whose wire size exceeded
    segment_size_threshold against a fresh empty log triggered rotation,
    which then tried to create a new segment at next_offset = 0 and failed
    with ErrorKind::AlreadyExists. The size cap is now correctly a soft
    target on segments with content; empty segments always accept the first
    write. Reachable under default config only at > 1 GiB single records;
    reachable under any custom small-threshold config.

Changed

  • Internal constant ZSTD_DECOMPRESS_MAX_SIZE renamed to DECOMPRESS_MAX_SIZE
    and now applies uniformly to LZ4 and Zstd.
  • README: Producer::send_batch moved from v0.2 roadmap to v0.1 list. The
    kafko-http /produce_batch bullet was dropped from the roadmap — it's
    an improvement for the measurement harness, not the library.