Skip to content

v0.2.0

Choose a tag to compare

@Vadimus1983 Vadimus1983 released this 29 May 14:01
· 6 commits to main since this release

Performance

  • LZ4 hot-path allocation reduced by ~99.997%. Compression::Lz4's encode
    path now amortizes its internal hash table across calls via lz4_flex 0.13's
    new compress_into_with_table API and a per-thread CompressTable. The
    per-record 8 KiB hash-table allocation that dominated heap traffic on
    LZ4-heavy workloads (~1.2 GiB cumulative across 300 K records on lz4_flex
    0.11) is now amortized to a single 8 KiB allocation per encoder thread for
    the process lifetime. Measured: 24.9 KiB total across 100 000 LZ4 sends
    in the kafko-bench lz4_sequential scenario (0.10% of total process
    allocation, down from 93%). LZ4 sequential throughput now tracks no-codec
    sequential throughput within 3% (164 K vs 160 K rec/s on the same path).

Added

  • Cargo features for compression codecs. Two new opt-in features —
    compression-lz4 and compression-zstd, plus the convenience
    compression-all — make lz4_flex and zstd optional dependencies. A
    default cargo add kafko no longer pulls either codec into the dep tree.
    Compression::Lz4 and Compression::Zstd remain visible in the public API
    regardless of feature flags so on-disk records written by another build are
    detectable and produce a friendly KafkoError::CompressionUnavailable
    rather than mis-decoding.
  • KafkoError::CompressionUnavailable(Compression) — error variant returned
    when encoding or decoding a record under a codec whose Cargo feature is not
    enabled. Display message names the missing feature.
  • Compression::is_available() -> bool — runtime feature detection for
    callers that want to fall back gracefully between codecs.
  • kafko-bench lz4_sequential scenario — mirrors the sequential scenario
    under Compression::Lz4 so the hotpath alloc table can attribute
    compression::compress heap traffic to the LZ4 path specifically. Gated
    behind the compression-lz4 feature.
  • [package.metadata.docs.rs] — docs.rs builds with all-features so both
    codec variants are documented in full on the rendered crate page.

Changed (breaking)

  • Record::encode_with signature. Previously -> usize; now
    -> Result<usize>. The error case is KafkoError::CompressionUnavailable,
    returned when encoding under a codec whose feature is not enabled. Callers
    using Producer::send / Producer::send_batch are unaffected — the
    Producer API was already Result-returning and propagates the new error
    naturally. Callers using Record::encode_with directly must add ? or
    .unwrap(). Record::encode() (the Compression::None convenience) is
    unchanged.
  • Default Cargo features. Previously lz4_flex and zstd were
    unconditional dependencies; now both are gated behind opt-in features. A
    cargo update kafko from 0.1.1 to 0.2.0 for a downstream pinned at
    ^0.1 will silently drop LZ4/Zstd support until features are added.
    To preserve v0.1.1 behaviour, change the dependency to:
    kafko = { version = "0.2", features = ["compression-all"] }

Internal

  • lz4_flex bumped from 0.11 to 0.13. The 0.11 line is in maintenance mode;
    0.13 ships the same security fixes as 0.11.6 plus the
    compress_into_with_table API that enables the per-thread hash-table reuse
    win above. Wire format unchanged; existing on-disk segments read back
    identically.
  • New LZ4_TABLE: RefCell<lz4_flex::block::CompressTable> thread-local in
    crates/kafko/src/compression.rs alongside the existing
    ZSTD_COMPRESSOR/ZSTD_DECOMPRESSOR thread-locals.