You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
LZ4 hot-path allocation reduced by ~99.997%.Compression::Lz4's encode
path now amortizes its internal hash table across calls via lz4_flex 0.13's
new compress_into_with_table API and a per-thread CompressTable. The
per-record 8 KiB hash-table allocation that dominated heap traffic on
LZ4-heavy workloads (~1.2 GiB cumulative across 300 K records on lz4_flex
0.11) is now amortized to a single 8 KiB allocation per encoder thread for
the process lifetime. Measured: 24.9 KiB total across 100 000 LZ4 sends
in the kafko-bench lz4_sequential scenario (0.10% of total process
allocation, down from 93%). LZ4 sequential throughput now tracks no-codec
sequential throughput within 3% (164 K vs 160 K rec/s on the same path).
Added
Cargo features for compression codecs. Two new opt-in features — compression-lz4 and compression-zstd, plus the convenience compression-all — make lz4_flex and zstd optional dependencies. A
default cargo add kafko no longer pulls either codec into the dep tree. Compression::Lz4 and Compression::Zstd remain visible in the public API
regardless of feature flags so on-disk records written by another build are
detectable and produce a friendly KafkoError::CompressionUnavailable
rather than mis-decoding.
KafkoError::CompressionUnavailable(Compression) — error variant returned
when encoding or decoding a record under a codec whose Cargo feature is not
enabled. Display message names the missing feature.
Compression::is_available() -> bool — runtime feature detection for
callers that want to fall back gracefully between codecs.
kafko-benchlz4_sequential scenario — mirrors the sequential scenario
under Compression::Lz4 so the hotpath alloc table can attribute compression::compress heap traffic to the LZ4 path specifically. Gated
behind the compression-lz4 feature.
[package.metadata.docs.rs] — docs.rs builds with all-features so both
codec variants are documented in full on the rendered crate page.
Changed (breaking)
Record::encode_with signature. Previously -> usize; now -> Result<usize>. The error case is KafkoError::CompressionUnavailable,
returned when encoding under a codec whose feature is not enabled. Callers
using Producer::send / Producer::send_batch are unaffected — the
Producer API was already Result-returning and propagates the new error
naturally. Callers using Record::encode_with directly must add ? or .unwrap(). Record::encode() (the Compression::None convenience) is
unchanged.
Default Cargo features. Previously lz4_flex and zstd were
unconditional dependencies; now both are gated behind opt-in features. A cargo update kafko from 0.1.1 to 0.2.0 for a downstream pinned at ^0.1 will silently drop LZ4/Zstd support until features are added.
To preserve v0.1.1 behaviour, change the dependency to:
kafko = { version = "0.2", features = ["compression-all"] }
Internal
lz4_flex bumped from 0.11 to 0.13. The 0.11 line is in maintenance mode;
0.13 ships the same security fixes as 0.11.6 plus the compress_into_with_table API that enables the per-thread hash-table reuse
win above. Wire format unchanged; existing on-disk segments read back
identically.
New LZ4_TABLE: RefCell<lz4_flex::block::CompressTable> thread-local in crates/kafko/src/compression.rs alongside the existing ZSTD_COMPRESSOR/ZSTD_DECOMPRESSOR thread-locals.