Skip to content

v0.4.0

Latest

Choose a tag to compare

@github-actions github-actions released this 29 Jul 17:13
e56fbd8

0.4.0 (2026-07-29)

⚠ BREAKING CHANGES

  • core: Chunk::text is now the method text(), Chunk::hash is a u64 with hash_hex() for the string form, and Chunker implementors provide chunk_shared(&Arc) instead of chunk(&str), which remains available as a defaulted method. Holding one chunk now keeps its whole source alive; clone text() if you need to drop the rest. The Python API is unchanged: chunk.hash still returns the same 16-digit hex string and JSON serialisation is byte-identical. chunk.hash_int is added for callers who would only parse the hex back into an integer.

Performance Improvements

  • core: SWAR scanning, shared-source chunks, parallel BPE and segmentation (d2bc5de)

Documentation

  • bench: regenerate benchmarks for the SWAR/arena work (667371b)