You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
core: Chunk::text is now the method text(), Chunk::hash is a u64 with hash_hex() for the string form, and Chunker implementors provide chunk_shared(&Arc) instead of chunk(&str), which remains available as a defaulted method. Holding one chunk now keeps its whole source alive; clone text() if you need to drop the rest. The Python API is unchanged: chunk.hash still returns the same 16-digit hex string and JSON serialisation is byte-identical. chunk.hash_int is added for callers who would only parse the hex back into an integer.
Performance Improvements
core: SWAR scanning, shared-source chunks, parallel BPE and segmentation (d2bc5de)
Documentation
bench: regenerate benchmarks for the SWAR/arena work (667371b)