Skip to content

v0.0.4

Choose a tag to compare

@chonknick chonknick released this 24 Mar 01:42

What's New

  • Vocab access APIs: id_to_token(), token_to_id(), get_vocab(), decode_batch() — in both Rust and Python
  • Padding & truncation: enable_padding(), enable_truncation(), no_padding(), no_truncation() — HuggingFace-compatible API
  • Encode returns Encoding: encode() and encode_batch() now return Encoding objects with ids, attention_mask, and type_ids
  • Batch pair encoding: encode_pair_batch() for parallel cross-encoder inference
  • 60 pre-built tokenizers: Published to tokiers HuggingFace org
  • Performance: Cached num_cpus(), zero-copy id_to_token() via Cow<str>
  • Cleanup: Removed 27 redundant examples, 10 unused dev-dependencies, -4500 lines net