Skip to content

TokenFlux++ 0.3.2 Release

Choose a tag to compare

View all tags
@TabNahida TabNahida released this 27 Feb 01:31
· 15 commits to main since this release
v0.3.2
b64739b

Release Notes

0.3.2 (2026-02-27)

Added

  • Python-first binding path: train(...) and tokenize(...) now accept Python-side file/URL lists directly, without requiring a temporary list file.
  • Runtime tokenizer binding: Added Tokenizer to load tokenizer.json, encode strings or batches, and emit Torch tensors directly from Python.
  • Stream-to-torch bridge: Added Tokenizer.tokenize_inputs_to_torch(...) to stream records from input files/URLs and return flat token buffers plus document offsets for Torch-side consumption.

Changed

  • GPT-style training defaults: Default unk_token, special_tokens, and tokenize-time eos_token now use <|endoftext|> by default.

Full Changelog: v0.3.0...v0.3.2

Assets 17
Loading