TokenFlux 0.2.0 Release

Release Notes

0.2.0 - 2026-02-26

Highlights

• Unified binary names:
  • TokenFluxTrain for tokenizer training
  • TokenFluxTokenize for corpus tokenization and sharding
• Added multi-model encoding support in TokenFluxTokenize:
  • BPE
  • WordPiece
  • Unigram
• Refactored training architecture into modular backends:
  • byte_bpe
  • bpe
  • wordpiece
  • unigram
• Improved large-file workflow with chunked concurrent processing and faster progress visibility.

Project Structure Changes

• Renamed legacy byte_bpe_* core files to tokenflux_* naming for consistency.
• Introduced per-backend training source files:
  • train_backend_byte_bpe.cpp
  • train_backend_bpe.cpp
  • train_backend_wordpiece.cpp
  • train_backend_unigram.cpp
• Kept orchestration separated from backend implementations.

Breaking Changes

• Old executable names are replaced:
  • byte_bpe_train -> TokenFluxTrain
  • prepare_shards -> TokenFluxTokenize
• Build targets were updated accordingly.

Migration Notes

1. Update scripts/CI to call TokenFluxTrain and TokenFluxTokenize.
2. Rebuild with xmake to pick up renamed sources and new targets.
3. If you parse logs programmatically, account for new model reporting in tokenization output.