TokenFlux++ 0.3.3 Release

Release Notes

Release date: 2026-03-02

Highlights

• Split large tokenizer source files for maintainability:

• tokenize_tokenizer.cpp was split into:

  • tokenize_tokenizer_parse.cpp
  • tokenize_tokenizer_load.cpp
  • tokenize_tokenizer_encode.cpp

• tokenize_pipeline.cpp was split into:

  • tokenize_pipeline_state.cpp
  • tokenize_pipeline_process.cpp
  • tokenize_pipeline.cpp (orchestration and run path)

• Added configurable EOS behavior for dataset tokenization:

  • New TokenizeArgs.add_eos (default: true)
  • New CLI flags: --add-eos and --no-eos
  • EOS is appended per document only when add_eos=true

• Clarified document boundary semantics in output metadata:

  • meta.json now includes:
    • add_eos
    • layout.document_stream (concatenated)
    • layout.doc_boundary (eos_token or none)

• Added user-facing API docs:

  • C++ API guide: docs/cpp_api.md
  • Python API guide: docs/python_api.md

• Added a C++ convenience API wrapper:

  • tokenizer/tokenflux_tokenizer_api.hpp

Cleanup

• Removed unused source file:

  • tokenizer/tokenize_tokenizer.cpp

• Checked tracked files for empty files:

  • No empty tracked files were found.

Versioning

• Version updated to 0.3.3 in:
  • pyproject.toml
  • xmake.lua
  • Python binding __version__

Validation

• TokenFluxTokenize, TokenFluxTrain, and tokenflux_cpp build paths compile successfully.
• Python unit tests pass (python -m unittest test_tokenizer.py).

Full Changelog: v0.3.2...v0.3.3