TokenFlux++ 0.3.4 Release

Release Notes

v0.3.4 (2026-03-06)

Highlights

• Added native C++ decode support (no HF dependency required at runtime for decoding).
• Added token analysis toolkit: TokenAnalyzer (tokenflux/analyzer.py).
• Added visualization toolkit: TokenVisualizer (tokenflux/visualizer.py).
• Updated Python bindings and docs for the new decode/analyze/visualize workflow.

Version

• Bumped package version to 0.3.4.

Validation

• Decode roundtrip and batch decode checks passed.
• Analyzer/visualizer basic workflow checks passed.

Full Changelog: v0.3.3...v0.3.4

Full Changelog: v0.3.3...v0.3.4

TokenFlux++ 0.3.3 Release

Release Notes

Release date: 2026-03-02

Highlights

• Split large tokenizer source files for maintainability:

• tokenize_tokenizer.cpp was split into:

  • tokenize_tokenizer_parse.cpp
  • tokenize_tokenizer_load.cpp
  • tokenize_tokenizer_encode.cpp

• tokenize_pipeline.cpp was split into:

  • tokenize_pipeline_state.cpp
  • tokenize_pipeline_process.cpp
  • tokenize_pipeline.cpp (orchestration and run path)

• Added configurable EOS behavior for dataset tokenization:

  • New TokenizeArgs.add_eos (default: true)
  • New CLI flags: --add-eos and --no-eos
  • EOS is appended per document only when add_eos=true

• Clarified document boundary semantics in output metadata:

  • meta.json now includes:
    • add_eos
    • layout.document_stream (concatenated)
    • layout.doc_boundary (eos_token or none)

• Added user-facing API docs:

  • C++ API guide: docs/cpp_api.md
  • Python API guide: docs/python_api.md

• Added a C++ convenience API wrapper:

  • tokenizer/tokenflux_tokenizer_api.hpp

Cleanup

• Removed unused source file:

  • tokenizer/tokenize_tokenizer.cpp

• Checked tracked files for empty files:

  • No empty tracked files were found.

Versioning

• Version updated to 0.3.3 in:
  • pyproject.toml
  • xmake.lua
  • Python binding __version__

Validation

• TokenFluxTokenize, TokenFluxTrain, and tokenflux_cpp build paths compile successfully.
• Python unit tests pass (python -m unittest test_tokenizer.py).

Full Changelog: v0.3.2...v0.3.3

TokenFlux++ 0.3.2 Release

Release Notes

0.3.2 (2026-02-27)

Added

• Python-first binding path: train(...) and tokenize(...) now accept Python-side file/URL lists directly, without requiring a temporary list file.
• Runtime tokenizer binding: Added Tokenizer to load tokenizer.json, encode strings or batches, and emit Torch tensors directly from Python.
• Stream-to-torch bridge: Added Tokenizer.tokenize_inputs_to_torch(...) to stream records from input files/URLs and return flat token buffers plus document offsets for Torch-side consumption.

Changed

• GPT-style training defaults: Default unk_token, special_tokens, and tokenize-time eos_token now use <|endoftext|> by default.

Full Changelog: v0.3.0...v0.3.2

TokenFlux 0.3.1 Release

Release Notes

0.3.1 (2026-02-27)

Changed

• Cross-platform remote fetcher: Replaced the Windows-only WinHTTP downloader with cpp-httplib for remote list and file materialization.
• Build dependency update: Added cpp-httplib as the HTTP client dependency for CLI and Python binding targets.

0.3.0 (2026-02-26)

Added

• Python binding: Added tokenflux_cpp.pyd with TrainConfig, TokenizeArgs, train(...), and tokenize(...) exposed through pybind11.
• Input list support: Added --data-list plus .env support for DATA_LIST / DATA_URL_LIST in both TokenFluxTrain and TokenFluxTokenize.
• Remote input materialization: Added native C++ handling for local paths, file://..., and remote http(s)://... entries from input lists.
• Shared train runtime: Added train_pipeline.* so the train path can be called from both CLI and Python binding.

Changed

• Single-pass streaming default: Train and tokenize now default to reading each file once, with file-count progress as the main progress indicator.
• Size-driven in-file splitting: Large files are now partitioned by derived byte targets during streaming instead of relying only on document-count batching.
• Progress unit clarity: Main processing progress now reports files for train/tokenize while merge progress still reports chunks.
• Input metadata: Tokenization meta.json now records both data_glob and data_list.
• Default prescan behavior: --prescan remains available, but default behavior is now --no-prescan.

Fixed

• Resume key stability: Unified path/URL normalization for local paths and file://... inputs in the completion cache path.
• List-relative resolution: Local list files now resolve relative entries from the list directory instead of the current working directory.

Technical

• Common input source layer: Added input_source.* to centralize list parsing, normalization, remote caching, and stat collection.
• Build updates: xmake.lua now builds the Python module and includes native remote input fetching support.

Migration Notes

• Default behavior changed: --prescan is now off by default for both train and tokenize.
• Existing output layout remains compatible: shards/, cache/completed.list, meta.json.
• Remote http(s) list/input fetching is available through the shared C++ HTTP client path.

TokenFlux 0.2.3 Release

Release Notes

0.2.3 (2026-02-26)

Added

• Pre-scan mode for TokenFluxTokenize: Added --prescan/--no-prescan option to pre-scan documents for stable ETA calculation and accurate total docs count.
• Completion record tracking: Enhanced caching mechanism with CompletionRecord structure to track completed files with source size and mtime for smarter resume behavior.
• Run signature validation: Added signature-based validation to detect resume state mismatches and prevent corrupted outputs.

Changed

• Improved cache semantics: --cache-max-entries now represents max in-memory token-piece cache entries per worker (was: per file).
• Enhanced resume behavior: --resume now reuses cache/completed.list for completed-file tracking instead of part files.
• Output layout update: Token output written directly to shards/ during tokenization with completion tracking in cache/completed.list.
• Better progress reporting: Pre-scan phase shows document count estimation before main tokenization for more reliable progress/ETA.
• Usage output improvements: Clarified help text for --encode-batch-size, --cache-max-entries, and --resume options.

Fixed

• Resume state validation: Added checks to detect and warn about resume state mismatches (completed tokens vs. expected tokens).
• Path normalization: Added normalized_path field to FileTask for consistent file tracking across platforms.

Technical

• Internal refactoring: Restructured TokenFluxTokenize.cpp with improved task queue management and completion tracking.
• Memory efficiency: Better control over per-worker cache entries and memory usage during tokenization.

Migration Notes

• Default behavior unchanged: --prescan and --resume are enabled by default.
• Output directory layout remains compatible: shards/, cache/completed.list, meta.json.
• If upgrading from 0.2.1 or earlier, consider running with --no-resume on first run to regenerate completion records.

TokenFlux 0.2.2 Release

Release Notes

0.2.2 (2026-02-26)

Changed

• Improved training progress reporting to update on processing interval, not only at chunk completion.
• Progress output now refreshes in-place on a single terminal line instead of printing many lines.
• ETA now prioritizes document throughput (docs/s) when total docs are known, with chunk-rate fallback.
• Progress shows fixed docs_done/total_docs when pre-scan is enabled.

Added

• Incremental in-chunk doc progress updates for smoother rate/ETA signals.
• Pre-scan reporting improvements for more stable startup progress behavior.
• Practical performance tuning guidance in README.md (threads, chunk size, queue, memory caps, and backend hints).

Notes

• Project version updated to 0.2.2 in xmake.lua.

TokenFlux 0.2.1 Release

Release Notes

Release Date: February 26, 2026

Changes

Refactoring

• Renamed all header files from .h to .hpp extension for better C++ convention compliance
• Updated all include statements across the codebase to reflect the new .hpp extension

Files Modified

• tokenflux_bpe.h → tokenflux_bpe.hpp
• tokenflux_config.h → tokenflux_config.hpp
• tokenflux_lib.h → tokenflux_lib.hpp
• train_backend.h → train_backend.hpp
• train_backend_common.h → train_backend_common.hpp
• train_frontend.h → train_frontend.hpp
• train_io.h → train_io.hpp
• trainers.h → trainers.hpp

Technical Details

This is a refactoring release with no functional changes. The header file extension change improves code clarity and follows modern C++ project conventions.

Compatibility

• No breaking changes to the API
• No changes to tokenizer behavior or output format
• Existing trained models remain compatible

TokenFlux 0.2.0 Release

Release Notes

0.2.0 - 2026-02-26

Highlights

• Unified binary names:
  • TokenFluxTrain for tokenizer training
  • TokenFluxTokenize for corpus tokenization and sharding
• Added multi-model encoding support in TokenFluxTokenize:
  • BPE
  • WordPiece
  • Unigram
• Refactored training architecture into modular backends:
  • byte_bpe
  • bpe
  • wordpiece
  • unigram
• Improved large-file workflow with chunked concurrent processing and faster progress visibility.

Project Structure Changes

• Renamed legacy byte_bpe_* core files to tokenflux_* naming for consistency.
• Introduced per-backend training source files:
  • train_backend_byte_bpe.cpp
  • train_backend_bpe.cpp
  • train_backend_wordpiece.cpp
  • train_backend_unigram.cpp
• Kept orchestration separated from backend implementations.

Breaking Changes

• Old executable names are replaced:
  • byte_bpe_train -> TokenFluxTrain
  • prepare_shards -> TokenFluxTokenize
• Build targets were updated accordingly.

Migration Notes

1. Update scripts/CI to call TokenFluxTrain and TokenFluxTokenize.
2. Rebuild with xmake to pick up renamed sources and new targets.
3. If you parse logs programmatically, account for new model reporting in tokenization output.

TokenFlux 0.1.0 Release

Support Byte Level BPE Train
Support Pre Tokenize
Support Format: json, jsonl, json.gz, json.xz, parquet