Release Notes

0.3.1 (2026-02-27)

Cross-platform remote fetcher: Replaced the Windows-only WinHTTP downloader with cpp-httplib for remote list and file materialization.
Build dependency update: Added cpp-httplib as the HTTP client dependency for CLI and Python binding targets.

Python binding: Added tokenflux_cpp.pyd with TrainConfig, TokenizeArgs, train(...), and tokenize(...) exposed through pybind11.
Input list support: Added --data-list plus .env support for DATA_LIST / DATA_URL_LIST in both TokenFluxTrain and TokenFluxTokenize.
Remote input materialization: Added native C++ handling for local paths, file://..., and remote http(s)://... entries from input lists.
Shared train runtime: Added train_pipeline.* so the train path can be called from both CLI and Python binding.

Single-pass streaming default: Train and tokenize now default to reading each file once, with file-count progress as the main progress indicator.
Size-driven in-file splitting: Large files are now partitioned by derived byte targets during streaming instead of relying only on document-count batching.
Progress unit clarity: Main processing progress now reports files for train/tokenize while merge progress still reports chunks.
Input metadata: Tokenization meta.json now records both data_glob and data_list.
Default prescan behavior: --prescan remains available, but default behavior is now --no-prescan.

Resume key stability: Unified path/URL normalization for local paths and file://... inputs in the completion cache path.
List-relative resolution: Local list files now resolve relative entries from the list directory instead of the current working directory.

Common input source layer: Added input_source.* to centralize list parsing, normalization, remote caching, and stat collection.
Build updates: xmake.lua now builds the Python module and includes native remote input fetching support.

Default behavior changed: --prescan is now off by default for both train and tokenize.
Existing output layout remains compatible: shards/, cache/completed.list, meta.json.
Remote http(s) list/input fetching is available through the shared C++ HTTP client path.