Release Notes

0.2.3 (2026-02-26)

Pre-scan mode for TokenFluxTokenize: Added --prescan/--no-prescan option to pre-scan documents for stable ETA calculation and accurate total docs count.
Completion record tracking: Enhanced caching mechanism with CompletionRecord structure to track completed files with source size and mtime for smarter resume behavior.
Run signature validation: Added signature-based validation to detect resume state mismatches and prevent corrupted outputs.

Improved cache semantics: --cache-max-entries now represents max in-memory token-piece cache entries per worker (was: per file).
Enhanced resume behavior: --resume now reuses cache/completed.list for completed-file tracking instead of part files.
Output layout update: Token output written directly to shards/ during tokenization with completion tracking in cache/completed.list.
Better progress reporting: Pre-scan phase shows document count estimation before main tokenization for more reliable progress/ETA.
Usage output improvements: Clarified help text for --encode-batch-size, --cache-max-entries, and --resume options.

Resume state validation: Added checks to detect and warn about resume state mismatches (completed tokens vs. expected tokens).
Path normalization: Added normalized_path field to FileTask for consistent file tracking across platforms.

Internal refactoring: Restructured TokenFluxTokenize.cpp with improved task queue management and completion tracking.
Memory efficiency: Better control over per-worker cache entries and memory usage during tokenization.

Default behavior unchanged: --prescan and --resume are enabled by default.
Output directory layout remains compatible: shards/, cache/completed.list, meta.json.
If upgrading from 0.2.1 or earlier, consider running with --no-resume on first run to regenerate completion records.