Repository navigation

TokenFlux 0.1.0 Release

TabNahida released this 26 Feb 01:18

· 39 commits to main since this release

2c7ebc4

Support Byte Level BPE Train
Support Pre Tokenize
Support Format: json, jsonl, json.gz, json.xz, parquet

Assets 2