Repository navigation

v0.0.2

github-actions released this 10 Mar 14:06

· 2 commits to master since this release

d931e2d

Added

add tokenizer compare mode
embed latest hf tokenizers
simplify builtin tokenizer selection

Assets 8