Skip to content

v0.0.7

Choose a tag to compare

View all tags
@chonknick chonknick released this 26 Mar 02:43
v0.0.7
dd0570f

What's new

  • 60/60 tokenizer accuracy — fixed the last 4 failing Unigram models (deepset-mxbai, Jina v3, Cohere multilingual v3/light)
  • Fixed Unigram Viterbi scoring for models without byte fallback tokens

Fix details

Unigram models without <0xXX> byte fallback tokens (like deepset-mxbai, Jina v3, Cohere multilingual) had unk_score=0.0, making <unk> "free" in Viterbi and causing incorrect segmentation. Now uses a heavy penalty (-100.0) for these models while preserving the actual model score for models with byte fallback (T5, XLM-R).

Assets 2
Loading