Skip to content

0.1.10: fix(tokenizer): update token count assertion and handle unknown tokens test(tokenizer)

Choose a tag to compare

View all tags
@ransaka ransaka released this 10 Aug 04:54
· 30 commits to main since this release
0.1.10
ed2e851

fix(tokenizer): update token count assertion and handle unknown tokens

  • test(tokenizer): update tests to reflect unknown token handling
  • docs: add note about temporarily unavailable modules
  • chore: update gitignore and disable romanizer tests
Assets 2
Loading