v0.9.0

walterbm-cohere released this 15 Aug 19:45

f5ef49c

Following: https://github.com/daulet/tokenizers/releases/tag/v0.9.0

What's Changed

feat: add option to retrieve offsets from tokenizer

Update to huggingface/tokenizers v0.20.0

Assets 9