Repository navigation
Grace submodule (parity with HF tokenizers) #62
Closed
edwardcapriolo
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
One of the subtle issues I have run into is that models like Qwen tokenize Strings in a way that is not compatible with the way deliverance does. It becomes hard to verify 1-1 and features like strip=left strip=right arent there.
For grace I am doing my best to port tokenizers 1-1.It is a big undertaking but I think we can get the majority of the way there without going to crazy
It isnt too easy to make python and java APIs look the same but Im off to an ok start.
All reactions