pip install -r requirements.txtpython utils/preprocess.py --load_tokenizerUse --load_tokenizer to load the tokenizer from the files if you have already saved one.
-
Notifications
You must be signed in to change notification settings - Fork 1
Scripts to pre train a Multi Tokenizer and also evaluate the performance.
License
aya-multitokenizer/multi-tokernizer-llm
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
Scripts to pre train a Multi Tokenizer and also evaluate the performance.