Skip to content

Releases: Yasuo414/Tokenizer

Release list

This is the first official version of Tokenizer as a Python Wheel with a pre-trained model.

This is the first official version of Tokenizer as a Python Wheel with a pre-trained model. Latest
Latest

Choose a tag to compare

View all tags
@Yasuo414 Yasuo414 released this 06 Apr 17:48
v1
e070be7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

!!!Warning!!! The pre-trained tokenizer is only trained on sample text. Please create a text corpus for use.

Assets 4
Loading