Skip to content

microsoft/Tokenizer

Error
Looks like something went wrong!

About

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

github.com/microsoft/Tokenizer/

Topics

ai tokenizer openai gpt llm

Resources

Readme

License

MIT license

Code of conduct

Code of conduct

Security policy

Security policy
Activity
Custom properties

Stars

196 stars

Watchers

8 watching

Forks

35 forks
Report repository

Releases 4

Microsoft.DeepDev.TokenizerLib 1.3.3 Release Latest
Jan 12, 2024
+ 3 releases

Packages

 
 
 

Contributors 15

Languages