A fast and lightweight implementation of a likelihood-based Byte Pair Encoding (BPE) tokeni. Unlike traditional BPE, this tokeni selects merge candidates based on a normalised likelihood score, improving tokenisation quality by prioritising statistically significant merges.
-
Notifications
You must be signed in to change notification settings - Fork 0
A fast and lightweight implementation of a likelihood-based Byte Pair Encoding (BPE) tokeniser. Unlike traditional BPE, this tokeniser selects merge candidates based on a normalized likelihood score, improving tokenisation quality by prioritising statistically significant merges.
WillKirkmanM/wordpiece
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
About
A fast and lightweight implementation of a likelihood-based Byte Pair Encoding (BPE) tokeniser. Unlike traditional BPE, this tokeniser selects merge candidates based on a normalized likelihood score, improving tokenisation quality by prioritising statistically significant merges.
Topics
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published