Skip to content

A fast and lightweight implementation of a likelihood-based Byte Pair Encoding (BPE) tokeniser. Unlike traditional BPE, this tokeniser selects merge candidates based on a normalized likelihood score, improving tokenisation quality by prioritising statistically significant merges.

Notifications You must be signed in to change notification settings

WillKirkmanM/wordpiece

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

WordPiece

A fast and lightweight implementation of a likelihood-based Byte Pair Encoding (BPE) tokeni. Unlike traditional BPE, this tokeni selects merge candidates based on a normalised likelihood score, improving tokenisation quality by prioritising statistically significant merges.

About

A fast and lightweight implementation of a likelihood-based Byte Pair Encoding (BPE) tokeniser. Unlike traditional BPE, this tokeniser selects merge candidates based on a normalized likelihood score, improving tokenisation quality by prioritising statistically significant merges.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages