issues Search Results · repo:github/rust-gems language:Rust
Filter by
3 results
(56 ms)3 results
ingithub/rust-gems (press backspace or delete to remove)Amazing work on bpe!
Wanted to see if there were any known Python bindings a la the snippets from tiktoken below?
import tiktoken
enc = tiktoken.get_encoding( o200k_base )
assert enc.decode(enc.encode( ...
coopslarhette
- 1
- Opened on Apr 28
- #61
Hello,
The BPE algorithm is capable of tokenizing any byte sequence, and LLMs generally accept any sequence of tokens and use
token dictionaries that can successfully represent any byte sequence, but ...
sharpobject
- 1
- Opened on Mar 8
- #51

Learn how you can use GitHub Issues to plan and track your work.
Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub IssuesProTip!
Press the /
key to activate the search input again and adjust your query.
Learn how you can use GitHub Issues to plan and track your work.
Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub IssuesProTip!
Restrict your search to the title by using the in:title qualifier.