Skip to content

issues Search Results · repo:github/rust-gems language:Rust

Filter by

3 results
 (50 ms)

3 results

ingithub/rust-gems (press backspace or delete to remove)

Amazing work on bpe! Wanted to see if there were any known Python bindings a la the snippets from tiktoken below? import tiktoken enc = tiktoken.get_encoding( o200k_base ) assert enc.decode(enc.encode( ...
  • coopslarhette
  • 1
  • Opened 
    on Apr 28
  • #61

Hello, The BPE algorithm is capable of tokenizing any byte sequence, and LLMs generally accept any sequence of tokens and use token dictionaries that can successfully represent any byte sequence, but ...
  • sharpobject
  • 1
  • Opened 
    on Mar 8
  • #51

what exactly do we do in this repo?
  • keys-i
  • Opened 
    on Jul 30, 2024
  • #11
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Press the
/
key to activate the search input again and adjust your query.
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Restrict your search to the title by using the in:title qualifier.
Issue search results · GitHub