Skip to content

Pull requests: huggingface/tokenizers

New pull request New
11 Open 593 Closed
11 Open 593 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

[Feature] support Assign token to update the content of a token
#1570 opened Jul 12, 2024 by ArthurZucker • Draft
1 task
1
1
[Fix metaspace prepending scheme]
#1568 opened Jul 11, 2024 by ArthurZucker • Draft
1
tokenizers
#1561 opened Jul 1, 2024 by Oleh8978 Loading…
Fast encode
#1560 opened Jun 20, 2024 by ArthurZucker • Draft
1
Bump braces from 3.0.2 to 3.0.3 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#1558 opened Jun 19, 2024 by dependabot bot Loading…
1
Bump ws from 8.8.1 to 8.17.1 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#1557 opened Jun 19, 2024 by dependabot bot Loading…
1
Add bytelevel normalizer to fix decode when adding tokens to BPE
#1555 opened Jun 18, 2024 by ArthurZucker Loading…
1
12
Add display capabilities to tokenizers objects
#1542 opened Jun 3, 2024 by ArthurZucker Loading…
26
feat: add support for pyarrow arrays as input Stale
#1535 opened May 18, 2024 by notjedi Loading…
1
12
feat: support custom regexes for GPT pre-tokenizer Stale
#1462 opened Feb 28, 2024 by gcampax Loading…
5
Merges cannot handle tokens containing spaces.
#909 opened Feb 16, 2022 by Narsil • Draft
15
ProTip! Follow long discussions with comments:>50.