-
Notifications
You must be signed in to change notification settings - Fork 915
Pull requests: huggingface/tokenizers
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
Bump webpack-dev-server from 4.10.0 to 5.2.1 in /tokenizers/examples/unstable_wasm/www
dependencies
Pull requests that update a dependency file
javascript
Pull requests that update Javascript code
#1792
opened Jun 6, 2025 by
dependabot
bot
Loading…
Expose
Encoding
attributes via the buffer protocol interface
#1789
opened Jun 4, 2025 by
mariosasko
Loading…
Add benchmark for deserializing large added vocab + optimizations
#1782
opened May 27, 2025 by
ArthurZucker
•
Draft
Implement
from_bytes
and read_bytes
Methods in WordPiece Tokenizer for WebAssembly Compatibility
#1758
opened Mar 31, 2025 by
sondalex
Loading…
Pre-tokenizers that support multi-word/non-whitespace BPE in single pass
#1753
opened Mar 22, 2025 by
mjbommar
Loading…
Add FxHash and ShortStringOptimization.
#1733
opened Feb 10, 2025 by
MeetThePatel
Loading…
3 of 4 tasks
[WIP] free speed/mem optimizations with ahash, dary_heap, and compact_str
#1618
opened Aug 21, 2024 by
mjbommar
Loading…
[Feature] support Assign token to update the content of a token
Stale
#1570
opened Jul 12, 2024 by
ArthurZucker
•
Draft
1 task
ProTip!
Add no:assignee to see everything that’s not assigned.