Skip to content

Issues: huggingface/tokenizers

ByteLevelBPETokenizer output seems weird
#203 opened Mar 24, 2020 by seyyaw
Open 2
Training a model from in-memory data
#198 by loicbarrault was closed Nov 28, 2020
Closed 1
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

40 Open 913 Closed
40 Open 913 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Tokenizer.from_bytes() not available in python bindings Feature Request
#1567 opened Jul 11, 2024 by RamvigneshPasupathy
2
Custom fast PreTokenizer, ported via PyO3 to Python
#1566 opened Jul 10, 2024 by vandrw
documentation of the pattern parameter in pre_tokenizers.Split is incorrect
#1565 opened Jul 10, 2024 by craigschmidt
1
Decode regression decoding performance
#1564 opened Jul 10, 2024 by daulet
7
Unable to sSet use_regex=False in BPE decoder & post_processor?
#1563 opened Jul 8, 2024 by jchwenger
Unable to Load Custom GPT2 Tokenizer - " data did not match any variant of untagged enum ModelWrapper at line 1 column 3193814" Error
#1562 opened Jul 5, 2024 by maghwa
Progress bar doesn't show in log file.
#1559 opened Jun 20, 2024 by amssljc
2
Encoding object stub doesn't include __len__
#1556 opened Jun 18, 2024 by thearchitector
2
Llama-3 offset-mapping needs fixing
#1553 opened Jun 14, 2024 by davidb-cerebras
5
[Bug?] Modifying normalizer for pretrained tokenizers don't consistently work Stale
#1552 opened Jun 12, 2024 by alvations
2
How to use TokenizerBuilder? Stale
#1549 opened Jun 7, 2024 by polarathene
4
"Solution" to memory hogging in train_new_from_iterator with a hack
#1546 opened Jun 4, 2024 by morphpiece
6
How can I get the mapping relationship between byte values and Unicode characters of the fast tokenizer?
#1545 opened Jun 4, 2024 by LuoKaiGSW
5
[BUG] Fast tokenizer does not deal with AddedTokens properly(no problem in Transformers python tokenizer impl.)
#1544 opened Jun 4, 2024 by MilkClouds
6
Memory leak for large strings
#1539 opened May 23, 2024 by noamgai21
8
How to allow the merging of consecutive newline tokens \n when training a byte-level bpe tokenizer?
#1534 opened May 18, 2024 by liuslnlp
3
Link to download the training text in docs/source/quicktour.rst is broken
#1526 opened May 9, 2024 by 14jdelap
5
How to write custom Wordpiece class? Stale
#1525 opened May 9, 2024 by xinyinan9527
3
Convert huggingface tokenizer into sentencepiece format Stale
#1524 opened May 7, 2024 by RRaphaell
3
Why the tokenizer is slower than tiktoken?
#1519 opened Apr 29, 2024 by BigBinnie
5
Loading tokenizer.model with Rust API Stale
#1518 opened Apr 28, 2024 by EricLBuehler
11
UnigramTrainer: byte_fallback is false. Feature Request training
#1515 opened Apr 25, 2024 by Moddus
3
BPE Trainer doesn't respect the vocab_size parameter when dataset size is increased Stale
#1514 opened Apr 25, 2024 by Abhinay1997
2
Breaking changes in v0.19.1 for tiktoken/llama3
#1512 opened Apr 24, 2024 by sanderland
6
LLamaTokenizer with use_fast=True / and use_fast=False causing memory leak when used with multiprocessing / dataset.map(num_proc)
#1495 opened Apr 15, 2024 by michaelfeil
9
ProTip! Updated in the last three days: updated:>2024-07-12.