[FIX] In CharBPETokenizer, when Vocab or merges is None, unk_token cannot be used.

[SeongBeomLEE]

Same Error #1120

Thanks.

before

char_level_bep.py

if vocab is not None and merges is not None:
    tokenizer = Tokenizer(
        BPE(
            vocab,
            merges,
            dropout=dropout,
            unk_token=str(unk_token),
            end_of_word_suffix=suffix,
        )
    )
else:
    tokenizer = Tokenizer(BPE())

main.py

from tokenizers import CharBPETokenizer

tokenizer = CharBPETokenizer(
    unk_token = "[UNK]",
    suffix = "</w>",
)

tokenizer.train(
    files = './vocab.txt',
    vocab_size = 1000,
    min_frequency = 1,
    special_tokens = ["[PAD]", "[BOS]", "[EOS]", "[UNK]", "[SEP]", "[CLS]", "[MASK]"],
)

line = "나는 😀 😃 😄 축구를 😀 😃 😄 좋아한다."
pieces = tokenizer.encode(line)
print("CharBPETokenizer", pieces.tokens, tokenizer.decode(pieces.ids))

output:

CharBPETokenizer ['나는</w>', '축구를</w>', '좋아한다</w>', '.</w>'] 나는 축구를 좋아한다 .

after

char_level_bep.py

if vocab is not None and merges is not None:
    tokenizer = Tokenizer(
        BPE(
            vocab,
            merges,
            dropout=dropout,
            unk_token=str(unk_token),
            end_of_word_suffix=suffix,
        )
    )
else:
    tokenizer = Tokenizer(BPE(unk_token=str(unk_token),))

main.py

from tokenizers import CharBPETokenizer

tokenizer = CharBPETokenizer(
    unk_token = "[UNK]",
    suffix = "</w>",
)

tokenizer.train(
    files = './vocab.txt',
    vocab_size = 1000,
    min_frequency = 1,
    special_tokens = ["[PAD]", "[BOS]", "[EOS]", "[UNK]", "[SEP]", "[CLS]", "[MASK]"],
)

line = "나는 😀 😃 😄 축구를 😀 😃 😄 좋아한다."
pieces = tokenizer.encode(line)
print("CharBPETokenizer", pieces.tokens, tokenizer.decode(pieces.ids))

output:

CharBPETokenizer ['나는</w>', '[UNK]', '[UNK]', '[UNK]', '축구를</w>', '[UNK]', '[UNK]', '[UNK]', '좋아한다</w>', '.</w>'] 나는 축구를 좋아한다 .

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[Narsil]

LGTM. Thank you for this