How to get consistent Roberta Tokenizer behavior between transformers and tokenizers ?

[galtay]

tokenizers.__version__: 0.9.4
transformers.__version__: 4.1.1

I'm trying to get the same tokenization from the tokenizers package and the transformers package and am running into issues.

The issues are around the Roberta post_processing (adding <s> to beginning and </s> to end) as well as the white space before <mask> tokens.

A full minimal example and its output is below,

"""
wget https://huggingface.co/roberta-base/resolve/main/merges.txt
wget https://huggingface.co/roberta-base/resolve/main/vocab.json
"""
import tokenizers
from tokenizers import ByteLevelBPETokenizer
from tokenizers.processors import RobertaProcessing
from tokenizers import AddedToken

import transformers
from transformers import RobertaTokenizer
from transformers import RobertaTokenizerFast


SPECIAL_TOKENS = []
SPECIAL_TOKENS.append(AddedToken("<s>"))
SPECIAL_TOKENS.append(AddedToken("<pad>"))
SPECIAL_TOKENS.append(AddedToken("</s>"))
SPECIAL_TOKENS.append(AddedToken("<unk>"))
SPECIAL_TOKENS.append(AddedToken("<mask>", lstrip=False))

print()
print("tokenizers.__version__: ", tokenizers.__version__)
print("transformers.__version__: ", transformers.__version__)
print()


model = "roberta-base"

trf = RobertaTokenizer.from_pretrained(model)
print("transformers-RobertaTokenizer")
print(trf)
print()


trf_fast = RobertaTokenizerFast.from_pretrained(model)
print("transformers-RobertaTokenizerFast")
print(trf_fast)
print()


tkn = ByteLevelBPETokenizer("vocab.json", "merges.txt")
tkn.add_special_tokens(SPECIAL_TOKENS)
tkn.post_processor = RobertaProcessing(
    sep=("</s>", tkn.token_to_id("</s>")),
    cls=("<s>", tkn.token_to_id("<s>")),
)

print("tokenizers-ByteLevelBPETokenizer")
print(tkn)
print()


text = "The <mask> and <pad> and <unk> tokens."

print("transformers-RobertaTokenizer")
print(trf.convert_ids_to_tokens(trf.encode(text)))
print()

print("transformers-RobertaTokenizerFast")
print(trf_fast.convert_ids_to_tokens(trf_fast.encode(text)))
print()

print("tokenizers-ByteLevelBPETokenizer")
print(tkn.encode(text).tokens)
print()

tokenizers.__version__:  0.9.4
transformers.__version__:  4.1.1

transformers-RobertaTokenizer
PreTrainedTokenizer(name_or_path='roberta-base', vocab_size=50265, model_max_len=512, is_fast=False, padding_side='right', special_tokens={'bos_token': AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'eos_token': AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'unk_token': AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'sep_token': AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'pad_token': AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'cls_token': AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'mask_token': AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=True)})

transformers-RobertaTokenizerFast
PreTrainedTokenizerFast(name_or_path='roberta-base', vocab_size=50265, model_max_len=512, is_fast=True, padding_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'sep_token': '</s>', 'pad_token': '<pad>', 'cls_token': '<s>', 'mask_token': AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False)})

tokenizers-ByteLevelBPETokenizer
Tokenizer(vocabulary_size=50265, model=ByteLevelBPE, add_prefix_space=False, lowercase=False, dropout=None, unicode_normalizer=None, continuing_subword_prefix=None, end_of_word_suffix=None, trim_offsets=False)

transformers-RobertaTokenizer
['<s>', 'The', '<mask>', 'Ġand', 'Ġ', '<pad>', 'Ġand', 'Ġ', '<unk>', 'Ġtokens', '.', '</s>']

transformers-RobertaTokenizerFast
['<s>', 'The', '<mask>', 'Ġand', 'Ġ', '<pad>', 'Ġand', 'Ġ', '<unk>', 'Ġtokens', '.', '</s>']

tokenizers-ByteLevelBPETokenizer
['The', 'Ġ', '<mask>', 'Ġand', 'Ġ', '<pad>', 'Ġand', 'Ġ', '<unk>', 'Ġtokens', '.']

[n1t0]

Hi @galtay

There are actually two things I can see that make the result different here.

1. The ByteLevelBPETokenizer doesn't let you specify the post_processor directly, and you have to do it on the internal Tokenizer:

tkn._tokenizer.post_processor = RobertaProcessing(
    sep=("</s>", tkn.token_to_id("</s>")),
    cls=("<s>", tkn.token_to_id("<s>")),
)

2. The lstrip option on the AddedToken exists specifically for this use case, so it should be True if you want to strip any white space on the left of it:

SPECIAL_TOKENS.append(AddedToken("<mask>", lstrip=True))

If you print the tokens, you might see that <mask> appears as <mask>, but it is actually the same token. Comparing IDs will confirm that.

Feel free to reopen if needed!

[galtay]

Thanks for taking a look!

Setting the postprocessor on the internal Tokenizer works as expected.

I still struggle to get the white space stripping around the <mask> token to be consistent. I've created a colab notebook that demonstrates the results with lstrip=True and lstrip=False for the tokenizer from the tokenizer package,

https://colab.research.google.com/drive/1R67kDyN-HzyIOcOeNVChaECc-O7rFXBm?usp=sharing

The output is the following (but the link should be open to the public)

tokenizers.__version__:  0.9.4
transformers.__version__:  4.1.1

transformers-RobertaTokenizer
==============================
PreTrainedTokenizer(name_or_path='roberta-base', vocab_size=50265, model_max_len=512, is_fast=False, padding_side='right', special_tokens={'bos_token': AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'eos_token': AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'unk_token': AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'sep_token': AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'pad_token': AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'cls_token': AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'mask_token': AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=True)})
bos_token AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True)
eos_token AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True)
unk_token AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True)
sep_token AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True)
pad_token AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=True)
cls_token AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True)
mask_token AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=True)


transformers-RobertaTokenizerFast
==============================
PreTrainedTokenizerFast(name_or_path='roberta-base', vocab_size=50265, model_max_len=512, is_fast=True, padding_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'sep_token': '</s>', 'pad_token': '<pad>', 'cls_token': '<s>', 'mask_token': AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False)})
bos_token '<s>'
eos_token '</s>'
unk_token '<unk>'
sep_token '</s>'
pad_token '<pad>'
cls_token '<s>'
mask_token AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False)


tokenizers-ByteLevelBPETokenizer-lstrip-true
==============================
Tokenizer(vocabulary_size=50265, model=ByteLevelBPE, add_prefix_space=False, lowercase=False, dropout=None, unicode_normalizer=None, continuing_subword_prefix=None, end_of_word_suffix=None, trim_offsets=False)

tokenizers-ByteLevelBPETokenizer-lstrip-false
==============================
Tokenizer(vocabulary_size=50265, model=ByteLevelBPE, add_prefix_space=False, lowercase=False, dropout=None, unicode_normalizer=None, continuing_subword_prefix=None, end_of_word_suffix=None, trim_offsets=False)

Test consistent output
==============================
transformers-RobertaTokenizer
['<s>', 'The', '<mask>', 'Ġand', 'Ġ', '<pad>', 'Ġand', 'Ġ', '<unk>', 'Ġtokens', '.', '</s>']

transformers-RobertaTokenizerFast
['<s>', 'The', '<mask>', 'Ġand', 'Ġ', '<pad>', 'Ġand', 'Ġ', '<unk>', 'Ġtokens', '.', '</s>']

tokenizers-ByteLevelBPETokenizer-lstrip-true
['<s>', 'The', ' <mask>', 'Ġand', 'Ġ', '<pad>', 'Ġand', 'Ġ', '<unk>', 'Ġtokens', '.', '</s>']

tokenizers-ByteLevelBPETokenizer-lstrip-false
['<s>', 'The', 'Ġ', '<mask>', 'Ġand', 'Ġ', '<pad>', 'Ġand', 'Ġ', '<unk>', 'Ġtokens', '.', '</s>']

In the lstrip-true case it nearly matches the transformers output, but there is still a single whitespace character to the left of the <mask> token. In the lstrip-false case, the token is trimmed of whitespace but it leaves in the Roberta token Ġ before the <mask> token (which transformer tokenizers don't do).

[galtay]

AH HA! OK, now I see what your final comment means,

If you print the tokens, you might see that <mask> appears as <mask>, but it is actually the same token. Comparing IDs will confirm that.

Yes, you are correct, the string representation of the token is different but the IDs are identical to the transformers output for lstrip=True.

This is the output with ids shown,

Test consistent output
==============================
transformers-RobertaTokenizer
[0, 133, 50264, 8, 1437, 1, 8, 1437, 3, 22121, 4, 2]
['<s>', 'The', '<mask>', 'Ġand', 'Ġ', '<pad>', 'Ġand', 'Ġ', '<unk>', 'Ġtokens', '.', '</s>']

transformers-RobertaTokenizerFast
[0, 133, 50264, 8, 1437, 1, 8, 1437, 3, 22121, 4, 2]
['<s>', 'The', '<mask>', 'Ġand', 'Ġ', '<pad>', 'Ġand', 'Ġ', '<unk>', 'Ġtokens', '.', '</s>']

tokenizers-ByteLevelBPETokenizer-lstrip-true
[0, 133, 50264, 8, 1437, 1, 8, 1437, 3, 22121, 4, 2]
['<s>', 'The', ' <mask>', 'Ġand', 'Ġ', '<pad>', 'Ġand', 'Ġ', '<unk>', 'Ġtokens', '.', '</s>']

tokenizers-ByteLevelBPETokenizer-lstrip-false
[0, 133, 1437, 50264, 8, 1437, 1, 8, 1437, 3, 22121, 4, 2]
['<s>', 'The', 'Ġ', '<mask>', 'Ġand', 'Ġ', '<pad>', 'Ġand', 'Ġ', '<unk>', 'Ġtokens', '.', '</s>']

Thanks again!