Tokenizer args

[Stasiche]

Hello! I can't understand how to set any args to BertTokenizer. Is it true that there is no way to set truncate, special tokens, etc.?

[cdpierse]

Hello! I can't understand how to set any args to BertTokenizer. Is it true that there is no way to set truncate, special tokens, etc.?

Hi, so right now I purposefully abstract that away as I have to do some case specific args to the tokenizers so allowing custom args could result in explanations not working. Can I ask what you are trying to achieve?

[LPerdu]

Hello Charles,

I use this library to explain some sampled predictions from production environment.

As such being able to specify easily the tokenizer arguments would be very useful to allow apples to apples comparison in my opinion (encodings should match exactly).

Thank you otherwise for this very useful tool.

[premthomas]

Hi Charles.

RuntimeError: The expanded size of the tensor (622) must match the existing size (512) at non-singleton dimension 1. Target sizes: [1, 622]. Tensor sizes: [1, 512]

I tried the following on the Tokenizer

• "model_max_length=512" and "max_len=512"
• "truncation=True"

Using BERT to try a Binary model I have trained. Using Transformer Interpret to check the reason behind the prediction.

Any suggestions?

[aaronrmm]

Here is what we ended up doing when running into the same error:

from functools import partial

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
tokenizer.encode = partial(tokenizer.encode, max_length=512, truncation=True)

There may be a better way, but this worked for now.