Is it possible to mimic trim_batch using new tokenizer strategies?

[sshleifer]

I am trying to replace the old workflow of
calling batch_encode_plus to make tensors of shape
(n_examples, model_max_length) and then calling trim_batch to reduce padding computation, with the new tokenizers kwargs.
Is this possible?
The following code does not seem to truncate inputs longer than 512 (the second assert breaks).

Attempt:

from transformers import BartTokenizer

tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
kw = dict(max_length=512, 
          pad_to_max_length=True, padding=True, return_tensors='pt', truncation='only_first')
batch = tokenizer(['tiny sentence 1', 'tiny_sentence2'],**kw)
assert batch.input_ids.shape[1] == 7, batch.input_ids.shape[1]
input_ids, mask = trim_batch(**batch, pad_token_id=tokenizer.pad_token_id)
assert input_ids.shape[1] == 7, batch.input_ids.shape[1]

batch_overflow = tokenizer(['tiny sentence 1'*1000, 'tiny_sentence2'], **kw)


assert batch_overflow.input_ids.shape[1] == 512, batch_overflow.input_ids.shape[1]

Traceback:

assert batch_overflow.input_ids.shape[1] == 512, batch_overflow.input_ids.shape[1]

AssertionError: 3002

Help much appreciated, @mfuntowicz @thomwolf

[thomwolf]

Hi @sshleifer you should read the detailed description on the tokenizers refactoring PR #4510 (comment)

Until it's added in the doc (will be soon), it's required reading for all core contributors of transformers.

[sshleifer]

Thanks. I read that, and am still somewhat confused about why I pass truncation=True and get entries that are longer than tokenizer.max_model_length. The PR description says:

Here is a simplified example:

from transformers import BartTokenizer
tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
assert tokenizer.model_max_length == 1024

# tokenizer.batch_encode_plus returns ids shaped (2, 1024)
batch_sentences = ['tiny sentence 1'*1000, 'tiny_sentence2']
ids = tokenizer.batch_encode_plus(batch_sentences, pad_to_max_length=True, max_length=tokenizer.model_max_length,
                                  truncation=True, return_tensors='pt').input_ids
assert ids.shape[1] <= tokenizer.model_max_length, ids.shape[1]

# tokenizer.__call__ returns ids shaped (2, 3002)
ids = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors='pt',
                max_length=tokenizer.model_max_length, ).input_ids
assert ids.shape[1] <= tokenizer.model_max_length, ids.shape[1]

[thomwolf]

I'll take a look