Fix list index out of range when padding nested empty lists

[qqaatw]

What does this PR do?

This PR fixes list index out of range error when nested empty lists are supplied.

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
padded = tokenizer.pad({"input_ids": [[], [], []] })
print(padded)

Traceback (most recent call last):
  File "test.py", line 6, in <module>
    padded = tokenizer.pad({"input_ids": [[], [], []] })
  File "/src/transformers/tokenization_utils_base.py", line 2730, in pad
    while len(required_input[index]) == 0:
IndexError: list index out of range

@LysandreJik

[LysandreJik]

What is the use-case of passing nested empty lists like this?

[qqaatw]

What is the use-case of passing nested empty lists like this?

I'm using tokenizer.pad to pad question answering labels like the below snippet,

# labels may look like this, where it should be padded to (batch_size, max_answers), 
# and sometimes the entire batch might have no answer being like this: `[[], [], []]`, so I proposed this PR.
start_position = [
    [3, 5, 10], # 3 answers
    [4], # 1 answer
    [], # no answer
]
end_position = [
    [4, 7, 14],
    [8],
    [],
]

Also, could you checkout this issue #13879 and see if that makes sense to you? (similar use-case as this PR.)

Thanks a lot!

[LysandreJik]

I would appreciate it greatly if you could take a look at this when you're back @SaulLu!

[SaulLu]

Thank you very much for your contribution @qqaatw !

What you propose makes sense to me! I just have a small request: could you add a test (probably in test_padding in the test_tokenization_common.py file)

[qqaatw]

I think this is an overlooking when copying test cases.

[SaulLu]

oh! Great catch! Thanks! 👍

[SaulLu]

That is perfect! Thanks for the addition! 🙌

[SaulLu]

oh! Great catch! Thanks! 👍