Actually the extra_id are from 0-99 and not from 1-100

[orena1]

a = tokenizer.encode("we got a <extra_id_99>", return_tensors='pt',add_special_tokens=True)
print(a)
>tensor([[   62,   530,     3,     9, 32000]])
a = tokenizer.encode("we got a <extra_id_100>", return_tensors='pt',add_special_tokens=True)
print(a)
>tensor([[   62,   530,     3,     9,     3,     2, 25666,   834,    23,    26,
           834,  2915,  3155]])

[codecov]

Codecov Report

Merging #5967 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #5967   +/-   ##
=======================================
  Coverage   78.51%   78.51%           
=======================================
  Files         146      146           
  Lines       26214    26214           
=======================================
  Hits        20581    20581           
  Misses       5633     5633           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ae67b24...e8e003f. Read the comment docs.

[orena1]

Hi @patrickvonplaten, can you have a look?

[LysandreJik]

Indeed!

Verified with

from transformers import T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained("t5-base")

for i in range(101):
    print(i, f"<extra_id_{i}>" in tokenizer.get_vocab())

which outputs True for [0, 99] and False for 100.

[LysandreJik]

Thansk @orena1 !

[LysandreJik]

pinging @patrickvonplaten for notification