Why the RoBERTa's max_position_embeddings size is 512+2=514?

[kugwzk]

❓ Questions & Help

When I see the code of Roberta, I have a question about the padding_idx = 1, I don't know very well. And the comment is still confused for me.

[julien-c]

What's your precise question?

[kugwzk]

What's your precise question?

the self.padding_idx's meaning in modeling_roberta.py

[BramVanroy]

It's the position of the padding vector. It's not unique to RoBERTa but far more general, especially for embeddings. Take a look at the PyTorch documentation.

[kugwzk]

It's the position of the padding vector. It's not unique to RoBERTa but far more general, especially for embeddings. Take a look at the PyTorch documentation.

I know that, but I confuse about why there is 1 and the <s> is 0, is it ignore and why the max_position_embeddings size is 512+2=514?

[BramVanroy]

Because that's their index in the vocab. The max_position_embeddings size is indeed 514, I'm not sure why. The tokenizer seems to handle text correctly with a max of 512. Perhaps someone of the developers can help with that. I would advise you to change the title of your topic.

transformers/transformers/tokenization_roberta.py

Lines 84 to 85 in ae50ad9

	self.max_len_single_sentence = self.max_len - 2 # take into account special tokens
	self.max_len_sentences_pair = self.max_len - 4 # take into account special tokens

[julien-c]

@LysandreJik can chime in if I’m wrong, but afaik max_position_embeddings is just the name of the variable that we use to encode the size of the embedding matrix. Max_len is correctly set to 512.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[morganmcg1]

Answer here in case anyone from the future is curious: facebookresearch/fairseq#1187

[dsantiago]

Answer here in case anyone from the future is curious: pytorch/fairseq#1187

@morganmcg1 Tks for this, was getting all kinds of CUDA errors because i setted max_position_embeddings=512, now that i setted 514 it's running ok...