GPTNeo: importing model with padded vocab size should truncate wte

[leogao2]

Environment info

• transformers version: latest from master

Who can help

@LysandreJik

Information

Model I am using (Bert, XLNet ...): GPTNeo

Script: https://github.com/huggingface/transformers/blob/master/src/transformers/models/gpt_neo/convert_gpt_neo_mesh_tf_to_pytorch.py

Some GPTNeo models are trained with a vocab size greater than the actual used vocab size (i.e 50304 in config when the actual vocab size is 50257) where all tokens after the first i.e 50257 are unused. These models cannot currently be converted using the script because there is no way to cut the extra embeddings out of wte.