RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.FloatTensor instead (while checking arguments for embedding)

[GeetDsa]

Environment info

• transformers version: 3.1.0
• Platform: Linux-4.19.0-10-amd64-x86_64-with-debian-10.5
• Python version: 3.6.10
• PyTorch version (GPU?): 1.3.1 (False)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help

Information

Model I am using (GPT2-large) for fine-tuning on custom data:

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

Trace:

File "gpt_language_generation.py", line 209, in
main()
File "gpt_language_generation.py", line 136, in main
trainer.train(model_path=None)
File "<conda_env>/lib/python3.6/site-packages/transformers/trainer.py", line 708, in train
tr_loss += self.training_step(model, inputs)
File "<conda_env>/lib/python3.6/site-packages/transformers/trainer.py", line 995, in training_step
outputs = model(**inputs)
File "<conda_env>/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "<conda_env>/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 731, in forward
return_dict=return_dict,
File "<conda_env>/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "<conda_env>/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 593, in forward
inputs_embeds = self.wte(input_ids)
File "<conda_env>/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "<conda_env>/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "<conda_env>/lib/python3.6/site-packages/torch/nn/functional.py", line 1484, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

model_class, tokenizer_class = GPT2LMHeadModel, GPT2Tokenizer
model = model_class.from_pretrained("gpt2-large")
tokenizer = tokenizer_class.from_pretrained("gpt2-large")
special_tokens_dict = {'bos_token': '<BOS>', 'eos_token': '<EOS>', 'pad_token': '<PAD>'}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))
input_text = ["a cat is sitting on the mat"]*100
train_dataset = tokenizer(input_text,add_special_tokens=True, truncation=True, max_length=64)
train_dataset = train_dataset["input_ids"]
eval_dataset = tokenizer(input_text,add_special_tokens=True, truncation=True, max_length=64)
eval_dataset = eval_dataset["input_ids"]
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer,mlm=False)
training_args = TrainingArguments(output_dir='./gpt_model/')
training_args.do_train = True
training_args.do_eval = True
training_args.per_device_train_batch_size = 32
trainer = Trainer(
           model=model,
            args=training_args,
            data_collator=data_collator,
            train_dataset=train_dataset,
            eval_dataset=eval_dataset,
            prediction_loss_only=True)

trainer.train(model_path=None) // The error occurs here
trainer.save_model()

Expected behavior

Expect the training to continue without an error.

[GeetDsa]

@patrickvonplaten , I did find out that the problem arises from data collator (DataCollatorForLanguageModeling). The returned tensors (tensor of indices to vocab) are not Long, which is creating the problem.