Trainer evaluation doesn't return eval loss for question-answering.

[avacaondata]

As mentioned in this issue:
"""
Just a note that I tried python run_squad_trainer.py --model_name_or_path bert-base-uncased --model_type bert --data_dir squad --output_dir /tmp/debug_squad/ --overwrite_output_dir --do_train --do_eval --evaluate_during_training --logging_steps 100.

For some reason I don't get any evaluation metric during training (I was expecting loss or eval_loss).

Originally posted by @borisdayma in #4829 (comment)
"""
I'm facing the same problem. I'm trying to train with Trainer class over a QA dataset different from SQUAD. Everything works fine, the model learns based on train loss. However, I haven't been able to get the eval loss. I hope these pieces of code show how I'm configuring Trainer. Can somebody tell me if I'm doing something wrong?

training_args = TrainingArguments(
    output_dir="./models/prueba_2",  
    per_gpu_train_batch_size=16,
    per_gpu_eval_batch_size=32,  
    num_train_epochs=10, 
    logging_steps=10,  
    save_steps=25, 
    do_eval=True,
    evaluate_during_training=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    data_collator=collator,
    prediction_loss_only=True,
    compute_metrics = EvalPrediction
)

I also tried without EvalPrediction in compute_metrics.
Thanks in advance !

[avacaondata]

I solved it editing trainer.py, in this line https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L765 . I added start_positions and end_positions to that "possible labels names" list, and it worked. Review the bug, please.

[patil-suraj]

Thanks @alexvaca0 !

cc @julien-c , @sgugger

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gungor2]

I solved it editing trainer.py, in this line https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L765 . I added start_positions and end_positions to that "possible labels names" list, and it worked. Review the bug, please.

I am having the same issue. Can you elaborate on how you solved this problem? I could not locate a possible label list and start/end positions?

[sgugger]

#7191 should solve the problem described here.

[mamoon115]

@sgugger How can we calculate validation and evaluation loss for question answering finetuning pipeline (run_qa.py)