[Feature Request] Extract Predictions from Trainer

[geblanco]

🚀 Feature request

Add the possibility to return predictions along with its example id in the new Trainer class.

Motivation

When working with extractive QA (i.e.: SQuAD), you get back the best predictions, but, the current example for running squad uses the old, plain training/eval script, without the new Trainer class.

Additionally, there are other tasks where predictions can be extremely useful (i.e.: Multiple Choice).

Adding such functionality in the Trainer class could solve this and unify both question answering and multiple choice examples.

Your contribution

I am familiarized with the code (both the Trainer class and the old train/eval script), so I could submit a PR with the new functionality.

[julien-c]

Unless I'm mistaken, Question-answering does support Trainer. Is a specific feature missing for your use case?

[geblanco]

Hi @julien-c!

At the time of writing the PR was not merged yet (and I was not aware of it, my bad). It is great!

In any case, the previous example (the one without trainer) outputted both predictions and nbest_predictions, which are crucial for error analysis.

The new example with trainer looses this ability (because the new dataset/trainer API does not support it). Adding this feature could benefit not only SQuAD, but other datasets too (i.e.: RACE).

What do you think on this regard?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dianags]

Any updates on this matter? I recently started using transformers and running the multiple-choice example I realized there's an output of the overall accuracy but I can't find the predictions. Currently going through the code to obtain them...

[geblanco]

Hi @dianags,

Might be a little late, but tired of waiting, I programmed my own solution that output predictions and some more data, the code can be found here

Best,

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.