What is response in model_predictions.jsonlines?

[pk1130]

Hey @EntilZha @jplalor! I was looking at model_predictions.jsonlines to interpret the evaluation after training and evaluating a 4PL IRT model on squad-pyirt.jsonlines. I understand that the prediction field in model_predictions.jsonlines is the probability that the model with id subject_id got the question with id example_id correct, calculated using the 4PL mathematical formula. What does the response field in the same file refer to? Does it refer to whether the model actually got the answer to the question correct? I am attaching a screenshot of the table generated below for better understanding. Please respond at your earliest convenience! Thanks!

[EntilZha]

Yup, that’s exactly it, it is whether the subject (squad model) got the question correct. In this case, that means if the EM score is 1 (either because the answer is correct or the model correctly abstained)

[pk1130]

Got it, thanks!