Sharing raw subjective ratings (or MOS + variance per image)

[subpic]

Hello,

I was wondering if you provided the raw ratings obtained in the experiment (rating corresponding to each user and image)? If these are not available, would it be possible to have access to the variance of the ratings for each image? It would help compare the reliability of the subjective experiment to that of other studies.

Thanks,
Vlad

[h4nwei]

Dear Vlad,

We have made the standard deviation available on the MEGA. I hope it is useful to you.

Best,
Hanwei

[subpic]

Dear Hanwei,

Thanks a lot for sharing the additional metadata! :)

I understand the SPAQ study was done in the lab, so the standard deviations should be low, but I found that 5 images have a standard deviation of 0 (06797.jpg, 06841.jpg, 06858.jpg, 07234.jpg, 07264.jpg). It means that at least 15 participants provided the exact same rating for each of those images. Given the 100 point scale used, this is very unlikely. Could you please help me check if that is really the case, or maybe there is something I missed or interpreted wrong?

Thanks,
Vlad

[h4nwei]

Dear Vlad,

Thank you for bringing it to our attention. I believe it contains several errors. I'll double-check everything and get back to you as soon as possible.

Best,
Hanwei

[h4nwei]

Dear Hanwei,

Thanks a lot for sharing the additional metadata! :)

I understand the SPAQ study was done in the lab, so the standard deviations should be low, but I found that 5 images have a standard deviation of 0 (06797.jpg, 06841.jpg, 06858.jpg, 07234.jpg, 07264.jpg). It means that at least 15 participants provided the exact same rating for each of those images. Given the 100 point scale used, this is very unlikely. Could you please help me check if that is really the case, or maybe there is something I missed or interpreted wrong?

Thanks, Vlad

Dear Vlad,

The revised standard deviation of each image is uploaded to https://docs.google.com/spreadsheets/d/1RxLsGq_3SPTkL3XVz6FlQJpPdjdavcwB/edit?usp=sharing&ouid=116447340964909555823&rtpof=true&sd=true. Thanks for pointing out it again.

Best,
Hanwei

[subpic]

Thanks, Hanwei! :)