Treat FFOE Video QA as a classifcation task on answers candidate set

[jiyt17]

Violet uses most commen answers as candidates, but there are other answers in the test set. How do you deal with them? Are they abandoned according to the txt_msvd.json?

[doc-doc]

I have checked the code and data. It seems that the authors just keep the Top-1k answers (MSVD-QA) and remove the unseen-answer samples . It is certainly not a standard and will give rise to unreasonably higher results than others. I thus doubt about the fairness of the comparison in the paper.
Please correct me if I am wrong, or othewise the authors need to revise their evaluation & results..

[tsujuifu]

Super thanks for the clarification!
After checking it, there is exactly a performance error for QAOE in the original VIOLET.
In our new Empirical-MVM, we have fixed it (treat all questions in the testing set), and the accuracy is correct.
We will release the code of Empirical-MVM soon 😊