Performance of DPG Model

[thinkrc]

Thanks for your work! I run the dpg_train.py and performed leave-one-person-out evaluation on MPIIGaze. What is the reasonable way to count the result, the mean of 15-fold final test results or the mean of 15-fold best live test results?

[swook]

Hi, I always pick an arbitrary stopping epoch or step that is consistent across all folds. The reason is that in the 15-fold evaluation setting used in prior papers, there was no validation set defined. This could not be defined as the final test performance is on different people and thus a true validation set should be formed of people neither in the training nor test set. Due to the size of MPIIGaze (15 people) this isn't feasible. The configuration in thus repository should include a good stopping step/epoch already. Best, Seonwook

…

On Fri, 12 Apr 2019, 06:45 thinkrc, ***@***.***> wrote: Thanks for your work! I run the dpg_train.py and performed leave-one-person-out evaluation on MPIIGaze. What is the reasonable way to count the result, the mean of 15-fold final test results or the mean of 15-fold best live test results? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#19>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAuI0ruJ8Lzgj33nUxBSi5Pk7dBxDH2kks5vgA9_gaJpZM4crUci> .

[thinkrc]

Thanks for your reply! The messages.log shows that the DPG in this repository runs final test at step 052499 across all folds. I calculated the mean angle error of 15 folds at 052499 step and reaches 4.90 degrees on MPIIGaze (not 4.63 degrees). I repeated the experiment three times and got similar results. Could you give some suggestions?

[swook]

Ach... I'm very sorry about that. I must have made a mistake in the process of making this available.

I can't promise that I can look into this ASAP, but I will try, and keep you updated.

[swook]

I went to check my experiment logs and found a discrepancy between my best configuration there, and the version committed to this repo...

Please find the changes required for a perf. boost at commit 7087b9e