Error in evaluation code

[ZijiaLewisLu]

Hi,

Thanks for sharing the code. I noticed the bg_class in the evaluation code is not properly set.

The default name of background class is set to background, which is true in GTEA yet need to be changed to SIL for breakfast and action_start and action_end for 50salads. It seems they are not changed for the results in the paper.

With the correct class name and the released model, I obtained a lower result

	F1@0.10	F1@0.25	F1@0.50
Breakfast	70.9	67.5	56.7
50salads	83.7	81.8	73.7

[ChinaYi]

Hi, there are some differences in the evaluation between (50salads, breakfast) and GTEA. For 50salads and breakfast, the action_start, action end and the SIL are evaluated as normal actions, while the background in GTEA is not. If you take a close look at the annotation file, you can find that action_start, action end, SIL only happens at the beginning and end of the video, while background happens in every two continuous normal actions in GTEA.

Actually, the eval code is adapted from MS-TCN, which is further adapted from ED-TCN(2017). The same evaluation manner has been used to ensure the comparability of results.

[ZijiaLewisLu]

Hi,

Thanks for clarification. I found the evaluation protocol of MSTCN is a bit confusing. I read its code before and at that time I assumed they changed the class name according to dataset and just did not take the trouble to auto-change it in the released code. Have you checked with MSTCN's author on that?