DAVIS 2016 evaluation error.

[Limingxing00]

Hi,

Thank you for the nice job!

When I load the pre-trained model, I test the performance on the davis-2016-evaluation. It is odd to obtain a bad score on the DAVIS 2016.

I also test the results mapping the output to 255 /1, referring to hkchengrex/Mask-Propagation/issues/33. It is useless.

+-----------+--------+----------+---------+--------+----------+---------+
| Method | J_mean | J_recall | J_decay | F_mean | F_recall | F_decay |
+-----------+--------+----------+---------+--------+----------+---------+
| STCN | 0.856 | 0.931 | 0.040 | 0.880 | 0.921 | 0.043 |
+-----------+--------+----------+---------+--------+----------+---------+

davis-2016-evaluation is poorly maintained, I would like to ask, do you know the cause of the problem?

[hkchengrex]

We already fixed that issue so it should not be a mapping problem.
Can you evaluate on our precomputed results and see if it is correct?

[Limingxing00]

I find the evaluation code is right.
The problem is inference on Davis 2016 (Davis 2017 is normal).

Except for the "Camel" folder, all other output results are normal, which is very strange. I am checking. It has been determined that "val.txt" contains "camel".

And I solve the problem by changing the num_work=1

test_loader = DataLoader(test_dataset, batch_size=1, shuffle=False, num_workers=1, pin_memory=True)

Maybe this is a synchronization problem. Thank you very much for your reply!

[bowen-upenn]

hkchengrex/XMem#41 (comment)