How to evaluate and compare with SOTA on test-dev2015 ??

[youngwanLEE]

I saw the description on 2015 Test set split. "It is no longer used and the evaluation servers are closed."

Then, How can we compare with State-of-the-arts which tested on test-dev2015 ?

To compare and research on coco dataset, we need to test-dev2015 annotations Because many papers have written and compared with other detectors on test-dev2015.

What do you think?

Thanks in advance.

[nightrome]

The test-dev and test-challenge sets between 2014/5 and 2017 are identical. The only difference is a different train/val split and that test-reserve and test-standard are now dropped.
So if you want to compare to an older paper, you can use the 2014/5 train/val splits and submit your result to the 2017 test-dev server...

[youngwanLEE]

I really appreciate for your reply.

Thank you so much :)

[youngwanLEE]

@nightrome ,
Whenever I upload my results file( detections_test-dev2017_Alg_results.zip which has the "detections_test-dev2017_Alg_results.json file), the server shows 'failed'.

I think my file name is correct.

Could you help me?

[nightrome]

Unfortunately I am not responsible/knowledgeable for/about the detection challenge. Maybe @tylin can help you. Don't you get a more specific issue?

[youngwanLEE]

@nightrome thanks for reply. I will open the new submission problem.

[shengyuan-tang]

etections_test-dev2017_Alg_results.json文件）时，服务器就会显示“失败”。

Could you share the code of generating .josn on COCO test-dev?Please.