Cannot reproduce results.

[tonysherbondy]

Thank you for the nice work! I'm having problems reproducing the results in your paper. I was hoping you can help.

I have done the following steps.

1. Download ICDAR15 video training and official test video dataset.
2. Prepare training and test dataset folder using: video2frames & convert_ICDAR15video_to_coco.
3. Download pretrain_coco.pth from your Baidu drive.
4. Train on ICDAR15 video using python -m torch.distributed.launch --nproc_per_node=8 --use_env main_track.py --output_dir ./output/icdar_tiv --dataset_file text --coco_path "${MY_DATA_DIR}/icdar_tiv" --batch_size 2 --with_box_refine --num_queries 300 --epochs 80 --lr_drop 40 --resume ./pths/pretrain_coco.pth.
5. Generate inferences using trained model on official test set: python main_track.py --eval --output_dir ./output/icdar_tiv_submit --resume ./output/icdar_tiv/checkpoint0079.pth --dataset_file text --coco_path "${MY_DATA_DIR}/icdar_tiv_test" --batch_size 1 --with_box_refine --num_queries 300
6. Zip up the results in output/icdar_tiv_submit/text/xml_dir.
7. Submit results to official ICDAR2015.

The resulting MOTA is 2.08% and very far from the expected ~45%. Note that the "Mostly Matched" is 842 matching reported results, so it seems that the object detection is working, but tracking is failing. Am I missing something from the code? Thanks for any help.

[tonysherbondy]

Ping.

[weijiawu]

Thank you for your attention to our work.
The following suggestions are for reference.

1. Since our paper is under review, we don't release complete code, e.g., the lacking of matcher.py, save_track.py, deformable_detrtrack_test.py. We don't know that you how to complement these codes. The complete code will be released after the review of the paper.
2. Maybe you can debug by visualization. (python3 track_tools/Evaluation_ICDAR15_video/vis_tracking.py)

[tonysherbondy]

Thank you for your response! I'll look forward to the full publication then.

…

On Mon, Sep 20, 2021, 9:32 AM weijiawu ***@***.***> wrote: Thank you for your attention to our work. The following suggestions are for reference. 1. Since our paper is under review, we don't release complete code, e.g., the lacking of matcher.py, save_track.py, deformable_detrtrack_test.py. We don't know that you how to complement these codes. The complete code will be released after the review of the paper. 2. Maybe you can debug by visualization. (python3 track_tools/Evaluation_ICDAR15_video/vis_tracking.py) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG3R2OQ2WCIRVDYWIJ36ILUC5OY7ANCNFSM5EHR4UAA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[tonysherbondy]

Hello! I see that the complete code has been released, but I still fail to reproduce results on ICDAR TIV challenge of 44% MOTA following steps I listed above. Am I missing something?

[weijiawu]

hello, I want to know the detailed performance for the reproduced results.

[tonysherbondy]

According to the official ICDAR test here. The results are:

MOTA | MOTP | IDF1 | MM | PM | ML
8.25% | 72.43% | 49.54% | 722 | 578 | 616

By setting track_thresh = 0.5 I get:
20.70% | 72.73% | 51.64% | 602 | 613 | 701

[weijiawu]

I have no idea about the performance.

Maybe you can debug by the visualization results? And I would update the weight of 44% MOTA, you can use it for checking.

And I also would check the code recently.

[JasmineRain]

I used the 44% MOTA weight to generate the results for ICDAR15, and the official test results are:
MOTA | MOTP | IDF1 | MM | PM | ML
43.35% | 73.11% | 57.85% | 701 | 533 | 682

These results are much lower than the leaderboard results of your model:
MOTA | MOTP | IDF1 | MM | PM | ML
54.36% | 73.70% | 57.83% | 992 | 495 | 429

Did you use any other tricks in the competition? (Or, are the results on the leaderboard generated by the 44% MOTA weight?)

---

哈喽，我用了你上传到网盘的44% MOTA的模型去生成ICDAR15的测试结果，上传之后分数差距较大，如上所示，请问你还用了其他的trick吗，还是说排行榜上的结果并不是这个44% MOTA的权重生成的？

[weijiawu]

hi, 这两个结果不是同一个算法跑出来的，是两种模型

[weijiawu]

I used the 44% MOTA weight to generate the results for ICDAR15, and the official test results are: MOTA | MOTP | IDF1 | MM | PM | ML 43.35% | 73.11% | 57.85% | 701 | 533 | 682

These results are much lower than the leaderboard results of your model: MOTA | MOTP | IDF1 | MM | PM | ML 54.36% | 73.70% | 57.83% | 992 | 495 | 429

Did you use any other tricks in the competition? (Or, are the results on the leaderboard generated by the 44% MOTA weight?)

哈喽，我用了你上传到网盘的44% MOTA的模型去生成ICDAR15的测试结果，上传之后分数差距较大，如上所示，请问你还用了其他的trick吗，还是说排行榜上的结果并不是这个44% MOTA的权重生成的？

hi, 这两个结果不是同一个算法跑出来的，是两种模型

[JasmineRain]

hi，论文里面report的MOVText数据集上效果的权重能分享一下吗

[weijiawu]

hi，论文里面report的MOVText数据集上效果的权重能分享一下吗

hi, icdar2015上能复现吗，icdar15上能复现的话，在BOVText直接训一下就行，很简单，权重可能没有了

[season-zhou]

Hi. I want to reproduce according to your instructions, and the final mota can only reach 0.32. Are there any tips that can help me improve this result?