Add mAP Metrics and validation

[zhiqwang]

Fixing #59

• Add mAP metric computation for COCO datasets

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@2125d06). Click here to learn what that means.
The diff coverage is 70.73%.

@@            Coverage Diff            @@
##             master      #89   +/-   ##
=========================================
  Coverage          ?   81.80%           
=========================================
  Files             ?        8           
  Lines             ?      775           
  Branches          ?        0           
=========================================
  Hits              ?      634           
  Misses            ?      141           
  Partials          ?        0           

Flag	Coverage Δ
unittests	81.80% <70.73%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
test/test_engine.py	91.60% <63.63%> (ø)
test/test_data_pipeline.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2125d06...70febca. Read the comment docs.

[masahi]

Hi @zhiqwang, do you happen to have a script to evaluate yolo5 on coco 2017 dataset? I also want to evaluate torchvision faster rcnn on the same dataset. I'm working on accuracy validation of TVM fp16 quantization.

[zhiqwang]

Hi @masahi , I have the evaluate script on COCO2017, I need to clean it and expect to upload it tomorrow.

[masahi]

@zhiqwang I was able to get reasonably looking numbers by modifying the input path to the data loader in test_vanilla_coco_evaluator at https://github.com/zhiqwang/yolov5-rt-stack/blob/947956f3c472bd4395272e261fe6f9d46bfd8b61/test/test_engine.py#L95 to point to my downloaded coco val 2017. Was that how you did it?

Another question: I tried to use the same eval script to test torchvision fasterrcnn. But I hit an error at this line https://github.com/zhiqwang/yolov5-rt-stack/blob/f4f2e6aebec97d72ec40051aac31f217fcf01ea0/yolort/data/coco_eval.py#L201 because the label there is 86 and contiguous_to_json_category has only 80 elements. Why is contiguous_to_json_category required? If I remove self.contiguous_to_json_category and directly use label[k], I got an accuracy number for faster rcnn, but it looks too good.

[zhiqwang]

Hi @masahi ,

Check #148 for the coco metrics evaluation CLI. And I add a data preprare function in there. Let's move the disscussion there.

https://github.com/zhiqwang/yolov5-rt-stack/blob/e1efb7c7fb88713c1d8db21584b32f09152f6283/tools/eval_metric.py#L45-L64

Why is contiguous_to_json_category required?

This is because the original categories number of COCO is 90, check the id in https://github.com/zhiqwang/yolov5-rt-stack/blob/f34194c/yolort/utils/builtin_meta.py for more details. On the other hand, ultralytics use a continuous map to [1-80] to train their model.

I changed the contiguous_to_json_category to category_id_maps in #148

https://github.com/zhiqwang/yolov5-rt-stack/blob/e1efb7c7fb88713c1d8db21584b32f09152f6283/yolort/data/coco_eval.py#L72-L77

and use --eval_type to control whether to switch this mapping.

https://github.com/zhiqwang/yolov5-rt-stack/blob/e1efb7c7fb88713c1d8db21584b32f09152f6283/tools/eval_metric.py#L93-L105