Improve mAP performance

[twsl]

What does this PR do?

Fixes #677

Work of @tkupek @OlofHarrysson @twsl
First steps to get performance on par with pycocotools/numpy

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

Merging #742 (a04dbab) into master (9c9c430) will decrease coverage by 8%.
The diff coverage is 97%.

@@          Coverage Diff           @@
##           master   #742    +/-   ##
======================================
- Coverage      91%    83%    -8%     
======================================
  Files         166    166            
  Lines        6817   6832    +15     
======================================
- Hits         6181   5651   -530     
- Misses        636   1181   +545     

[tkupek]

The performance improvements are great and really necessary.

However, CUDA calculations are still by a factor 9-10 slower than CPU. Computations are not CUDA optimized and I don't know if it's possible (or how much effort it is).

Running metric on 10 samples on cuda:0
Total time: 8.060346603393555
Time per sample 0.8060346603393554

Running metric on 10 samples on cpu
Total time: 0.98065185546875
Time per sample 0.098065185546875

Until then, I suggest to make the mAP metric CPU only, to avoid that somebody uses the slow CUDA version.
I would move everything to CPU, as we did it with the pycocotools implementation.

@twsl @Borda @OlofHarrysson @SkafteNicki

[Borda]

Until then, I suggest to make the mAP metric CPU only, to avoid that somebody uses the slow CUDA version.
I would move everything to CPU, as we did it with the pycocotools implementation.

sounds reasonable to me 🐰

[tkupek]

I ran a benchmark on a real-works use-case with 1088 samples and ~10 bounding boxes per sample.

Pycocotools (previous implementation): 74.91s
Current implementation: 1742.07s
This branch: 140.79s

IMHO this should be merged and released ASAP to make the metric usable again on GPUs.
Further improvements should be done to get closer to the pycocotools benchmark.

@twsl @OlofHarrysson @Borda

[SkafteNicki]

@tkupek @OlofHarrysson @twsl thanks for really trying to improve performance for this metric.
I agree that we should merge this and do a small release.
However, out hole GPU testing pipeline is down at the moment so no PRs are currently getting merged :(

[SkafteNicki]

@Borda please approve and merge.
Passes locally for multi-gpu support:

[Borda]

Pycocotools (previous implementation): 74.91s
Current implementation: 1742.07s
This branch: 140.79s

Further improvements should be done to get closer to the pycocotools benchmark.

@twsl @tkupek do you think that you can make further improvements to get speed parity with cocotooks?

[tkupek]

Unfortunately, I don't have any concrete ideas in mind, nor the time to look at it right now.

[twsl]

I might give it another shot after i handed in my thesis next week. But cant promise any results or timeline