Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement] Speedup AVATest #784

Merged
merged 40 commits into from
Apr 7, 2021

Conversation

kennymckormick
Copy link
Member

@kennymckormick kennymckormick commented Apr 5, 2021

Speed up reading csv and adding ground-truth info during ava testing.
We also remove the difficult & group flags, which are not used in ava evaluation.
The evaluation results are exactly the same.

Old (7m 52s):
==> 1.55786 seconds to Reading detection results
==> 34.0824 seconds to Reading detection results
==> 52.8981 seconds to Convert groundtruth
==> 366.015 seconds to convert detections
==> 17.5152 seconds to run_evaluator

New (5m 58s):
==> 1.38209 seconds to Reading detection results
==> 28.1567 seconds to Reading detection results
==> 0.860567 seconds to Convert groundtruth
==> 310.64 seconds to convert detections
==> 17.4251 seconds to run_evaluator

Another way to speed up testing is to increase the action_thr in test_cfg, for example, increasing it from 0.00 to 0.002 can shorten the test time by 2mins, with no performance drop (exactly the same performance):
action_thr 0.002 (4m 51s):
==> 1.41658 seconds to Reading detection results
==> 20.2866 seconds to Reading detection results
==> 0.792911 seconds to Convert groundtruth
==> 254.963 seconds to convert detections
==> 13.6164 seconds to run_evaluato

HaodongDuan and others added 30 commits October 16, 2020 17:35
@codecov
Copy link

codecov bot commented Apr 5, 2021

Codecov Report

Merging #784 (d462764) into master (78eb446) will decrease coverage by 0.20%.
The diff coverage is 37.87%.

❗ Current head d462764 differs from pull request most recent head f8d3f07. Consider uploading reports for the commit f8d3f07 to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##           master     #784      +/-   ##
==========================================
- Coverage   85.37%   85.16%   -0.21%     
==========================================
  Files         130      130              
  Lines        9401     9413      +12     
  Branches     1580     1589       +9     
==========================================
- Hits         8026     8017       -9     
- Misses        970      997      +27     
+ Partials      405      399       -6     
Flag Coverage Δ
unittests 85.15% <37.87%> (-0.22%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmaction/apis/train.py 10.18% <4.76%> (-3.46%) ⬇️
.../evaluation/ava_evaluation/per_image_evaluation.py 82.85% <83.33%> (+1.03%) ⬆️
...tion/ava_evaluation/object_detection_evaluation.py 71.34% <100.00%> (+2.72%) ⬆️
mmaction/core/evaluation/ava_utils.py 81.74% <100.00%> (+1.44%) ⬆️
mmaction/version.py 58.33% <100.00%> (ø)
mmaction/datasets/pipelines/augmentations.py 94.85% <0.00%> (+0.50%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78eb446...f8d3f07. Read the comment docs.

Copy link
Contributor

@congee524 congee524 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
And test it on LFB, the computing time reduced to 120.4 seconds

@innerlee innerlee merged commit a2cbd11 into open-mmlab:master Apr 7, 2021
@kennymckormick kennymckormick deleted the speedup_avatest branch April 9, 2021 06:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants