[Improve] Change default value of average-clips

[kennymckormick]

Set 'average_clips' in the test_cfg of each config file.
You can still set this parameter during testing and it will override the default value in the config file.

[codecov]

Codecov Report

Merging #232 into master will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #232   +/-   ##
=======================================
  Coverage   85.15%   85.15%           
=======================================
  Files          81       81           
  Lines        5267     5267           
  Branches      846      846           
=======================================
  Hits         4485     4485           
  Misses        644      644           
  Partials      138      138           

Flag	Coverage Δ
#unittests	85.15% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmaction/datasets/base.py	68.33% <ø> (ø)
mmaction/datasets/pipelines/formating.py	93.85% <ø> (ø)
mmaction/version.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fafca4e...cdcfb9b. Read the comment docs.

[dreamerlin]

The default value of average_clips should be "prob" rather than "score" for 3D model. "prob" will get higher performance for 3D. And for 2D recognizers, just set None for average_clips is OK.

[innerlee]

Do u need to change model zoo performance when changing configs from None to 'something'?

[kennymckormick]

The default value of average_clips should be "prob" rather than "score" for 3D model. "prob" will get higher performance for 3D. And for 2D recognizers, just set None for average_clips is OK.

Sorry that I made the mistake, the default value should be 'prob'.

[kennymckormick]

Do u need to change model zoo performance when changing configs from None to 'something'?

Emm, Yes. Since the original default value of 'average_clips' is 'score', so some ckpts trained by me should be re-tested. @SuX97, did you explicitly set 'average_clips' to 'prob' during testing previously?

[dreamerlin]

Do u need to change model zoo performance when changing configs from None to 'something'?

Emm, Yes. Since the original default value of 'average_clips' is 'score', so some ckpts trained by me should be re-tested. @SuX97, did you explicitly set 'average_clips' to 'prob' during testing previously?

The performance of the ckpt is test by setting 'average_clips' to 'prob', so there is no need to re-test.

[kennymckormick]

Do u need to change model zoo performance when changing configs from None to 'something'?

Emm, Yes. Since the original default value of 'average_clips' is 'score', so some ckpts trained by me should be re-tested. @SuX97, did you explicitly set 'average_clips' to 'prob' during testing previously?

The performance of the ckpt is test by setting 'average_clips' to 'prob', so there is no need to re-test.

My bad, 3D checkpoints trained by me is tested with default setting (average_clips = 'score'). So the reported performance might be a little bit lower. I will create another PR to fix them.

[innerlee]

the perf table can be updated here too, so that configs and reported accuracies are always matched.

[innerlee]

As long as the settings are consistent with the reported accuracies in modelzoo, i'm ok.

I'm a little confused by the use of None. Here are two possible meanings

• invalid. this argument does not make sense for this method
• don't care and use whatever the default value.

If it belongs to the second case, then the direction of this pr is correct: configs should be clear and concrete.

[dreamerlin]

As long as the settings are consistent with the reported accuracies in modelzoo, i'm ok.

I'm a little confused by the use of None. Here are two possible meanings

• invalid. this argument does not make sense for this method
• don't care and use whatever the default value.

If it belongs to the second case, then the direction of this pr is correct: configs should be clear and concrete.

It belongs to the second case. For 3D recognizers, the average_clips matters since there is no consensus op to average the segments and average_clip does this thing. For some 2D recognizers, if it uses consensus op and doesn't reshape for temporal dimension (like TSM in 3crop + twice sample setting, which will be supported soon), the average_clips doesn't matter.

[kennymckormick]

As long as the settings are consistent with the reported accuracies in modelzoo, i'm ok.
I'm a little confused by the use of None. Here are two possible meanings

• invalid. this argument does not make sense for this method
• don't care and use whatever the default value.

If it belongs to the second case, then the direction of this pr is correct: configs should be clear and concrete.

It belongs to the second case. For 3D recognizers, the average_clips matters since there is no consensus op to average the segments and average_clip does this thing. For some 2D recognizers, if it uses consensus op and doesn't reshape for temporal dimension (like TSM in 3crop + twice sample setting, which will be supported soon), the average_clips doesn't matter.

I agree. @innerlee