[Improvement] set default batch_size of evaluation and testing to 1 #250

dreamerlin · 2020-10-14T07:02:31Z

This PR sets default batch_size of evaluation and testing as 1 to avoid OOM and manual config changes in most cases, since it used the training batch_size as default when videos_per_gpu=cfg.data.get('videos_per_gpu', 2)

codecov · 2020-10-14T07:13:53Z

Codecov Report

Merging #250 (3a4aecd) into master (17a6f25) will increase coverage by 3.92%.
The diff coverage is 80.84%.

@@            Coverage Diff             @@
##           master     #250      +/-   ##
==========================================
+ Coverage   82.67%   86.59%   +3.92%     
==========================================
  Files          95       98       +3     
  Lines        6867     6901      +34     
  Branches     1126     1113      -13     
==========================================
+ Hits         5677     5976     +299     
+ Misses        980      708     -272     
- Partials      210      217       +7

Flag	Coverage Δ
unittests	`86.58% <80.84%> (+3.92%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmaction/apis/inference.py	`81.81% <ø> (ø)`
mmaction/apis/train.py	`15.00% <0.00%> (ø)`
mmaction/datasets/audio_dataset.py	`73.68% <ø> (-10.10%)`	⬇️
mmaction/datasets/audio_feature_dataset.py	`73.68% <ø> (-10.10%)`	⬇️
mmaction/datasets/image_dataset.py	`83.33% <ø> (ø)`
mmaction/datasets/rawframe_dataset.py	`88.05% <ø> (+9.72%)`	⬆️
mmaction/datasets/rawvideo_dataset.py	`93.75% <ø> (+71.02%)`	⬆️
mmaction/datasets/video_dataset.py	`66.66% <ø> (-15.88%)`	⬇️
mmaction/models/losses/hvu_loss.py	`87.50% <ø> (ø)`
mmaction/datasets/pipelines/loading.py	`89.59% <46.15%> (+3.44%)`	⬆️
... and 31 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17a6f25...68a4168. Read the comment docs.

innerlee · 2020-10-14T08:45:54Z

what does most cases refers to?

dreamerlin · 2020-10-14T08:50:38Z

what does most cases refers to?

It means the cfg.data.videos_per_gpu is set for training but too large for testing or evaluation (since the testing and evaluation may need more crops), which will cause OOM, and users may need to manually change the config setting for cfg.data.videos_per_gpu or cfg.data.val_dataloader. So we set the videos_per_gpu to 1 to avoid this case though it is slow. And for multi-batch testing, users can manually change the cfg.data.val_dataloader and cfg.data.test_dataloader according to their hardware environment.

innerlee · 2020-10-14T08:53:47Z

but videos_per_gpu support val and test

innerlee · 2020-10-14T08:55:37Z

see https://github.com/open-mmlab/mmaction2/blob/master/configs/localization/bmn/bmn_400x100_2x8_9e_activitynet_feature.py#L69

dreamerlin · 2020-10-14T08:57:40Z

see https://github.com/open-mmlab/mmaction2/blob/master/configs/localization/bmn/bmn_400x100_2x8_9e_activitynet_feature.py#L69

U mean it is better to change the config file by setting the val_dataloader or test_dataloader rather than use 1 batch_size as default?

innerlee · 2020-10-14T09:03:20Z

but too large for testing or evaluation

sure, this solves the problem, isn't it?

mmaction/apis/train.py

tools/test.py

dreamerlin · 2020-10-31T02:21:36Z

@kennymckormick @innerlee

innerlee · 2020-10-31T13:30:36Z

tools/test.py

@@ -52,6 +52,10 @@ def parse_args():
        help='override some settings in the used config, the key-value pair '
        'in xxx=yyy format will be merged into config file. For example, '
        "'--cfg-options model.backbone.depth=18 model.backbone.with_cp=True'")
+    parser.add_argument(
+        '--multi-batches',


what does multi-batch mean

It will determine whether to set batch as cfg.data.videos_per_gpu first.

But cfg.data.videos_per_gpu may raise OOM, so this args provide a option for users.

kennymckormick · 2020-11-27T05:35:54Z

tools/test.py

@@ -128,7 +132,8 @@ def main():
    # build the dataloader
    dataset = build_dataset(cfg.data.test, dict(test_mode=True))
    dataloader_setting = dict(
-        videos_per_gpu=cfg.data.get('videos_per_gpu', 2),
+        videos_per_gpu=cfg.data.get('videos_per_gpu', 1)
+        if args.multi_batches else 1,
        workers_per_gpu=cfg.data.get('workers_per_gpu', 0),


Better to set the default value of 'workers_per_gpu' as 1 (for better testing speed).

Would higher value be better?

kennymckormick · 2020-11-27T05:37:56Z

@innerlee This PR is of high priority, old codes have bugs when performing localization testing.

Signed-off-by: lizz <lizz@sensetime.com>

innerlee · 2020-11-27T07:20:37Z

Notes:

I have changed the defaults to small values.
If one want more control, use val_dataloader=dict(videos_per_gpu=1000) or test_dataloader=dict(videos_per_gpu=1000)

dreamerlin requested review from innerlee and kennymckormick October 14, 2020 07:02

kennymckormick reviewed Oct 14, 2020

View reviewed changes

mmaction/apis/train.py Outdated Show resolved Hide resolved

tools/test.py Outdated Show resolved Hide resolved

innerlee mentioned this pull request Oct 16, 2020

Cuda out of Memory when doing Demo #257

Closed

dreamerlin requested a review from kennymckormick October 30, 2020 09:59

dreamerlin added need test and removed need test labels Oct 31, 2020

dreamerlin added 2 commits October 31, 2020 10:19

set default batch_size of evaluation and testing to 1

7a61741

use multi batches

2ebda9b

dreamerlin force-pushed the default branch from 81d3758 to 2ebda9b Compare October 31, 2020 02:20

innerlee reviewed Oct 31, 2020

View reviewed changes

dreamerlin requested a review from innerlee November 19, 2020 09:54

kennymckormick reviewed Nov 27, 2020

View reviewed changes

Simple change

6852b1d

Signed-off-by: lizz <lizz@sensetime.com>

dreamerlin requested a review from kennymckormick November 27, 2020 07:14

hm

68a4168

Signed-off-by: lizz <lizz@sensetime.com>

innerlee approved these changes Nov 27, 2020

View reviewed changes

innerlee merged commit 52cfdee into open-mmlab:master Nov 27, 2020

dreamerlin deleted the default branch December 14, 2020 02:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Improvement] set default batch_size of evaluation and testing to 1 #250

[Improvement] set default batch_size of evaluation and testing to 1 #250

dreamerlin commented Oct 14, 2020

codecov bot commented Oct 14, 2020 •

edited

Loading

innerlee commented Oct 14, 2020

dreamerlin commented Oct 14, 2020 •

edited

Loading

innerlee commented Oct 14, 2020

innerlee commented Oct 14, 2020

dreamerlin commented Oct 14, 2020

innerlee commented Oct 14, 2020

dreamerlin commented Oct 31, 2020

innerlee Oct 31, 2020

dreamerlin Nov 10, 2020

dreamerlin Nov 10, 2020

innerlee Nov 27, 2020

kennymckormick Nov 27, 2020

innerlee Nov 27, 2020

kennymckormick commented Nov 27, 2020

innerlee commented Nov 27, 2020

[Improvement] set default batch_size of evaluation and testing to 1 #250

[Improvement] set default batch_size of evaluation and testing to 1 #250

Conversation

dreamerlin commented Oct 14, 2020

codecov bot commented Oct 14, 2020 • edited Loading

Codecov Report

innerlee commented Oct 14, 2020

dreamerlin commented Oct 14, 2020 • edited Loading

innerlee commented Oct 14, 2020

innerlee commented Oct 14, 2020

dreamerlin commented Oct 14, 2020

innerlee commented Oct 14, 2020

dreamerlin commented Oct 31, 2020

innerlee Oct 31, 2020

Choose a reason for hiding this comment

dreamerlin Nov 10, 2020

Choose a reason for hiding this comment

dreamerlin Nov 10, 2020

Choose a reason for hiding this comment

innerlee Nov 27, 2020

Choose a reason for hiding this comment

kennymckormick Nov 27, 2020

Choose a reason for hiding this comment

innerlee Nov 27, 2020

Choose a reason for hiding this comment

kennymckormick commented Nov 27, 2020

innerlee commented Nov 27, 2020

codecov bot commented Oct 14, 2020 •

edited

Loading

dreamerlin commented Oct 14, 2020 •

edited

Loading