[Feature] Support Mixup and Cutmix for Recognizers.

[irvingzhang0512]

Support Mixup and Cutmix for Recognizers.

TODO

• codes of mixup/cutmix/register/recognizers
• unittest
• demo config for tsn
• changelog
• visualize input results(gif)
• ckpt

Discussions

• Since we are using soft-label, do we still need to calculate top1/top5 accuracy during training(not val)?
• For now, we CANNOT inherit model settings and set train_cfg. A sample config is listed as follows. TypeError will be raised.

_base_ = [
    '../../_base_/models/tsm_r50.py'
]

model = dict(
    train_cfg=dict(
        blending=dict(type="CutmixBlending", num_classes=400, alpha=.2)))

And error msg is TypeError: train_cfg={'blending': {'type': 'CutmixBlending', 'num_classes': 400, 'alpha': 0.2}} in child config cannot inherit from base because train_cfg is a dict in the child config but is of type <class 'NoneType'> in base config. You may set delete=True to ignore the base config

visualize examples

mixup

cutmix

Results

• model: tsm-r50 1x1x8 50e
• dataset: sthv1

configs	top1 acc(efficient/accuracy)	top5 acc(efficient/accuracy)
mmaction2 model zoo	45.58 / 47.70	75.02 / 76.12
testing with model zoo ckpt	45.47 / 47.55	74.56 / 75.79
training with default config	45.82 / 47.90	74.38 / 76.02
mixup alpha=1.0	45.68 / 47.73	74.26 / 76.82
mixup alpha=0.2	46.35 / 48.49	75.07 / 76.88
cutmix alpha=0.2	45.92 / 47.46	75.23 / 76.71

[codecov]

Codecov Report

Merging #681 (8ada1c6) into master (ced53ac) will increase coverage by 0.11%.
The diff coverage is 98.59%.

❗ Current head 8ada1c6 differs from pull request most recent head 749de3e. Consider uploading reports for the commit 749de3e to get more accurate results

@@            Coverage Diff             @@
##           master     #681      +/-   ##
==========================================
+ Coverage   84.59%   84.71%   +0.11%     
==========================================
  Files         130      131       +1     
  Lines        9357     9426      +69     
  Branches     1578     1583       +5     
==========================================
+ Hits         7916     7985      +69     
  Misses       1042     1042              
  Partials      399      399              

Flag	Coverage Δ
unittests	84.70% <98.59%> (+0.11%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmaction/datasets/blending_utils.py	98.27% <98.27%> (ø)
mmaction/datasets/__init__.py	100.00% <100.00%> (ø)
mmaction/datasets/registry.py	100.00% <100.00%> (ø)
mmaction/models/heads/base.py	95.00% <100.00%> (+8.15%)	⬆️
mmaction/models/recognizers/base.py	66.66% <100.00%> (+2.12%)	⬆️
mmaction/core/evaluation/accuracy.py	92.27% <0.00%> (-0.91%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ced53ac...749de3e. Read the comment docs.

[irvingzhang0512]

@innerlee This pr is ready for review. Training models now.

[dreamerlin]

@congee524 Please help to check the config error

[irvingzhang0512]

Since our kinetics400/sthv2 are somehow different, I train cutmix/mixup on sthv1.

sthv1 tsm-r50-1x1x8 results:

configs	top1 acc(efficient/accuracy)	top5 acc(efficient/accuracy)
mmaction2 model zoo	45.58 / 47.70	75.02 / 76.12
testing with model zoo ckpt	45.47 / 47.55	74.56 / 75.79
training with default config	45.82 / 47.90	74.38 / 76.02
mixup alpha=1.0	45.68 / 47.73	74.26 / 76.82
mixup alpha=0.2	46.35 / 48.49	75.07 / 76.88
cutmix alpha=0.2	45.92 / 47.46	75.23 / 76.71

ckpt/json/log is here: https://wp.cethik.com:443/link/65C665F9E03EE42D9D479A70DACA53FA

Could anyone help to upload these?

[kennymckormick]

Since our kinetics400/sthv2 are somehow different, I train cutmix/mixup on sthv1.

sthv1 tsm-r50-1x1x8 results:

configs top1 acc(efficient/accuracy) top5 acc(efficient/accuracy)
mmaction2 model zoo 45.58 / 47.70 75.02 / 76.12
testing with model zoo ckpt 45.47 / 47.55 74.56 / 75.79
training with default config 45.82 / 47.90 74.38 / 76.02
mixup alpha=1.0 45.68 / 47.73 74.26 / 76.82
mixup alpha=0.2 46.35 / 48.49 75.07 / 76.88
cutmix alpha=0.2 45.92 / 47.46 75.23 / 76.71
ckpt/json/log is here: https://wp.cethik.com:443/link/65C665F9E03EE42D9D479A70DACA53FA

Could anyone help to upload these?

Sure

[kennymckormick]

@innerlee Please check this PR.

[innerlee]

duplicate

[innerlee]

remove

[innerlee]

Thanks!