[Improvement] Set RandAugment as Imgaug default transforms. #585

irvingzhang0512 · 2021-02-01T11:49:58Z

Use imgaug to reimplement RandAugment.

According to VideoMix, RandAugment helps a little.

Results

sthv1 & tsm-r50, 8V100, 50epochs

configs	top1 acc(efficient/accuracy)	top5 acc(efficient/accuracy)
mmaction2 model zoo	45.58 / 47.70	75.02 / 76.12
testing with model zoo ckpt	45.47 / 47.55	74.56 / 75.79
training with default config	45.82 / 47.90	74.38 / 76.02
flip	47.10 / 48.51	75.02 / 76.12
randaugment	47.16 / 48.90	76.07 / 77.92
flip+randaugment	47.85/50.31	76.78/78.18

Kinetics400, 8 V100, test with 256x256 & three crops

Models	top1/5 accuracy	Training lost(epoch 100)	training time
TSN-R50-1x1x8-Vanilla	70.74%/89.37%	0.8	2days 12hours
TSN-R50-1x1x8-RandAugment	71.07%/89.40%	1.3	2days 22hours
I3D-R50-32x2x1-Vanilla	74.48%/91.62%	1.1	3days 10hours
I3D-R50-32x2x1-RandAugment	74.23%/91.45%	1.5	4days 10hours

codecov · 2021-02-01T12:18:56Z

Codecov Report

Merging #585 (2e25cc3) into master (681920e) will increase coverage by 0.04%.
The diff coverage is 100.00%.

❗ Current head 2e25cc3 differs from pull request most recent head 6d90b43. Consider uploading reports for the commit 6d90b43 to get more accurate results

@@            Coverage Diff             @@
##           master     #585      +/-   ##
==========================================
+ Coverage   85.44%   85.49%   +0.04%     
==========================================
  Files         130      130              
  Lines        9336     9339       +3     
  Branches     1564     1564              
==========================================
+ Hits         7977     7984       +7     
+ Misses        962      959       -3     
+ Partials      397      396       -1

Flag	Coverage Δ
unittests	`85.48% <100.00%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmaction/datasets/pipelines/augmentations.py	`94.85% <100.00%> (+0.52%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 681920e...6d90b43. Read the comment docs.

irvingzhang0512 · 2021-02-02T06:19:28Z

@innerlee This pr is ready for review.
It would be great if someone could train kinetics-400 with/without RandAugment.

mmaction/datasets/pipelines/augmentations.py

innerlee · 2021-02-04T10:56:25Z

@congee524 would you like to do a comparison with this aug on/off on tsn&i3d?

congee524 · 2021-02-04T11:08:08Z

@congee524 would you like to do a comparison with this aug on/off on tsn&i3d?

oh, hh，I forgot about it. （ﾉ´д｀）

Because of the shortage of resources, it may start this Saturday.

congee524 · 2021-02-08T03:34:52Z

#492

irvingzhang0512 · 2021-02-08T03:58:54Z

I did some tests myself, with tsn_r50_video_1x1x8_100e_kinetics400_rgb.py(test with 256 ThreeCrop)

Training loss(epoch 100) is quite large. Maybe more epochs and tuning lr could help to get better results.

Models	top1/5 accuracy	Training lost(epoch 100)	training time
TSN-R50-1x1x8-Vanilla	70.74%/89.37%	0.8	2days 12hours
TSN-R50-1x1x8-RandAugment	71.07%/89.40%	1.3	2days 22hours
I3D-R50-32x2x1-Vanilla	74.48%/91.62%	1.1	3days 10hours
I3D-R50-32x2x1-RandAugment	74.23%/91.45%	1.5	4days 10hours

congee524 · 2021-02-08T04:00:44Z

I did some tests myself, with tsn_r50_video_1x1x8_100e_kinetics400_rgb.py

Results(test with 256 ThreeCrop)

Models top1/5 accuracy
Vanilla 70.74%/89.37%
RandAugment 71.07%/89.40%

iφ(*￣0￣)， I‘m working on i3d

irvingzhang0512 · 2021-02-08T04:05:34Z

I did some tests myself, with tsn_r50_video_1x1x8_100e_kinetics400_rgb.py
Results(test with 256 ThreeCrop)
Models top1/5 accuracy
Vanilla 70.74%/89.37%
RandAugment 71.07%/89.40%

iφ(*￣0￣)， I‘m working on i3d

Hope randaugment could help.
Unfortunately, the server must be shut down during the holidays(tears), otherwise I can train more models...

congee524 · 2021-02-09T02:56:52Z

@irvingzhang0512 Hi, when I was training the ckpt of i3d with imgaug, I found that the training speed was very slow because the CPU was running at full load. Is this method CPU intensive?

irvingzhang0512 · 2021-02-09T04:18:47Z

@irvingzhang0512 Hi, when I was training the ckpt of i3d with imgaug, I found that the training speed was very slow because the CPU was running at full load. Is this method CPU intensive?

Yes. It takes me 3 days to train tsn + 1x1x8 + randaugment with 8 V100 gpus.
I do notice that during training data_time increases. But I forget how much time it takes to train tsn without randaugment.

irvingzhang0512 · 2021-02-23T03:14:28Z

Update some training results. RandAugment doesn't improve accuracy for I3D. Training more epochs may help.

Models	top1/5 accuracy	Training lost(epoch 100)	training time
I3D-R50-32x2x1-Vanilla	74.48%/91.62%	1.1	3days 10hours
I3D-R50-32x2x1-RandAugment	74.23%/91.45%	1.5	4days 10hours

innerlee · 2021-02-23T03:19:23Z

My bet is movie transition fx #436 (comment) will be more helpful

congee524 · 2021-02-23T03:30:07Z

Update some training results. RandAugment doesn't improve accuracy for I3D. Training more epochs may help.

Models top1/5 accuracy Training lost(epoch 100) training time
I3D-R50-32x2x1-Vanilla 74.48%/91.62% 1.1 3days 10hours
I3D-R50-32x2x1-RandAugment 74.23%/91.45% 1.5 4days 10hours

Could you provide your training log? I have trained 66 epoch Maybe the performance will be better

irvingzhang0512 · 2021-02-23T03:41:55Z

I3D_Vanilla.log
I3D_RandAugment.log

FYI, there is a huge gap between our kinetics-400...

congee524 · 2021-02-23T03:48:35Z

I3D_Vanilla.log
I3D_RandAugment.log

FYI, there is a huge gap between our kinetics-400...

It looks similar. Maybe I can stop the program :-P

irvingzhang0512 · 2021-02-23T03:51:58Z

It looks similar. Maybe I can stop the program :-P

Since our datasets are quite different, it's better to upload ckpts trained by your dataset.

congee524 · 2021-02-23T04:40:31Z

It looks similar. Maybe I can stop the program :-P

Since our datasets are quite different, it's better to upload ckpts trained by your dataset.

OK, about 4 days left.

congee524 · 2021-03-02T06:47:57Z

@irvingzhang0512 My result is 72.82%/90.64%. Maybe your ckpt should be uploaded.

irvingzhang0512 · 2021-03-02T06:56:37Z

@irvingzhang0512 My result is 72.82%/90.64%. Maybe your ckpt should be uploaded.

Worse than model without randaugment?
Since our datasets are different, my ckpt makes no sense in current mmaction2 model zoo.

congee524 · 2021-03-02T07:06:08Z

@irvingzhang0512 My result is 72.82%/90.64%. Maybe your ckpt should be uploaded.

Worse than model without randaugment?
Since our datasets are different, my ckpt makes no sense in current mmaction2 model zoo.

Yes, from 73.27 to 72.82.
Is the resolution of your dataset short-side 320? I use short-side 256.

irvingzhang0512 · 2021-03-02T07:08:44Z

I use short-side 256. But our training&val samples(videos) are quite different.

#415 (comment)

irvingzhang0512 · 2021-03-02T07:10:20Z

Yes, from 73.27 to 72.82.
Is the resolution of your dataset short-side 320? I use short-side 256.

I'll try randaugment in other datasets(sth) this month

irvingzhang0512 · 2021-03-17T01:22:44Z

kindly ping @congee524

training tsm-r50 on sthv1 with randaugment, get very good results. ckpt/json/log is here

Maybe this ckpt is enough for this pr?

congee524 · 2021-03-23T05:12:23Z

@irvingzhang0512 Sorry for my late reply...

When I use Imgaug with default transforms, The training time has increased a lot (about 18 days on 8 V100), which may be due to weak CPU performance. But the performance didn't improve (72.49/90.65).

congee524 · 2021-03-23T05:29:24Z

Maybe Kinetics is large enough, so the image enhancement effect is very small, and simple randmultiscale, flip is enough.

For sthv1 and shtv2, since we didn't apply flip on them, Imgaug performs well.

I have found that the results of imgaug and flip with label transform on sthv1 are similar.

irvingzhang0512 · 2021-03-24T01:01:45Z

Maybe Kinetics is large enough, so the image enhancement effect is very small, and simple randmultiscale, flip is enough.

For sthv1 and shtv2, since we didn't apply flip on them, Imgaug performs well.

I have found that the results of imgaug and flip with label transform on sthv1 are similar.

Maybe we can add sthv1/tsm-r50/randaugment ckpt/log/json to model zoo instead of i3d-r50/kinetics/randaugment.

BTW, train sthv1 & tsm-r50 with flip & randaugment, get 47.85/50.31 top-1 accu1 and 76.78/78.18

congee524 · 2021-03-24T01:20:09Z

I'll test your ckpt on our sthv1 now. (ง •_•)ง

congee524 · 2021-03-24T02:06:44Z

Maybe Kinetics is large enough, so the image enhancement effect is very small, and simple randmultiscale, flip is enough.
For sthv1 and shtv2, since we didn't apply flip on them, Imgaug performs well.
I have found that the results of imgaug and flip with label transform on sthv1 are similar.

Maybe we can add sthv1/tsm-r50/randaugment ckpt/log/json to model zoo instead of i3d-r50/kinetics/randaugment.

BTW, train sthv1 & tsm-r50 with flip & randaugment, get 47.85/50.31 top-1 accu1 and 76.78/78.18

BTW, could you provide the ckpt of tsm_50_flip_randaugment ？

irvingzhang0512 · 2021-03-24T02:10:18Z

~~https://wp.cethik.com:443/link/204D82F1146AF450E22C4E2BCEF67774~~
https://wp.cethik.com:443/link/AB7BD549AF8760E17B9576A9C97C2C08

congee524 · 2021-03-24T02:13:23Z

tsm_r50_randaugment on our dataset: 47.07/48.90 75.81/77.64

irvingzhang0512 · 2021-03-24T02:21:43Z

I do know there is a gap between our sthv1 datasets, which is odd. We could directly download sthv1 rawframes from official website. So we are supposed to have exact the same dataset...

configs	top1 acc(efficient/accuracy)	top5 acc(efficient/accuracy)
mmaction2 model zoo	45.58 / 47.70	75.02 / 76.12
testing with model zoo ckpt	45.47 / 47.55	74.56 / 75.79
training with default config	45.82 / 47.90	74.38 / 76.02

congee524 · 2021-03-24T02:47:29Z

@dreamerlin may help to upload the ckpt/log/json :p

congee524 · 2021-03-24T02:48:27Z

https://wp.cethik.com:443/link/204D82F1146AF450E22C4E2BCEF67774

It seems not the tsm_50_flip_randaugment.

irvingzhang0512 · 2021-03-24T02:50:03Z

https://wp.cethik.com:443/link/204D82F1146AF450E22C4E2BCEF67774

It seems not the tsm_50_flip_randaugment.

My mistake,
randaugment: https://wp.cethik.com/#/link/858EA8B5B02FAB1D8231201F559A1BBF
randaugment + flip: https://wp.cethik.com:443/link/AB7BD549AF8760E17B9576A9C97C2C08

congee524 · 2021-03-24T03:37:05Z

tsm_r50_flip_randaugment on our dataset:
47.75/50.16 76.52/77.89

configs/recognition/tsm/tsm_r50_flip_randaugment_1x1x8_50e_sthv1_rgb.py

mmaction/datasets/pipelines/augmentations.py

dreamerlin · 2021-03-24T05:33:24Z

https://wp.cethik.com:443/link/204D82F1146AF450E22C4E2BCEF67774

It seems not the tsm_50_flip_randaugment.

My mistake,
randaugment: https://wp.cethik.com/#/link/858EA8B5B02FAB1D8231201F559A1BBF
randaugment + flip: https://wp.cethik.com:443/link/AB7BD549AF8760E17B9576A9C97C2C08

uploaded

congee524 · 2021-03-24T06:18:02Z

@innerlee This PR is ready.

innerlee · 2021-03-25T04:32:14Z

Thanks!

irvingzhang0512 added 3 commits February 1, 2021 19:46

first commit for image randaugment

bff8a8e

fix test

a0cc324

update changelog

f292395

irvingzhang0512 mentioned this pull request Feb 1, 2021

[PR Welcome] More data augmentation methods #436

Open

6 tasks

innerlee reviewed Feb 2, 2021

View reviewed changes

mmaction/datasets/pipelines/augmentations.py Outdated Show resolved Hide resolved

use percentage params for TranslateX &S TranslateY

99399c0

innerlee approved these changes Feb 4, 2021

View reviewed changes

innerlee added the waiting for ckpts label Feb 4, 2021

delete blank line

550e09c

fix config

80aa4c8

irvingzhang0512 added 3 commits March 24, 2021 11:10

Merge branch 'master' into imgaug-randaugment

724d998

add tsm-r50 flip + randaugment sthv1

75d0a7b

remove useless annotations

77e8565

congee524 reviewed Mar 24, 2021

View reviewed changes

configs/recognition/tsm/tsm_r50_flip_randaugment_1x1x8_50e_sthv1_rgb.py Outdated Show resolved Hide resolved

mmaction/datasets/pipelines/augmentations.py Outdated Show resolved Hide resolved

congee524 reviewed Mar 24, 2021

View reviewed changes

mmaction/datasets/pipelines/augmentations.py Outdated Show resolved Hide resolved

Update README.md

1aa8754

fix

6d90b43

congee524 approved these changes Mar 24, 2021

View reviewed changes

congee524 removed the waiting for ckpts label Mar 24, 2021

innerlee merged commit 8a79e52 into open-mmlab:master Mar 25, 2021

irvingzhang0512 deleted the imgaug-randaugment branch March 25, 2021 06:25

irvingzhang0512 mentioned this pull request Aug 3, 2021

Data Augmentation for Video #1064

Closed

[Improvement] Set RandAugment as Imgaug default transforms. #585

[Improvement] Set RandAugment as Imgaug default transforms. #585

Conversation

irvingzhang0512 commented Feb 1, 2021 • edited

Results

codecov bot commented Feb 1, 2021 • edited

Codecov Report

irvingzhang0512 commented Feb 2, 2021

innerlee commented Feb 4, 2021

congee524 commented Feb 4, 2021

congee524 commented Feb 8, 2021

irvingzhang0512 commented Feb 8, 2021 • edited

congee524 commented Feb 8, 2021

irvingzhang0512 commented Feb 8, 2021

congee524 commented Feb 9, 2021

irvingzhang0512 commented Feb 9, 2021 • edited

irvingzhang0512 commented Feb 23, 2021

innerlee commented Feb 23, 2021

congee524 commented Feb 23, 2021

irvingzhang0512 commented Feb 23, 2021

congee524 commented Feb 23, 2021

irvingzhang0512 commented Feb 23, 2021

congee524 commented Feb 23, 2021

congee524 commented Mar 2, 2021

irvingzhang0512 commented Mar 2, 2021

congee524 commented Mar 2, 2021

irvingzhang0512 commented Mar 2, 2021

irvingzhang0512 commented Mar 2, 2021

irvingzhang0512 commented Mar 17, 2021

congee524 commented Mar 23, 2021

congee524 commented Mar 23, 2021

irvingzhang0512 commented Mar 24, 2021

congee524 commented Mar 24, 2021

congee524 commented Mar 24, 2021

irvingzhang0512 commented Mar 24, 2021 • edited

congee524 commented Mar 24, 2021

irvingzhang0512 commented Mar 24, 2021

congee524 commented Mar 24, 2021

congee524 commented Mar 24, 2021

irvingzhang0512 commented Mar 24, 2021 • edited

congee524 commented Mar 24, 2021

dreamerlin commented Mar 24, 2021

congee524 commented Mar 24, 2021

innerlee commented Mar 25, 2021

irvingzhang0512 commented Feb 1, 2021 •

edited

codecov bot commented Feb 1, 2021 •

edited

irvingzhang0512 commented Feb 8, 2021 •

edited

irvingzhang0512 commented Feb 9, 2021 •

edited

irvingzhang0512 commented Mar 24, 2021 •

edited

irvingzhang0512 commented Mar 24, 2021 •

edited