[Feature] Support Imgaug for augmentations in the data pipeline.

[irvingzhang0512]

Support imgaug in dataset pipeline.

• TODO
  • changelog & docs.
  • tests.

Some of these codes are modified from mmdet.

Any suggestions about this PR?

[codecov]

Codecov Report

Merging #492 (283a0e1) into master (910d2fb) will increase coverage by 0.09%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #492      +/-   ##
==========================================
+ Coverage   84.67%   84.76%   +0.09%     
==========================================
  Files         121      121              
  Lines        8526     8579      +53     
  Branches     1398     1416      +18     
==========================================
+ Hits         7219     7272      +53     
  Misses        953      953              
  Partials      354      354              

Flag	Coverage Δ
unittests	84.75% <100.00%> (+0.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmaction/datasets/pipelines/__init__.py	100.00% <ø> (ø)
mmaction/datasets/pipelines/augmentations.py	94.81% <100.00%> (+0.37%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 910d2fb...283a0e1. Read the comment docs.

[innerlee]

This library looks interesting.

Do you know the typical use cases? Any mixed usage of imgaug and normal augmentation pipelines?

[irvingzhang0512]

Imgaug (and most of the other data augmentation libs) is desined for single image augmentation(eg. blur/rotate/crop/pad/resize...), along with all kinds of labels(detection, segmentation, keypoints). So when dealing ava dataset, no need to take care of bbox separately

• The usage is quite simple:
  • step1: find our favourite augmenters in imgaug docs.
  • step2: create transforms.
  • step3: add imgaug pipeline before FormatShape.

pipeline = [
    dict(
        type='SampleFrames',
        clip_len=1,
        frame_interval=1,
        num_clips=16,
    ),
    dict(type='RawFrameDecode'),
    dict(type='Resize', scale=(-1, 256)),
    dict(
        type='MultiScaleCrop',
        input_size=224,
        scales=(1, 0.875, 0.75, 0.66),
        random_crop=False,
        max_wh_scale_gap=1,
        num_fixed_crops=13),
    dict(type='Resize', scale=(224, 224), keep_ratio=False),
    dict(type='Flip', flip_ratio=0.5),
    dict(type='Imgaug', transforms='default'),
    # dict(type='Imgaug', transforms=[dict(type='Rotate', rotate=(-30, 30))]),
    dict(type='FormatShape', input_format='NCHW'),
    dict(type='Collect', keys=['imgs', 'label'], meta_keys=[]),
    dict(type='ToTensor', keys=['imgs', 'label'])
]

However, imgaug pipeline only generates/modifies 4 keys imgs/img_shape/gt_bboxes/proposals. Normal pipelines will also generate keys like crop_bbox, flip_direction, etc...

I was planing to create a config file, using imgaug to replace all normal augmentation pipelines. But imgaug doesn't support all mmaction2 augmentation pipelines, especially the crop/resize.

After this pr is merged, I'll also create a pr to support albumentations, which is also a third party data augmentation library and more popular nowadays.

[irvingzhang0512]

FYI, mmdet pr about albumentations

[SuX97]

Very interesting. If you have trained some models that benefit from this augmentation, could you please share some with us?

BTW, there are many other augmentation packages such as https://github.com/hassony2/torch_videovision, https://github.com/torchvideo/torchvideo, https://github.com/DemisEom/SpecAugment that might be helpful.

Especially this one https://github.com/piergiaj/pytorch-i3d/blob/master/videotransforms.py, which applies augmentation by volume slicing rather than for looping. @innerlee Will it be faster?

[innerlee]

no

[SuX97]

no

@innerlee , I am not sure about that. I have done a toy test:

import numpy as np

def tiktok(func):
	import time
	def wrapper(*args, **kwargs):
		tik = time.time()
		ret = func(*args, **kwargs)
		tok = time.time()
		print(f'{func.__name__} takes {tok - tik}s.')
		return ret
	return wrapper

@tiktok
def slice(imgs):
	for i in range(1000):
		_ = imgs[:, 5:219, 5:219, :]

@tiktok
def for_loop(imgs):
	for i in range(1000):
		out = [img[5:219, 5:219, :] for img in imgs]
		#out = np.array(out)

imgs = np.random.randn(32, 256, 256, 3)
slice(imgs)
for_loop(imgs)

This gives a result of:
slice takes 0.0004169940948486328s.
for_loop takes 0.018939733505249023s.

Since we do the for-loop slicing and convert it into numpy array afterward, when taking out = np.array(out) into account, it yields:
slice takes 0.0003859996795654297s.
for_loop takes 8.275573968887329s.

[irvingzhang0512]

I haven't tried these augmenters to train kinetics-400. When training our own datasets, blur/rotate helps.

My original intention is to try everything. For now, speed is not my priority. So I prefer 3rd party libraries with more augmenters(imgaug and albumentations). If we can all benefit from certain augmenters, we can implements those in the future.

BTW, albumentations doesn't use loop. I'll take a close look at related source codes later.

[SuX97]

you may need to pull the new master and rebase to it.

[irvingzhang0512]

kindly ping @dreamerlin @SuX97

[irvingzhang0512]

@innerlee any suggestions for this pr?

[innerlee]

Quite busy this week, will take a look next Wed. Thanks for the patience

[irvingzhang0512]

kindly ping @innerlee

[innerlee]

It will be great to add an example (runnable) config in tsn/ or i3d/ folder

[irvingzhang0512]

Found a serious bug: training failed when using imgaug and multi-gpu. Works fine when training with single-gpu(i used to do). Will try to fix this in two weeks.

Traceback (most recent call last):
  File "./tools/train.py", line 178, in <module>
    main()
  File "./tools/train.py", line 174, in main
    meta=meta)
  File "/ssd/zhangyiyang/mmaction2/mmaction/apis/train.py", line 156, in train_model
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs, **runner_kwargs)
  File "/home/user/.conda/envs/zyy_pytorch1.6/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 125, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/user/.conda/envs/zyy_pytorch1.6/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 47, in train
    for i, data_batch in enumerate(self.data_loader):
  File "/home/user/.conda/envs/zyy_pytorch1.6/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 291, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "/home/user/.conda/envs/zyy_pytorch1.6/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 737, in __init__
    w.start()
  File "/home/user/.conda/envs/zyy_pytorch1.6/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/user/.conda/envs/zyy_pytorch1.6/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/user/.conda/envs/zyy_pytorch1.6/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/user/.conda/envs/zyy_pytorch1.6/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/user/.conda/envs/zyy_pytorch1.6/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/user/.conda/envs/zyy_pytorch1.6/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/user/.conda/envs/zyy_pytorch1.6/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle module objects

[innerlee]

try putting import in call function instead of init

[innerlee]

@congee524 Please train a checkpoint for these two configs, and upload them in a separate pr, thx!