use tensorpack dataflow to improve augmentation

[katierupp]

I have added a different variation of the pose_dataset.py file that uses Tensorpack DataFlow to apply more types of augmentation during training to help the network generalize better. In addition to scaling and cropping, these types of augmentation include rotation, saturation, brightness, contrast, blur, and noise. All of the augmentation parameters are configurable through pose_cfg.yaml, so they can be optimized for any dataset.

We have tested augmentation on our own data, and there is a significant reduction in test error, as well as a noticeable visual improvement in the network's performance. The tests were run on a camera view that the network was not trained on. A figure of a bar chart is attached below to demonstrate how this augmentation compares to the previous implementation. Training time is comparable to the original version of DeepLabCut.

These changes were tested on Ubuntu 16.04. The output from testscript.py with the original pose_dataset.py, as well as the output from testscript.py using the tensorpack dataset can be found at https://gist.github.com/katierupp/893c3d314845c7eedfaad0837bb58f95.

.

[MMathisLab]

@katierupp thanks very much! - Mackenzie

[AlexEMG]

Looks good, thanks for your contribution! Will test this soon.
I suppose: tensorpack should be added to the setup.py.

[katierupp]

@AlexEMG Sounds great, thank you! I just added the compatible version of tensorpack to setup.py.

[AlexEMG]

Thanks for that contribution!

I moved the defaults directly into https://github.com/AlexEMG/DeepLabCut/blob/master/deeplabcut/pose_estimation_tensorflow/dataset/pose_dataset_tensorpack.py
rather than https://github.com/AlexEMG/DeepLabCut/blob/master/deeplabcut/pose_cfg.yaml
as I think that way it will be easier to read. Also we will release another loader soonish to push the envelope on robustness of DLC. We'll have to think about how to make the parameter setting easily accessible though, I suppose. For now the parameters can be set in https://github.com/AlexEMG/DeepLabCut/blob/master/deeplabcut/pose_estimation_tensorflow/dataset/pose_dataset_tensorpack.py and of course specifically in the corresponding pose_config.yaml file before training. In fact,
pose_dataset_tensorpack.py will fall back to default parameters if they are not defined in pose_config.yaml and one is using dataset_type:'tensorpack'

I also added a testscript that https://github.com/AlexEMG/DeepLabCut/blob/master/examples/testscript_openfielddata_augmentationcomparison.py

This is a test script to compare the loaders. tensorpack allows much more choices for augmentation.
This script creates one identical split for the openfield test dataset and trains it with the
standard loader and the tensorpack loader for 50k iterations.

My results were:

With standard loader:

Training iterations:	%Training dataset	Shuffle number	 Train error(px)	 Test error(px)	p-cutoff used	Train error with p-cutoff	Test error with p-cutoff
10000	80	2	2.64	3.11	0.4	2.64	3.11
20000	80	2	2.26	2.72	0.4	2.26	2.72
30000	80	2	1.71	2.28	0.4	1.71	2.28
40000	80	2	1.88	2.61	0.4	1.88	2.61
50000	80	2	1.86	2.32	0.4	1.86	2.32
60000	80	2	1.92	2.42	0.4	1.92	2.42
70000	80	2	2.38	3.04	0.4	2.38	3.04
80000	80	2	1.55	2.34	0.4	1.55	2.34
90000	80	2	1.5	2.27	0.4	1.5	2.27
100000	80	2	1.52	2.34	0.4	1.52	2.34

With tensorpack loader:

Training iterations:	%Training dataset	Shuffle number	 Train error(px)	 Test error(px)	p-cutoff used	Train error with p-cutoff	Test error with p-cutoff
10000	80	3	2.35	2.91	0.4	2.35	2.91
20000	80	3	3.28	3.51	0.4	3.28	3.51
30000	80	3	1.57	2.24	0.4	1.57	2.24
40000	80	3	3.54	4.17	0.4	3.54	4.17
50000	80	3	1.76	2.74	0.4	1.76	2.74
60000	80	3	2.85	3.39	0.4	2.85	3.39
70000	80	3	3.88	4.71	0.4	3.88	4.71
80000	80	3	1.2	2.06	0.4	1.2	2.06
90000	80	3	2.2	3.07	0.4	2.2	3.07
100000	80	3	1.06	1.96	0.4	1.06	1.96

[katierupp]

@AlexEMG Looks good, thank you!

[Luke3D]

Hi everyone, I am using DLC '2.1.6.2' and just started playing with the image augmenters (standard, tensor-pack and imgaug). After training on my dataset I noticed both the train and test error for tensor-pack are really high as compared to the other augmenters.

These are the parameters I have set for imgaug and tensor-pack
if shuffle == 2: #tensorpack
edits = {'rotate_max_deg_abs':30, 'noiseratio':0.05, 'blur_max_window_size':10}
elif shuffle == 1: # imgaug
edits = {"rotation": 30, "motion_blur": True}

This is the loss and learning rate during training

I also tried to train more the Tensorpack augmented net (280k iterations). While the loss decreased to eventually plateau as the other ones, the results got worse.

One basic question is whether the label positions are updated accordingly with the transformed image (i.e. if an image is rotated by 25 degrees, are the labels x-y shifted to the new position)?

Secondly, is there an interface / function in DLC to visualize a batch of transformed images that are applied to the training set?
Thanks!