Query regarding input transformation

[Viresh-R]

Hey,
I guess input and ema_input transformed versions of the same images, right ?(

mean-teacher/pytorch/main.py

Line 208 in 618c844

for i, ((input, ema_input), target) in enumerate(train_loader):

)

If so, did you guys experiment with using the same input for both model and ema_model ? Does using the same input lead to drop in performance ?

Thanks !

[tarvaina]

Hi!

At that point in code the noise has been already applied. The input and ema_input incorporate different samples from the same noise distribution.

For example, the CIFAR-10 image translations are drawn from uniform distribution [-4, 4] on both axes. Thus, on line 208, an individual image in input might have been translated two pixels to the left whereas the corresponding image in ema_input might be one pixel up and three pixels left.

We did not experiment using the same noise on both sides. You can explore it yourself if you want to by moving the image augmentation steps before TransformTwice in PyTorch code here:

mean-teacher/pytorch/mean_teacher/datasets.py

Lines 38 to 43 in 618c844

	train_transformation = data.TransformTwice(transforms.Compose([
	data.RandomTranslateWithReflect(4),
	transforms.RandomHorizontalFlip(),
	transforms.ToTensor(),
	transforms.Normalize(**channel_stats)
	]))

Please let us know what happens if you end up running this experiment. I have been wondering about it and meant to do it but never actually did.

[tarvaina]

So something like:

train_transformation = transforms.Compose([
    data.RandomTranslateWithReflect(4), 
    transforms.RandomHorizontalFlip(), 
    data.TransformTwice(transforms.Compose([ 
        transforms.ToTensor(), 
        transforms.Normalize(**channel_stats) 
    ])
]) 

[Viresh-R]

Thanks for your reply ! I will let you know if I end up doing the experiment.

I would think that the performance of Mean Teacher won't drop drastically, since the teacher and student output would still be very different, as the parameters of the two nets are very different. Perhaps approaches like Pi-model(https://arxiv.org/pdf/1610.02242.pdf) would suffer more( where input transformation and dropout are the only sources of variability).