normalization

[twmht]

the normalization step might be wrong.

https://github.com/hysts/pytorch_image_classification/blob/master/dataloader.py#L134

the correct order is
https://github.com/kuangliu/pytorch-cifar/blob/master/main.py#L35

Any idea?

[hysts]

Hi, thank you for pointing it out. But, I'm not sure if one of them is "correct" and the other is "wrong." I guess the effect of the normalization order is negligible and it's just a matter of taste.

That being said, since you brought it up, I checked some of the official implementations of papers, and found that both are used.

Implementations of ResNet, DenseNet, and ResNeXt apply normalization first and then apply zero padding, horizontal flip and random crop.
https://github.com/facebook/fb.resnet.torch/blob/master/datasets/cifar10.lua#L45-L49
https://github.com/liuzhuang13/DenseNet/blob/master/datasets/cifar10.lua#L45-L49
https://github.com/facebookresearch/ResNeXt/blob/master/datasets/cifar10.lua#L45-L49

Implementations of PyramidNet, Cutout and RandomErasing apply zero padding, random crop and horizontal flip and then apply normalization.
https://github.com/dyhan0920/PyramidNet-PyTorch/blob/master/train.py#L99-L104
https://github.com/uoguelph-mlrg/Cutout/blob/master/train.py#L74-L77
https://github.com/zhunzhong07/Random-Erasing/blob/master/cifar.py#L106-L112

Implementation of Wide ResNet is a bit different. It applies reflection padding, random crop and horizontal flip and then apply normalization.
https://github.com/szagoruyko/wide-residual-networks/blob/master/pytorch/main.py#L66-L77

So, it seems there's no "correct" order of normalization after all.

[twmht]

@hysts

You normalize the tensor with mean and std first, for example, in mnist, mean is 0.1307 and std is 0.3081, mean and std are already normalized to [0,1].

suppose a pixel is 240, in pytorch, it produces (240-0.1307)/0.3081. doesn't it look weird?

it seems that the order should be normalize the tensor to [0,1] first, and then normalize by mean and std.

for example,

it should change

transform = torchvision.transforms.Compose([
            transforms.normalize(self.mean, self.std),
            transforms.to_tensor(),
        ]) 

to

transform = torchvision.transforms.Compose([
            transforms.to_tensor(),
            transforms.normalize(self.mean, self.std),
        ]) 

normalize the tensor with mean and std after normalize to [0,1]

[hysts]

Oh, I completely misunderstood your point. Thanks for the clarification.

Sorry, I think my implementation was a bit confusing, and that leads to your concern.

I implemented transforms.py myself, and its transforms.normalize and transforms.to_tensor are different from torchvision.transforms.Normalize and torchvision.transforms.ToTensor.

When using torchvision.transforms, an image is divided by 255 with ToTensor, and Normalize subtracts mean and divides it by standard deviation.

But, in my implementation, normalize first divides the image with 255, and then subtracts mean and divides it by standard deviation. to_tensor of my implementation only swaps channel order and makes it torch.tensor.
https://github.com/hysts/pytorch_image_classification/blob/master/transforms.py#L7-L27

[twmht]

@hysts

it's my mistake. thank you.