Added center as top-left for shear X/Y ops for autoaugment

[vfdev-5]

Fixes #5204

Description:

• Added center as top-left for shear X/Y ops for autoaugment

@datumbox do you think we should add something else here to be fully consistent ?

Here is how affine with center=(0, 0) behaves for shear X and Y vs using PIL as in TF:

from PIL import Image
import math
import torchvision # >= 0.11
import numpy as np

from torchvision.transforms import functional as F
interpolation = torchvision.transforms.InterpolationMode.NEAREST
fill = None

img = Image.fromarray(np.arange(3 * 128 * 128, dtype='uint8').reshape(128, 128, 3))
magnitude = .4

# shear_x as seen in torchvision https://github.com/pytorch/vision/blob/b5aa0915fe16e82ee4c24919032b4e7afae3ae1b/torchvision/transforms/autoaugment.py#L17
im_torch = F.affine(img, angle=0.0, translate=[0, 0], scale=1.0, shear=[math.degrees(magnitude), 0.0],                    
                    interpolation=interpolation, fill=fill, center=[0, 0])

# shear_x as seen in https://github.com/automl/trivialaugment/blob/3bfd06552336244b23b357b2c973859500328fbb/aug_lib.py#L156 and https://github.com/tensorflow/models/blob/fd34f711f319d8c6fe85110d9df6e1784cc5a6ca/research/autoaugment/augmentation_transforms.py#L290
im_pil = img.transform(img.size, Image.AFFINE, (1, magnitude, 0, 0, 1, 0))

print("Original image")
img.show()
print("Torchvision Shear X with center as top-left")
im_torch.show()
print("Shear X using PIL as TF")
im_pil.show()

# shear_x as seen in torchvision https://github.com/pytorch/vision/blob/b5aa0915fe16e82ee4c24919032b4e7afae3ae1b/torchvision/transforms/autoaugment.py#L17
im_torch = F.affine(img, angle=0.0, translate=[0, 0], scale=1.0, shear=[0.0, math.degrees(magnitude)],                    
                    interpolation=interpolation, fill=fill, center=[0, 0])

# shear_x as seen in https://github.com/automl/trivialaugment/blob/3bfd06552336244b23b357b2c973859500328fbb/aug_lib.py#L156 and https://github.com/tensorflow/models/blob/fd34f711f319d8c6fe85110d9df6e1784cc5a6ca/research/autoaugment/augmentation_transforms.py#L290
im_pil = img.transform(img.size, Image.AFFINE, (1, 0, 0, magnitude, 1, 0))


print("Torchvision Shear Y with center as top-left")
im_torch.show()
print("Shear Y using PIL as TF")
im_pil.show()

cc @SamuelGabriel

EDIT:

Links to TF implementation uses Pillow with shear parametrized such that fixed point is top-left:

• https://github.com/tensorflow/models/blob/885fda091c46c59d6c7bb5c7e760935eacc229da/research/autoaugment/augmentation_transforms.py#L273-L290

However, official vision ops refers as well autoaugment paper and does shear with TF raw ops, the only difference is that they do fill_mode="reflect" by default:

• https://github.com/tensorflow/models/blob/885fda091c46c59d6c7bb5c7e760935eacc229da/official/vision/beta/ops/augment.py#L504-L512
• https://github.com/tensorflow/models/blob/885fda091c46c59d6c7bb5c7e760935eacc229da/official/vision/beta/ops/augment.py#L180-L181
• https://github.com/keras-team/keras/blob/89879e2c76e86c685e44c47a6cdb82f7e645c142/keras/layers/preprocessing/image_preprocessing.py#L654

Output for magnitude = .4 shear X with TF from official vision ops:

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit 6412fb0 (more details on the Dr. CI page):

---

• 3/3 failures introduced in this PR

---

3 failures not recognized by patterns:

Job	Step	Action
cmake_macos_cpu	curl -o conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
sh conda.sh -b
source $HOME/miniconda3/bin/activate
conda install -yq conda-build cmake
packaging/build_cmake.sh
	🔁 rerun
binary_linux_conda_py3.7_cu115	packaging/build_conda.sh	🔁 rerun
binary_linux_conda_py3.7_cu111	packaging/build_conda.sh	🔁 rerun

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[datumbox]

@vfdev-5 Thanks a lot for the investigations and for sending the PR.

We had long discussions with Victor about this because it's a tough call but I'll summarize here the reasons why we believe we should merge this BC-breaking change. After looking into the original implementations of AutoAugment and RandAugment, it became clear that their Shear method was using the top-left as the center. Unfortunately TorchVision didn't support this option until it was raised by the author of TrivialAugment @SamuelGabriel at #5204. This means that our implementation deviated from the canonical and thus we consider this discrepancy a bug.

Though I don't believe that this discrepancy had major effect on the overall accuracy of the models that we trained previously, I think it's worth aligning TorchVision's behaviour with the rest of the ecosystem as the majority of frameworks and libraries we checked all implement the Shear operator with center the top-left corner.

I recommend leaving this PR unmerged for a couple of days to allow for people to comment. If there are no objections we can merge it early next week.