`F.pad` in `FixedSizeCrop` needs `int` not `float`

[rvandeghen]

🐛 Describe the bug

The augment FixedSizeCrop in the detection recipe is filled with mean=(123.0, 117.0, 104.0) but when we call the padding function:

vision/references/detection/transforms.py

Line 356 in 2921174

img = F.pad(img, padding, self.fill, self.padding_mode)

we should use int not float as it produces the following error:

~/anaconda3/envs/pt1.11-tv0.12-cu10.2/lib/python3.7/site-packages/torchvision/transforms/functional.py in pad(img, padding, fill, padding_mode)
    483         _log_api_usage_once(pad)
    484     if not isinstance(img, torch.Tensor):
--> 485         return F_pil.pad(img, padding=padding, fill=fill, padding_mode=padding_mode)
    486 
    487     return F_t.pad(img, padding=padding, fill=fill, padding_mode=padding_mode)

~/anaconda3/envs/pt1.11-tv0.12-cu10.2/lib/python3.7/site-packages/torchvision/transforms/functional_pil.py in pad(img, padding, fill, padding_mode)
    166             return image
    167 
--> 168         return ImageOps.expand(img, border=padding, **opts)
    169     else:
    170         if isinstance(padding, int):

~/anaconda3/envs/pt1.11-tv0.12-cu10.2/lib/python3.7/site-packages/PIL/ImageOps.py in expand(image, border, fill)
    402     else:
    403         palette = None
--> 404     out = Image.new(image.mode, (width, height), color)
    405     if palette:
    406         out.putpalette(palette.palette)

~/anaconda3/envs/pt1.11-tv0.12-cu10.2/lib/python3.7/site-packages/PIL/Image.py in new(mode, size, color)
   2668         im.palette = ImagePalette.ImagePalette()
   2669         color = im.palette.getcolor(color)
-> 2670     return im._new(core.fill(mode, size, color))
   2671 
   2672 

TypeError: integer argument expected, got float

We can either cast the tuple of float to a tuple of int with

self.fill = tuple(map(int, fill))

in the __init__ or change the value in the presets script.

Versions

[pip3] numpy==1.21.5
[pip3] torch==1.11.0
[pip3] torchvision==0.12.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.2.89              hfd86e86_1  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py37h7f8727e_0  
[conda] mkl_fft                   1.3.1            py37hd3c417c_0  
[conda] mkl_random                1.2.2            py37h51133e4_0  
[conda] numpy                     1.21.5           py37he7a7128_1  
[conda] numpy-base                1.21.5           py37hf524024_1  
[conda] pytorch                   1.11.0          py3.7_cuda10.2_cudnn7.6.5_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchvision               0.12.0               py37_cu102    pytorch

cc @vfdev-5 @datumbox

[datumbox]

Thanks for reporting.

PIL only supports ints. This is why you get this error.

@vfdev-5 Do you think we should do this casting within the low-level kernel of PIL? What are your thoughts?

[rvandeghen]

@datumbox I did the change from my side but I guess it should also be done in the detection recipe.

[datumbox]

@rvandeghen Let's get Victor's input on where is the best place to put this. Then we can see if a PR is needed and what to modify. :) I assume if that's the direction we will go, you would be OK to send a PR?

[vfdev-5]

Looks like detection recipe is passing float values here :

vision/references/detection/presets.py

Line 6 in 2921174

def __init__(self, *, data_augmentation, hflip_prob=0.5, mean=(123.0, 117.0, 104.0)):

but F.pad (stable API) requires ints or tuple of ints for PIL RGB images :

vision/torchvision/transforms/functional.py

Line 434 in 2921174

def pad(img: Tensor, padding: List[int], fill: int = 0, padding_mode: str = "constant") -> Tensor:

(we can pass float values for F type of PIL Image I suppose, but these images are single channel )

vision/torchvision/transforms/functional.py

Lines 451 to 455 in 2921174

	fill (number or tuple): Pixel fill value for constant fill. Default is 0.
	If a tuple of length 3, it is used to fill R, G, B channels respectively.
	This value is only used when the padding_mode is constant.
	Only number is supported for torch Tensor.
	Only int or tuple value is supported for PIL Image.

If it works with mean=(123, 117, 104), I'd change it in the recipe. On the other hand tensors support only numbers...

EDIT:
However, for transforms using grid sampler, it was implemented fill as a tuple for tensors:

vision/torchvision/transforms/functional_tensor.py

Lines 589 to 600 in 2921174

	# Fill with required color
	if fill is not None:
	mask = img[:, -1:, :, :] # N * 1 * H * W
	img = img[:, :-1, :, :] # N * C * H * W
	mask = mask.expand_as(img)
	len_fill = len(fill) if isinstance(fill, (tuple, list)) else 1
	fill_img = torch.tensor(fill, dtype=img.dtype, device=img.device).view(1, len_fill, 1, 1).expand_as(img)
	if mode == "nearest":
	mask = mask < 0.5
	img[mask] = fill_img[mask]
	else: # 'bilinear'
	img = img * mask + (1.0 - mask) * fill_img

[rvandeghen]

From my point of view, I would also change in the the recipe with mean=(123, 117, 104) because the other reference of mean is used with the ssd augmentation which also recasts the values in int

vision/references/detection/transforms.py

Lines 169 to 172 in 2921174

	def _get_fill_value(self, is_pil):
	# type: (bool) -> int
	# We fake the type to make it work on JIT
	return tuple(int(x) for x in self.fill) if is_pil else 0

I tried the change and it works for both augmentations.

If it's ok for you, I will make the PR.

[datumbox]

I suspect this has already been fixed by #5596. See my comment at #5981 (comment)