Adjust Gaussian blur padding mode

[stephen-huan]

🚀 The feature

Currently torchvision.transforms.v2.GaussianBlur takes a kernel_size and sigma but has no means of adjusting the padding_mode (as in Conv2d). I propose either (1) adding such an argument or (2) changing the default padding mode from "reflect" to "symmetric".

Happy to submit a PR if there's interest.

Motivation, pitch

The padding mode "reflect" is not intensity preserving, that is, the average pixel value can change after blurring (even though the kernel sums to 1). This would be fixed if the padding mode "symmetric" was used. This mirrors opencv's GaussianBlur(), which defaults to BORDER_REFLECT_101, and instead BORDER_REFLECT must be used for the intensity preserving behavior. Instead, scipy's scipy.ndimage.gaussian_filter defaults to the confusingly named "reflect" (pytorch's "symmetric") and pytorch's "reflect" is instead "mirror".

In summary,

• torch "reflect" / opencv BORDER_REFLECT_101/ scipy "mirror": not intensity preserving, default for torch and opencv.
• torch "symmetric" / opencv BORDER_REFLECT/ scipy "reflect": intensity preserving, default for scipy.

I find the scipy behavior more intuitive and I see no reason to prefer "reflect" over "symmetric" as a default. If we want to maintain strict equivalence to opencv, then we should add an argument to customize the padding like opencv and scipy.

import cv2 as cv
import numpy as np
import torch
from torchvision.transforms.v2.functional import gaussian_blur
from scipy.ndimage import gaussian_filter

if __name__ == "__main__":
    img = np.zeros((8, 8))
    img[[0, -1], :] = 1
    img[:, [0, -1]] = 1
    img /= np.sum(img)
    print(f"original: {np.sum(img):.6f}")

    blur = gaussian_filter(img, 1, radius=2)
    print(f"   scipy: {np.sum(blur):.6f}")

    blur = gaussian_filter(img, 1, radius=2, mode="mirror")
    print(f"   scipy: {np.sum(blur):.6f}")

    blur = cv.GaussianBlur(img, (5, 5), 1, borderType=cv.BORDER_REFLECT)
    print(f"  opencv: {np.sum(blur):.6f}")

    blur = cv.GaussianBlur(img, (5, 5), 1)
    print(f"  opencv: {np.sum(blur):.6f}")

    x = torch.tensor(np.expand_dims(img, 0))
    blur = gaussian_blur(x, [5, 5], [1, 1])
    print(f"   torch: {torch.sum(blur):.6f}")

original: 1.000000
   scipy: 1.000000
   scipy: 0.731235
  opencv: 1.000000
  opencv: 0.731235
   torch: 0.731235

somewhat related: opencv/opencv#9863

Alternatives

No response

Additional context

No response

[NicolasHug]

Thank you for the detailed feature request @stephen-huan . That sounds reasonable, although for backward-compatibility of results, we should preserve the existing reflect padding mode as the default. TorchVision's default values for transforms aim to mimic PIL's implementation, so I suspect -but haven't checked - that this is why the existing default is reflect. Happy to consider a PR that adds a new padding_mode parameter.

[stephen-huan]

That sounds reasonable, although for backward-compatibility of results, we should preserve the existing reflect padding mode as the default.

Sounds good.

TorchVision's default values for transforms aim to mimic PIL's implementation, so I suspect -but haven't checked - that this is why the existing default is reflect.

Correct me if I'm wrong, but the existing tests test against opencv.

vision/test/test_transforms_v2.py

Lines 3924 to 3927 in 342eb92

	REFERENCE_GAUSSIAN_BLUR_IMAGE_RESULTS = torch.load(
	Path(__file__).parent / "assets" / "gaussian_blur_opencv_results.pt",
	weights_only=False,
	)

As stated above opencv defaults to (something equivalent to torch's) reflect.

Happy to consider a PR that adds a new padding_mode parameter.

Sounds good, I'll ping you when it's ready.

[stephen-huan]

Oh and while we're talking about modifications to Gaussian blur, would a PR adding the separable kernel optimization be considered? (i.e., since the 2-d Gaussian kernel is the outer product of 1-d kernels, it can be performed as a row and column convolution for a complexity of O(K N^2) instead of O(K^2 N^2) for a K x K kernel and N x N image). This makes a legitimate performance difference for very large kernels (necessary to faithfully capture very large sigma) and is performed by opencv (and scipy) as well. There should be no user-visible changes (other than improved performance).

We can also use fft to accelerate the convolution, leading to something like a O((K^2 + N^2) log(K^2 + N^2) time algorithm. No mainstream library does this to the best of my knowledge, and I am not sure exactly why.

[NicolasHug]

Correct me if I'm wrong, but the existing tests test against opencv

You're right, generally torchvision mimics PIL, but there are rare exceptions. I looked at the original PR that implemented GaussianBlur and I couldn't find a reason for that. It's possible that PIL didn't implement GaussianBlur at the time, but that's unlikely since python-pillow/Pillow#961 is from 2014.

For the separable kernel optimization: do you mind opening a separate issue for this? It can be in scope, as long as the changes can be transparent for the user, and we'd have to confirm the expected speed-ups via benchmarks. Thanks!