PIL

[AlexanderShashkin]

I would like to add a possibility to resize in a "PIL nearest neighbor style".
I've started with the issue
#9096

Than try it by myself and see differencies:

Left is a original binary mask, middle is a PIL resize and right is OpenCV resize.
Both libraries were launched with nearest neighbor algorithm.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

• I agree to contribute to the project under OpenCV (BSD) License.
• To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
• The PR is proposed to proper branch
• There is reference to original bug report and related work
• There is accuracy test, performance test and test data in opencv_extra repository, if applicable
  Patch to opencv_extra has the same branch name.
• The feature is well documented and sample code can be built with the project CMake

force_builders=linux,docs

[asmorkalov]

@AlexanderShashkin Thanks for the contribution. Could you describe the purpose of PIL-style resize? What is your use-case? Also please take a look CI bot failures.

[dkurt]

We definitely know that there is difference between OpenCV and PIL bilinear resize. Is that the same for NN interpolation? I think there are difference only with specific configurations like non integer scale factor (i.e. 10x11 to 13x14) or with downsampling. So please provide an example when OpenCV NN resize does not match PIL's one. Thanks!

/cc @l-bat

[asmorkalov]

@AlexanderShashkin thanks for detailed description.

[AlexanderShashkin]

We definitely know that there is difference between OpenCV and PIL bilinear resize. Is that the same for NN interpolation? I think there are difference only with specific configurations like non integer scale factor (i.e. 10x11 to 13x14) or with downsampling. So please provide an example when OpenCV NN resize does not match PIL's one. Thanks!

/cc @l-bat

@AlexanderShashkin Thanks for the contribution. Could you describe the purpose of PIL-style resize? What is your use-case? Also please take a look CI bot failures.

Sorry for bot failures, this is because of test file that I included to show the difference. I've deleted it for now and than I will add test.

[mshabunin]

It would be nice to have slightly more verbose description in the documentation, something like differs from INTER_NEAREST in the way it calculates pixel coordinates: center of the square instead of top left corner. For example, pixel (0, 0) will have coordinates (0.5, 0.5) in this mode.

[mshabunin]

Won't this new enum value interfere with flags? INTER_NEAREST_PIL = 9 // 1001 will imply INTER_LINEAR = 1 with WARP_FILL_OUTLIERS = 8 // 1000 flag set. Maybe it is better to add new flag: WARP_SQUARE_PIXELS or WARP_SHIFT_COORDINATES with value 32? Or maybe INTER_NEAREST_PIL should have value 6?

[AlexanderShashkin]

We definitely know that there is difference between OpenCV and PIL bilinear resize. Is that the same for NN interpolation? I think there are difference only with specific configurations like non integer scale factor (i.e. 10x11 to 13x14) or with downsampling. So please provide an example when OpenCV NN resize does not match PIL's one. Thanks!

/cc @l-bat

@AlexanderShashkin Thanks for the contribution. Could you describe the purpose of PIL-style resize? What is your use-case? Also please take a look CI bot failures.

Sorry for bot failures, this is because of test file that I included to show the difference. I've deleted it for now and than I will add test.

We definitely know that there is difference between OpenCV and PIL bilinear resize. Is that the same for NN interpolation? I think there are difference only with specific configurations like non integer scale factor (i.e. 10x11 to 13x14) or with downsampling. So please provide an example when OpenCV NN resize does not match PIL's one. Thanks!

/cc @l-bat

Yes, these are really differencies in NN algorithm as well. I've added pictures in a descriprion of this PR. The original binary mask ias 256x256 and than it was downscaled and upscaled with thae factor of 16.
I've implemented NN algorithm as I've found in PIL sources (in Python) that binary images are resized only with NN algorithm no matter which flag was given to the function (bilinear or whatever):
if self.mode in ("1", "P"):
resample = NEAREST
Binary images is "1" and for them only NEAREST in PIL

Now my solution works only for grayscale and squared images, I'm going to support not only squared and add test.
Could you please advice, which tests and docs will be enough?

[mshabunin]

You support only CV_8UC1. What about other types of Mat? You should at least add Not implemented assertion.

[AlexanderShashkin]

I've found in resize.cpp similar situation:
if (interpolation == INTER_LINEAR_EXACT && (_src.depth() == CV_32F || _src.depth() == CV_64F))
interpolation = INTER_LINEAR; // If depth isn't supported fallback to generic resize

and add my code:
if (interpolation == INTER_NEAREST_PIL && _src.depth() != CV_8UC1)
interpolation = INTER_NEAREST; // INTER_NEAREST_PIL is supported only for grayscale images

If image is not a mask (binary) or grayscale generic INTER_NEAREST will be used. I suppose it could be better than Not implemented assertion. What do you think?
In case NotImplemented is a demand - I'll do it.

[mshabunin]

Case with INTER_LINEAR_EXACT is slightly different, it is same algorithm as INTER_LINEAR but with stable calculation method which produces same results on all platforms. While INTER_NEAREST_PIL is separate algorithm and can produce completely different results so the assertion is more appropriate.

[AlexanderShashkin]

OK, I've added
if (interpolation == INTER_NEAREST_PIL && _src.depth() != CV_8UC1)
CV_Error(Error::StsNotImplemented, "INTER_NEAREST_PIL supports only CV_8UC1");

[vpisarev]

does not look very fast.

const char* srcrow = src.ptr<char>(yi);
char* dstrow = dst.ptr<char>(y);
for(x = 0; x < dsize.width; x++)
    dstrow[x] = srcrow[x_ofs[x]];

should be a bit better

[mshabunin]

This functionality must be tested.

[AlexanderShashkin]

I've added tests in test_resize_pil.cpp

[mshabunin]

I think it is better to use synthetic image, maybe gradient?

[AlexanderShashkin]

Yes, I've added tests with gradient.png file

[AlexanderShashkin]

@mshabunin, I've added test with gradient.png but my tests haven't passed due to absence of this file in test folder. Here is my code:

const string root = cvtest::TS::ptr()->get_data_path();
const string filename = root + "gradient.png";
Mat img;
ASSERT_NO_THROW(img = imread(filename, IMREAD_GRAYSCALE));
ASSERT_FALSE(img.empty());

and the error is:
Value of: img.empty()
Actual: true
Expected: false

Should I add tests with another image or it is possible to add gradient.png in test data folder as well?

[mshabunin]

I meant that you can generate source image like here on the fly.

[AlexanderShashkin]

OK, I've done it in test_resize_pil.cpp

[mshabunin]

Perhaps it makes sense to show results of both NEAREST_PIL and NEAREST to show difference, otherwise the sample does not have any value. Or, even better, all modes and flags should be visualized with lena.jpg and/or some synthetic images - that would be a great example.

[dkurt]

Maybe it's even better to have Python sample instead of C++ which can run both PIL and OpenCV and compare the difference between, INTER_NEAREST, potential INTER_NEAREST_PIL and PIL?

[AlexanderShashkin]

Yes, sure. I've created resize_pil.py in which I've demonstrated:

1. Difference between cv.INTER_NEAREST and PIL NEAREST
2. Equivalence betwen cv.INTER_NEAREST_PIL and PIL NEAREST
   I've used lena.jpg

[AlexanderShashkin]

@AlexanderShashkin thanks for detailed description.

@asmorkalov , could you please remove tags needs docs and need tests as I've added them both

[asmorkalov]

@AlexanderShashkin good job! done.

[asmorkalov]

@mshabunin @alalek Are you ready to merge the PR?

[mshabunin]

I agree with @catree , we need to think about better naming for this mode. Maybe it should be another flag?

[catree]

Yes, probably another flag would be better. To be able to do something like:

dst = cv.resize(img, new_size, \
    interpolation=cv.INTER_NEAREST + cv.CENTER_PIXEL_COORDINATES)
dst = cv.warpAffine(img, M, new_size, \
    interpolation=cv.INTER_LINEAR_EXACT + \
    cv.CENTER_PIXEL_COORDINATES + cv.WARP_FILL_OUTLIERS)
//same thing for cv.warpPerspective, cv.remap, ....?

Not sure, but I think pixel coordinates convention should matter also for getRotationMatrix2D(), and probably some other functions.

---

Comparison between OpenCV, Pillow, scikit-images and Matlab NN interpolation mode.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# Python 2/3 compatibility
from __future__ import print_function

import numpy as np
import cv2 as cv
print('OpenCV:', cv.__version__)
import PIL
from PIL import Image
print('PIL:', PIL.__version__)
import skimage
from skimage.transform import resize
print('scikit-image:', skimage.__version__)

def resize_compare(size, new_size):
    img = np.arange(size[0]*size[1], dtype=np.uint8).reshape(size)
    print('\nimg:\n', img)

    # OpenCV
    img_resize_OpenCV = cv.resize(img, (new_size[1], new_size [0]), interpolation=cv.INTER_NEAREST)
    print('\nOpenCV:\n', img_resize_OpenCV)

    # Pillow
    im = Image.fromarray(img)
    img_resize_PIL = im.resize((new_size[1], new_size [0]), resample=Image.NEAREST)
    print('\nPIL:\n', np.array(img_resize_PIL))

    # scikit-image
    img_resize_sk = resize(img, new_size, order=0, preserve_range=True).astype(np.uint8)
    print('\nskimage:\n', img_resize_sk)

size = (3,5)
new_size = (5,7)
resize_compare(size, new_size)

size = (2,3)
new_size = (4,6)
resize_compare(size, new_size)

clearvars;
clc;

iw = 5;
ih = 3;
ow = 7;
oh = 5;

img = reshape(uint8(0:iw*ih-1), [iw, ih])'
img_resize = imresize(img, [oh, ow], 'nearest')

iw = 3;
ih = 2;
ow = iw*2;
oh = ih*2;

img = reshape(uint8(0:iw*ih-1), [iw, ih])'
img_resize = imresize(img, [oh, ow], 'nearest')

Results:

OpenCV: 4.3.0-dev
PIL: 7.1.2
scikit-image: 0.14.2

img:
 [[ 0  1  2  3  4]
 [ 5  6  7  8  9]
 [10 11 12 13 14]]

OpenCV:
 [[ 0  0  1  2  2  3  4]
 [ 0  0  1  2  2  3  4]
 [ 5  5  6  7  7  8  9]
 [ 5  5  6  7  7  8  9]
 [10 10 11 12 12 13 14]]

PIL:
 [[ 0  1  1  2  3  3  4]
 [ 0  1  1  2  3  3  4]
 [ 5  6  6  7  8  8  9]
 [10 11 11 12 13 13 14]
 [10 11 11 12 13 13 14]]

skimage:
 [[ 0  1  1  2  3  3  4]
 [ 0  1  1  2  3  3  4]
 [ 5  6  6  7  8  8  9]
 [10 11 11 12 13 13 14]
 [10 11 11 12 13 13 14]]

img:
 [[0 1 2]
 [3 4 5]]

OpenCV:
 [[0 0 1 1 2 2]
 [0 0 1 1 2 2]
 [3 3 4 4 5 5]
 [3 3 4 4 5 5]]

PIL:
 [[0 0 1 1 2 2]
 [0 0 1 1 2 2]
 [3 3 4 4 5 5]
 [3 3 4 4 5 5]]

skimage:
 [[0 0 1 1 2 2]
 [0 0 1 1 2 2]
 [3 3 4 4 5 5]
 [3 3 4 4 5 5]]

img =

  3×5 uint8 matrix

    0    1    2    3    4
    5    6    7    8    9
   10   11   12   13   14


img_resize =

  5×7 uint8 matrix

    0    1    1    2    3    3    4
    0    1    1    2    3    3    4
    5    6    6    7    8    8    9
   10   11   11   12   13   13   14
   10   11   11   12   13   13   14


img =

  2×3 uint8 matrix

   0   1   2
   3   4   5


img_resize =

  4×6 uint8 matrix

   0   0   1   1   2   2
   0   0   1   1   2   2
   3   3   4   4   5   5
   3   3   4   4   5   5

Looks like output of Pillow, scikit-images and Matlab match.

[homm]

PIL has the same problems with arithmetic which you try to fix and is abandoned for a long time. The geometry for affine transformations was fixed in Pillow. Please consider change the name of the PR and correct all comments to prevent misleading.

[AlexanderShashkin]

Yes, probably another flag would be better. To be able to do something like:

dst = cv.resize(img, new_size, \
    interpolation=cv.INTER_NEAREST + cv.CENTER_PIXEL_COORDINATES)

I agree with @catree , we need to think about better naming for this mode. Maybe it should be another flag?

@mshabunin ,

• what do you think about cv.CENTER_PIXEL_COORDINATES? If it is accepted, I can change the flag and tests for resize.

• as @catree has also mentioned some other functions, probably we could start thinking about them as well

[AlexanderShashkin]

PIL has the same problems with arithmetic which you try to fix and is abandoned for a long time. The geometry for affine transformations was fixed in Pillow. Please consider change the name of the PR and correct all comments to prevent misleading.

@homm yes, we're definetely talking about Pillow. I've seen your comment to the issue
#10146
and reproduced your example from https://stackoverflow.com/questions/49764662/nearest-neighbor-image-resizing-wrong-using-scipy-and-pil/49784207
INTER_NEAREST_PIL gives exactly the same result, so I've resized
{1, 2, 3,
4, 5, 6,
7, 8, 9}
to
[ 1, 1, 1, 2, 2, 2, 3, 3, 3;
1, 1, 1, 2, 2, 2, 3, 3, 3;
1, 1, 1, 2, 2, 2, 3, 3, 3;
4, 4, 4, 5, 5, 5, 6, 6, 6;
4, 4, 4, 5, 5, 5, 6, 6, 6;
4, 4, 4, 5, 5, 5, 6, 6, 6;
7, 7, 7, 8, 8, 8, 9, 9, 9;
7, 7, 7, 8, 8, 8, 9, 9, 9;
7, 7, 7, 8, 8, 8, 9, 9, 9]

[asmorkalov]

@mshabunin @dkurt Friendly reminder. I re-triggered CI build to update status.

[mshabunin]

@asmorkalov , we need design decision on adding a flag for shifted pixel coordinates mode and its naming (see #17068 (comment)). Implementations for other resize modes and same flag for other functions (e.g. warpAffine) can be added later (should it be named differently for each function? or should it be placed in separate enum?).

[vpisarev]

@AlexanderShashkin, thank you for the patch!
@dkurt, @mshabunin, @asmorkalov, my vote is for INTER_NEAREST_PIL. It would be nice to support color images and speedup the inner loop, but maybe the patch can be integrated and then the code can be polished further.

[homm]

my vote is for INTER_NEAREST_PIL

Again, PIL originally has the same issue. It was fixed only in Pillow

[AlexanderShashkin]

@AlexanderShashkin, thank you for the patch!
@dkurt, @mshabunin, @asmorkalov, my vote is for INTER_NEAREST_PIL. It would be nice to support color images and speedup the inner loop, but maybe the patch can be integrated and then the code can be polished further.

@vpisarev , thanx! I can add a support for color images when naming is approved

[AlexanderShashkin]

my vote is for INTER_NEAREST_PIL

Again, PIL originally has the same issue. It was fixed only in Pillow

yes, sure, as I've mentioned before, we're definetely talking about Pillow and there is no that issue in my INTER_NEAREST_PIL implementation

[homm]

and there is no that issue in my INTER_NEAREST_PIL implementation

And you still call it PIL…

[catree]

Please disregard my comments in this thread above.

From my point of view:

---

About adding a flag for pixel center coordinates:

• unfortunately not gonna happen given the amount of work and how everything is possibly entangled between C++, SIMD code, OCL, Cuda, resize code and related functions like warpAffine, ... (too much work to make everything consistent)
• did a quick test between Pillow, scikit-image and Matlab and none of these can produce a consistent result
• maybe all of these methods use center pixel coordinates and the difference comes from rounding method?
• I am definitively not qualified to speak about these resize and pixel coordinates issues

---

About using INTER_NEAREST_PILLOW. I just strongly disagree to name it using the name of a library. Moreover when the NN resize method is trivial.
I can only propose INTER_NEAREST_2 instead, and in the documentation add something like: "produce same results than PIL, scikit-image or Matlab like NN resize method".

Speaking about documentation, it would be great to add the formula to iterate over the pixels. From the code I have no idea how it works. I believe the correct formula to take into account pixel center coordinates in the naive implementation is the following:

u_dst = (u_src + 0.5) * scale_u - 0.5;
v_dst = (v_src + 0.5) * scale_v - 0.5;

Plus the rounding / border methods.
This would be very useful for learning purpose for OpenCV users.

---

About this PR:

• honestly I see no urge to have this PR merged,
• especially if only grayscale is supported
• I see some for loop variables declared outside. For me this is useless and not recommended in C++ but I might be wrong. Since I am using github code to learn best practices, and I believe other people too, it would be great if more efforts toward code quality would be made.

---

Finally my last words.

Whatever solution, flag name is chosen, from this statement:

once a code is inside into OpenCV, it stays for years

Please check that the user will not be able to shoot himself in the foot:

• e.g. the enum value INTER_NEAREST_PIL = 6 for correct use in combination with other flags maybe 6 works but the idea is there
• support of this flag with (possibly) related functions like other warping functions
• different mat type

[alalek]

To exclude mess with common "interpolation" enum (shared between other warp functions) and avoid creation of God functions (like cvtColor) we should:

• use different separate name, like resizePillow()
• which has own interpolation/filter modes (PIL.Image.NEAREST, PIL.Image.BOX, PIL.Image.BILINEAR, PIL.Image.HAMMING, PIL.Image.BICUBIC or PIL.Image.LANCZOS)

Unification of resize pixel centers in 0 / 0.5 should be done as a separate process (through OE).

[vpisarev]

@AlexanderShashkin, @alalek, @homm,

I played a bit with it and think and we need just one INTER_NEAREST that works well enough in all cases. Whether it will be a universal formula or some adaptive one, depending on the combination of the input and output size – we need to see.

Here are some partial cases where the behaviour is either obvious or just standard convention that people rely on:

1. 2x decimation. The formula should return even points, starting with 0: [0, 2, 4, ...]. The suggested formula gives [1, 3, 5, ...], which could also be fine, but it's incompatible with the previous behaviour, and as far as I remember, for some algorithms it's crucial difference. The formula needs to be corrected, or special formula should be used in this case.
2. decimation n to 1 where n is odd. I think, in this case most reasonable behaviour is to choose the middle of each segment of n pixels. The new formula does it correctly.
3. decimation n*2-1 to n. This is the case in the reported issue. The new formula does it correctly, the old one is incorrect.
4. decimation n*2+1 to n. The new formula seems to do it well.
5. zoom 1 to n where n is integer. The formula should, of course, give segments of equal size. The new formula provides it.

Also, preferably the code should not use floating-point arithmetics to give the same results on any hardware.

I found that the following algorithm (in Python) satisfies all conditions (1-5):

sw = 5
dw0 = 3
fx = (dw0+0.)/sw
dw = int(round(sw*fx))
shift = 16
ifx = (sw*(1 << shift) + dw//2)//dw
ifx0 = ifx//2-1

print ([(ifx0+ifx*x) >> shift for x in range(0, dw)])

where sw is input image size (we consider width, height is processed in the same way), fx is the supplied scale factor, dw0 is the desired image size, from which user computed fx and passed it to the function. We first find dw – the output image size, and after that use solely fixed-point arithmetics.
The formula is basically equivalent to the proposed, with the only small difference how we compute the initial offset. We subtract a tiny 1/65535 from it, and this is enough to give [0, 2, 4, ...] sequence in the case of 2x decimation. This formula also gives correct results for the 3x5 => 5x7 image upscaling sample.

Maybe in general that ifx0 needs to be corrected, depending on the combination of sw and dw.

Probably, this should be put it into the current implementation of INTER_NEAREST (after OpenCV 4.4 release).

[asenyaev]

jenkins cn please retry a build

[aclark4life]

I love the terminology discussion here … I think it's fair to continue to refer to Pillow internals as PIL in variable names, particularly since the Pillow distribution still contains the PIL package. For example I recall we deprecated a PILLOW_VERSION variable in favor of PIL.__version__.