Non-deterministic behavior in cv2.distanceTransform()

[SherifN]

System Information

OpenCV python version: 4.5.5
Operating System: Ubuntu 16.04
Python version: 3.9.15

Detailed description

I am using cv2.distanceTransform in a deep learning application and have noticed non-deterministic training results in pytorch lightning. After setting manual seeds for all random number generators in my setup, the training was still not deterministic so I decided to have a closer look at the used data. I use cv2.distanceTransform in the ground truth data generation in my training and have noticed variation in the output using the same input.

It's possible to mitigate the problem by rounding the results to one decimal, although this is not the best solution ...

result = np.round(cv2.distanceTransform(feat_map, distanceType=cv2.DIST_L2, maskSize=cv2.DIST_MASK_PRECISE), decimals=1)

Is there a better way to fix this problem?

Steps to reproduce

import numpy as np
import cv2

cv2.setNumThreads(1)

num_tests = 100
shape = (800, 800)
for i in range(num_tests):
    feat_map = np.full(shape=shape, fill_value=255, dtype=np.uint8)
    # Creating a random line
    random_x_start = np.random.randint(0, 399)
    random_x_end = np.random.randint(400, 799)
    random_y_start = np.random.randint(0, 399)
    random_y_end = np.random.randint(400, 799)
    X, Y = np.mgrid[random_x_start:random_x_end:1, random_y_start:random_y_end:1]
    line = np.vstack((X.flatten(), Y.flatten())).transpose().reshape(-1, 1, 2)
    cv2.polylines(feat_map, [line], isClosed=False, color=0)
    result_1 = cv2.distanceTransform(feat_map, distanceType=cv2.DIST_L2, maskSize=cv2.DIST_MASK_PRECISE)
    result_2 = cv2.distanceTransform(feat_map, distanceType=cv2.DIST_L2, maskSize=cv2.DIST_MASK_PRECISE)
    np.testing.assert_array_equal(result_1, result_2)

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)

[dkurt]

Cannot reproduce the problem with opencv-python==4.5.5.62 and opencv-python==4.5.5.64. Also tried latest 4.x branch with disabled IPP (as there is IPP code in https://github.com/opencv/opencv/blob/4.x/modules/imgproc/src/distransform.cpp).

Do you still have problem on the latest 4.8.0 release? Can you share your CPU arch and details from cat /proc/cpuinfo?

[a-mos]

Cannot reproduce the problem with opencv-python==4.5.5.62 and opencv-python==4.5.5.64. Also tried latest 4.x branch with disabled IPP (as there is IPP code in https://github.com/opencv/opencv/blob/4.x/modules/imgproc/src/distransform.cpp).

Do you still have problem on the latest 4.8.0 release? Can you share your CPU arch and details from cat /proc/cpuinfo?

Got same problem when fix number of threads to 1 (cv2.setNumThreads(1)).
Using all threads, current (4.8.0) implementation return deterministic results.

I use AMD Ryzen Threadripper 1900X CPU

[dkurt]

@a-mos, thanks! Can confirm an error on 4.x branch with IPP enabled (WITH_IPP=OFF works fine).

[dkurt]

@opencv-alalek, @mshabunin, @eplankin, can you recommend how to assign the bug to IPP team? C++ reproducer:

int cols = 10, rows = 10;
std::vector<Ipp32f> ref(cols * rows, 0);
std::vector<Ipp32f> dst(cols * rows, 0);
for (int i = 0; i < 2; ++i) {
    std::vector<Ipp8u> src(cols * rows, 255);
    src[0] = 0;

    IppStatus status;
    IppiSize roi = { cols, rows };
    Ipp8u *pBuffer;
    int bufSize=0;

    status = ippiTrueDistanceTransformGetBufferSize_8u32f_C1R(roi, &bufSize);

    pBuffer = (Ipp8u *)ippMalloc_L( bufSize );
    if (i == 0) {
        status = ippiTrueDistanceTransform_8u32f_C1R(src.data(), cols, ref.data(), cols*sizeof(float), roi, pBuffer);
    } else {
        status = ippiTrueDistanceTransform_8u32f_C1R(src.data(), cols, dst.data(), cols*sizeof(float), roi, pBuffer);
    }
    ippFree( pBuffer );
}

for (int y = 0; y < rows; ++y) {
    for (int x = 0; x < cols; ++x) {
        if (ref[y * cols + x] != dst[y * cols + x]) {
            printf("(x=%d, y=%d) diff %e\n", x, y, fabs(ref[y * cols + x] - dst[y * cols + x]));
        }
    }
}

(x=7, y=0) diff 4.768372e-07

[dkurt]

This is why OpenCV calls ippiTrueDistanceTransform_8u32f_C1R with num threads 1 but uses own implementation otherwise:

opencv/modules/imgproc/src/distransform.cpp

Lines 780 to 784 in 2c1ec42

	#if IPP_DISABLE_PERF_TRUE_DIST_MT
	// IPP uses floats, but 4097 cannot be squared into a float
	if((cv::getNumThreads()<=1 || (src.total()<(int)(1<<14))) &&
	src.rows < 4097 && src.cols < 4097)
	#endif

Origin of the macro is pretty old: 35c7216

[dkurt]

Some analysis on non-deterministica values location:
Description - white pixels are 255, black pixels are zeros, red pixels is where distanceTransform returns non-deterministic values. So all non-red pixels are stable:

As you may see:

1. The problematic pixels located not far from the zero pixels
2. Non-deterministic behavior reproduced only by x-axis - there is no unstable results by the y-axis

Code

import numpy as np
import cv2

cv2.setNumThreads(1)

np.random.seed(123)

num_tests = 100
shape = (200, 200)
for i in range(num_tests):
    feat_map = np.full(shape=shape, fill_value=255, dtype=np.uint8)
    # Creating a random line
    random_x_start = np.random.randint(0, shape[1] / 2 - 1)
    random_x_end = np.random.randint(shape[1] / 2, shape[1] - 1)
    random_y_start = np.random.randint(0, shape[0] / 2 - 1)
    random_y_end = np.random.randint(shape[0] / 2, shape[0] - 1)

    # 1.
    # X, Y = np.mgrid[random_x_start:random_x_end:1, random_y_start:random_y_end:1]
    # line = np.vstack((X.flatten(), Y.flatten())).transpose().reshape(-1, 1, 2)
    # cv2.polylines(feat_map, [line], isClosed=False, color=0)

    # 2.
    # cv2.circle(feat_map, (100, 150), 6, (0, 0, 0))

    # 3.
    cv2.line(feat_map, (90, 10), (17, 170), (0, 0, 0))

    result_1 = cv2.distanceTransform(feat_map.copy(), distanceType=cv2.DIST_L2, maskSize=cv2.DIST_MASK_PRECISE)
    result_2 = cv2.distanceTransform(feat_map.copy(), distanceType=cv2.DIST_L2, maskSize=cv2.DIST_MASK_PRECISE)

    feat_map = np.stack((feat_map, feat_map, feat_map)).transpose(1, 2, 0)

    diff = np.abs(result_1 - result_2)
    if np.max(diff) > 0:
        feat_map[diff > 0] = [0, 0, 255]
        cv2.imwrite("img4.png", feat_map)
        print('saved')
        break

I also do not recommend remove IPP part because it shows better performance:

Geometric mean (ms)

                                                                      Name of Test                                                                          ipp     ipp      ipp    
                                                                                                                                                            no      no        no    
                                                                                                                                                          threads threads  threads  
                                                                                                                                                                    no        no    
                                                                                                                                                                    ipp      ipp    
                                                                                                                                                                   dist      dist   
                                                                                                                                                                              vs    
                                                                                                                                                                             ipp    
                                                                                                                                                                              no    
                                                                                                                                                                           threads  
                                                                                                                                                                          (x-factor)
distanceTransform::DistanceTransform_Test::(640x480, DIST_L2, DIST_MASK_PRECISE, CV_8U)                                                                    2.732   5.640     0.48   
distanceTransform::DistanceTransform_Test::(640x480, DIST_L2, DIST_MASK_PRECISE, CV_32F)                                                                   2.688   5.538     0.49   
distanceTransform::DistanceTransform_Test::(800x600, DIST_L2, DIST_MASK_PRECISE, CV_8U)                                                                    4.295   9.001     0.48   
distanceTransform::DistanceTransform_Test::(800x600, DIST_L2, DIST_MASK_PRECISE, CV_32F)                                                                   4.289   8.965     0.48   
distanceTransform::DistanceTransform_Test::(1024x768, DIST_L2, DIST_MASK_PRECISE, CV_8U)                                                                  10.188  16.042     0.64   
distanceTransform::DistanceTransform_Test::(1024x768, DIST_L2, DIST_MASK_PRECISE, CV_32F)                                                                 10.206  16.354     0.62   
distanceTransform::DistanceTransform_Test::(1280x1024, DIST_L2, DIST_MASK_PRECISE, CV_8U)                                                                 12.947  23.042     0.56   
distanceTransform::DistanceTransform_Test::(1280x1024, DIST_L2, DIST_MASK_PRECISE, CV_32F)                                                                13.045  23.091     0.56 

[dkurt]

After #24984 problem remains actual

[eplankin]

@dkurt, a bug to IPP was submitted.