External Interface (OpenCV) with OpenMP enabled produce partial image

[VisualFox]

Component

DIPlib 3.5.2
tested also with DIPlib 3.4.1

Describe the bug

Calling a DipLib function from OpenCV using one of these three methods (method #3 is tested with and without copy) described here: https://github.com/DIPlib/diplib/blob/master/examples/external_interfaces/opencv_with_dip.cpp
with OpenMP enabled (DIP_ENABLE_MULTITHREADING=ON) will randomly produce an OpenCV image (cv::Mat) with only partially proceeded channels (channel partially filled with 0). Disabling OpenMP (DIP_ENABLE_MULTITHREADING=OFF) resolve the issue.

This behavior with OpenMP enabled seems random, not all channels are affected all the time. Multiple call of the same DipLib function (for example dip::Gauss) will only sometime produce the issue, but in my test the same binary will always produce the exact same defect in the exact order and place.

This behavior is also present with DIPlib 3.4.1 (I have not tried any other version).

To Reproduce

cv::Mat DipGauss(const cv::Mat& src, const dip::FloatArray& sigmas, const dip::UnsignedArray& derivativeOrder, const dip::String& method, const dip::StringArray& boundaryCondition, const dip::dfloat truncation) {
    dip_opencv::ExternalInterface external;

    const dip::Image src_dip = dip_opencv::MatToDip(src);
    dip::Image output_dip = external.NewImage();
    dip::Gauss(src_dip, output_dip, sigmas, derivativeOrder, method, boundaryCondition, truncation);

    return external.GetMat(output_dip);
}

int main(int argc, char** argv) {
    cv::Mat src = cv::imread("_PATH_TO _IMAGE_FILE", -1);
    cv::imshow("src", src);

    cv::Mat blurredA = DipGauss(src, {1.0}, {0}, "best", {"mirror"}, 3);
    cv::imshow("blurred 1.0", blurredA);

    cv::Mat blurredB = DipGauss(src, {3.0}, {0}, "best", {"mirror"}, 3);
    cv::imshow("blurred 3.0", blurredB);

    cv::Mat dx = DipGauss(src, {1.0}, {1, 0}, "best", {"mirror"}, 3);
    cv::imshow("dx", dx);

    cv::Mat dy = DipGauss(src, {1.0}, {0, 1}, "best", {"mirror"}, 3);
    cv::imshow("dy", dy);

    cv::Mat dxx = DipGauss(src, {1.0}, {2, 0}, "best", {"mirror"}, 3);
    cv::imshow("dxx", dxx);

    cv::Mat dyy = DipGauss(src, {1.0}, {0, 2}, "best", {"mirror"}, 3);
    cv::imshow("dyy", dyy);
}

System information:

Ubuntu 24.04
C++ compiler: GNU 13.3.0
OpenCV: 4.10.0
OpenMP: version 4.5 as reported by CMake (libomp-dev/noble 1:18.0-59~exp2 amd64)

[VisualFox]

Let me know if you need any extra information.

[crisluengo]

I'm not really sure what is causing your exact weird result, so I don't know if this will fix it or not.

Notice that your function DipGauss() returns a floating-point Mat object, which cv::imshow scales weirdly (floating point images are assumed to be in the range [0,1], whereas the smoothed images are in the range [0,255], and the derivatives in the range [-255,255]. I added a line

dip::ContrastStretch(output_dip, output_dip, 0, 100, 0, 1);

after the call to dip::Gauss to help the OpenCV display function. Alternatively, you can scale the input, but even then the derivatives will not display correctly because approximately half the pixels have a negative value, which will all be shown as black.

---

But you said that turning OpenMP off solved the issue, which makes me assume the code you posted is not your full code, and the contrast is not the problem.

I could not reproduce your issue on macOS, using Clang. I tried reproducing using GCC, but it won't link the Homebrew OpenCV (which is built with Clang).

It is possible that there's some issue with both OpenCV and DIPlib using OpenMP, but so far I've only seen issues when mixing libraries that use the Clang OpenMP with libraries that use the Intel OpenMP. And the result is usually a crash...

Could you help me debug this issue?

• Do you see the same problem if you build with OpenMP enabled, but call dip::SetNumberOfThreads(1) at the start of your program?
• Do you see this result using your example program as-is? Or are you calling DIPlib functions from within a multi-threaded environment?
• Does your version of OpenCV use pthreads, TBB or OpenMP for multi-threading?
• Do you see the same issues without OpenCV?

#include "diplib.h"
#include "diplib/linear.h"
#include "diplib/simple_file_io.h"
#include "dipviewer.h"

int main() {
   dip::Image src = dip::ImageRead("_PATH_TO_IMAGE_FILE");
   dip::viewer::Show(src, "src");

   dip::Image blurredA = dip::Gauss(src, {1.0}, {0});
   dip::viewer::Show(blurredA, "blurred 1.0");

   dip::Image blurredB = dip::Gauss(src, {3.0}, {0});
   dip::viewer::Show(blurredB, "blurred 3.0");

   dip::Image dx = dip::Gauss(src, {1.0}, {1, 0});
   dip::viewer::Show(dx, "dx");

   dip::Image dy = dip::Gauss(src, {1.0}, {0, 1});
   dip::viewer::Show(dy, "dy");

   dip::Image dxx = dip::Gauss(src, {1.0}, {2, 0});
   dip::viewer::Show(dxx, "dxx");

   dip::Image dyy = dip::Gauss(src, {1.0}, {0, 2});
   dip::viewer::Show(dyy, "dyy");

   dip::viewer::Spin();
}

Thank you!

[VisualFox]

Thank you so much for the quick reply and clear instructions on how to approach debugging. Sorry took me a while to reply as I wanted to get everything right and not waste any more of your time.

I'm going to close this bug report as I'm now convince that is not a issue coming from DIPlib nor from its interface with OpenCV but with some odd incompatibility (that I don't understand) between DIPlib and OpenCV only when OpenMP is used by both libraries.

For a quick workaround if someone else is experiencing the same issue:

• don't compile OpenCV with OpenMP if you intent to use it with DIPLib. OpenCV by default don't use OpenMP, so that should not be an issue, but instead rely on some native solution per OS (@see https://docs.opencv.org/4.x/db/d05/tutorial_config_reference.html under "Parallel processing")
• if you really need OpenCV compiled with OpenMP, disable OpenMP on DipLib (dip::SetNumberOfThreads(1) or DIP_ENABLE_MULTITHREADING=OFF in cmake)

@crisluengo I will followed up with more details but I really don't think you should spend more time on it - only if you are curious. Thank you again!

[VisualFox]

@crisluengo

But you said that turning OpenMP off solved the issue, which makes me assume the code you posted is not your full code, and the contrast is not the problem.

I did omit the bit about the image conversion/format when I submitted the example, sorry about that. I do convert the cv::Mat to floating-point and I scale it to [0,1] prior to pass it to DIPLib, but to rule this one out I replaced highgui (cv::imshow) by dip::viewer and I saw the same artifacts.

I could not reproduce your issue on macOS, using Clang. I tried reproducing using GCC, but it won't link the Homebrew OpenCV (which is built with Clang).

OpenCV by default is not compiled with OpenMP. On mac it use Grand Central Dispatch (and other native solution on other OS) so that explain why you weren't able to reproduce the issue.

It is possible that there's some issue with both OpenCV and DIPlib using OpenMP, but so far I've only seen issues when mixing libraries that use the Clang OpenMP with libraries that use the Intel OpenMP. And the result is usually a crash...

I think you were spot on with the issue with OpenMP. I compile both libraries so they use the same OpenMP implementation. I don't experience any crash just these strange missing part on some channels. Also I was at first under the impression that the issue was always reproducible (set at compile time) but in fact it's a runtime issue (the same image will not always trigger the issue) which is why it took me so long to test everything as I have to run the tests multiple times. What I noticed is that by just linked both libraries together without even using the bridging between DipLib and OpenCV (just by declaring a cv::Mat somewhere in my code) it will trigger the issue at runtime (but with far less frequency).

Do you see the same problem if you build with OpenMP enabled, but call dip::SetNumberOfThreads(1) at the start of your program?

Disabling multi-threading dip::SetNumberOfThreads(1) solve the issue even with both libraries compiled with OpenMP. Oddly enough, reducing the number of threads on DIPLib end dip::SetNumberOfThreads(dip::GetNumberOfThreads()-4) seem to solve the issue as well.

Do you see this result using your example program as-is? Or are you calling DIPlib functions from within a multi-threaded environment?

As-is not in multi-threaded environment.

Does your version of OpenCV use pthreads, TBB or OpenMP for multi-threading?

OpenMP - I wasn't able to test it with TBB but using pthreads (which is the default on linux) resolved the issue.

Do you see the same issues without OpenCV?

No, a pure DIPLib program (like the one you provided) worked flawless. Oddly I can also use OpenMP in my own program without an issue with DIPLib as long as OpenCV is not using OpenMP.

Thank you again for your help and this great library!

[crisluengo]

Thank you so much for reporting back with so much useful information!

If you’re compiling OpenCV yourself, I will assume you’re using the same compiler (and therefore OpenMP version) as you’re using for DIPlib. There really should not be any incompatibilities there, there is no reason not to use multiple libraries with OpenMP together.

Your output matches what you’d get if some subset of threads did not do their job. Plus the fact that it happens sometimes, and doesn’t happen if you use fewer threads than the default. All of this is highly puzzling and confusing, I have never seen such a thing happen before.

There must be something OpenCV is doing to OpenMP (some configuration setting maybe?) that is causing this. I will look for some time to peruse the OpenCV source code and see if I can learn how they use OpenMP.

[crisluengo]

I did find a bunch of posts where OpenMP code becomes single-threaded when OpenCV is linked in. That seems related to this issue: opencv/opencv#25717

But that causes fewer threads to be used, rather than some threads not doing their work.

[VisualFox]

Thank you so much for reporting back with so much useful information!

You are very welcome! I wish I was able to provide you more details and some clues about this one but I'm not too familiar with OpenMP. I only use it as it's convenient for parallelizing loops but it's a bit like magic at this point.

If you’re compiling OpenCV yourself, I will assume you’re using the same compiler (and therefore OpenMP version) as you’re using for DIPlib.

Yes, correct I do use the same exact toolchain and OpenMP version for both DipLib and OpenCV, also same machine and same OS.

Your output matches what you’d get if some subset of threads did not do their job.

That was my impression as well, but that as far as my guess went. I did notice something related to that. Let me run some more test but I think I get to dip::SetNumberOfThreads(dip::GetNumberOfThreads()-4)) because if I was only removing 1 or 2 thread the artifacts were still present but they were "shrinking" (a screenshot will explain that better).

Plus the fact that it happens sometimes, and doesn’t happen if you use fewer threads than the default. All of this is highly puzzling and confusing, I have never seen such a thing happen before.

I totally agree! I guess I can try to run the debugger again but I'm not sure what I should be looking for, it's not like I get a segmentation fault or some other clear error. Maybe some nested OpenMP call? But I'm not sure how that will show up in the debugger so I may not recognize it...

There must be something OpenCV is doing to OpenMP (some configuration setting maybe?) that is causing this. I will look for some time to peruse the OpenCV source code and see if I can learn how they use OpenMP.

Thank you but really I think this one is quite obscure and such an edge case... as OpenCV with OpenMP is not the default and the odd of using this particular configuration with DIPLib is really small (well it happen to me but it's mostly because I was experimenting with OpenCV and I forgot to recompile it) and for my own use case I plan to eventually fully switch to DIPLib which fit my need better :)

I did find a bunch of posts where OpenMP code becomes single-threaded when OpenCV is linked in. That seems related to this issue: opencv/opencv#25717

Running the app a few minutes later made a couple more threads used.
The more I waited, the more performant the app got.

No sure if that related but the reason why I didn't catch that the issue was happening during runtime it's because if I run the application once I get the artifacts immediately but if I re-run the same binary a bit later (to check something with dip::viewer for example) it may go away, at this point generally I was getting frustrated and recompiled the application...
After I did run the test multiple time but generally it was just after another. I'm going to run it once, and wait a minute and run it again and see if the artifacts went away...

[VisualFox]

I can also try this:

Regarding OpenCV affecting OpenMP, I've found only one thing to try - set OPENCV_FOR_OPENMP_DYNAMIC_DISABLE environment variable to 1.

[VisualFox]

...and it works! set OPENCV_FOR_OPENMP_DYNAMIC_DISABLE environment variable to 1 resolve the issue.

@crisluengo I have no idea how you were able find a link between this issue and the OpenCV bug's report, I read the report and nothing will have gave me a hint, so kudo to you! Regardless it was the perfect storm as I was doing some heavy load (compiling OpenCV, DIPLib) before running my application... and the more frustrated I was the more I recompiled...

No idea why it have this effect on DIPLib

[crisluengo]

I didn't think that the OpenCV bug report was the same issue, but it was related in that OpenCV changed the number of threads used by the program. I like to link to any issues that might be related because they might provide information. I am surprised, but also very happy, that this is actually the same issue. So thank you so much for figuring this out! Because it's not just the rare occurrence of an OpenCV compiled with OpenMP, it's any program that enables this dynamic thread count mode.

I just learned something about OpenMP. I thought that the number of threads requested, as long as you don't request more than omp_get_max_threads(), is the number of threads you actually get. Turns out that this is not the case!
When dyn-var = true (i.e. when someone called omp_set_dynamic(true)), you can get any number of threads between 1 and the number requested (see OpenMP documentation):

if (dyn-var = true) and (ThreadsRequested ≤ThreadsAvailable)
then 1 ≤ number of threads ≤ThreadsRequested;

In DIPlib, code first determines how many threads to make, then divides the work (or image) up into that many blocks, and then starts the parallel section. If the number of threads created is less than the number that the code wanted to get, some of that work doesn't get done. This is a defect in the way I implemented the parallel execution. Usually people use a #pragma omp parallel for, where all the work gets done, no matter how many threads were actually created. I wanted a different model because things can get a bit more efficient that way.

The easy fix is to throw an exception if the number of threads doesn't match the expected number. The error message can tell people not to use dynamic thread counts.

The more involved fix is to first create threads, then figure out how to divide up the work among the current number of threads, in the master thread, while the other threads idle. That idling shouldn't be an issue, it's not a lot of work to do that division.

I'm re-opening until I find the time to fix this.

[crisluengo]

for my own use case I plan to eventually fully switch to DIPLib which fit my need better :)

Great to hear!