OpenCL / Local Contrast / Local Laplacian - issue with AMD/ROCM: 'amplified effect'

[arigit]

This is an issue reported in redmine 1 year ago, didn't get any traction, bringing it here since bug reporting seems to have moved and it may be found by other people affected.

This issue affects Darktable when used with AMD-based GPUs, OpenCL and the ROCM opensource driver which is the currently supported OpenCL driver for all new AMD discrete and integrated GPUs. It's been proven to affect both Polaris and Vega GPUs.

This issue does not happen with the legacy closed-source amdgpu-pro OpenCL driver (that many still use).

Original issue: https://redmine.darktable.org/issues/12423

Describe the bug

The result of applying Local Contrast > Local Laplacian is totally different with OpenCL "on" and OpenCL "off", for the exact same settings of the module. The more % of Detail is set, the more striking is the difference.

This issue does NOT happen for Local Contrast > Bilateral Grid, it is specific to Local Laplacian. When I select Bilateral Grid, the result is the same regardless of whether OpenCL is enabled or not.

It looks like the Local Contrast > Local Laplacian OpenCL implementation has a problem. See the attached JPG snapshots, taken with the exact same settings of the model, one with OpenCL "on" and one with OpenCL "off". Basically LocalContrast+LocalLaplacian is unusable (and destroys the image) when OpenCL is on.

It looks like with OpenCL enabled, the "Detail" effect of the Local Contrast Filter is like grossly amplified.

Example:
Local Contrast + Local Laplacian, OpenCL = "off"

Local Contrast + Local Laplacian, OpenCL = "on"

No other other OpenCL kernel from darktable has problems with the official ROCM driver from AMD in my setup (I have been using ROCM OpenCL with darktable for 1.5 years now, meaning thousands of pictures developed, using profiled denoise, filmic, retouch, basecurve, etc etc).

My workaround in order to keep OpenCL "on" has been to remove the LocalLaplacian kernel.

sudo mv /usr/share/darktable/kernels/locallaplacian.cl /usr/share/darktable/kernels/locallaplacian.cl.temporarilyRemovedDueToROCM

The issue happens with both current stable and github master (Darktable 3.0RC2), the issue was first found on DT 2.4 / Ubuntu 18.10, stock Kernel 4.18, and rocm-opencl 1.6 from AMD. It has been reproducible 100% of the time since then, with DT 2.6, and now DT3.0, and dozens of ROCM releases (all the way to the current 2.10, kernel 5.3)

There are no errors at all shown in the logs by darktable when compiling the kernel and when using it:

0.280302 [opencl_init] compiling program `locallaplacian.cl' ..
0.280608 [opencl_load_program] loaded cached binary program from file `/home/ariel/.cache/darktable/cached_kernels_for_gfx803/locallaplacian.cl.bin'
0.280611 [opencl_load_program] successfully loaded program from `/usr/share/darktable/kernels/locallaplacian.cl'
0.281739 [opencl_build_program] successfully built program
0.281746 [opencl_build_program] BUILD STATUS: 0

While it is possible that the bug is in the ROCM driver (since the same OCL kernel works in nvidia and in the old amdgpu-pro driver), in order to have a rocm developer engaged we need some more details on what is wrong with the driver; we have this ROCM bug opened with them:

ROCm/ROCm#704

however no detail = no action.

It is also possible that some peculiarity of the locallaplacian DT OCL kernel code is triggering the problem - hopefully the problem could alternatively be worked around via refactoring Darktable's Local Laplacian OpenCL code

Platform (please complete the following information):

• OS: Ubuntu 19.10
• OpenCL activated
• GPU: AMD RX-560; Driver: rocm-opencl 2.10
  Same issue reported with AMD Vega 56

[blitzgneisserin]

I think this is similar to #3460 and #3418
Note that I have an Nvidia GPU, sometimes it is also an issue without OpenCL

[arigit]

This issue though is 100% reproducible and specific to the ROCM open source OpenCL driver from AMD. It doesn't happen when using the closed-source OpenCL driver that's still around (although not for much longer since AMD is shifted all development effort to open source) and it doesn't happen when using bilateral grid for local contrast. There is something in the locallaplacian kernel from DT that triggers this problem. All other openCL kernels are fine.

[blitzgneisserin]

@arigit Did you read my bug reports carefully? Did you check what happens when you hide the side panels and the thumbnails with the tab key? I think it is the same thing and 100% reproducible. Maybe it does not happen with the closed source AMD driver but with Intel and Nvidia it does. Anyway it is strange that it does not happen when you hide the side panels and above all the thumbnails. I think the original code was written by hanatos. I don't know why he is so quiet. He could at least say that he has the intention to fix this eventually.

[arigit]

@blitzgneisserin I did. Here's a sample I just took:

Open CL off, Local Contrast off

OpenCL off, Local Contrast (local laplacian) on, panels on:

OpenCL off, Local Contrast on, panels off (turned off by hitting "tab")

OpenCL on, Local Contrast on, panels on

OpenCL on, Local Contrast on, panels off

In my case the panel state does not seem to make a difference. Turning OpenCL on makes LocalLaplacian destroy the image.

[blitzgneisserin]

Ok. Although the result looks a bit different I suspect that the reason for this behavior is the same thing. @arigit Did you try to "replace" local contrast by the equalizer (preset clarity)? How does it behave?

[aurelienpierre]

Could you try my patch by compiling my https://github.com/aurelienpierre/darktable/tree/fix-locallaplacian-padding branch ?

[aurelienpierre]

Also

I think the original code was written by hanatos. I don't know why he is so quiet. He could at least say that he has the intention to fix this eventually.

Hanatos has a full time job and 2 kids under 5 years.

[arigit]

@aurelienpierre thanks a bunch for jumping in!

I'm trying to build your branch on ubuntu 19.10.
The build errors-out in the cmake checking part.

I installed all dependencies like so:

sudo apt build-dep darktable

and also: llvm-dev

The error:

-- Found intltool-merge
-- Found desktop-file-validate
CMake Error at /usr/lib/llvm-9/lib/cmake/llvm/LLVMExports.cmake:1323 (message):
  The imported target "yaml-bench" references the file

     "/usr/lib/llvm-9/bin/yaml-bench"

  but this file does not exist.  Possible reasons include:

  * The file was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and contained

     "/usr/lib/llvm-9/lib/cmake/llvm/LLVMExports.cmake"

  but not all the files it references.

Call Stack (most recent call first):
  /usr/lib/llvm-9/cmake/LLVMConfig.cmake:245 (include)
  CMakeLists.txt:273 (find_package)
  CMakeLists.txt:281 (find_llvm)


-- Configuring incomplete, errors occurred!

I've been googling to find a solution to this but no luck. Any hint on what could be wrong in my build environment?

[blitzgneisserin]

Sorry for my earlier comment.

[arigit]

@aurelienpierre given that @blitzgneisserin issue got fixed, if your patch could get merged into DT master, I will be able to find a binary package in OBS (the opensuse build service) since it will automatically be built in a few hours time - and then test it

[blitzgneisserin]

@arigit no, I am sorry, it is not fixed. I just forgot to switch on OpenCL. I just realized on the 3rd check.

[blitzgneisserin]

@arigit I compiled it according to the instructions on darktable.org. That works.

[aurelienpierre]

So the results are still off ?