New lens correction not working when auto-applied

[lintujuh]

Describe the bug/issue
The latest development build doesn't apply the lens correction parameters correctly on Windows.

To Reproduce

1. Open image where the distortion is visible
2. Activate lens correction
3. See error
   
   

Expected behavior
With 4.0.1

Which commit introduced the error
Didn't check, but assume 1e819b6

Platform

• darktable version :4.1.0+744~g873413e3c
• OS : Windows 10
• Memory : 32GB
• Graphics card : NVIDIA T500/4GB
• Graphics driver : NVIDIA 517.40
• OpenCL installed : yes
• OpenCL activated : no impact
• CMAKE_BUILD_TYPE : release

Additional context

• Can you reproduce with another darktable version(s)? works correctly on the Linux version (Fedora)
• Camera: Olympus E-M1MarkII
• Lens: M.Zuiko ED 12-40mm f/2.8 Pro

[sgotti]

@lintujuh

3. See error

Do you mean the image not corrected or is there also an error reported that I can't find in your report?

I don't have windows to test and I'm not sure what can be the difference between windows and Linux in that change but I'll take a deeper look at the changes in the PR.

Have you tried with a nightly build made before PR #12714 ?

[lintujuh]

I mean the image is not corrected. The latest nightly I have had before is darktable 4.1.0+681~g19825f35c, and that works correctly.

One strange thing I noticed is, that when I added only one image into the library (I have my database in :memory:), dt opened the image in darkroom and it was corrected. When I added multiple images, and selected one from the list, then the correction was not applied. The correction was still properly applied to the preview window. Switching the correction on and off, there is a slight change in the corner.

off/on (better visible on the screen when you toggle the module on/off)

Edit: added the log file (-d all).
darktable-log.txt

[paolodepetrillo]

This is not related to the bug, but would you be able to provide this raw file and the camera jpeg if available? I am trying to figure out the distortion correction tags embedded in the Olympus raw files and this image would be useful for testing because of the strong distortion and straight lines. Thanks!

[sgotti]

@lintujuh I started a win vm but I don't have your issues with the latest nightly build.

From your first snapshot looks like the corrections are applied for the module (see the message corrections done: all) but you don't see them.

off/on (better visible on the screen when you toggle the module on/off)

This looks really strange and looks like a caching/opencl artifact...

Does it have the same behavior if you try disabling opencl?

[lintujuh]

One thing I forgot to mention is that I have the lens correction module enabled as a preset. I didn't think that yesterday, and I did a quick test, and managed to reproduce the behaviour also on Linux.

I created a duplicate of the image, and deleted the history stack. I have on my Linux box the same presets as on Windows, including enabling lens correction automatically. When I opened the duplicate first time I got the same results as on Windows. It looks like the lens correction is activated, and the camera+lens identified correctly, but no visual correction. Also the minor changes in the corner are visible when you toggle the lens correction module.

[TurboGit]

@sgotti : Given the last comment it seems to me that a possible cause is a bug in the legacy_params() routine. To check that you can:

• create a lens preset with 4.0.1
• start dt master (preset will be migrated)
• check if the preset still works

[sgotti]

@TurboGit I tried it, created an auto apply preset with dt 4.0.1 (with customized parameters), started darktable master, upgraded the db. Imported some images (without any sidecar xmp), the preset was correctly applied and visible. Darktable logged:

[imageop_init_presets] updating 'lens' preset 'test preset' from version 5 to version 6
to:'gz04eJxjZGBgYGeAAG/jJnsGhgYg9nA2Nt7ssOLKO0dGoLhfZnZ+noKLSTHDwACI/Y5uusEKQGZ2fpGCuYGukYFBbq5Cmr6RnoW7gquLQliQgqengqGZEa1c0WCPihkYAD+HGOE='

@lintujuh

• Can you provide some more detailed steps to reproduce it? Can you share the preset and also an image to test it? I imagine that the preset should be already updated to version 6 (you said you used library in memory db, but presets are in the data.db in your config dir and if you haven't changed the configdir the presets in data.db should be updated).
• If you reset the module or manually apply the preset does the correction is correctly shown?

[TurboGit]

Darktable logged:

All good on this side then!

[lintujuh]

@sgotti here is the example raw and my data.db file. If I create a duplicate, discard the history stack, and open the image in darkroom toggling the module on/off, it has only very minor impact to the corners (when it has been first auto-applied). raw and data_db.zip

@paolodepetrillo here are the raw and corresponding SOOC JPG. PB040842.zip

[lintujuh]

I forgot the detailed steps, but there aren't many. When I have the module activated in the presets, it's enough to open the image in darkroom, and see they the correction is not applied.

[jenshannoschwalm]

I can confirm the issue on fedora linux too, using today's master and opencl on or off. I also have an auto-applied lens specific preset already active.
lens_Zuiko 12-40mm.zip

Also saying "corrections done: all" but not done. Not in darkroom, not in preview and not while exported.

I can "fix" the issue immediately by changing one of the lens parameters (scale, TCA overwrite). Discard history brings back the issue.

[sgotti]

@lintujuh @jenshannoschwalm Thanks for the information, I'll try to reproduce/debug/fix after the (busy) weekend (if someone else isn't faster than me).

If I understand your info correctly this issue happens when an auto apply preset is applied on a new image (or on history reset).
If you change some params or reset the lens module it works correctly.

Probably the title should be updated.

[TurboGit]

The good news is that I may have a fix for this in #12783.

The bad news is that previous version has broken the preset for sure. So to test this you need to restore a database (data.db) as it was before you first use the new lens module.

[TurboGit]

@sgotti : Please double check that I did not mess something in my PR. TIA.

[lintujuh]

Is it enough to delete and recreate the entry? I don't think I have a backup available.

[TurboGit]

@lintujuh : Yes, if you recreate all will be ok, but sadly we won't know if the fix I did is sufficient or not.

[gi-man]

Pascal, what do you mean by restoring the data.db? Why not create a new auto preset for the module?

[sgotti]

@TurboGit @jenshannoschwalm After a lot of debugging I discovered that there was an error passing arguments to the specific method commit params. The style had params modified = 0 and camera = PENF, these are reset since modified is 0 on reload defaults for the real camera/lens of the current image (E-M1 MarkII). But due to the issue the old params were passed.

PR #12796

The good thing is that probably the legacy params not initializing new fields is not the problem (but wrong anyway) causing this issue and the styles doesn't need to be recreated.

[TurboGit]

Pascal, what do you mean by restoring the data.db? Why not create a new auto preset for the module?

This is to reproduce the issue. You can create a new preset but this won't help testing that the new code put in place is fully correct.

[lintujuh]

I recreated my data.db with dt 4.0.1 and copied that to my 4.1 folder. Everything seems to work okay.

[TurboGit]

Thanks a lot for this test. We can definitely say that this issue is fixed.