styles: enable built-in styles, add sampling of camera-JPEG styles

[ralfbrown]

This PR adds startup code to load styles from share/darktable/styles/ which are not yet present in the database, and also provides a sampling of styles to approximately match the look of camera JPEGs for a few models from each of the major manufacturers. Further, it updates the styles quick-access button in the darkroom to produce a full menu hierarchy on popup rather than collapsing everything below the top level into single huge submenus.

The provided styles are all under a common darktable camera styles top level to avoid name conflicts. This name is localizable.

This setup will allow us to accept user-contributed camera-match styles and ship them with future version of darktable.

[wpferguson]

Tested.

It works.

Observations:

• If you don't want them, you have to delete them one by one
• If you don't want them and delete them they show up again the next time you start darktable because they are "system" files and you have to be a superuser to delete them. Maybe we should just copy them to the users darktable config folder and import from there? That way the user could get rid of them if they wanted to.
• Maybe change the top level name to "Camera Styles"?
• I tried a couple of the styles on Play Raw images and they give a result that is pretty close to the JPEG preview.

[ralfbrown]

I didn't use a name like "camera styles" because we'd want to internationalize that, which isn't possible atm. That would of course be better.

You can remove multiple styles at once by selecting them all before clicking on "remove", though you can't remove an entire virtual folder of styles by selecting just the top-level name.

Copying to the user config folder before importing would have the same issue of re-appearing, though only when clearing the folder or using a new one with --configdir. Is it really so bad to have one top-level entry in the styles list? After all, we have a lot of built-in presets which can't be removed either. Maybe a preference to hide that hierarachy? I'll have to look at the code to see if there's a reasonable way to determine whether the database is being newly created and only import then, because the only other way I can think of making them permanently deletable is not to automatically import them, in which case a majority of users will never discover them....

The results should look pretty close, as I tuned them up on the raw.pixls.us sample images. The biggest difference I'm seeing in the settings is the exposure compensation to match midtones to where the camera's tone curve puts them, which ranges from +0.3 EV to +1.6 EV, with most being +0.8 to +1.1. The styles are built with exposure, color balance rgb (saturation/vibrance), filmic rgb (global contrast), and local contrast modules.

[wpferguson]

though you can't remove an entire virtual folder of styles by selecting just the top-level name

I can vouch for that :-)

Copying to the user config folder before importing would have the same issue of re-appearing

If they are in the config folder the user owns them and should be able to delete them

Is it really so bad to have one top-level entry in the styles list?

If you are never going to use it? I could live with it (and if I got really annoyed I could fix it :-D). Others?

The results should look pretty close

Colors looked pretty much spot on in my small sampling. I did notice the exposure changes.

It should make the look like jpg crowd happy

EDIT:

I could make a lua script to go with it that runs on import and auto applies the "correct" style to the image.

[ralfbrown]

I could make a lua script to go with it that runs on import and auto applies the "correct" style to the image.

That would be a great addition, and would really make the look-like-jpeg crowd happy.

BTW, it's occurred to me that only importing the styles to an empty config runs into the problem that users upgrading from a previous version will never get the styles unless they manually import them from the system folder.... I think if we want them auto-installed but user-removable, we need to flag them as user-removed in the database to keep them from being re-imported.

[wpferguson]

Here's a script that you can play with. Works on import and has shortcuts for apply to collection/selection

apply_camera_style.zip

[wpferguson]

Updated to handle ICLE-9 series

apply_camera_style.zip

[ralfbrown]

I took a look at the code (currently don't have Lua setup in dt).

When I was generating and naming the styles, I wasn't thinking in terms of automatic name matching. It makes sense for us to discuss this so that the script won't need updating any time new styles are added... The main issue is of course the X-series styles; it wouldn't be difficult to just duplicate them for each known model, though that would mean potentially more maintenance in the future. I was also thinking of adding a "generic" style for each manufacturer (like I have at the top level) once I have a better sense of the typical settings for the maker. That would be a useful fallback when available.

Other than the Ricoh/Pentax normalization, the maker normalization could be handled by storing the lowercase version of the maker as the key in the lookup table (the original case is already present in the stored value at the bottom level).

Finally, shouldn't the required API version be the current 9.x rather than 7.0.0?

[wpferguson]

I'm letting the style naming/camera mapping thing rattle around in my head to see if I come up with a better way to handle it

Here's a dump of model and maker info from my play raw collection (nice diverse set of images for testing).

modelmaker.zip

Implementing a generic for each maker should be easy enough, just an else to the style check.

Finally, shouldn't the required API version be the current 9.x rather than 7.0.0?

Maybe. Right now it would run on anything greater than or equal to 7.0.0 (dt 3.6.0). I didn't look at the styles to see if they are using any modules that require a specific version of darktable. Here's the API <->dt version list

// 1.6 was 2.0.1
// 1.6.1 was 2.0.2
// 2.0.0 was 3.0.0
// 2.2.0 was 4.0.0 ( removed the ugly yield functions make scripts incompatible)
// 2.4.0 was 5.0.0 (going to lua 5.3 is a major API bump)
// 3.2.0 was 6.0.0 (removed facebook, flickr, and picasa from types.dt_imageio_storage_module_t)
// 3.6.0 was 7.0.0 (added naming to events, selections, and actions)
// 3.8.0 was 8.0.0 (moved to lua 5.4 and added some events)
// 4.2.0 was 9.0.0 (view toolbox functions and snapshot filename removed)
// 4.4.0 was 9.1.0 (added mimic and dt_lua_image_t changes)
// 4.6.0 was 9.2.0 (added change_timestamp to dt_image_t)
// 4.8.0 was 9.3.0 (added button and box widget enhancements)

Let me know where to set it.

EDIT:

If we're not using any really new modules, then users from earlier versions could just install the styles and still benefit from the scripts.

[wpferguson]

Idea: Use the model name as a search specification, i.e.:

EOS R series becomes EOS R? series
D7000 series becomes D7?00 series

brackets around multiple choice

EOS [7D,7DmkII] or EOS 5D Mark [I,II,III,IV]

Specifics become before generics, i.e.:

ILCE-9 Mark 3
ILCE-9? series

I should be able to read that and build the lookup table from it. That way whenever the styles change I should just be able to pick it up without a code change.

EDIT:

Keep the naming to what image.exif_model returns or query the models table in the database. I'll use the models table to figure out who likes to put their name on the model so I strip it out.

[wpferguson]

Here's a sorted model list from my play raws

sorted_model_list.txt

[ralfbrown]

Idea: Use the model name as a search specification, i.e.:

Done. I verified that brackets and question marks in the name don't cause problems (they get replaced by underscores when generating the export filename). The bracket notation might cause a bit of confusion for less technical users, but it shouldn't be a big deal.

Specifics become before generics, i.e.:

Standard procedure :)

Keep the naming to what image.exif_model returns or query the models table in the database. I'll use the models table to figure out who likes to put their name on the model so I strip it out.

Should be OK for the majors (Nikon never had a D60 despite using D40/D50/D70/D80/D90, presumably because Canon had a D60 first). Might be one or two collisions among the smaller manufacturers.

[ralfbrown]

Maybe. Right now it would run on anything greater than or equal to 7.0.0 (dt 3.6.0). I didn't look at the styles to see if they are using any modules that require a specific version of darktable. Here's the API <->dt version list

If we're not using any really new modules, then users from earlier versions could just install the styles and still benefit from the scripts.

It's not an issue of how new the module is, but of the module's parameter block version. So far, I've only used local contrast, filmic rgb, color balance rgb, and exposure. Filmic is using v7, which is definitely newer than dt 3.6 - 4.2 or 4.4? But even if I were using e.g. v5, dt would be writing v7-compatible parameter blocks.

[wpferguson]

filmic V7 was dt 4.4. Set API to 9.1.0