RAW: Add File Extension Blacklists for Darktable and RawTherapee

[citadella]

I was receiving the following errors in my logs when scanning these Fuji RAW files:

time="2021-06-07T00:34:44+02:00" level=error msg="location: record not found (find cell s2:14cab798d68c)"
time="2021-06-07T00:34:45+02:00" level=error msg="location: Error 1366: Incorrect string value: '\xC5\x9Fikta...' for column photoprism.places.place_city at row 1 (create place s2:14cab7b3667c)"
time="2021-06-07T00:34:45+02:00" level=error msg="location: Error 1366: Incorrect string value: '\xC4\x9F\xC4\xB1' for column photoprism.cells.cell_name at row 1 (create cell s2:14cab7b3667c)"

time="2021-06-07T01:54:24+02:00" level=error msg="thumbs: thumbnail not cached"
time="2021-06-07T01:54:24+02:00" level=error msg="thumbs: thumbnail not cached"

After chasing this down a bit it seemed it may have been a permissions issue, but Michael suggested turning off Darktable and it seems RawTherapee does indeed recognize these files:

2021-06-07 12:00:57 DEBU camera: cache hit for fujifilm-x100s
2021-06-07 12:00:57 INFO media: 11 thumbnails created for FujiRAWlg-02073 [1.361047004s]
2021-06-07 12:00:56 DEBU metadata: a05be2c98993488a46e5fc1e045

So it seems to be an issue with Darktable. Darktable was converting the files into .jpgs and I could download the .jpgs from within PP, but they would not display as thumbnails anywhere.

[citadella]

Have confirmed it's working within RawTherapee, so it appears to be an issue exclusively with Darktable:

2021-06-07 12:51:51 DEBU index: updated yaml file FujiRAWlg-00877.yml
2021-06-07 12:51:51 INFO converting Unsorted Recovered Pics/Fuji RAW LARGE/FujiRAWlg-00880.RAF to jpg
2021-06-07 12:51:51 DEBU lens: cache hit for zz
2021-06-07 12:51:51 DEBU camera: cache hit for fujifilm-x100s
2021-06-07 12:51:51 DEBU index: image classification took 148.85892ms
2021-06-07 12:51:51 DEBU index: FujiRAWlg-00878.RAF.jpg was modified (new size 3921779, old size 6234504, new timestamp 1623063110, old timestamp 1622999034)
2021-06-07 12:51:51 INFO index: updated main raw file “Unsorted Recovered Pics/Fuji RAW LARGE/FujiRAWlg-00878.RAF”
2021-06-07 12:51:51 DEBU photo: using label Fujirawlg to create title for pquai2j2f0xirhgk
2021-06-07 12:51:51 INFO index: updated related jpg file FujiRAWlg-00876.RAF.jpg
2021-06-07 12:51:51 DEBU media: no new thumbnails created for FujiRAWlg-00876 [52.859µs]
2021-06-07 12:51:51 DEBU index: updated yaml file FujiRAWlg-00876.yml

[mtoupsUNO]

I have mixed feelings about this. I have tried both darktable and rawtherapee with my Fujifilm RAF files and they both have problems.

Darktable only recently added support for my camera, so previously I needed to disable it. Rawtherapee had support first, but it is buggy (see Beep6581/RawTherapee#6180) so it is generating bad jpgs.

I would prefer to have the ability to choose between them, using the "Disable Darktable" and "Disable Rawtherapee" checkboxes in Advanced Settings. It looks like 07c711a would always disable darktable for RAF even if I decided to disable Rawtherapee also.

It may be sensible to default to Rawtherapee instead of Darktable, but please allow the user to override this default using Advanced Settings. Thanks!

[lastzero]

Does it work with the version of Darktable that comes with our latest Docker image?

[mtoupsUNO]

No it does not, when using docker image photoprism:preview which includes darktable 3.4.1.

My Fujifilm camera is supported in darktable starting with version 3.6.0 (released about 1 month ago). I verified support for these RAF files with the standalone darktable package.

I thought I could test out the new darktable pretty easily by upgrading it within the photoprism docker image. But as it stands, this line

photoprism/internal/photoprism/convert.go

Line 189 in 5076af2

} else if c.conf.DarktableEnabled() && fileExt != fs.CanonCr3Ext && fileExt != fs.FujiRawExt {

means that I can't get photoprism to use darktable on my RAF files, even if I explicitly disable RawTherapee and leave Darktable enabled.

If I could override this with a config option (without recompiling photoprism, building a new docker image, etc), that would make it much easier for me to test. For now, I was able to work around this by going back to photoprism:latest which predates 07c711a

Using the older photoprism, I updated darktable to 3.6.0 and then verified that photoprism does work correctly with the new darktable on my Fujifilm RAF files. This confirms that darktable is definitely preferable to RawTherapee in my case. This is why I ask for you to let me choose which converter to use (which was true prior to 07c711a). Thanks!

[lastzero]

It should work out of the box without additional configuration, so just removing this or asking the user to configure it first won't do. We'll find a solution for our stable release. Note you can change the CLI command names (not the parameters though) to implement more complex logic eg in a wrapper script.

[lastzero]

We've added comma-separated file extension blacklists for both converters so that you can configure what to use for which RAW format:

--darktable-blacklist BLACKLIST          

      Comma-separated file extension BLACKLIST (default: "raf,cr3") [$PHOTOPRISM_DARKTABLE_BLACKLIST]

--rawtherapee-blacklist BLACKLIST        

      Comma-separated file extension BLACKLIST [$PHOTOPRISM_RAWTHERAPEE_BLACKLIST]

Does this work for you?

[citadella]

Aye, should do!