CLI: Create thumbs and convert files in deterministic order

[Rolf-Smit]

1. Problem:

When running photoprism thumbs I can see in the debug logs that thumbnails are being created for files in folders that are otherwise ignored by the indexer (due to the .ppignore files).

log lines like these appear:

media: created 11 thumbnails for photos/should-be-ignored/some-photo-2.jpg [3.649349277s]

I can also see files being created in the cache folder for the images that should be ignored.

2. Reproduce:

Setup an ignored folder using a .ppignore file (put it inside of the folder you want to ignore), then run photoprism thumbs.

Example contents:

# Ignore all files
*

3. Expected behavior:

Thumbnails should not be created for files that are supposed to be ignored by the .ppignore files.

4. Environment:

4a. Software

• PhotoPrism Build Number: 230211-fd635c0ce (preview)
• PhotoPrism Architecture: x86_64 (AMD64)
• Database Type & Version: MariaDB 10.7
• Operating System Types & Versions: Linux (QNAP 5.10.60-qnap)

4b. Hardware

• Device / Processor Type: Intel® Celeron® J4125
• Physical Memory & Swap Space in GB: 16GB
• Storage Type: HDD (RAID) with SSD read cache

[lastzero]

That's expected (and documented to my knowledge) as you can't ignore JPEGs (primary file format) nor all files (required to use the application).

[Rolf-Smit]

you can't ignore JPEGs (primary file format)

You're saying the .ppignore can't ignore JPEGs, or the thumbs command can't? I assume the latter, but why is that? Directories/files ignored by the .ppignore file are not indexed, so what is the use of thumbnails for those?

In my situation in have about 60.000 files that I want PhotoPrism to ignore (contained within several subdirectories), the .ppignore files work great for that. However I have to regenerate the thumbs due to some corruption issues (missing thumbnails), and now I end up with an additional 660.000 or so useless thumbnails.

If this is working as intended then maybe it would be nice if we could run the thumbs command only on a certain sub directory within the originals folder (or sidecar folder for that matter?).

[lastzero]

• It is correct that specifying a path to the thumbs command is not currently supported. However, this could be changed.
• You should alternatively be able to add the folder to an ignore file, not in the directory itself as *, but with its name one level higher.
• Ignoring folders by adding an ignore file containing only * would be a new feature, which we may be able to include in a future release. However, this is the first time someone has asked for it.

[lastzero]

When I checked the problem in our code, I found that the thumbs and convert commands processed files in a non-deterministic order. This was intentional, but could cause the .ppignore file to load after the first file had been processed, and was therefore unreliable.

The last commit fixes this behavior by processing the files deterministically (and a bit slower). In addition, you can now specify a subfolder when using the thumbs command.

[lastzero]

I've started a new preview build for testing! It should be available in about one hour.

[Rolf-Smit]

Just tested, .ppignore files now work as expected. I also tested giving the thumbs command a specific folder, that also works fine!

I assume * (ignore all files in folder) is a .ppignore feature that is working as intended as well?

Anyhow: BUG resolved!

Thanks @lastzero

[lastzero]

Indeed, the wildcard works as well. Just not for related files, e.g. if you exclude JPEGs but not RAW files. In this case PhotoPrism still uses JPEG sidecar files.