Storage migration complains about non-existent sidecars

[mmomjian]

The bug

Non-existent sidecar files are referenced in the database.

The OS that Immich Server is running on

Docker on Debian 12

Version of Immich Server

v1.93.3

Version of Immich Mobile App

n/a

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

n/a

Your .env content

n/a

Reproduction steps

1. Import photos from Google Photos using a storage template
2. Change the template, and run a migration job in the server jobs
3. view logs of immich-microservices

Additional information

The immich-microservices logfiles will contain lines like:

[Nest] 8  - 01/28/2024, 6:37:33 PM     LOG [StorageCore] Attempting to finish incomplete move: upload/library/<uuid>/<oldpath>/HEIC.xmp => upload/library/<uuid>/<newpath>/IMG_0157.HEIC.xmp
[Nest] 8  - 01/28/2024, 6:37:33 PM    WARN [StorageCore] Unable to complete move. File does not exist at either location.

I can confirm based on backups that this file did not have an XMP associated with it. When reviewing storage-template.service.ts , it looks like it might try to move a sidecar regardless of if it exists or not - not really sure.

Postgres reports:

immich=# select "originalPath", "sidecarPath" from assets WHERE "originalPath" LIKE '%Newport%IMG_0157.HEIC%';
                                         originalPath                                          |                                            sidecarPath                                      
-----------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------
 upload/library/uuid/path/IMG_0157.HEIC | upload/library/uuid/path/IMG_0157.HEIC.xmp
(1 row)

These images were uploaded using immich-go, so it's possible that it somehow create a false XMP path? I'm unsure how that all works.

[bo0tzz]

cc @simulot, I think this may have been an immich-go bug in the past, right?

[mmomjian]

On further research, I have 5 images (out of ~50k) that seem to have an unmigrated sidecar file from upload/upload. My upload/ folder does not contain any files, just empty folders. It's not clear to me that an issue in immich-go would lead to this kind of database error - whatever the source of the missing XMP files, shouldn't Immich check for the existence of said file before loading it into the DB?

immich=# select "originalPath", "sidecarPath" from assets WHERE "sidecarPath" LIKE '%upload/upload%';
                                            originalPath                                             |                                              sidecarPath           
-----------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------
 upload/library/e46649fa-6d49-4b4d-b923-3450743816ee/IMG/redacted/IMG_2993+1.HEIC        | upload/upload/e46649fa-6d49-4b4d-b923-3450743816ee/2d/0e/2d0e693e-fb6c-47ad-ab3d-8571594bb1e2.HEIC.xmp

It seems to appear that both of these issues would be fixed (sort of) by draft PR #6293 - this would not fix the missing data, but it would fix the reference to non-existent files.

[simulot]

immich-go used to create XMP files from google photo import. As far as I know this is fixed since ... long. (i need to dig into commits)

But in this case, I think that the immich server just check the existence of an XMP files

[jrasm91]

I believe we handle deleted xmp files now.