Index: Extract original filenames from Exiftool JSON

[joachimtingvold]

After import, when going through photos to add them to albums, I noticed that a photo was not found (initially by looking at original_name).

After looking further, I found that the originals photo was present, but missing the original_name column;

c28ec8ddbf6f075cdff8466ee98dfcb3  /storage/photos/archive/6D/2013/10/12/IMG_5023.CR2
c28ec8ddbf6f075cdff8466ee98dfcb3  originals/2013/10/20131012_155750_4286DBCD.cr2

root@photoprism1:/storage/photoprism# exiftool originals/2013/10/20131012_155750_4286DBCD.cr2 | grep -i date
File Modification Date/Time     : 2022:01:03 01:15:42+01:00
File Access Date/Time           : 2022:01:05 21:31:00+01:00
File Inode Change Date/Time     : 2022:01:03 01:15:42+01:00
Modify Date                     : 2013:10:12 15:57:50
Date/Time Original              : 2013:10:12 15:57:50
Create Date                     : 2013:10:12 15:57:50
Create Date                     : 2013:10:12 15:57:50.00
Date/Time Original              : 2013:10:12 15:57:50.00
Modify Date                     : 2013:10:12 15:57:50.00

MariaDB [photoprism]> select photo_uid, photo_path, photo_name, original_name, taken_at, taken_at_local, place_src, created_at from photos where photo_name like '%4286DBCD%';
+------------------+------------+--------------------------+---------------+---------------------+---------------------+-----------+---------------------+
| photo_uid        | photo_path | photo_name               | original_name | taken_at            | taken_at_local      | place_src | created_at          |
+------------------+------------+--------------------------+---------------+---------------------+---------------------+-----------+---------------------+
| pr5404t2mayh2gc3 | 2013/10    | 20131012_155750_4286DBCD |               | 2013-10-12 13:57:50 | 2013-10-12 15:57:50 | estimate  | 2022-01-03 01:00:29 |
+------------------+------------+--------------------------+---------------+---------------------+---------------------+-----------+---------------------+
1 row in set (0.134 sec)

Photos imported just before, or just after, has original_name as expected;

MariaDB [photoprism]> select photo_uid, photo_path, photo_name, original_name, taken_at, taken_at_local, place_src, created_at from photos where created_at like '%2022-01-03 01:00%';
+------------------+------------+--------------------------+------------------------+---------------------+---------------------+-----------+---------------------+
| photo_uid        | photo_path | photo_name               | original_name          | taken_at            | taken_at_local      | place_src | created_at          |
+------------------+------------+--------------------------+------------------------+---------------------+---------------------+-----------+---------------------+
| pr54041l7d327eh2 | 2013/11    | 20131119_212433_95595609 | 6D/2013/11/19/IMG_5419 | 2013-11-19 20:24:33 | 2013-11-19 21:24:33 | estimate  | 2022-01-03 01:00:01 |
| pr5404810mct4ixc | 2013/11    | 20131119_212433_FE04DDE0 | 6D/2013/11/19/IMG_5420 | 2013-11-19 20:24:33 | 2013-11-19 21:24:33 | estimate  | 2022-01-03 01:00:08 |
| pr5404k34w44u3nh | 2013/11    | 20131119_212434_616A971C | 6D/2013/11/19/IMG_5421 | 2013-11-19 20:24:34 | 2013-11-19 21:24:34 | estimate  | 2022-01-03 01:00:20 |
| pr5404okxznncmsx | 2013/11    | 20131119_212434_61396CBE | 6D/2013/11/19/IMG_5422 | 2013-11-19 20:24:34 | 2013-11-19 21:24:34 | estimate  | 2022-01-03 01:00:24 |
| pr5404t2mayh2gc3 | 2013/10    | 20131012_155750_4286DBCD |                        | 2013-10-12 13:57:50 | 2013-10-12 15:57:50 | estimate  | 2022-01-03 01:00:29 |
| pr540512dmitjb8p | 2013/11    | 20131119_212435_9C28778F | 6D/2013/11/19/IMG_5423 | 2013-11-19 20:24:35 | 2013-11-19 21:24:35 | estimate  | 2022-01-03 01:00:37 |
| pr540553a9xs31ea | 2013/11    | 20131119_212437_531F5E22 | 6D/2013/11/19/IMG_5424 | 2013-11-19 20:24:37 | 2013-11-19 21:24:37 | estimate  | 2022-01-03 01:00:41 |
| pr5405i2mcc8byx0 | 2013/11    | 20131119_212439_6715786F | 6D/2013/11/19/IMG_5425 | 2013-11-19 20:24:39 | 2013-11-19 21:24:39 | estimate  | 2022-01-03 01:00:54 |
| pr5405m1pz738sdu | 2013/11    | 20131119_212441_17E975D7 | 6D/2013/11/19/IMG_5426 | 2013-11-19 20:24:41 | 2013-11-19 21:24:41 | estimate  | 2022-01-03 01:00:58 |
+------------------+------------+--------------------------+------------------------+---------------------+---------------------+-----------+---------------------+
9 rows in set (0.108 sec)

The original_name is also gone from the files table;

MariaDB [photoprism]> select photo_uid, file_name, original_name from files where photo_uid='pr5404t2mayh2gc3';
+------------------+------------------------------------------+---------------+
| photo_uid        | file_name                                | original_name |
+------------------+------------------------------------------+---------------+
| pr5404t2mayh2gc3 | 2013/10/20131012_155750_4286DBCD.cr2     |               |
| pr5404t2mayh2gc3 | 2013/10/20131012_155750_4286DBCD.cr2.jpg |               |
+------------------+------------------------------------------+---------------+
2 rows in set (0.000 sec)

Looking in the logs, I found the following;

The database entry was created at 2022-01-03 01:00:29, with error log entry at 2022-01-03 01:15:48. The image is visible just fine within PP; two stacked files (raw+jpeg) with proper info.

I'm not sure where the missing original_name stems from, since there was a time difference of ~15 minutes between the import and the error (which is probably because the error is caused by the indexing). I would assume that the import is what would generate the original_name, as the indexer has no way of knowing this (as far as I know, at least).

[joachimtingvold]

There seems to be more photos without original_name, and all of the photos are imported (i.e. none of the photos are indexed straight from the originals/ folder). I have not looked into the below results in more detail.

MariaDB [photoprism]> select count(*) from photos where original_name='';
+----------+
| count(*) |
+----------+
|      389 |
+----------+
1 row in set (0.067 sec)

[lastzero]

Did you try the Development Preview? Seems a problem with color profile detection, which was refactored.

[joachimtingvold]

No, this was imported using photoprism:latest. But the error (read color metadata) would be part of the index, and not the import, yes? If so, I can't understand how that would impact original_name, as that would be set during import? (as a pure index of originals would not reveal the original filename in any way).

[lastzero]

Import just moves the file, the index operation is the same then. There's only one additional parameter for the original name which is empty when indexing existing originals.

[joachimtingvold]

But the import must add the info to the original_name column? If not, index would have no way of knowing this info?

[lastzero]

It knows because the name is passed to the index function. Before indexing is has been finished successfully, there is no index entry and thus no row / column.

[joachimtingvold]

So if indexing fails (like it seems to have done here), the original_name info is basically lost forever?

[lastzero]

Yep, unless we add yet another sidecar file just for this. Logs might of course also contain infos. That's why indexing should be robust and why the color profile detection was improved today. It's a new feature and it wasn't clear what could trigger an error. Would be great to get failing files for testing as we have none.

[lastzero]

JSON files in the cache folder might also contain the original name.

[joachimtingvold]

root@photoprism1:/storage/photoprism# sha1sum originals/2013/10/20131012_155750_4286DBCD.cr2
54ef3374bd601168e9f107b90f264acf8085cceb  originals/2013/10/20131012_155750_4286DBCD.cr2
root@photoprism1:/storage/photoprism# grep SourceFile storage/cache/json/5/4/e/54ef3374bd601168e9f107b90f264acf8085cceb_exiftool.json
  "SourceFile": "/photoprism/import/6D/2013/10/12/IMG_5023.CR2",

Is that created by the import? If so, maybe the indexer could fetch it from there? It would only be relevant if the import-called index fails, and it suceeds at a later point during normal index.