index: flagged 1 broken photo as hidden

[dml33]

Hi.
I am running 211118 release and, when I do a purge, the following message appears:

INFO[2021-12-03T23:18:50Z] purge: removing missing files in originals
INFO[2021-12-03T23:18:50Z] index: flagged 1 broken photo as hidden [9.034638ms]
INFO[2021-12-03T23:18:50Z] index: found no orphan files [1.832252ms]
INFO[2021-12-03T23:18:50Z] index: updating counts
INFO[2021-12-03T23:18:50Z] index: updating covers
INFO[2021-12-03T23:18:51Z] purged 0 files and 0 photos in 1.173315206s
INFO[2021-12-03T23:18:51Z] closed database connection

How could I know which one is that "broken photo"? Why is it broken? In the Hidden section "No pictures found" appears. So, I do not have any information about it.

Could you help me? Thanks!

[lastzero]

Probably nothing to worry about. Broken means it's an orphan index entry without JPEG or something, maybe after a file has been stacked. You may try to enable debug mode to see more.

[dml33]

Thanks for the comment. I enabled debug, and there is no extra information about it. For example:

2021-12-04 13:35:58 INFO index: updating covers

2021-12-04 13:35:58 DEBU counts: updated 31 labels [14.572391ms]

2021-12-04 13:35:58 DEBU counts: updated 10 subjects [24.011028ms]

2021-12-04 13:35:58 DEBU counts: updated 16 places [3.590393ms]

2021-12-04 13:35:58 INFO index: updating counts

2021-12-04 13:35:58 DEBU moments: 22 albums saved as yaml files

2021-12-04 13:35:58 DEBU moments: analyzing 638 photos and 37 videos, with threshold 10

2021-12-04 13:35:58 INFO index: flagged 1 broken photo as hidden [13.086765ms]

2021-12-04 13:35:58 DEBU metadata: starting routine check

2021-12-04 13:35:58 DEBU faces: updated 0 markers, recognized 0 faces, 0 unknown [19.972262ms]

2021-12-04 13:35:58 DEBU faces: found no unmatched markers

2021-12-04 13:35:58 DEBU faces: found no new faces [4.304245ms]

2021-12-04 13:35:58 DEBU faces: skipped clustering

2021-12-04 13:35:58 DEBU faces: found no clusters to be merged [2.870817ms]

2021-12-04 13:35:58 DEBU faces: found no ambiguous subjects [118.586724ms]

2021-12-04 13:35:58 DEBU markers: found no missing subjects [1.027778ms]

2021-12-04 13:35:58 DEBU markers: found no invalid references [4.774204ms]

2021-12-04 13:35:58 DEBU faces: found no orphan markers [20.229978ms]

2021-12-04 13:35:58 DEBU metadata: running facial recognition

[lastzero]

see

photoprism/internal/query/photo.go

Line 175 in 403eb0d

log.Infof("index: flagged %s as hidden [%s]", english.Plural(int(res.RowsAffected), "broken photo", "broken photos"), time.Since(start))

Maybe should change the message to make it less scary. Are you using SQLite or MySQL? AFAIK SQLite will always report the total number of matching rows, independent of if they were changed. So you may get the same message over and over again.

[lastzero]

Could also just be a deleted file. Try running the purge & cleanup commands.

[dml33]

Hi.

I am using sqlite:

root@Emby2:/ # pkg info | grep sql
py27-sqlite3-2.7.18_7          Standard Python binding to the SQLite3 library (Python 2.7)
py36-sqlite3-3.6.15_7          Standard Python binding to the SQLite3 library (Python 3.6)
py37-sqlite3-3.7.12_7          Standard Python binding to the SQLite3 library (Python 3.7)
py38-sqlite3-3.8.12_7          Standard Python binding to the SQLite3 library (Python 3.8)
py39-sqlite3-3.9.7_7           Standard Python binding to the SQLite3 library (Python 3.9)
sqlite3-3.35.5_3,1             SQL database engine in a C library

The purge and cleanup commands report:

root@Emby2:/ # photoprism --assets-path=/var/db/photoprism/assets --storage-path=/var/db/photoprism/storage --originals-path=/var/db/photoprism/storage/originals --import-path=/var/db/photoprism/storage/import purge
INFO[2021-12-04T13:27:52Z] Help us make a difference and become a sponsor today!
INFO[2021-12-04T13:27:52Z] Visit https://docs.photoprism.org/funding/ to learn more.
DEBU[2021-12-04T13:27:53Z] migrate: found 0 previous migrations
WARN[2021-12-04T13:27:53Z] migrate: creating 20211121-094727 failed with UNIQUE constraint failed: migrations.id [665.328µ            s]
WARN[2021-12-04T13:27:53Z] migrate: creating 20211124-120008 failed with UNIQUE constraint failed: migrations.id [485.896µ            s]
INFO[2021-12-04T13:27:53Z] purge: removing missing files in originals
DEBU[2021-12-04T13:27:53Z] index: found no photos without primary file [7.516063ms]
INFO[2021-12-04T13:27:53Z] index: flagged 1 broken photo as hidden [9.577244ms]
INFO[2021-12-04T13:27:53Z] index: found no orphan files [1.682907ms]
INFO[2021-12-04T13:27:53Z] index: updating counts
DEBU[2021-12-04T13:27:53Z] counts: updated 16 places [3.487522ms]
DEBU[2021-12-04T13:27:53Z] counts: updated 10 subjects [17.826895ms]
DEBU[2021-12-04T13:27:53Z] counts: updated 31 labels [14.677037ms]
INFO[2021-12-04T13:27:53Z] index: updating covers
DEBU[2021-12-04T13:27:53Z] covers: updated 8 albums [102.321701ms]
DEBU[2021-12-04T13:27:53Z] covers: updated 5 folders [25.960763ms]
DEBU[2021-12-04T13:27:53Z] covers: updated 4 months [31.036144ms]
DEBU[2021-12-04T13:27:53Z] covers: updated 35 labels [282.629864ms]
DEBU[2021-12-04T13:27:53Z] covers: updated 5 subjects [22.430892ms]
INFO[2021-12-04T13:27:53Z] purged 0 files and 0 photos in 1.179806178s
INFO[2021-12-04T13:27:53Z] closed database connection

root@Emby2:/ # photoprism --assets-path=/var/db/photoprism/assets --storage-path=/var/db/photoprism/storage --originals-path=/var/db/photoprism/storage/originals --import-path=/var/db/photoprism/storage/import cleanup
INFO[2021-12-04T13:28:23Z] Help us make a difference and become a sponsor today!
INFO[2021-12-04T13:28:23Z] Visit https://docs.photoprism.org/funding/ to learn more.
DEBU[2021-12-04T13:28:24Z] migrate: found 0 previous migrations
WARN[2021-12-04T13:28:24Z] migrate: creating 20211121-094727 failed with UNIQUE constraint failed: migrations.id [673.14µs            ]
WARN[2021-12-04T13:28:24Z] migrate: creating 20211124-120008 failed with UNIQUE constraint failed: migrations.id [541.882µ            s]
INFO[2021-12-04T13:28:24Z] index: found no orphan files [1.695944ms]
INFO[2021-12-04T13:28:24Z] removed 0 index entries and 0 thumbnails in 344.269116ms
INFO[2021-12-04T13:28:24Z] closed database connection

No clues about the broken photo.

I updated to 20211130 release yesterday night. I have just noticed the migrate WARNINGs. What are they?

[lastzero]

Did you use a development version meanwhile? It means the migration already exists, so it can't be created. For the info related to marking photos as hidden / broken, it means nothing other than that Sqlite doesn't return the true number of affected rows.

[lastzero]

@graciousgrey can test again with Sqlite next week, not working on the weekend 👍

[dml33]

I updated from https://github.com/photoprism/photoprism/tree/211018-e200f322 to https://github.com/photoprism/photoprism/tree/211130-13cfcf6d

I did not use any other release between.

Is there any way to remove that migration or force it to be carried out (if nothing gets broken 😆)

[lastzero]

I'm also not working on the weekend. Would be great if you can find out more!

Here is the code:

• https://github.com/photoprism/photoprism/blob/develop/internal/migrate/migrations.go#L68

Here are the migrations:

• https://github.com/photoprism/photoprism/tree/develop/internal/migrate/sqlite

[lastzero]

The code comment says "should not happen". Seems like the existing IDs can't be read of the DB. Without spending more time than I already did on my "free" weekend, I can't tell you more right now.

[lastzero]

The only actual problem you seem to have is that the log levels are too high for SQLite. Migrations were executed. Rows were already updated.

[lastzero]

Found what's causing this: The database dialect isn't named "sqlite" but "sqlite3", so the strings didn't match. Validations are pretty strict and logging verbose, so mistakes like these don't go unnoticed for long. Should be fixed with the latest changes. Started a new Development Preview build for testing! Thanks for reporting it 👍

[lastzero]

Log levels for routine index maintenance have been reduced as well to avoid confusion.

[dml33]

Hi, I have just installed g211215 93b26f19 and the index: flagged 1 broken photo as hidden message has disappeared 👍
But, still the warnings about the migrations remain:

root@Emby2:/usr/ports/science/libtensorflow1 # photoprism --assets-path=/var/db/photoprism/assets --storage-path=/var/db/photoprism/storage --originals-path=/var/db/photoprism/storage/originals --import-path=/var/db/photoprism/storage/import cleanup
INFO[2022-01-03T19:52:11Z] Help us make a difference and become a sponsor today!
INFO[2022-01-03T19:52:11Z] Visit https://docs.photoprism.app/funding/ to learn more.
WARN[2022-01-03T19:52:12Z] migrate: creating 20211121-094727 failed with UNIQUE constraint failed: migrations.id [582.58µs]
WARN[2022-01-03T19:52:12Z] migrate: creating 20211124-120008 failed with UNIQUE constraint failed: migrations.id [458.335µs]
INFO[2022-01-03T19:52:26Z] index: found no orphan files [1.367655ms]
INFO[2022-01-03T19:52:26Z] removed 0 index entries and 0 thumbnails in 14.400460065s
INFO[2022-01-03T19:52:26Z] closed database connection

root@Emby2:/usr/ports/science/libtensorflow1 # photoprism --assets-path=/var/db/photoprism/assets --storage-path=/var/db/photoprism/storage --originals-path=/var/db/photoprism/storage/originals --import-path=/var/db/photoprism/storage/import purge
INFO[2022-01-03T19:53:15Z] Help us make a difference and become a sponsor today!
INFO[2022-01-03T19:53:15Z] Visit https://docs.photoprism.app/funding/ to learn more.
WARN[2022-01-03T19:53:15Z] migrate: creating 20211121-094727 failed with UNIQUE constraint failed: migrations.id [624.054µs]
WARN[2022-01-03T19:53:15Z] migrate: creating 20211124-120008 failed with UNIQUE constraint failed: migrations.id [524.683µs]
INFO[2022-01-03T19:53:15Z] purge: removing missing files in originals
INFO[2022-01-03T19:53:16Z] index: found no orphan files [1.246371ms]
INFO[2022-01-03T19:53:16Z] index: updating counts
INFO[2022-01-03T19:53:16Z] index: updating covers
INFO[2022-01-03T19:53:16Z] purged 0 files and 0 photos in 1.418775607s
INFO[2022-01-03T19:53:16Z] closed database connection

Is it a way to make them go if they are, in fact, inocuos? Just for cleanness and avoid confusion.

Thanks for this awesome program!

[lastzero]

Can you check the migrations table? Does the dialect column say sqlite or sqlite3?

[dml33]

Hi.
It says sqlite:

[lastzero]

Should be sqlite3, try updating it manually to see if it works!

[lastzero]

Might as well remove the column in future versions. For now, it was good to discover the inconsistency.

[dml33]

It works! The warning has dissappeared:

root@Emby2:/var/db/photoprism/storage # photoprism --assets-path=/var/db/photoprism/assets --storage-path=/var/db/photoprism/storage --originals-path=/var/db/photoprism/storage/originals --import-path=/var/db/photoprism/storage/import cleanup
INFO[2022-01-04T08:08:11Z] Help us make a difference and become a sponsor today!
INFO[2022-01-04T08:08:11Z] Visit https://docs.photoprism.app/funding/ to learn more.
INFO[2022-01-04T08:08:12Z] index: found no orphan files [1.271157ms]
INFO[2022-01-04T08:08:12Z] removed 0 index entries and 0 thumbnails in 399.248132ms
INFO[2022-01-04T08:08:12Z] closed database connection

Thanks!