Exif: Add cycle detection to prevent infinite loops

[iwinstar]

I've installed PhotoPrism by docker-compose in my Synology server. While index one of those Pictures.zip cause photoprism docker very busy and be killed after several minutes. After ignore them photoprism work back to normal.

I'm using version 210505-d3e53a89-Linux-x86_64. Could I try latest release or wait for you check this problem? Thanks

Here is my options.yml config.

Debug: false
DisableBackups: false
DisableDarktable: true
DisableFFmpeg: true
DisableRawtherapee: true
JpegQuality: 95
ThumbFilter: blackman
ThumbSize: 720
ThumbSizeUncached: 720
ThumbUncached: true

[lastzero]

Please try our latest release and make sure your Synology station meets the hardware requirements. You need at least 4 GB of RAM plus Swap. Try reducing the number of workers to 1.

[iwinstar]

I give photoprism docker 12 GB RAM to use, so hardware requirements shouldn't be a problem. I also tried reducing PHOTOPRISM_WORKERS to 1 but still useless.

[iwinstar]

It has successfully indexed over 40000 pictures before those two.

[lastzero]

Are these files somehow special, like very large or RAW?

[iwinstar]

I have uploaded them. https://github.com/photoprism/photoprism/files/6529695/Pictures.zip
just common jpeg, not very large.

[lastzero]

I also see you have tweaked the thumb settings. Are you sure this isn't because of high CPU load when viewing search results while indexing?

[iwinstar]

No. I'm sure no one is viewing.

[lastzero]

Try again using our latest release from yesterday. We've updated a lot of dependencies. If it still happens, I'll take a look at your files later today.

[iwinstar]

Thanks, I will try latest docker image soon.

[iwinstar]

Still get killed using latest version 210523-b1856b9d-Linux-x86_64. It even can't indexed any new picture...
I'll roll back to 210505-d3e53a89-Linux-x86_64.

[lastzero]

What database are you using? Any other special settings?

[lastzero]

Maybe the disk is full?

[iwinstar]

Database: MariaDB 10.4.19
Disk still has 9.3TiB free space
I have executed "photoprism purge" and "photoprism cleanup" command before start indexing.
I checked webpage keep saying "Indexing photo-1512923874408-340b0f3cb7e6.jpg…" but no any update msg or docker log.

[iwinstar]

I changed all table default charset to utf8mb4, and serval column collation utf8mb4_unicode_ci to fix database "1366 Incorrect String Value" error. such as
photoprism.details.subject
photoprism.photos.photo_title
photoprism.photos.photo_description

[iwinstar]

Photoprism docker has used more than 7.61 GB RAM.
docker log keep print like "INFO[2021-05-24T06:04:23Z] photo: won't update title, xxxxxxx was modified" before get killed.

[iwinstar]

After rollback to version 210505-d3e53a89 and ignore those two picture I uploaded, index work fine.

[iwinstar]

I directly stopped the photoprism container and create new one with latest image, never click "cancel index" button through webpage. Is this why I couldn't use the latest release to continue indexing pictures?

[lastzero]

Excellent catch! Seems to be an issue with our JPEG Exif parser. Hope Dustin can help with this.

[lastzero]

@iwinstar May we add your samples to our test file archive at https://dl.photoprism.org/samples/?

[iwinstar]

Of course. Excellent work, Thanks.

[lastzero]

Reading Exiftool's warning message, it becomes clear why go-exif may get stuck in a loop extracting all data:

$ exiftool -g yourfile.jpg 
---- ExifTool ----
ExifTool Version Number    : 12.16
Warning                    : IFD1 pointer references previous IFD0 directory

See https://exiftool.org/forum/index.php?topic=3524.0

Money quote:

Question 1: What does this mean? (Sorry I could not find it anywhere.)

It means that the EXIF information is corrupted, and some may have been lost.

Looks very much like the same issue as in #1463, same warning there.

[rickysarraf]

With the current devel tip, I am getting a crash on PhotoPrism when doing a full index. The crash has been consistently reproduced for the last 1 week or so.

photoprism_1  | time="2021-08-17T10:17:21Z" level=debug msg="moments: updated “Water, Ships & Boats” (public:true label:lakeside|water)"
photoprism_1  | time="2021-08-17T10:17:26Z" level=debug msg="moments: 934 albums saved as yaml files"
photoprism_1  | time="2021-08-17T10:17:26Z" level=info msg="faces: found 137 new markers"
photoprism_1  | panic: sync: negative WaitGroup counter
photoprism_1  | 
photoprism_1  | goroutine 7357 [running]:
photoprism_1  | sync.(*WaitGroup).Add(0xc00484d280, 0xffffffffffffffff)
photoprism_1  |         /usr/local/go/src/sync/waitgroup.go:74 +0x147
photoprism_1  | sync.(*WaitGroup).Done(...)
photoprism_1  |         /usr/local/go/src/sync/waitgroup.go:99
photoprism_1  | github.com/mpraski/clusters.(*dbscanClusterer).nearestWorker(0xc001f72ff0)
photoprism_1  |         /go/pkg/mod/github.com/mpraski/clusters@v0.0.0-20171016094157-18104487c312/dbscan.go:251 +0x230
photoprism_1  | created by github.com/mpraski/clusters.(*dbscanClusterer).startNearestWorkers
photoprism_1  |         /go/pkg/mod/github.com/mpraski/clusters@v0.0.0-20171016094157-18104487c312/dbscan.go:228 +0xdb
mariadb_1     | 2021-08-17 10:17:32 33 [Warning] Aborted connection 33 to db: 'photoprism' user: 'photoprism' host: '172.17.0.3' (Got an error reading communication packets)
mariadb_1     | 2021-08-17 10:17:33 30 [Warning] Aborted connection 30 to db: 'photoprism' user: 'photoprism' host: '172.17.0.3' (Got an error reading communication packets)
mariadb_1     | 2021-08-17 10:17:34 32 [Warning] Aborted connection 32 to db: 'photoprism' user: 'photoprism' host: '172.17.0.3' (Got an error reading communication packets)
mariadb_1     | 2021-08-17 10:17:34 31 [Warning] Aborted connection 31 to db: 'photoprism' user: 'photoprism' host: '172.17.0.3' (Got an error reading communication packets)
photoprism_photoprism_1 exited with code 0

[lastzero]

Not related. Has to do with facial recognition. Do you have it enabled? How many workers do you have configured? The cluster library doesn't seem to like it...

[rickysarraf]

Not related. Has to do with facial recognition. Do you have it enabled? How many workers do you have configured? The cluster library doesn't seem to like it...

Thanks Michael. I have now created a new issue for it.
#1478

I don't recollect seeing any explicit switch for Facial Recognition. But yes, my past 2 weeks rebuilds are more about tracking the recent developments around

[lastzero]

Our go-mod-outdated command didn't show new major versions of related image structure libraries, so we didn't upgrade those. The new versions use go-exif v3 instead of v2 which solves the issue. We'll build a new development preview for testing as soon as possible.