panic: runtime error: index out of range [-9223372036852667758]

[en-ver]

The app is installed using docker-compose on Debian GNU/Linux 11 (bullseye)
My CPU is AMD PRO A8-8650B R7
The app starts well but after about 30 sec I start initial scan the container stops with the following error:

panic: runtime error: index out of range [-9223372036852667758]

goroutine 1088 [running]:
github.com/disintegration/imaging.(*scanner).scan(0xc001c7e300, 0x0, 0x5d9, 0xb00, 0x5da, {0xc001e2c000, 0xdf, 0x2c00})
/go/pkg/mod/github.com/disintegration/imaging@v1.6.2/scanner.go:202 +0x810
github.com/disintegration/imaging.resizeHorizontal.func1(0xc000386098)
/go/pkg/mod/github.com/disintegration/imaging@v1.6.2/resize.go:114 +0xbd
github.com/disintegration/imaging.parallel.func1()
/go/pkg/mod/github.com/disintegration/imaging@v1.6.2/utils.go:33 +0x63
created by github.com/disintegration/imaging.parallel
/go/pkg/mod/github.com/disintegration/imaging@v1.6.2/utils.go:31 +0xdc

Please advise on how to get it working.

[lastzero]

Did you see the bug issue template? Could it be that some information we need is missing?

[en-ver]

Steps to reproduce the behavior:

1. Install using docker compose
2. Run library scan
3. The app goes offline
4. See container error in portainer

No changes - all default settings

There is no file which stops the indexing - everytime they are different. Those files that last time passed indexing successfully, next time may stop it.

_photoprism_photoprism_1_logs.txt

I use:

• the last docker-compose image (app and mariadb)
• PhotoPrism® 220121-2b4c8e1f-Linux-x86_64
• Debian GNU/Linux 11 (bullseye)
• Chrome for Mac
• HP EliteDesk 705 G2 Small Form Factor PC
• Physical Memory: 8GB
• Swap Space - 1GB (in GB)
• Storage Type: 120GB SSD + 1TB RAID1
• No reverse proxy, VPN, firewall

[lastzero]

So it happens with every file? Our demo and all other users don't seem to have a problem with every file, so it's unclear how to reproduce it.

[lastzero]

I'd suspect some sort of metadata issue.

[en-ver]

I didn't say "every" :) It may stop on "any" file.
If I restart the scan it may stop on the file which was successfully indexed on the previous try.

I'd suspect some sort of metadata issue.

What do you mean? How to check it?

[lastzero]

Could be the color profile, orientation, config options like JPEG quality and size... About everything that influences thumbnail generation.

[en-ver]

Ok. To validate this hypothesis it would be enough to scan the library with no thumbnail generation. Is there any setting to try to disable the thumbnail generation?

[lastzero]

I'm in bed already. Quickly checked the related code and it happens in the image.YCbCr code block, see Wikipedia:

YCbCr, Y′CbCr, or Y Pb/Cb Pr/Cr, also written as YCBCR or Y′CBCR, is a family of color spaces used as a part of the color image pipeline in video and digital photography systems. Y′ is the luma component and CB and CR are the blue-difference and red-difference chroma components.

Could it be that you have such files? If yes, does it happen with them? Would you be willing to share them for debugging?

[en-ver]

Here is one more finding
In the web interface in the error section under the library section, I see errors like on the picture attached

YCbCr, Y′CbCr, or Y Pb/Cb Pr/Cr, also written as YCBCR or Y′CBCR, is a family of color spaces used as a part of the color image pipeline in video and digital photography systems. Y′ is the luma component and CB and CR are the blue-difference and red-difference chroma components.

I don't know. The photos were captured for the last 10-15 years. I was dreaming to have something self-hosted and now trying to get it working :)

[lastzero]

You can check the metadata, for example with exiftool. Since RGB (RED, GREEN, BLUE) encoded files work for everyone, it probably doesn't happen with them. So spending a day testing if everything works as it always did doesn't make a lot of sense and won't help you.

[srett]

It might be worth a shot running memtest86+ if you don't have ECC memory. If things sometimes work and sometimes don't, it's been faulty RAM more than once for me in the past ;-) Granted, more often than not it wasn't, but then again it's a low effort thing to rule out, except if you run a dozen important services on that machine that you don't want to have any downtime.

[en-ver]

What I tried to do to allocate the root cause:

1. moved pictures from sw raid1 to local ssd and unmounted it to eliminate the risk of raid failures
2. removed 2 of 4 ram modules and changed them after the index stop to another 2 modules to eliminate the issue with RAM module
3. indexed different folders captured on different devices to eliminate the color issue mentioned by @lastzero

... no luck! any ideas?

[lastzero]

To me, it looks like unexpected data in JPEG files. Either there's a bug in the parser (which is not from us directly) or some files are not standard compliant. Without test data, it's mostly fruitless guesswork for us.

[lastzero]

Could well be a bug in the disintegration/imaging library. Did you check their GitHub Issues already?

[en-ver]

Without test data, it's mostly fruitless guesswork for us.

@lastzero I can provide the file but I don't know which one for sure is a problematic one. There is no one file or group of files that breaks the process. I can provide the random one and you'll waste your time investigating something irrelevant.

Could well be a bug in the disintegration/imaging library. Did you check their GitHub Issues already?

No, I didn't yet. WIll check it tomorrow.

[lastzero]

Anyhow, we need a way to reproduce this. There's always a chance it's a local issue. Guessing what it is doesn't work or at least requires more time than we have.

[en-ver]

Well, here are some findings after hours of experiments :)

I set up Debian on my very old and very slow laptop and run Photoprism there. And it does index the files with no issues. Which means that there is no issue with the pictures.

Also, I found out that I can't install Ubuntu on my brand new PC/server. It end-ups with OS installation error. I assume there is some hardware issue. Tomorrow I'll change it to another PC with the same configuration and check if I can run Photoprism and index files there.

... to be continued :)

[lastzero]

Added this to https://docs.photoprism.app/getting-started/troubleshooting/#fatal-server-errors

[lastzero]

Thank you very much for your feedback! I am relieved that it is not a problem with the JPEG parser. I'll move this to Discussions then 👍

[en-ver]

Ok. Here are the outcomes of my journey.
The container still keeps failing on another PC with the same configuration.
But I found out that it doesn't stop if you limit the CPU usage for photoprism container using "cpus: 1" parameter in docker-compose file.
In this case, the container uses about 25% of each of the four cores of my CPU.
If I add more (for example 1.5) it stops after some time of indexing.

I didn't get the reason, but now it works.

Probably it makes sense to add this workaround somewhere in the troubleshooting tutorial or even the docker-compose file.

[lastzero]

Feel free to make specific suggestions / send a pull request. Spent the whole week updating and expanding the docs, especially the troubleshooting guide. The performance section has been completely reworked, the developer guide now also contains much more information, e.g. about thumbnails.

[en-ver]

Update: I finally found the root cause of the issue.
The CPU overheat!
My PC CPU doesn't manage the fan speed depending on the temp and/or cores load.
As a result, the CPU is overheating during the import and index process and CPU starts throttling.
After I increased the speed of CPU fan the temperature lowered and now the container works great (with no fails and stops).

[lastzero]

We already had that on our list 😇