Index: Log underlying error when MIME type detection fails

[drewp]

Many photos from my Nikon 5000 SLR camera are plain JPEG files, not JFIF files. This appears to fail to ingest with a misleading error message. Either the error message should be improved or, ideally, the support should be fixed.

Repro

Run docker.io/photoprism/photoprism:250707 in k8s on linux x86_64.

Make a test dir like this:

/my/pic/jpeg-test% l
total 8711
-rw-rw-r-- 1 drewp drewp    7298 Aug  4 21:07 jfif2.jpg
-rw------- 1 drewp drewp 4355064 Aug  4 21:04 plain.jpg
-rw------- 1 drewp drewp 4355064 Aug  4 21:03 plain.JPG

Run the file command:

/my/pic/jpeg-test% file *
jfif2.jpg: JPEG image data, JFIF standard 1.01, resolution (DPCM), density 39x39, segment length 16, baseline, precision 8, 235x64, components 1
plain.jpg: JPEG image data, Exif standard: [TIFF image data, big-endian, direntries=11, manufacturer=NIKON CORPORATION, model=NIKON D5000, orientation=upper-left, xresolution=180, yresolution=188, resolutionunit=2, software=Ver.1.00 , datetime=2009:07:31 01:52:34, GPS-Data], baseline, precision 8, 4288x2848, components 3
plain.JPG: JPEG image data, Exif standard: [TIFF image data, big-endian, direntries=11, manufacturer=NIKON CORPORATION, model=NIKON D5000, orientation=upper-left, xresolution=180, yresolution=188, resolutionunit=2, software=Ver.1.00 , datetime=2009:07:31 01:52:34, GPS-Data], baseline, precision 8, 4288x2848, components 3

Run ffprobe:

/my/pic/jpeg-test% ffprobe -hide_banner plain.jpg 
Input #0, image2, from 'plain.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 871012 kb/s
  Stream #0:0: Video: mjpeg (Baseline), yuvj422p(pc, bt470bg/unknown/unknown), 4288x2848, 25 fps, 25 tbr, 25 tbn

See the mounted files from inside the k8s container:

photoprism@250707:/photoprism/originals/jpeg-test$ ls -ll
ls: plain.JPG: Permission denied
ls: plain.jpg: Permission denied
total 8711
-rw-rw-r-- 1 user-501 photoprism    7298 Aug  5 04:07 jfif2.jpg
-rw------- 1 user-501 photoprism 4355064 Aug  5 04:03 plain.JPG
-rw------- 1 user-501 photoprism 4355064 Aug  5 04:04 plain.jpg

On the PP web ui, reindex /jpeg-test. The logs follow:

1  level=info msg="indexing completed in 286 s"
2  level=info msg="indexing files in jpeg-test"
3  level=debug msg="exiftool: extracting metadata from jpeg-test/jfif2.jpg"
4  level=debug msg="cache: created b47fe917f2b1593d5eafe5877fed947769be406a_exiftool.json"
5  level=warning msg="index: skipped jpeg-test/plain.JPG because it has an invalid extension (unknown media type)"
6  level=warning msg="index: skipped jpeg-test/plain.jpg because it has an invalid extension (unknown media type)"
7  level=info msg="media: generated 8 thumbnails for jpeg-test/jfif2.jpg [98.181021ms]"
8  level=info msg="media: jfif2.jpg was taken at 2025-08-05 04:07:57 +0000 UTC (file mod time)"

Log line 5 tricked me into thinking there was case sensitivity on the filenames, but I no longer think that's the issue.

However, it also doesn't seem to be the upstream mimetype lib's fault. Here's a result from https://github.com/gabriel-vasile/mimetype :

go mod init example
go get github.com/gabriel-vasile/mimetype

% cat go.mod
module example

go 1.24.2

require (
	github.com/gabriel-vasile/mimetype v1.4.9 // indirect
	golang.org/x/net v0.39.0 // indirect
)

main.go

package main

import (
	"fmt"
	"os"

	"github.com/gabriel-vasile/mimetype"
)

func main() {
	filePath := os.Args[1]
	file, _ := os.Open(filePath)
	mime, _ := mimetype.DetectReader(file)
	fmt.Printf("The MIME type of the file is: %s\n", mime.String())
}

Output:

% go run main.go /my/pic/jpeg-test/jfif2.jpg 
The MIME type of the file is: image/jpeg

% go run main.go /my/pic/jpeg-test/plain.jpg     
The MIME type of the file is: image/jpeg

% go run main.go /my/pic/jpeg-test/plain.JPG
The MIME type of the file is: image/jpeg

% go run main.go /etc/passwd                
The MIME type of the file is: text/plain; charset=utf-8

[lastzero]

Thanks for reporting this! We will be happy to look into the issue when you provide us with sample images for testing:
https://www.photoprism.app/contact

[drewp]

(2nd try- more exact match to the camera)

/my/pic/jpeg-test/DSC_0034.JPG: JPEG image data, Exif standard: [TIFF image data, big-endian, direntries=11, manufacturer=NIKON CORPORATION, model=NIKON D5000, orientation=upper-left, xresolution=180, yresolution=188, resolutionunit=2, software=Ver.1.00 , datetime=2009:07:18 13:45:43, GPS-Data], baseline, precision 8, 4288x2848, components 3

[drewp]

I confirm that I get this with the shareable sample pic too:

photoprism-59c6f989db-8cwgj photoprism 2025-08-04T22:39:47.331413162-07:00 time="2025-08-05T05:39:47Z" level=warning msg="index: skipped jpeg-test/DSC_0034.JPG because it has an invalid extension (unknown media type)"

[lastzero]

Did you build PhotoPrism from source or are you using a custom integration? 🤔

If so, required dependencies might be missing...

I don't have any issues indexing this file unless GitHub modified your original image file when you uploaded it:

DEBU[2025-08-05T05:51:55Z] exiftool: extracting metadata from 'Issues/JPEG files that are not JFIF get skipped - GitHub Issue 5149.jpeg' 
DEBU[2025-08-05T05:51:55Z] cache: created 3f2075b7d0f927e0cfe206c2a125204ba7c59d72_exiftool.json 
DEBU[2025-08-05T05:51:55Z] metadata: no gps tags in 'JPEG files that are not JFIF get skipped - GitHub Issue 5149.jpeg' (exif gps-info) 
INFO[2025-08-05T05:51:57Z] media: generated 13 thumbnails for 'Issues/JPEG files that are not JFIF get skipped - GitHub Issue 5149.jpeg' [1.471662272s] 
INFO[2025-08-05T05:51:57Z] media: 'JPEG files that are not JFIF get skipped - GitHub Issue 5149.jpeg' was taken at 2009-07-18 13:45:43 +0000 UTC (meta) 
INFO[2025-08-05T05:51:57Z] tensorflow: loading nasnet                   
DEBU[2025-08-05T05:52:00Z] media: 'Issues/JPEG files that are not JFIF get skipped - GitHub Issue 5149.jpeg' has content type 'image/jpeg' 
DEBU[2025-08-05T05:52:00Z] photo: 'Issues/JPEG files that are not JFIF get skipped - GitHub Issue 5149' has new title 'JPEG Files That Are Not JFIF Get Skipped GitHub Issue' [1.234326ms] 
INFO[2025-08-05T05:52:00Z] photo: created sidecar file 'Issues/JPEG files that are not JFIF get skipped - GitHub Issue 5149.yml' 
INFO[2025-08-05T05:52:00Z] index: added main jpg file 'Issues/JPEG files that are not JFIF get skipped - GitHub Issue 5149.jpeg'

These are the file details displayed in the UI after indexing:

[lastzero]

When using our latest/preview Docker image (or the development environment, which uses the same Ubuntu base image), the magic data for MIME type detection should be located here:

# ls -alh /usr/lib/file/magic.mgc 

-rw-r--r-- 1 root root 8.2M Mar 31  2024 /usr/lib/file/magic.mgc

[lastzero]

I suggest trying to reproduce this on our public demo to see if it works there:

https://demo.photoprism.app/library/browse

[drewp]

The github jpeg above is identical to my file.

Im running your docker image docker.io/photoprism/photoprism:250707 with one more docker layer that makes a photoprism user:

% cat Dockerfile.photoprism
FROM docker.io/photoprism/photoprism:250707
RUN deluser photoprism && \
  deluser user-1050 && \
  adduser \
  --no-create-home \
  --uid 1050 \
  --ingroup photoprism \
  --disabled-password \
  --disabled-login \
  photoprism
USER photoprism

Then I run it with settings in deploy.yaml, which I'll attach.
deploy.yaml.txt

On my photoprism container that runs that image, I don't find any magic.mgc file:

photoprism@250707:/$ ls /usr/lib/file
ls: cannot access '/usr/lib/file': No such file or directory
photoprism@250707:/$ find / -xdev -name magic.mgc
find: '/root': Permission denied
find: '/var/log/private': Permission denied
find: '/var/cache/apt/archives/partial': Permission denied
find: '/var/cache/ldconfig': Permission denied
find: '/var/cache/private': Permission denied
find: '/var/lib/private': Permission denied
find: '/etc/credstore.encrypted': Permission denied
find: '/etc/ssl/private': Permission denied
find: '/etc/credstore': Permission denied
find: '/run/sudo': Permission denied

But, https://github.com/gabriel-vasile/mimetype/blob/master/internal/magic/image.go#L13 suggests that the upstream detector lib doesn't use system magic files?

I see the upload at https://demo.photoprism.app/library/albums/artisl61ivvteyk1/view

What do you think is going on?

[drewp]

Is there a different file detection code path between upload-via-web-page and put-in-originals-folder-and-reindex ?

[drewp]

Meanwhile, looking through /library/errors, I see cases like this:
index: skipped phonecam/dn202012/20201209_194518~2.heic because it has an invalid extension for media type 'image/jpeg'

I consider it a bug that the file is skipped, since everyone agrees it's a jpeg:

✨ dash(pts/27):/my/pic% vipsheader phonecam/dn202012/20201209_194518\~2.heic
phonecam/dn202012/20201209_194518~2.heic: 2297x3172 uchar, 3 bands, srgb, jpegload

✨ dash(pts/27):/my/pic% file phonecam/dn202012/20201209_194518\~2.heic
phonecam/dn202012/20201209_194518~2.heic: JPEG image data, Exif standard: [TIFF image data, big-endian, direntries=4, height=0, datetime=2020:12:09 19:50:15, width=0], baseline, precision 8, 2297x3172, components 3

✨ dash(pts/27):/my/pic% ffprobe -hide_banner phonecam/dn202012/20201209_194518\~2.heic
Input #0, jpeg_pipe, from 'phonecam/dn202012/20201209_194518~2.heic':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 2297x3172 [SAR 1:1 DAR 2297:3172], 25 fps, 25 tbr, 25 tbn

[lastzero]

Since it is working just fine in our development environment and public demo, the "bug" may be caused by the Kubernetes runtime environment, file system permission issues, other permission or privilege issues, or the configuration options you are using? 🤔 You could e.g. comment out the PHOTOPRISM_DISABLE_ flags to see if that makes a difference.

If so, we can look further into this! However, my primary suspicion is that it's some kind of permission issue...

[lastzero]

You are right that /usr/lib/file/magic.mgc seems to be missing in our production images. I only found it in the development base image. However, our demo instance uses the "preview" production image, and it's still working, so this might not be the cause.

You could try manually installing this package with the apt command (i.e. run apt update && apt install libmagic-mgc) in a terminal to see if it makes a difference:

• https://packages.ubuntu.com/plucky/libmagic-mgc