HEIC: Converted JPG has wrong orientation

[sionicion]

iPhone 12 mini running iOS 14.4 and Nextcloud 3.3.0.5 Beta (auto-uploads pictures/videos to originals folder)

Took a motion picture, first noticed Photoprism needs manual reindex after I upload new media to originals folder from Nextcloud, still have to figure that out. Then I noticed my most recent picture was a blurred image, turns out Photoprism had made the primary image a preview JPG of the mov file, learned how to change the primary to the converted JPG of the HEIC file.

But now I noticed the converted JPG of the HEIC is rotated, I downloaded the HEIC from Photoprism and that is not rotated, so not sure how it got rotated during the conversion. I think for now I might just experiment with using the compatibility iOS camera option so Photoprism won't have to convert to JPG in the first place.

[inthreedee]

Then I noticed my most recent picture was a blurred image, turns out Photoprism had made the primary image a preview JPG of the mov file

For that one part of your issue, see here:
#926

I can’t speak to the rotation issue though.

[lastzero]

See #1052 (comment)

We need orientation tags of all related files to debug this. The HEIC converter is external software, so there might or might not be issues. Going to improve primary JPEGs selection based on quality & size, see #926.

[lastzero]

If you need JPEGs, just use them straight away so that you don't need extra storage and transcoding. Most software doesn't support HEIC yet or just with tradeoffs like in this case.

[lastzero]

I'd be great to have test images for debugging... you're welcome to mail them to hello@photoprism.app

Please let us know if we have permission to add them to our public test suite to avoid regressions 👍 No problem if not, we'll help you anyway.

[lastzero]

As it turns out, heif-convert (the reference implementation) has a bug that rotates the image but keeps the orientation in the Exif header - thus all images get rotated twice UNLESS Exif is not supported. Obviously we support Exif.

Apparently HEIF support may be compiled in imagemagick, but their compiled 7.x version doesn't include it. Ubuntu still ships 6.x. Fail.

We then tried to compile the latest version from https://github.com/strukturag/libheif on our own:

$ heif-convert 20210131_IMG_8219.heic 20210131_IMG_8219.heic.jpeg
File contains 1 images
Could not decode image: 0: Unsupported feature: Unsupported codec

The whole point of the tool is to read this file format. Unsupported Codec? Maybe JPEG? It's pretty new and probably needs extra work 👯

[lastzero]

Once this dependency is fixed, it should work just fine. It will also work when installed directly on a Mac as we use Apple's sips command there, see https://ss64.com/osx/sips.html.

The refactoring lays the foundation for further improvements like #464.

[lastzero]

Found another NodeJS-based converter, but it was 12 times slower and didn't preserve any Exif data at all.

Ended up implementing a wrapper for heif-convert that sets the orientation of all JPEG output files to 1. Part of today's update.

[melyux]

I'm still having this issue where the .heic files in portrait orientation are always rotated 90 degrees into landscape when converted to JPEG for preview.

[lastzero]

Could you provide us with a test file and specify which version of PhotoPrism you are using? Note that we offer a number of different Docker images, for example based on Ubuntu or Debian. Also note that previously created JPEGs will not be replaced after upgrading to a (hopefully) fixed version, so you will need to recreate them.

[melyux]

Sure, is there an email I can send a sample file to? I saw a couple floating around.

I'm using the photoprism/photoprism:latest image, and I first installed it today, so it's a brand new library and installation. The only change I made is turn Dynamic Previews on, but there doesn't seem to be a difference before/after turning that on/off.

[lastzero]

Yes, it is described on our website: https://photoprism.app/contact

Sample image and video files can be sent to samples@photoprism.app. Please include the format and GitHub issue number or other helpful reference in the subject line, and let us know if we have permission to upload your files to dl.photoprism.app/samples for future use.

It might also be possible to attach files to comments on GitHub Issues, or you could add a link to a file uploaded to your own server and service like Google Drive.

[melyux]

Sent an email with two original HEIC images, a converted JPEG version of one of them, and screenshots of the orientations shown in the PhotoPrism UI. Hope that helps

[lastzero]

Unfortunately, we could not reproduce this. Note that even if you use an image with the :latest tag, Docker does not automatically download new images for you and the current image can theoretically be out of date.

You can perform a manual upgrade as described in our docs at https://docs.photoprism.app/getting-started/updates/#docker-compose.

Some tools like Portainer also set a custom PATH variable that can change the behavior of our Docker images so that the wrong converter is used (we use a wrapper script that must be called instead of the binary in /usr/bin).

[joachimtingvold]

I had this issue as well. Can confirm that updating to latest :preview (for me that was 220919-cc8bab446) fixes the issue. I'm running default image: photoprism/photoprism:preview, which seems to be running Ubuntu Jammy. It's running libheif 1.12.0 at least.

After the upgrade, I did a photoprism convert --force 2022/09 (just to speed it up, as only the last few days had issues), which seems to have produced proper sidecar files (where the orientation attributes in the EXIF data is correct). Still waiting for the thumbs to re-create, but so far so good.

Could we maybe get a way to limit photoprism thumbs to a certain directory? So we could do photoprism thumbs --force 2022/09 to limit it to the 2022/09 path?

[lastzero]

Generally yes, that might be possible. It would help a lot if you could test heif-convert on Debian 11 and 12. That's because Ubuntu will eventually get the same version as Debian 12, or even newer. Meaning if there is a problem, you will have the same issue soon again.

As thumbs get generated based on hashes, I don't think the force option is needed which should accelerate the update significantly.

[joachimtingvold]

Converted sidecar images (from original HEIC) was re-created when using photoprism convert --force 2022/09 (confirmed by downloading the raw sidecar image). Followed up with a photoprism thumbs (without --force flag), and it did not create new thumbnails for the pictures that had their sidecar re-created. Hence the thumbnails (including the preview images) are still wrongly orientated.

[lastzero]

Did you update the index in between? Without looking at the code, I can't tell you where the hashes come from but it seems entirely possible that these are retrieved from the index database and thus the new preview hashes won't get recognized without indexing.

[joachimtingvold]

@lastzero, I did a photoprism index --cleanup followed by photoprism thumbs. It did regenerate some thumbnails, but not the ones that are wrongly orientated. Any other suggestions?

[lastzero]

@joachimtingvold The latest heif-convert version should finally fix all these issues:

• HEIF: heif-convert must be updated to prevent empty JPEGs #2726

It may take a while before a new stable version is available for you, as it also includes many other changes that need to be tested.

[joachimtingvold]

@lastzero , excellent. Can confirm that 221027-6a33ae2e9 followed by photoprism index --cleanup + photoprism convert -f + photoprism thumbs -f seems to have fixed the issues.

[lastzero]

@joachimtingvold As that looks like a preview version, you should upgrade to today's stable version after reading the full release notes for Build 221102-905925b4d 👍

[martin-g-it]

Hi, it appears this fix is not applied when running PhotoPrism without Docker (ie. as Proxmox LXC). Is that correct?
Running Build 230106-e58fee0fb

[graciousgrey]

@martin-g-it we do not offer LXC images for PhotoPrism yet and can't help much with third party installations. A potential issue could be that your installation uses a different version of the library that is used for heif conversion.

[martin-g-it]

Hi @graciousgrey, it's not really a third party installation I build up PhotoPrism by;
git clone
https://github.com/photoprism/photoprism.git &>/dev/null
cd photoprism
git checkout release &>/dev/null
make all &>/dev/null
./scripts/build.sh prod /opt/photoprism/bin/photoprism &>/dev/null
cp -r assets/ /opt/photoprism/ &>/dev/null

I expect the conversion of heic > jpg being done by:
https://github.com/photoprism/photoprism/blob/develop/scripts/dist/heif-convert.sh

Or did I miss any other place where I should look?

[graciousgrey]

@martin-g-it

it's not really a third party installation

Building from source results in custom builds.

Make sure you follow all steps in the Dockerfile and install the same build tools and dependencies, for example the exact same version of libheif, as those tiny little differences are what makes your build custom. It's not all in the source we provide.

[martin-g-it]

The libheif is all OK, it's just the orientation that is incorrect when .HEIC files were converted to .jpg.

I expect because the convert_jpeg.go converts using heif-convert without the exif correction (like done in the Docker script). https://github.com/photoprism/photoprism/blob/727b3eb55255e79fdff64943648ced1f8f3fae42/internal/photoprism/convert_jpeg.go line 171:
result = append(result, exec.Command(c.conf.HeifConvertBin(), "-q", c.conf.JpegQuality().String(), f.FileName(), jpegName))
I ended up making a wrapper around heif-convert (copying /usr/bin/heif-convert > /usr/bin/heif-convert-orig and set /usr/bin/heif-convert with below script):

#!/bin/bash
heif-convert-orig "$@"
exiftool -overwrite_original -P -n  -Orientation=1 "${@: -1}"

This resulted in .HEIC files being converted/imported correctly.. :-)

Small disadvantage though; while importing .HEIC files; the sidecar folder size is growing due to the converted .HEIC.jpg files.
Luckily you can use config option PHOTOPRISM_SIDECAR_PATH to place the sidecar files on eg. a mounted NAS location.

Another option (or workaround) is to convert the .HEIC files upfront before importing into PhotoPrism library;
Convert HEIC to jpg:
for i in *.HEIC; do heif-convert "$i" "${i%.*}.heic.jpg"; done
Set orientation correct:
for i in *.heic.jpg; do exiftool -overwrite_original -P -n '-ModifyDate<FileModifyDate' -Orientation=1 $i; done

[lastzero]

What do you mean by "libheif is all OK"? Is it the same version we use or the one that e.g. comes with the Linux distribution you use? This is the correct version with bug fixes:

👉 https://dl.photoprism.app/dist/libheif/

[martin-g-it]

Is it the same version we use or the one that e.g. comes with the Linux distribution you use?

Maybe thats the thing.. I took the package installed via:
apt install -y libheif-examples exiftool

Running Debian GNU/Linux 11 (bullseye)

This one works fine converting .HEIC into .jpg. It's just the orientation that is wrong (which is how I ended up in this thread) and which can be corrected by the exiftool command.

I tried the version you've referred, but ran into libjpeg.so.8 is missing errors..

[lastzero]

It's the metadata and orientation that we've fixed (pulled from upstream and built, to be exact).

[lastzero]

Also, Bullseye is outdated compared to Ubuntu Jammy plus Debian doesn't ship libjpeg, only libjpeg-turbo, so you'd have to build it yourself. That's the reason we use Docker - it's impossible to get it right otherwise.

[martin-g-it]

Right, makes sense now... Thanks for the swift reply! Will try to upgrade to Bookworm or run a fresh install on that release to see whether this will resolve.

[lastzero]

The version we are using, and that has proven to be bug free, is v1.13.0. As far as I remember, all versions that shipped with Debian had problems. Also, libjpeg-turbo, which ships with Debian, does not support ICC profiles in combination with libheif, so libheif had to add code to handle this. It is not known if this code works without any bugs and does the same as our solution.

[lapo-luchini]

Just as an extra data point, running 221118-e58fee0 on FreeBSD (compiled using this port) and using system libheif-1.14.0_1 seems to still have the bug, even though it is marked as solved in libheif's 1.14.0 release log.

[lapo-luchini]

I meant to go deeper in this (I still have wrong rotation out of HEIF files) and I wonder if this is right:

# original HEIF
File Modification Date/Time     : 2023:03:24 21:12:52+01:00
Orientation                     : 0
Components Configuration        : 1 2 3 0
Exposure Compensation           : 0
HEVC Configuration Version      : 1
Min Spatial Segmentation IDC    : 0
Rotation                        : 270
# generated JPEG
File Modification Date/Time     : 2023:03:26 16:43:02+02:00
Orientation                     : 1
Components Configuration        : 1 2 3 0
Exposure Compensation           : 0

Orientation 0 + rotation 270 ⇒ orientation 1.
I'm not sure about these values, but it looks correct to me…?

Still… (using libheif-1.15.1)

[lastzero]

No, that's the problem. Exif Orientation and QuickTime rotation add up, causing the wrong orientation in the end. You can either update libheif or use a wrapper script to fix this. Both can be found in our code base.

[lapo-luchini]

But the photo shows like this, I think "rotation 270" is right; orientation 0 is unknown, assuming 1 as default adding 270 rotation I should get an 8 in output, right?

(for some reason my Xiaomi Mi 11T seems to only change quicktime rotation, not orientation in HEICs — I guess this makes my issue totally different from the rest of this issue)
I'm on FreeBSD, not Linux, so I prefer to stick to binary libheif as found in official packages, but that's not even the problem probably. I will try and work around it it with a wrapper, thanks!

[lastzero]

Note that browsers show JPEG images and not native QuickTime or HEIC. So the Exif Orientation field should be used. JPEG readers typically don't know much about video metadata.

[lastzero]

FreeBSD ports should be more up-to-date than most Linux distributions. The libheif version that finally worked for us was released last year, although newer versions could theoretically have a regression. Unfortunately, we have very little time to focus on this given all the other work.

[lapo-luchini]

Just in case anyone with Xiaomi phone is interested: this is the produced metadata with the 4 possible orientation of the phone:

% exiftool -m -n -p '$filename $imagewidth $imageheight Orientation: $orientation Rotation: $rotation' *.HEIC
IMG_20230329_090439.HEIC 3000 4000 Orientation: 0 Rotation: 270
IMG_20230329_090443.HEIC 4000 3000 Orientation: 1 Rotation:
IMG_20230329_090446.HEIC 3000 4000 Orientation: 0 Rotation: 90
IMG_20230329_090450.HEIC 4000 3000 Orientation: 0 Rotation: 180

Image is physically rotated, the already applied rotation is shown in QuickTime data (only for HEIC, for JPG it is not done), and EXIT orientation is either 1 (normal) or 0 (undefined) for all 3 other rotations.
I think the best approach is to overwrite all the metadata madness like this (as the image is pre-rotated, it shows ok):

exiftool -r -if '$model eq "21081111RG"' -exif:orientation#=1 -quicktime:rotation=0 .

My only regret is that this way the "original rotation" is lost, but I can't keep that unless I de-rotate the actual image too, and I don't want to touch the image itself, only metadata. I'll maybe add it as a custom EXIF meta, just for the sake of it.

[lastzero]

Glad you got this sorted out! Keep in mind that the behavior changes depending on the libheif version.