HEIF format support in OpenCV

[zzq889]

This is a feature request, not a bug.

Detailed description

https://github.com/strukturag/libheif

I'd like to access HEIF files directly without conversion. Is it possible to add a build flag by using libheif?

[clausmichele]

I would also point to https://github.com/nokiatech/heif for more details on the format and the original source code of the reader/writer.

[cgrabmann]

Hi @alalek
Are there any plans on adding support soon?

By now HEIC is the default image format on new iOS versions, Android 10 added support for HEIC, Windows and Mac also do support the format.

It would be great if we could just load them in opencv directly as well.

[GitUser2000]

Hi @alalek,
I would like to suggest to raise the priority, it's iOS default format and supported both by Windows and Mac.

[asmorkalov]

Some research on HEIF format and support options

General

HEIF is not single file format like JPEG or PNG, but container like AVI for video and may contain single image, sequence of images, images derived from other images like expose stacks and extra data like sound, text, meta information. The format is designed for several codec types, including JPEG, H264, H265, AV1 and the list can be extended in future. HEIC just HEIF branch with default codec HEVC, ITU-T H.265.

Existing solutions

1. libheif (https://github.com/strukturag/libheif/). GNU LGPL v3 HEIF implementation that can utilize h265 codec from the same author (hosted as dedicated project on Github) and dav1d for AVIF branch (AV1 codec). Concerns: LGPL v3 license and patents for codec part. It introduces issues with vendering to OpenCV repository for cross build and official binaries distribution.

2. HEIF solution from Nokia (https://github.com/nokiatech/heif). The project covered by Nokia specific license for non-commercial use. Codecs are not included. Concerns: no codec implementation, license.

[asmorkalov]

Nokia HEIF conformance data: https://github.com/nokiatech/heif_conformance

[asmorkalov]

Related work on HEIF support in FFmpeg: https://trac.ffmpeg.org/ticket/6521

[Wansh619]

If this issue still open can you assign it to me?

[asmorkalov]

Yes, sure.

[Wansh619]

@asmorkalov Can you provide me some resources as i am a absolute beginner in open source and guide me what I have to learn for working on this issue?

[lyksdu]

Some research on HEIF format and support options

General

HEIF is not single file format like JPEG or PNG, but container like AVI for video and may contain single image, sequence of images, images derived from other images like expose stacks and extra data like sound, text, meta information. The format is designed for several codec types, including JPEG, H264, H265, AV1 and the list can be extended in future. HEIC just HEIF branch with default codec HEVC, ITU-T H.265.

Existing solutions

1. libheif (https://github.com/strukturag/libheif/). GNU LGPL v3 HEIF implementation that can utilize h265 codec from the same author (hosted as dedicated project on Github) and dav1d for AVIF branch (AV1 codec). Concerns: LGPL v3 license and patents for codec part. It introduces issues with vendering to OpenCV repository for cross build and official binaries distribution.
2. HEIF solution from Nokia (https://github.com/nokiatech/heif). The project covered by Nokia specific license for non-commercial use. Codecs are not included. Concerns: no codec implementation, license.

Can we just use libheif(with LGPL license)? I wonder if the two Licenses conflict. If conflict, we should search for another solution

[lyksdu]

It seems that only link libheif dynamic libraries is OK under the LGPL license.
PS:
1, video codecs' licenses are always annoying
2, how OpenCV img_codecs deal with webp/heic which contains multiple images

[asmorkalov]

Multi-frame images are not supported by imread/imwrite for now. OpenCV reads single frame only. cv::VideoCapture should be used/implemented for sequences.

[damonmaria]

Just to add this would be very useful for our use case. We deal with large numbers of big images (100MB PNGs) that we need to store lossless. WebP doesn't do well compressing these (only 10% improvement over PNG but takes 10x longer to encode) whereas HEIC can store them half the size (yes, lossless) while not being much slower than PNG encoding.

[KartikAiyer]

I'm trying to use the HEIF library. I want to convert the primary image in the heif file into a cv::Mat. I'm not exactly sure how to go about doing that. I'm using the decode_image method on the primary image with a colorspace of RGB and chroma setting of interleaved RGB. Not sure how to map the resulting image into a cv::Mat

[alanwilter]

Is it a dead end?

[jeffheaton]

This would be very useful.

[roticv]

HEIF is a container format -- HEIC is using HEIF with HEVC/H265 while AVIF is using HEIF with AV1. AVIF is already implemented in #23596 and so it sounds like the remaining work is to implement HEIC support (i.e. specifically HEIF with HEVC/H265).

The problem is that there doesn't look like there is any non-LGPL open source library for HEVC/H265 decoding. The options that I have found at libdeh265 and ffmpeg.

It does sound like using ffmpeg to decode HEIC makes sense especially since https://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=a0821345eb31b727d93c9c3ed7d74d2774c73afa (where some heic support was added to ffmpeg).

I can help to make a patch if there are some agreement about what's the best way forward :)