Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for ISOBMFF Files (AVIF, HEIF, CR3) #1066

Closed
cryptomilk opened this issue Nov 23, 2019 · 28 comments
Closed

Add support for ISOBMFF Files (AVIF, HEIF, CR3) #1066

cryptomilk opened this issue Nov 23, 2019 · 28 comments
Assignees
Labels
request
Milestone

Comments

@cryptomilk
Copy link
Collaborator

@cryptomilk cryptomilk commented Nov 23, 2019

Describe the bug

Hello!

I've started to add support for the AVIF file format [2] in darktable using libavif [1]. I've worked with the maintainer so that it works nicely with cmake.

However I'm not able to load the image file in darktable, because exiv2 is not able to read the exif data and then fails. You can find my darktable code here [3].

If you need help or want to do some pair programming, let me know. It isn't hard to just read the exif data blob.

[1] https://github.com/AOMediaCodec/av1-avif
[2] https://github.com/AOMediaCodec/libavif/
[3] https://github.com/cryptomilk/darktable/tree/master-avif

@cryptomilk cryptomilk added bug request and removed bug labels Nov 23, 2019
@piponazo piponazo self-assigned this Nov 23, 2019
@cryptomilk
Copy link
Collaborator Author

@cryptomilk cryptomilk commented Nov 26, 2019

I've finished the implementation and can export and import AVIF images now!

@cryptomilk
Copy link
Collaborator Author

@cryptomilk cryptomilk commented Dec 30, 2019

AVIF support landed in master and will be available with 3.0.1.

@boardhead
Copy link
Collaborator

@boardhead boardhead commented Dec 31, 2019

Perhaps I'm missing something (likely), but the latest update I see to master was 20 days ago.

@cryptomilk
Copy link
Collaborator Author

@cryptomilk cryptomilk commented Dec 31, 2019

I'm talking about darktable master -> https://github.com/darktable-org/darktable

@boardhead
Copy link
Collaborator

@boardhead boardhead commented Jan 1, 2020

Ah

@novomesk
Copy link

@novomesk novomesk commented Feb 21, 2020

I have a question for exiv2 developers:
Would you allow another library as dependency to support this feature?
Or the code must be completely independent, without additional libraries?
If external dependency is possible, then read-only support should be trivial. After calling avifDecoderParse() in libavif, size and metadata are available already (pixel data were not decompressed yet).

@D4N
Copy link
Member

@D4N D4N commented Feb 22, 2020

I have a question for exiv2 developers:
Would you allow another library as dependency to support this feature?

I have no objections to additional dependencies, provided that they are sane to build, relatively stable and don't drag us into a licensing nightmare like HEIF does.

Or the code must be completely independent, without additional libraries?

No, we have already some dependencies for reading well known formats.

If external dependency is possible, then read-only support should be trivial. After calling avifDecoderParse() in libavif, size and metadata are available already (pixel data were not decompressed yet).

Sounds good. So how is the licensing & patent situation around AVIF?

@cryptomilk
Copy link
Collaborator Author

@cryptomilk cryptomilk commented Feb 22, 2020

AVIF is a royalty-free codec. The patents are free to use. I've added libavif to openSUSE and currently working to get it into Fedora.

@boardhead
Copy link
Collaborator

@boardhead boardhead commented Feb 23, 2020

Note that AVIF is nearly identical to HEIF as far as structure and metadata is concerned.

@novomesk
Copy link

@novomesk novomesk commented Feb 23, 2020

Here is an example how to print EXIF and XMP metadata from .AVIF file using libavif:
avif_metadata.cpp.txt

How to compile:
g++ avif_metadata.cpp -lavif -lexiv2 -o avif_metadata

How to use (example):
./avif_metadata Tomsk_with_thumbnails.avif

Example of the output:

[Tomsk_with_thumbnails.avif],   1280x720        8 bit   images: 1
EXIF data length: 58 bytes
Exif.Image.Copyright                         0x8298 Ascii      31  https://www.elecard.com/videos
No XMP metadata in Tomsk_with_thumbnails.avif

Test file(s) download:
https://github.com/AOMediaCodec/av1-avif/tree/master/testFiles/Microsoft

My plug-in for GIMP 2.99 is able to save XMP metadata too. Windows installer here: https://www.reddit.com/r/GIMP/comments/f6ro3q/gimp_2991_for_windows_64bit_with_avif_support/

@clanmills clanmills assigned clanmills and unassigned piponazo Mar 27, 2020
@clanmills clanmills changed the title [FEATURE] Please add support for AVIF file format Add support for ISOBMFF Files (AVIF, HEIF, CR3) Mar 27, 2020
@clanmills clanmills added this to the v0.27.3 milestone Mar 27, 2020
@clanmills
Copy link
Collaborator

@clanmills clanmills commented Mar 27, 2020

I'll take ownership of this for Exiv2 v0.27.3

@D4N has expressed concern about patent infringement if we use a library to read heif files. We can totally avoid patent infringement by writing our own code to read/write the metadata. This has be done for many file formats and typically requires about 500 lines of code.

For Exiv2 v0.28 and later, the team may prefer to use a library.

@cryptomilk
Copy link
Collaborator Author

@cryptomilk cryptomilk commented Mar 27, 2020

I think it is a good idea to not require a library for that!

@D4N
Copy link
Member

@D4N D4N commented Mar 28, 2020

We can totally avoid patent infringement by writing our own code to read/write the metadata

That's unfortunately not how patents work.

@clanmills
Copy link
Collaborator

@clanmills clanmills commented Mar 28, 2020

You cannot have a patent on an ISO standard. We are reading an ISO standard file. When I have the code ready, I will get legal advice from the person who advised us about the commercial license.

@phako
Copy link
Contributor

@phako phako commented Apr 6, 2020

You cannot have a patent on an ISO standard. We are reading an ISO standard file.

I think ISO/IEC 13818 (MPEG-2) and ISO/IEC 14496-10 (H.264/MPEG4-10) tell a different story.

@clanmills
Copy link
Collaborator

@clanmills clanmills commented Apr 6, 2020

Is it impossible to support HEIF? Everybody is being so negative. Is there a solution, or should I give up?

@novomesk
Copy link

@novomesk novomesk commented Apr 6, 2020

I believe you can parse the ISOBMFF container and find location of the metadata but you should not decompress the h265-hevc data.
If someone is super scared of renamed QuickTime's container from 2001, it is known fact that Apple joined Alliance of Open Media, and the AOM members granted usage of their patents (in AOM related software).
Many people are able to find metadata using sequential search (I usually found them at the end of AVIF files I worked with)

@clanmills
Copy link
Collaborator

@clanmills clanmills commented Apr 6, 2020

Thank You @novomesk. You are confirming my assertion. It is legal to parse an ISOBMFF container which is a linked list of self contained boxes.

The boxes which contain metadata are written in their native format. XMP is written in XML, Exif in Tiff, IPTC and ICC are binary blocks.

Exiv2 has no code to deal with pixels. Preview images in metadata are normally stored as embedded JPEGs (which do not have embedded metadata). Exiv2 simply does a binary copy of those previews without any further parsing.

I don't recall hearing of AOM. I believe ISO will never ratify a standard which includes technology which must be licensed from the owner (even if the owner requires no payment). I believe you are saying that AOM put the QuickTime file format into the public domain and ISOBMFF is the ISO Standardization of that format.

It's entirely possible for an ISOBMFF format to include boxes which have data encoded in a proprietary format such a h265-hevc. Decoding such a box may be a patent infringement, and Exiv2 will have no such code. The boxes of interest to Exiv2 are encoded using public standards.

My plan is to implement my own ISOBMFF parser to read/write metadata boxes. All other boxes will be skipped on reading, and binary copied to output when writing.

When I have the code written and tested, I will get a professional legal opinion about what I intend to publish.

I would very much appreciate help with testing from Team Exiv2 and the wider open-source community.

@phako
Copy link
Contributor

@phako phako commented Apr 7, 2020

I'm not saying to not support it. I'm saying make it completely disablable so that those who care about not stepping into mucky waters or care about not supporting any non-free formats can safely remove the support.

@clanmills
Copy link
Collaborator

@clanmills clanmills commented Apr 7, 2020

This project is cancelled.

@clanmills
Copy link
Collaborator

@clanmills clanmills commented Apr 7, 2020

This project is cancelled until somebody sorts out the legal challenges. I have expressed my point of view about this and been met with hostility. This hostility has not been backed by evidence or reasoned argument. I have offered to get professional advice before publishing. Exiv2 v0.27.3 is on track for RC1 and will not have support for ISOBMFF.

@AurelC2G
Copy link

@AurelC2G AurelC2G commented Apr 7, 2020

I'm really confused by this: HEIF itself is just a container, you are fine to read the ISOBMFF boxes (such as the ones containing exif metadata) as long as you don't decode the encoded image bytes.

AVIF is just a HEIF file containing images encoded using the AV1 codec (as opposed to HEIC, which are HEIF files containing images encoded with HEVC). Decoding metadata from either would be fine, the problems only start once you decode the HEVC data itself (av1, on the other hand, is open and there's no problems at all for AVIF: https://aomedia.org/the-alliance-for-open-media-statement/)

@1div0
Copy link
Collaborator

@1div0 1div0 commented Apr 7, 2020

Depeche Mode: Wrong!

@cryptomilk
Copy link
Collaborator Author

@cryptomilk cryptomilk commented Apr 7, 2020

Please stop spreading FUD!

@johnny-bit
Copy link

@johnny-bit johnny-bit commented Jun 1, 2020

In what jurisdiction is the legal advice needed? I can personally pay for legal advice regarding ISOBMFF files it that would mean possibility to read data from them. I have very little patience for FUD spreading, patent trolling etc.

@clanmills
Copy link
Collaborator

@clanmills clanmills commented Jun 1, 2020

I also offered to get legal advice. I will never work on ISOBMFF.

I'll be 70 in January. I'm working on a book Image Metadata and Exiv2 Architecture. When that is finished, I'm done with Exiv2. I am willing be the release engineer for one more release. Either Exiv2 v0.28 or Exiv2 v0.27.4.

@clanmills
Copy link
Collaborator

@clanmills clanmills commented Jun 17, 2020

Here's the summary of the Sunday's meeting on Zoom. #318 (comment)

@ruxandra-valcu
Copy link

@ruxandra-valcu ruxandra-valcu commented Jun 18, 2020

Tried a bit of Karen-ness in Canon support, let's see if it works.

Hello!
I'm considering upgrading to an EOS R6 (from my current 800D) but I won't be able to use this, or any other camera with CR3 format, until my favourite photo editor, Darktable, supports it on Linux. Currently, progress on this support is stalled by people being worried by legal issues (see darktable-org/rawspeed#121 , and #1066 for the legal issues in the decoder library), I understand you may not have the resources to create an open SDK for the CR3 format, but could you at least confirm to these guys that it's fine to parse the CR3 format as they're parsing CR2 or anything else? As things stay now, people aren't buying Canon mirrorless because of lack of full Linux support.

P.S: I'm aware of the Adobe DNG converter but that introduces unnacceptable brittleness in my workflow and is therefore not a workable solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
request
Projects
None yet
Development

No branches or pull requests