proposal: image: add generic metadata support for jpeg, gif, png

[drswork]

Proposal

Currently the standard image libraries provide no way to read or write most of the metadata associated with an image. I'd like to change that so that the standard image loading libraries surface the metadata in images they read and allow code to annotate images with metadata.

Each image format has its own metadata. Additionally there are two common image metadata formats (eXif and XMP) which are each used in multiple image formats. (XMP is used in GIF, JPEG, and PNG files, while eXif is used in PNG and JPEG files) The formats themselves also have some interesting quirks which make excessive cleverness unwise. (There are three(!) separate key/value stores attached to PNG files, for example, and the format doesn't disallow a key being used multiple times in the different, or even the same, kv store)

With this in mind, the three guiding principles behind this proposal are:

1. Metadata should be minimally processed on read and write
2. It should be possible to read an image and immediately write it out in the same format, with the output having semantically identical metadata to the original.
3. It should be drop-in compatible with the existing image interfaces.

Point 1 means that PNG's three key/value stores will be exposed as three separate stores, and can't be a plain string/string map. It also means that code which wishes to pull information out of a PNG needs to know to look in both the XMP and eXif data.

Point 2 means we need to at least record any metadata that we don't currently know how to interpret so it can be written back out, and the images that are returned from decoding an image need to have its metadata silently attached.

Point 3 means we can't add methods to existing interfaces, since it's possible there are other packages that implement them. It also means we can't alter the calling conventions of RegisterFormat() in image.

This proposal should make it possible for third-party image libraries to participate in the new standard if they choose.

This proposal has some decisions which are either potentially controversial or potentially fragile. The ones I can think of are noted in the Potential Issues section

NOTE: "Metadata" means any data not directly related to the pixels in an image on disk. For example PPI and rotation, for example, are metadata, while alpha channel is not. (Rotation is arguably pixel data the way that alpha channel is, but since it massively affects the translation of disk pixels to image pixels we'll classify it as metadata for now)

Changes to current packages

Some of the standard packages will be changed. The changes are backwards-compatible.

image

• New registration function RegisterFormatExtended

  type Format struct {
    Name           string
    MagicString    string
    Decode         func(io.Reader) (Image, error)
    DecodeConfig   func(io.Reader) (Image, error)
    DecodeMetadata func(io.Reader) (*metadata.Metadata, error)
    GetMetadata    func(Image) (applies bool, *metadata.Metadata, error)
  }

  RegisterFormatExtended(*Format)

The new registration format includes a function to decode the metatada, as well as a format-specific function to return the parsed metadata for an image.

• RegisterFormat marked deprecated

It will still work, but internally will just construct a Format struct and call RegisterFormatExtended.

• GetMetadata added as registered function

This function will be passed an Image. Because there's no way to tell what type the image is, the metadata getting code must iterate through each format's get function. The function sets applies to false if it doesn't apply. Iteration short-circuits at the first function that says it does apply.

image/gif

• Metadata type, and supporting types, added

type Metadata struct {
  Comment     []string
  XMPMetadata *xmp.XMP
  AppBlocks   []AppBlock
}  

type AppBlock struct {
  AppID [8]byte
  AppAuthCode [3]byte
  BlockData []byte
}

• Metadata added to Options

type Options struct {
    NumColors int
    Quantizer draw.Quantizer
    Drawer    draw.Drawer
    Metadata  *Metadata // Added
}

image/png

• Metadata added. Note that for the initial proposal not all the chunks are proposed to be exported. Any chunk the library can't process is stored in the unexported field, which is only accessible to the library's reader.

type Metadata struct {
  KVText           []struct{string, string}
  CompressedKVText []struct{string, string}
  UnicodeKVText    []struct{string, string}
  ChangeTime       *time.Time
  XMPMetadata      *xmp.XMP
  ExifMetadata     *exif.Exif
  unexported       map[string][]byte // For currently uninterpreted chunks
}

• Encode function gets optional Options parameter:

func Encode(w io.Writer, m image.Image, o *Options) error

type Options {
  Metadata *Metadata
}

image/jpeg

• Metadata added

type Metadata struct {
  ExifMetadata *exif.Exif
  XMPMetadata  *xmp.XMP
  unexported []byte // For other uninterpreted data
}

• Metadata added to Options struct

type Options struct {
    Quality  int
    Metadata *Metadata
}

New packages

We add a new general-purpose metadata package to hold the metadata get function. We also add packages for the two shared metadata formats, XMP and eXif.

images/metadata

  interface Metadata {
    // Make sure the Metadata struct is actually valid, as each format has
    // its own quirks and restrictions.
    Validate() error
    // Here mostly to disambiguate from all the other interfaces with just
    // Validate in them.
    IsImageLibraryMetadata()
  }

  // Returns the filetype-specific metadata. Type-switch to tell what kind.
  Get(image.Image) (*Metadata, error)

Get will internally iterate through the registered GetMetadata functions until it finds one that matches.

The Validate method allows code to make sure the metadata struct is valid for the format before writing it out. This is important since many of the fields have quirks and limitations -- for example the key in png's key/value store must be ISO 8859-1 and between 1 and 79 characters.

images/metadata/exif

The exif package decodes metadata in eXif format.

  type Exif {
    // Handwave on the internals until the proposal is generally deemed OK
  }

  // Decode a binary blob, without any leading or trailing markers, as exif
  func Decode([]byte) (*Exif, error)

images/metadata/xmp

The xmp package decodes metadata in XMP format. (Or, if you prefer, ISO 16684-1:2019 as it's an official ISO standard)

  type XMP {
    // Handwave on the internals until the proposal is generally deemed OK
  }

  // Decode a binary blob, without leading or trailing markers, as xmp
  func Decode([]byte) (*XMP, error)

Potential Issues

The proposal has some decisions which are either potentially controversial or potentially fragile. They are noted

Iterating through metadata get functions is fragile

I can't think of a better way, without having the libraries tag the returned Image with a type, which the API doesn't support that I can see.

Adding an optional *Options parameter to png.Encode

This has to be done by actually adding a ...*Options parameter to the signature and then complaining if multiple ones are passed. This requires runtime signature validation, which isn't great

Adding *Metadata to existing Options parameters may cause unexpected behaviour

jpeg.Options has an integer quality parameter, so it can't actually be left off. We'd have to interpret 0 == default. gif.Options has an integer color count parameter, and we'd have to interpret 0 == default.

No access to uninterpreted metadata fields and chunks

This is intentional. It means that writers can't add their own chunks to the output, and it means that readers can't implement their own interpretation code. This is definitely inconvenient, but it means that when the additional chunks are interpreted by the standard library there won't potentially be two ways to access the same data.

Once the implementation is deemed complete then read/write access to these additional chunks shoud be added.

Optional metadata entites are stored as pointers to things.

This allows us to interpret nil pointers as elided entities and not require a HasFoo functions for each element. Not everyone is fond of this style.

No translation between image format metadata

If you read a jpeg in, then write it as a gif, the metadata isn't translated. This is intentional, since there's currently no clear 1:1 translation between file format metadata information.

[odeke-em]

Thank you for this proposal @drswork and welcome to the Go project!

We shall take a look at this perhaps during Go1.14, and I'll also kindly ping our image expert @nigeltao.

[drswork]

Sure. Just to be clear, I'm not asking anyone else to implement this -- I'm perfectly happy to do the work. I just want to make sure that the API and semantics are mostly correct before starting anything.

[smasher164]

This may also enable a general solution to #11420 and #20613, since color-space information is also metadata.

[drswork]

I fully expect to surface the color space information as metadata. Actually applying it to the image is well beyond my image manipulation skills so I'll leave that part to someone else.

[rsc]

/cc @nigeltao

[nigeltao]

Thanks for the detailed write-up. A few quick observations:

• ICCP metadata isn't mentioned?

• I don't understand how Format.GetMetadata is supposed to work. Is it for decoding, encoding, or both? If it's decoding, Decode will return you e.g. an *image.RGBA. How are you going to squeeze some metadata out of an *image.RGBA?

Similarly, you say:

Get will internally iterate through the registered GetMetadata functions until it finds one that matches.

but Get takes an image.Image. How does a png.Format or a jpeg.Format know that it applies to an *image.Gray?

• Is there no way to decode both the image and metadata in one pass, without having to rewind the io.Reader (which might require arbitrarily large buffering)? Yes, DecodeConfig doesn't let you do this either, but in hindsight, it might have been nice for DecodeConfig to return some sort of way to continue a partial decoding without having to rewind the io.Reader.

• You also say:

Each image format has its own metadata.

And they also implement metadata.Metadata (right?), but if I have a value of type metadata.Metadata, is there a good way to ask it for its Exif (if it has one)?

• Do you need a Metadata.Validate method? It's not like you're going to pass a jpeg.Metadata to png.Encode, right? It seems like you could have a (non-exported) validate(*Metadata) function in the png package, which only took a png.Metadata.

• You also say:

Adding an optional *Options parameter to png.Encode

That'll break any existing code that looks like:

var encode func(io.Writer, image.Image) error = png.Encode

---

In general, there's a lot of promising ideas here, but I also think that landing on the right design will take longer than e.g. 3 months of iterative feedback, and 3 months is the "feature development" part of the 6-monthly Go release cycle (https://github.com/golang/go/wiki/Go-Release-Cycle). I'm hesitant to add something in e.g. the 1.14 release cycle that, 9 months later, we realize could work much better with a different API design, but by then we're then stuck by backwards compatibility constraints.

For example, if an encoded image contains megabytes of some flavor of metadata, but all we want is an EXIF orientation, will we still have to decode all of the metadata into an in-memory representation up front? It's not exactly the same, but there might be some food for API thought from #5465 "The Request struct... has all its fields publicly exposed, most of which require memory allocation".

I'd advise forking the stdlib's image/* packages for now, and adding these new features there. Once we get some good iterations and usage on that one, then we can consider adding stdlib API that we're going to have to support for many years.

[drswork]

ICCP metadata isn't mentioned?

Yeah, at the time I wrote the proposal I wasn't aware of any of the details of ICCP. Now that I am (I gave the spec a scan over the weekend after writing a bunch of png chunk decoders) I admit I'm inclined to punt and surface it as a binary blob, leaving it to someone else to decode.

While on the one hand it's not great since it isn't decoded, on the other at least the data's surfaced so it can be decoded, so it's a step up from the current state of things.

I don't understand how Format.GetMetadata is supposed to work. Is it for decoding, encoding, or both? If it's decoding, Decode will return you e.g. an *image.RGBA. How are you going to squeeze some metadata out of an *image.RGBA?

and

but Get takes an image.Image. How does a png.Format or a jpeg.Format know that it applies to an *image.Gray?

This bit of API awkwardness is because image.Image is an interface, so I can't add anything to it without breaking things. (I would've preferred to just hang a Metadata field off image.Image, but alas that's not an option)

My assumption here is that image/{jpeg,gif,png} would hang a real metadata field off whatever concrete type they returned, and when you called .GetMetadata it would query the thing it was handed to see if it had 's metadata hanging off it. That was something I figured was an implementation detail and was handwaving past to focus on the user-facing API, but I should've been a bit more concrete here.

Each image format has its own metadata.

And they also implement metadata.Metadata (right?), but if I have a value of type metadata.Metadata, is there a good way to ask it for its Exif (if it has one)?

Because the three image formats have such... interestingly divergent ideas of what metadata to store and how to store it, I hadn't planned on having the different metadata types be particularly interchangeable. In concrete terms it means that if you're handed a metadata.Metadata there's very little you can do with it without a type switch and casting it to a {png,gif,jpeg}.Metadata.

In the few rare cases where the different formats hold the same data then the fields that hold it should be named the same, but that's a matter of convention more than anything. (Also I'm not sure it's a good idea anyway, since the way the data is stored causes issues. XMP metadata is in both GIF and PNG files, but PNG stores it as a text blob with a particular key in its key/value store and a good case could be made that pulling it out and decoding it hides an essential, if very annoying, property of the data)

Do you need a Metadata.Validate method? It's not like you're going to pass a jpeg.Metadata to png.Encode, right? It seems like you could have a (non-exported) validate(*Metadata) function in the png package, which only took a png.Metadata.

Oh, yes, this is definitely required. It's not to validate that you're handing jpeg metadata to the png encoder, but more that the information in the metadata is valid in the first place. The specs have some firm(ish) ideas about data validation -- for example, in png files it's bogus to have both iCCP and sRGB data, the keys in its key/value store have to be latin-1, and they can't be more than 79 characters. It'd be lovely if each type in the metadata structure was strictly specified such that you can't possibly create bogus metadata, but that seems infeasible.

I am also 100% sure every single data validation requirement (well, maybe not the png key length, though I wouldn't be surprised if that's a good attack vector for some decoders) has been hilariously violated by software out in the wild, and as such I'd actually argue that encode should write out bogus metadata when handed it, since that metadata may well have come from an existing image. I don't think it's a good idea to not be able to write out data we just read (opinions differ here, of course), so encode needs to not complain

Is there no way to decode both the image and metadata in one pass, without having to rewind the io.Reader (which might require arbitrarily large buffering)? Yes, DecodeConfig doesn't let you do this either, but in hindsight, it might have been nice for DecodeConfig to return some sort of way to continue a partial decoding without having to rewind the io.Reader.

It's actually much, much easier to decode the metadata and image in a single pass. Honestly DecodeMetadata is in there mostly because DecodeConfig exists. I assumed there were cases where code only wanted the configuration info and not fully decode the image, and I'm 100% good with dropping it. (I can see a few cases where only wanting the metadata would be useful, but I'm good with requiring folks to decode the image for now and consider adding the API in later)

Adding an optional *Options parameter to png.Encode

That'll break any existing code that looks like:

var encode func(io.Writer, image.Image) error = png.Encode

Bah. That was the least bad option I could come up with. I'm open to alternatives -- adding another encode function, the way image/gif has done, was my Plan B, with some kind of metadata attachment function being the backup plan for that.

In general, there's a lot of promising ideas here, but I also think that landing on the right design will take longer than e.g. 3 months of iterative feedback, and 3 months is the "feature development" part of the 6-monthly Go release cycle (https://github.com/golang/go/wiki/Go-Release-Cycle). I'm hesitant to add something in e.g. the 1.14 release cycle that, 9 months later, we realize could work much better with a different API design, but by then we're then stuck by backwards compatibility constraints.

I fully expect the result will be sub-optimal in some way, sure. APIs always are. (Heck, the current API, with each format having different Encode function signatures, is a clear example there) That's part of the reason why I wrote this up for discussion before starting in on coding, to try and work out a reasonable API.

I have the free time to work on this, though I realize it may not be my time constraints that are the issue for the API design.

For example, if an encoded image contains megabytes of some flavor of metadata, but all we want is an EXIF orientation, will we still have to decode all of the metadata into an in-memory representation up front? It's not exactly the same, but there might be some food for API thought from #5465 "The Request struct... has all its fields publicly exposed, most of which require memory allocation".

I'm sympathetic to lazy decoding, sure. In regards to #5465 is the concern the amount of memory allocated or the number of objects created? Having function calls intermediate access to metadata elements isn't without cost either, though I can certainly see deferring, say, XMP decoding until something actually fetches the XMP data.

This does kinda argue for a DecodeMetadata() call of some sort, FWIW. :)

I'd advise forking the stdlib's image/* packages for now, and adding these new features there. Once we get some good iterations and usage on that one, then we can consider adding stdlib API that we're going to have to support for many years.

I'm unclear here -- are you suggesting making a branch and flddling with image/ in that branch, or actually releasing cloned version of the image packages under a different name and see how that goes?

[drswork]

Also, semi-separately, is there a better way to iterate on the API than github issue comments? At work I'd spin up a google doc for this but that doesn't seem viable here. (I suppose I still can, though that shuts most folks out)