Optional 'progressive' download frame

[jakearchibald]

The lack of progressive rendering seems like a missing feature in AVIF. Sure, it's possible to do a top-to-bottom decode, but that isn't as good as multi-pass, and it sounds like the implementation is progressive.

Could another image be added to the start file, marked up in some way to indicate that it's optional for the browser to render it? Although it should render it if it doesn't have the rest of the file yet.

This part could be a different resolution to the main image, be encoded at a different quality, or be visually different (eg blurred).

For example, the cat image I used in the demo is 96 kB for full quality, but an acceptable preview can be done in 5 kB by halving the resolution and turning the quantizers to max.

I don't think this is the same as a thumbnail as a blurry version or a low-quality version would be unacceptable there.

[wellington1993]

I think that idea could be an essential step to make that format popular in the Web usage.
And could be "optional", don't affecting other usages of the format.

Nice suggestion! @jakearchibald

[wantehchang]

Encoding AVIF images for progressive rendering is being discussed in the AOM Still Picture Subgroup. It seems that both the HEIF container format and the AV1 bitstream format have the necessary features (HEIF's multi-layer images and AV1's spatial layers) to support this, but someone needs to work out the details and generate a multi-layer AVIF image as a proof of concept. Leo Barnes reported his findings in an email on this topic last week. I don't know HEIF well enough to understand Leo's email, but it looks like he figured out the HEIF part of the puzzle. So we are making progress!

[kornelski]

I'd like to note that Cloudflare is interested in this feature. We have an HTTP/2 server that understands structure of image formats, and multiplexes network streams in order to deliver progressive previews of all images in parallel as soon as possible. We have an AVIF converter already, and would like to implement progressive encoding.

[jakearchibald]

I created a demo by loading the 5k 'preview' version, then changing the src to the full version https://jakearchibald.com/2020/avif-has-landed/#progressive-avif-demo.

Here's a the full demo, although it's Chrome-only https://jakearchibald.com/2020/avif-has-landed/demos/loading/

[paperboyo]

Just a comment from a mere user: while I think it’s a great suggestion (and the demo clearly shows the benefit, thanks!), I think ideally it would be properly* included in the standard (maybe in AV2, if impossible for AV1?).

While including an extra image to be used as a “fake” progressive has its benefits, it does increase the overall file size and presents its own problems: if the make-up of this (resolution reduction ratio to original, extra compression, bluriness, etc) is left entirely for the users to decide on, there are questions around best practice of what that make-up should be (extra 5K may be OK for a 100KB image, but not so for a 20KB image). Also, it may bring some visual chaos to the web that will be hard to come back from (not a fan of a plethora of placeholder image techniques, myself): what would prevent users from sticking a company logo as this extra frame?

*as I understand (and I don’t!), “proper” progressive would use the original image in a progression of standard (although tweakable) renders and wouldn’t use an extra frame. This way, connection quality naturally eliminates (perceptually) the intermediate stage, while in the “fake” progressive, there would be a flash of this unneeded really extra image (?). I hope the video-first nature of AV standard doesn’t prevent it from becoming properly progressive one day.

[kornelski]

@paperboyo progressive is impossible in AV1. It's too soon to predict how adoption and upgrade paths will look like for AV2 and AVIF. For example, WebP remained stuck with the old VP8 despite the world moving on to VP9, so it's possible that AVIF, at least on the Web, will live and die with AV1.

Progressive rendering is cheap in the old JPEG, because JPEG doesn't have prediction modes and filters that depend on exact pixels of neighboring blocks. AV1 (and all modern video codecs) has such dependencies. Removing compression features that prevent progressive rendering could make compression worse, and overall it may not be better than adding an extra fake image.

[kornelski]

@wantehchang It looks like the mailing lists are closed and only members (presumably big corporations) can join? As an individual implementer, can I participate in these discussions?

[paperboyo]

Boo, thanks for the explanation, @kornelski. Then, as a user, I vote for the users to have as little control over the make up of this extra frame as (in)humanly possible, let’s just decide this in the standard and disallow encoders any “creative” control over that.

It would be rigid, not take connection quality into account and so on, but still better than having to write tools to strip the oversized extra frame, tools to replace some company logos with your own, tools to remove company logos from images one got from the web, tools to update the extra frame other tool forgot to update upon resaving, tools to stick multiple AVIFs with exactly the same main image but differently-sized extra frames into a srcset to be served depending on network conditions and so on. And better than having to write (and read) all the blog posts about how to use them tools and what’s best practice with this extra frame, instead of having to read (and write) blog posts about the art and science of optimising the proper images.

Or, maybe, I’m wrong. Maybe I’m unnecessarily pessimistic. I, for one, wouldn’t abuse them, no way! 😜

[verlok]

Good idea!
If it's not feasible in the AV1 file forma, we could just use the small 5k file as placeholder while we load the full image, just like @jakearchibald did on his demo.
BUT it would be better if it could be embedded in the file format itself.

[wantehchang]

@wantehchang It looks like the mailing lists are closed and only members (presumably big corporations) can join? As an individual implementer, can I participate in these discussions?

Hi Kornel,

Yes, only employees of AOM members can join the mailing lists of AOM working groups. I think companies of all sizes can join AOM, but the details don't seem to be published. If you work for a company, you can have the company fill out this application form to find out more: http://aomedia.org/membership/

One thing I can do is to ask Leo Barnes to post his progress in this issue.

[leo-barnes]

Hi,
I'll try to put together a summary of the discussion during/after the weekend. There are a bunch of different use-cases for progressive decoding and a bunch of options available depending on which tradeoffs you want to make.

[leo-barnes]

This is going to be a pretty long comment given that this is still under very active discussion as of now. The conclusion so far is that the HEIF/MIAF/AVIF specifications as well as AV1 has pretty much everything in them that is needed. The missing parts are mainly clarifications on exactly how it should work and recommendations/best practices on how files should be structured as well as how parsers and decoders should interpret them.

Use-cases

To start with, it's helpful to outline the two scenarios where progressive decoding is currently in use for JPEGs:

1. When downloading an image with the intent of displaying it at full resolution and wanting to show progressive updates if the connection is slow.
2. When decoding an image at less than full resolution, which may allow not loading the full file. This is a more general use-case and may happen both while downloading an image and when it's already downloaded.

Personally I don't consider the first scenario (which I'll call S1) to be extremely important these days. You can get more or less the same functionality by simply displaying the thumbnail until the full image has been downloaded.

Worth noting is that there has been very little scientific study on whether people actually prefer progressive or sequential decoding over slow connections. Facebook has run multiple internal experiments over the years to answer the question, but the conclusions were kind of mixed. In the cases where users preferred progressive rendering it couldn't be determined if this was actually due to the progressive updates or simply due to the progressive images being smaller in size and therefore downloading faster.

S1 is only really an issue when you have very slow download speeds. This is still the case in some places in the world, but is becoming less and less of an issue. Added to that, most monitors are not actually very high resolution. Some googling seems to indicate that most monitors tend to have 1920x1080 pixels or less. So only small to medium sized images will ever really be displayed at full resolution. For small images you will probably not really notice progressive updates. If you immediately want to show something on screen, you can use the thumbnail and simply display the full image once it's fully downloaded.

This leaves medium sized (typically smaller than 1080p) images as the main scenario where progressive updates would potentially be noticeable on slow connections. Here you can again show the thumbnail on screen until the full image has been decoded, but it may be low resolution enough that it won't look very pretty. But neither will only the first scans of a progressive JPEG.

Which brings us to the second scenario (S2). This is the case any time you want to display a large image on a small screen (or part of a screen). For typical camera images, this is almost always the case unless the user has a 4K display (or unless the user has zoomed in, in which case you only want to download/decode that region of the image rather than the full image). For a view that is showing a wall of pictures it will almost be guaranteed to be S2 unless the content provider has downscaled thumbnails that are shown instead.

With progressive JPEGs, you would in this case usually not need all the scans in the file. When downloading, this is very beneficial since you don't need to download the full file. When in local storage, decoding is faster and uses less memory.

The reason why it's important to outline S1 and S2 is that there are multiple ways of accomplishing progressive decoding. Some of them are better for S1, and some of them are better for S2.

Solutions

With the use-cases out of the way, how would we actually do progressive decoding in AVIF? There are basically three ways:

1. Use thumbnails.
   This is very simple and something that MIAF (on which AVIF is based) recommends all main images in files to have. Thumbnails solve S1 reasonably well in that something can typically be displayed very quickly. The main drawbacks of thumbnails are:

   • The thumbnail may be too small to be useful for S2.
   • There is some coding redundancy since the thumbnail is not used for prediction in the main image.
   • Thumbnails are assumed to be lower resolution than the main image. You can't (or are not really supposed to) have thumbnails that are the same resolution but lower quality.

2. Use independent images in an 'altr' group.
   This is basically exactly what jakearchibald is asking for. You have two items in the file. The first is the lower quality and/or lower resolution image, the second is the full quality image. This can work for both S1 and S2.
   Both items are placed in an 'altr' group with the (full quality) item placed first in the list. Decoders are supposed to pick the first item in the 'altr' group that it can decode and matches it's requirements. If the first item in the list has not yet been downloaded but the second has, the decoder is supposed to use that instead. Which in this case means that the lower quality item is used until the full quality item is available.
   The main drawback of this method is that the different items in the 'altr' group are independent which leads to some pretty severe coding redundancy if the lower quality/resolution items are large.
   This was also discussed with browser developers when MIAF was initially written. The conclusion from those teams were that they in general preferred using separate files rather than have multiple independent assets in the same file.

3. Multi-layer coding (but no extra info on the container level)
   AV1 supports multi-layer coding both in the form of scalable quality and scalable resolution. This means you can create an elementary stream that gives you progressive refinement either in the form of layers of progressively better quality, or layers of progressively larger resolution. If such a stream is placed in an AVIF container, you would be able to satisfy S1 in much the same way that progressive JPEGs do.
   But since no extra information has been added to the container, it doesn't know which parts of the elementary stream contains which layers or which potential alternative representations are available. This makes it relatively tricky to satisfy S2. It can probably be done for a single non-derived image item if the AVIF parser can also parse and understand the properties of the AV1 elementary stream, but it's going to be very non-trivial for grids. The container also doesn't know up front which parts of the elementary stream to download and would instead have to sequentially download and parse in chunks until it has enough data.
   The main drawbacks of multi-layer coding are:

   • Coding complexity. Compared to a single layer, it's going to take more cycles to decode a multi-layer stream.
   • For other codecs, multi-layer support is pretty limited. It's normative in AV1 and existing software implementations support it so this should not be an issue.

4. Multi-layer coding with all/some layers exposed on the container level
   HEIF has the 'lsel' (LayerSelector) property which allows you to create individual items for the possible outputs of a multi-layer stream. This means that you can create an explicit 'altr' group containing the different layers in an elementary stream. The parser/decoder decodes the layers that are available.
   The benefits of exposing the layers on the container level are:

   • The layer locations are specified in the container, so the parser knows exactly which parts of the file need to be downloaded. This is great for S2.
   • This works well for derived items in the form of grids. One grid can be created of the first layer of all tiles, another of the second layer. Both grids are placed in an 'altr' group. The parser can easily reason about which layers and tiles are available without having to parse any of the individual AV1 streams.
     The main drawback compared to 3 is that the AVIF container writer needs to be a bit more complicated.

The main part missing from 4 is exactly how 'lsel' is supposed to interact with AV1. @cconcolato has opened issue #103 to track that.

I personally think 4 is the best way of accomplishing progressive refinement since it allows for more use cases. The container parser should ideally not need to be able to parse codec elementary streams to reason about a file. I would prefer a slightly more complicated AVIF writer to a much more complicated AVIF reader.

[jakearchibald]

Thanks for the update!

S1 is only really an issue when you have very slow download speeds. This is still the case in some places in the world, but is becoming less and less of an issue.

I think there's still a benefit at 3g-like speeds. Back when commuting was a thing, I'd frequently end up on the train's wifi, which was terrible, or on mobile, which was spotty/slow during the commute. This was a train to London, so I don't think this is just an emerging markets issue.

The conclusion from those teams were that they in general preferred using separate files rather than have multiple independent assets in the same file.

I'm not sure how that works on the web. If I ask the server for two files, the 'preview' version and the full version, the server would need to be smart enough to know they're part of the same overall image and send them in sequence. Otherwise, you can end up receiving the preview after the full version. Or, the server might chunk them and send them in parallel, but bandwidth spent on the full version rather than the preview is kind of a waste. Servers are pretty bad at serving content in the most optimal order today.

The client could avoid this by requesting each in sequence, but that creates a delay as it has to create a separate request once it's received the preview.

With a single file, you don't have the sequencing issues, since the order of bytes in the file dictates that. It also means the browser can make decisions like "I don't need to decode the preview, because I already have the whole file".

The layer locations are specified in the container, so the parser knows exactly which parts of the file need to be downloaded.

If I've understood this correctly, I worry about the performance. It means the browser would have to:

1. Start downloading the image.
2. See it's an AVIF, start parsing.
3. Realise the bit of the image it needs is elsewhere in the file.
4. Terminate the stream.
5. Make range request for the bit of the image it does need.

I think in many cases this would be slower than downloading the whole file.

For the web to benefit from it, I think the data needs to be delivered in a single response, where the data is ordered lowest-resolution/quality first.

[cconcolato]

The conclusion from those teams were that they in general preferred using separate files rather than have multiple independent assets in the same file.

I'm not sure how that works on the web. If I ask the server for two files, the 'preview' version and the full version, the server would need to be smart enough to know they're part of the same overall image and send them in sequence. Otherwise, you can end up receiving the preview after the full version. Or, the server might chunk them and send them in parallel, but bandwidth spent on the full version rather than the preview is kind of a waste. Servers are pretty bad at serving content in the most optimal order today.

I think Leo was referring to this part of the MIAF specification:

Master image items that are not the primary item and not required for generating the output image of the
primary item may be present, but should be present only if it is expected that the application environment
could make use of multi-branded files efficiently. It is recommended that alternative coding formats be
represented by a choice of image file in a higher-level context (e.g. the picture element in HTML5).

It don't think it was meant for progressive decoding.

Anyway, to avoid any confusion, we're looking at clarifying how to do 3 and/or 4. In both cases, we're talking about having a single file. In both cases, the bits can be ordered such that you don't have sequencing issues. The difference are that:

• in 3, writers would be simple (just copy a layered image into an item), clients would be simple, but limited (one download until you have enough).
• in 4, writers are a bit more complex (they have to parse the layered image and signal layers at the container level), clients can still do the simple case but can use that information in cases where the client wants to do multiple byte-range requests to fetch the data (e.g. displaying only the high quality in a part of a large panorama or NASA image).

For the web to benefit from it, I think the data needs to be delivered in a single response, where the data is ordered lowest-resolution/quality first.

Then you would simply rely on (3) or the basic feature of (4).