Document how to serve webp / jpg based on accept header

[sparhami]

Two things need to be done:

Have a better way to load webp with amp img. We don't want to wait for it to fail to load before falling back to jpg.

Change the optimizer output both webp and jpg. There may be other formats to consider.

[aghassemi]

related: #3517

[jridgewell]

CDN already optimizes images to webp:

$ curl -H 'Accept: */*' 'https://static01-nyt-com.cdn.ampproject.org/i/s/static01.nyt.com/images/2019/03/15/opinion/15stephensWeb/15stephensWeb-jumbo.jpg' -s | head -c 50
����JFIF��C

$ curl -H 'Accept: */*, image/webp' 'https://static01-nyt-com.cdn.ampproject.org/i/s/static01.nyt.com/images/2019/03/15/opinion/15stephensWeb/15stephensWeb-jumbo.jpg' -s | head -c 50
RIFF��WEBPVP8 ����*�>Q(�F#���#2)��

[cathyxz]

Hi @jridgewell ! Yup, I think we can word this better, but basically what @sparhami and I were talking about is actually how many browsers (FF, Safari) don't support webp, and therefore when webp is provided as default to <amp-img>, we end up fetching that once, failing to render, and then showing the fallback in jpg. There is probably a very nice bite-sized project here to upgrade <amp-img> to more gracefully handle this for browsers without webp support. I'm not actually super sure what the cache does here with regards to non-optimized images based on different browsers. Is the cache smart enough to send webp only to browsers that support it?

[sparhami]

Since the AMP cache already handles this, maybe it would be better let developers know about the same pattern (serving webp based on accept header) for non-AMP cache (but still optimized) pages rather than trying to implement this in AMP.

[jridgewell]

I think this is a server concern and not AMP's. It doesn't matter what bytes are sent down for http://example.com/image.jpg, and the browser doesn't care that the URL ends in .jpg. It'll send an Accept header like image/*,*/*;q=0.8 (or image/webp,image/*,*/*;q=0.8 if it supports webp), and the server just needs to respond with image bytes and the appropriate Content-Type header for the bytes.

[cathyxz]

So it looks like the assumption

In the case of the AMP Cache, the origin can't respond selectively with different image types based on the browser request headers, so the decision of image variant would need to be something the browser selects.
in #3517 is no longer true.

#18599 (short-circuiting webp on ios) still provides a benefit, but it looks like the main action item here then is just to document this and suggests user to follow the AMP Cache's lead server-side.

The AMP Cache is so smart. =)

[sparhami]

One problem with the current approach is that we are doing a lossy -> lossy image conversion in the AMP cache to generate the webp version. It would be better for page authors to be able to run the image conversion based on a raw source image, or provide a lossless webp on the page. There is no reasonable way to do that currently, since simply using a lossless webp in the source document will not work correctly for all browsers visiting the origin page.

[jridgewell]

Publishers can link to lossless PNG files, which will also be converted to webp images in the cache.

[sparhami]

Publishers can link to lossless PNG files, which will also be converted to webp images in the cache.

But that wouldn't be very good bandwidth wise for users visiting the original (non-cached) page. Same with loseless webp actually.

[jridgewell]

The server can decide what to serve to whoever requests the file. Having endpoints as 1-1 with some file is the wrong way to accomplish differentiated serving, you can serve multiple files from the same endpoint. It's not much different than the brotli/gzip compressed responses being served from the same URL (in this case, it just happens to be a lossy transformation that can happen).

1. If UA matches AMP GoogleBot, serve lossless PNG or lossless webp
2. If Accept includes webp, serve lossless webp
3. Else, serve lossy png

[sparhami]

The server can decide what to serve to whoever requests the file. Having endpoints as 1-1 with some file is the wrong way to accomplish differentiated serving, you can serve multiple files from the same endpoint. It's not much different than the brotli/gzip compressed responses being served from the same URL (in this case, it just happens to be a lossy transformation that can happen).

1. If UA matches AMP GoogleBot, serve lossless PNG or lossless webp
2. If Accept includes webp, serve lossless webp
3. Else, serve lossy png

Does the AMP cache generate lossy jpgs out of PNGs? That does not seem like the right thing to be doing. PNGs are a lossless format (and intentionally so). For example, you may use it for encoding a bitmaps. If the AMP cache is re-encoding PNGs into lossy formats for serving, that should be fixed.

I suppose it could check the file extension from the request path, and if it is something like jpg, assume that this is not really supposed to be a lossless PNG, and only then re-encode the image. Is this something that actually exists, that we can recommend to developers?

I think there is still a wider problem that not all serving platforms support this. You are turning something that was essentially static file serving into something dynamic. How does this work with various (non-AMP) caches?

[jridgewell]

Does the AMP cache generate lossy jpgs out of PNGs?

No, lossless will only be converted to optimized lossless. The image must already be lossy to be converted to an optimized lossy.

[sparhami]

Right, we want to create lossy images, using a lossless source, rather than doing a lossy to lossy conversion (not just for webp, but also for the different compression levels for srcset). We currently cannot do this.

[jridgewell]

Right, we want to create lossy images, using a lossless source, rather than doing a lossy to lossy conversion

I see essentially 4 different image intents:

1. Ship lossless PNG
2. Ship lossy JPG
3. Ship lossless WebP
4. Ship lossy WebP

If the publisher intended to ship a lossless image to the user, they can send either PNG or WebP. The cache will optimize these to compressed lossless images.

If the publisher intended to ship a lossy image to the user, they can send JPG or (lossy) WebP. The cache will optimize these to compressed lossy images.

The case we're talking about here is when the publisher wants to ship a lossless WebP to a user or a lossy JPG. But why wouldn't they have just sent a high quality, but still lossy, WebP?

[sparhami]

The case we're talking about here is when the publisher wants to ship a lossless WebP to a user or a lossy JPG. But why wouldn't they have just sent a high quality, but still lossy, WebP?

To clarify: when we are creating compressed lossy images, we should start from a lossless source, rather than a lossy one.

[jridgewell]

Are you sure? We already compress a JPG twice (it starts as lossy and we optimize it). Why wouldn't compressing a high-quality lossy WebP be acceptable?

[sparhami]

Are you sure? We already compress a JPG twice (it starts as lossy and we optimize it). Why wouldn't compressing a high-quality lossy WebP be acceptable?

For example, the developer may not give us a super high quality source to begin with. They may assume that generating something for the largest mobile screen size is sufficient. However, in the mean time, we have decided to automatically lightbox (and allow pan-zoom) on all images in the page over a certain size. The ideal resolution for our zoomed in mode will be higher than the largest resolution they put on the page. We cannot do a lossy to lossy conversion where we increase the quality.

I think it would be better to be explicit about what is the original source image, so we can create the right compression levels.

[jridgewell]

I am wondering which type will be cached by AMP cache and if AMP cache will cache the webp,

I believe the Googlebot (the Google AMP cache's fetcher) will advertise WebP support, so you should be serving WebP to the cache.

what will happen in a browser that does not support webp if we aren't using the nested fallback image.

Depending on the user's browser, we'll either serve the WebP or transform it to an acceptable format (either lossless PNG if the WebP is lossless, or JPG if the WebP is lossy).

[PutziSan]

So it is intended that AMP can only be rendered on the server side?
Actually not (I thought), but here it is assumed that it would be, why?

The cases you thereby exclude from use (that I can think of):

• static HTML files
• static site generators (hugo, gatsby, jekyll, ...)
• server-side rendered pages that are behind a CDN, which cache the dynamic output (e.g. zeits now edge caching or AWS Cloudfront)

I find this approach questionable.

note:
The fallback method is (in my eyes) no substitute: that ios users need twice as many requests to load the images on the site is unacceptable if performance is one of the most important things. (and I have built in AMP to show a page to the user as fast as possible...)

[jridgewell]

So it is intended that AMP can only be rendered on the server side?

The goal is to serve the same content to every browser: <amp-img src="example.jpg" />. But example.jpg can be any image, and your CDN server is free to serve a webp response based on the content negotiation headers.

This means we get the benefit of static HTML pages, rendered by static CMSs and served by static serves. The CDN in front of your image has the ability to optimize the image responses for you, automatically.

[sebastianbenz]

Stumbled over this as well. We currently suggest using fallback for webp support in our docs which is imo an anti-pattern as the image is being downloaded twice in browsers not supporting webp. If there are no concerns I will remove this from the documentation.

However, I agree with @caroqliu and @sparhami that it'd be nice to have proper client-side support for webp.

[jridgewell]

We currently suggest using fallback for webp support in our docs which is imo an anti-pattern as the image is being downloaded twice in browsers not supporting webp. If there are no concerns I will remove this from the documentation.

Yes, we should remove that. The recommended approach should be to serve webp when responding to the image request, if webp is advertised in the Accept request header.

that it'd be nice to have proper client-side support for webp.

There was discussion about this in ampproject/amp-wp#3082. As far as I'm aware, there is no good way to determine webp/formatX support on the browser side.

[sebastianbenz]

@jridgewell did we ever consider generating a picture element inside an amp-img? This would make it easy to add webp support.

[westonruter]

did we ever consider generating a picture element inside an amp-img? This would make it easy to add webp support.

The picture element is supported by all browsers other than IE11: https://caniuse.com/#feat=picture

For IE11, the fallback img inside of the picture could still be used.

[jridgewell]

I believe picture wasn't used because we'd have to implement <amp-picture> to get the lazy-loading and control we need. And it falls back to the same "how do we select the codec?" issue described here.

[westonruter]

One option would be for amp-img to allow source children, similar to amp-video. So what about this:

<amp-img width="550" height="368" alt="Mountains">
  <source src="/static/inline-examples/images/mountains.webp" type="image/webp">
  <source src="/static/inline-examples/images/mountains.jpg" type="image/jpeg">
</amp-img>

When such an amp-img lacks src then instead of adding an img to the shadow DOM, it could instead add a picture like so when the image needs to be lazily-loaded:

<amp-img width="550" height="368" alt="Mountains" class="i-amphtml-element i-amphtml-layout-fixed i-amphtml-layout-size-defined i-amphtml-layout" i-amphtml-layout="fixed" style="width: 550px; height: 368px;">
  <source src="/static/inline-examples/images/mountains.webp" type="image/webp">
  <source src="/static/inline-examples/images/mountains.jpg" type="image/jpeg">
  <picture decoding="async" class="i-amphtml-element i-amphtml-layout-fixed i-amphtml-layout-size-defined i-amphtml-layout">
    <source src="/static/inline-examples/images/mountains.webp" type="image/webp">
    <source src="/static/inline-examples/images/mountains.jpg" type="image/jpeg">
  </picture>
</amp-img>

Then the codec selection would still be left up to the browser.

[jridgewell]

I think that could be workable. Are you interested in bringing this up for a design review?

[westonruter]

I am interested but I probably won't have time to do so until Q3 or Q4 in the worst case. So if someone gets to it first, good on them.

[jridgewell]

/cc @ampproject/wg-ui-and-a11y #21912 (comment)

[cathyxz]

Happy to see this old bug picked up again! I'm going to remove my assignment from this issue since I probably won't be working on it anytime soon. =)

[jridgewell]

I found https://github.com/jakearchibald/jakearchibald.com/blob/86fc3ccb22359b2fed177344d40e2554569ba909/shared/demos/2020/avif-has-landed/DecodedImg/index.tsx#L26-L47 today, which demonstrates how to use a native <picture> and <source> element to detect image codec support without a network request, which is amazing. We could build a amp-picture component around this, for older Edge and IE support. Using a native <picture> like #21912 (comment) would still be best for newer browsers.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.