New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"decode" attribute on <img> #1920

Closed
smfr opened this Issue Oct 18, 2016 · 98 comments

Comments

@smfr

smfr commented Oct 18, 2016

Decoding of large images can block the main thread for hundreds of milliseconds or more, interrupting fluid animations and user interaction. Currently, there's no way for a web author to specify that they want an image to be decoded asynchronously, so there are scenarios where it is impossible to avoid UI stalls.

To solve this problem we propose an "async" attribute on image elements. This attribute is a hint to the UA that the author has requested asynchronous decoding. This implies that if the UA paints an image after the "load" event has fired, but before the image has been decoded, the UA is allowed to not paint the image (rather than block on decoding it).

To notify authors when a decoded image frame is available, we propose firing a new event, "ready", on the image element. This would allow authors who require a fully-decoded image, in content that is sensitive to UI stalls, to wait for the "ready" event before doing something that brings the image into view (such as a CSS transition).

Some images repeatedly decode frames, for example, animated GIFs. In addition, the UA can throw away the decoded frame for a still image, requiring a re-decode. In these cases, we propose that the "ready" event only fires once, the first time a frame is available for display.

ISSUES:
"async" is currently used to imply async loading, and it's possible that we'd want to use it in this sense for images too. Maybe call the new attribute "asyncDecode" or something else.

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

@grorg

This comment has been minimized.

Show comment
Hide comment
@grorg

grorg Oct 18, 2016

Collaborator

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

We could maybe follow CSS animations/transitions here. Make a new "imageDecoded" or "imageReady" event (I suck at names) and fire it on the target that has the new CSS property.

I wonder if you'll ever want some parts of the element to have async images, and other parts to not. e.g. borders vs background. If so, it seems like it would have to go in the new image() function. (It already has tags for rtl and ltr).

Collaborator

grorg commented Oct 18, 2016

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

We could maybe follow CSS animations/transitions here. Make a new "imageDecoded" or "imageReady" event (I suck at names) and fire it on the target that has the new CSS property.

I wonder if you'll ever want some parts of the element to have async images, and other parts to not. e.g. borders vs background. If so, it seems like it would have to go in the new image() function. (It already has tags for rtl and ltr).

@grorg

This comment has been minimized.

Show comment
Hide comment
@grorg

grorg Oct 18, 2016

Collaborator

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

I do think it is confusing that async here is a slightly different meaning from script (which is "Set this Boolean attribute to indicate that the browser should, if possible, execute the script asynchronously." )

Or does it?... Would you consider image decoding to be the equivalent of script executing?

Collaborator

grorg commented Oct 18, 2016

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

I do think it is confusing that async here is a slightly different meaning from script (which is "Set this Boolean attribute to indicate that the browser should, if possible, execute the script asynchronously." )

Or does it?... Would you consider image decoding to be the equivalent of script executing?

@grorg

This comment has been minimized.

Show comment
Hide comment
@grorg

grorg Oct 18, 2016

Collaborator

Of course, the HTML5 specification has a more complete definition of script async.

Collaborator

grorg commented Oct 18, 2016

Of course, the HTML5 specification has a more complete definition of script async.

@smfr

This comment has been minimized.

Show comment
Hide comment
@smfr

smfr Oct 18, 2016

Actually reading that, I think 'async' here is quite a good match.

smfr commented Oct 18, 2016

Actually reading that, I think 'async' here is quite a good match.

@duanyao

This comment has been minimized.

Show comment
Hide comment
@duanyao

duanyao Oct 18, 2016

Isn't off main thread image decoding already possible in browsers without a spec change? E.g.:
https://bugzilla.mozilla.org/show_bug.cgi?id=716140
https://bugs.webkit.org/show_bug.cgi?id=90375

Does "async" also means lazy download of the image file?

duanyao commented Oct 18, 2016

Isn't off main thread image decoding already possible in browsers without a spec change? E.g.:
https://bugzilla.mozilla.org/show_bug.cgi?id=716140
https://bugs.webkit.org/show_bug.cgi?id=90375

Does "async" also means lazy download of the image file?

@smfr

This comment has been minimized.

Show comment
Hide comment
@smfr

smfr Oct 18, 2016

Isn't off main thread image decoding already possible
Possible, yes, but the main thread may still have to block on the decoder thread if it is still decoding an image when the main thread wants to paint the image. If a UA chooses not to block, then the user gets a a blank or partial image even though the load event fired (I'm not sure what Gecko does, and Mac/iOS WebKit have used main thread decoding for the past few years).

This proposal allows an author to present images with a guarantee both that the main thread is never blocked on image decoding, and that the image is ready to display when the "ready" event fires.

Does "async" also means lazy download of the image file?

No. It's purely about decoding.

smfr commented Oct 18, 2016

Isn't off main thread image decoding already possible
Possible, yes, but the main thread may still have to block on the decoder thread if it is still decoding an image when the main thread wants to paint the image. If a UA chooses not to block, then the user gets a a blank or partial image even though the load event fired (I'm not sure what Gecko does, and Mac/iOS WebKit have used main thread decoding for the past few years).

This proposal allows an author to present images with a guarantee both that the main thread is never blocked on image decoding, and that the image is ready to display when the "ready" event fires.

Does "async" also means lazy download of the image file?

No. It's purely about decoding.

@duanyao

This comment has been minimized.

Show comment
Hide comment
@duanyao

duanyao Oct 18, 2016

Once a UA implemented off main thread decoding, there is little reason for them to keep the old behavior, so still no need to specify "async" in HTML. Currently there seems no UA which (intentionally) blocks layout/paint on image download/decoding.

As to "ready" event, I think the situation can be compicated, because UA might drop decoded images that are no longer visible (out of viewport or in background tabs) to save memory. For completeness, "unload" event may be also needed.

duanyao commented Oct 18, 2016

Once a UA implemented off main thread decoding, there is little reason for them to keep the old behavior, so still no need to specify "async" in HTML. Currently there seems no UA which (intentionally) blocks layout/paint on image download/decoding.

As to "ready" event, I think the situation can be compicated, because UA might drop decoded images that are no longer visible (out of viewport or in background tabs) to save memory. For completeness, "unload" event may be also needed.

@smfr

This comment has been minimized.

Show comment
Hide comment
@smfr

smfr Oct 18, 2016

Currently there seems no UA which (intentionally) blocks layout/paint on image download/decoding.

First, we're not talking about downloading here. This proposal doesn't change anything about how image downloading works.

Second, Mac and iOS WebKit block painting on image decoding (I work on WebKit for Apple, BTW). That's the problem we're trying to address. If other UAs have solved this problem, I would be interested to hear how.

smfr commented Oct 18, 2016

Currently there seems no UA which (intentionally) blocks layout/paint on image download/decoding.

First, we're not talking about downloading here. This proposal doesn't change anything about how image downloading works.

Second, Mac and iOS WebKit block painting on image decoding (I work on WebKit for Apple, BTW). That's the problem we're trying to address. If other UAs have solved this problem, I would be interested to hear how.

@duanyao

This comment has been minimized.

Show comment
Hide comment
@duanyao

duanyao Oct 18, 2016

I am confused because I observed that UAs may display images increamentally on a slow connection, so I think decoding is not blocking painting, at least in such situation.

So does "async" also gurantee that the image is not visible when the downloading/decoding is in progress?

duanyao commented Oct 18, 2016

I am confused because I observed that UAs may display images increamentally on a slow connection, so I think decoding is not blocking painting, at least in such situation.

So does "async" also gurantee that the image is not visible when the downloading/decoding is in progress?

@smfr

This comment has been minimized.

Show comment
Hide comment
@smfr

smfr Oct 18, 2016

UAs can decode and paint an image that is not fully downloaded, yes. It may appear partial or low-resolution in this state, but the decode itself can still take time (100ms or more). I think the "ready" event should fire when the first full-resolution frame is available.

smfr commented Oct 18, 2016

UAs can decode and paint an image that is not fully downloaded, yes. It may appear partial or low-resolution in this state, but the decode itself can still take time (100ms or more). I think the "ready" event should fire when the first full-resolution frame is available.

@annevk

This comment has been minimized.

Show comment
Hide comment
@annevk

annevk Oct 18, 2016

Member

@igrigorik @yoavweiss may have opinions.

I think the API should be <img>.ready returning a promise. No opinions on the attribute, though we might in the future also want something that influences when the image is downloaded (e.g., an indication to the user agent that it's fine to load the image later).

Member

annevk commented Oct 18, 2016

@igrigorik @yoavweiss may have opinions.

I think the API should be <img>.ready returning a promise. No opinions on the attribute, though we might in the future also want something that influences when the image is downloaded (e.g., an indication to the user agent that it's fine to load the image later).

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Oct 18, 2016

Member

Let's set aside the bikeshedding on the name and style of the event/promise for now, and try to get some multi-implementer interest in the base feature of using a new attribute to control image decoding. I think the key question is:

Second, Mac and iOS WebKit block painting on image decoding (I work on WebKit for Apple, BTW). That's the problem we're trying to address. If other UAs have solved this problem, I would be interested to hear how.

For example, a UA could choose to simply make all images have this "async" behavior and draw placeholders. Presumably this might not be great for authors (why, exactly?), thus this proposal to make it opt-in, if I am understanding correctly. If we were to, in certain circumstances, flip the default to async, this is starting to sound intervention-ish; /cc @RByers @ojanvafai

Another aspect that this is related to is off-main-thread image decoding. Some of the recent work on ImageBitmap has enabled this to be done in JavaScript; see https://developers.google.com/web/updates/2016/03/createimagebitmap-in-chrome-50. But this requires awkward contortions because then you have to draw it onto a canvas, instead of using a simple <img>. I am unsure to what extend this async proposal overlaps---would this allow off-main-thread image decoding? Is it in some sense sugar over the ImageBitmap technique that makes it less weird? /cc @junov since he's our ImageBitmap guy, both for Chrome and for this spec.

My sense is that this area is of definite interest to Chrome, and as such they're good candidates for second-vendor interest. I've tried to pepper @-mentions of people who might know more above. Any thoughts on Mozilla or Edge?

Member

domenic commented Oct 18, 2016

Let's set aside the bikeshedding on the name and style of the event/promise for now, and try to get some multi-implementer interest in the base feature of using a new attribute to control image decoding. I think the key question is:

Second, Mac and iOS WebKit block painting on image decoding (I work on WebKit for Apple, BTW). That's the problem we're trying to address. If other UAs have solved this problem, I would be interested to hear how.

For example, a UA could choose to simply make all images have this "async" behavior and draw placeholders. Presumably this might not be great for authors (why, exactly?), thus this proposal to make it opt-in, if I am understanding correctly. If we were to, in certain circumstances, flip the default to async, this is starting to sound intervention-ish; /cc @RByers @ojanvafai

Another aspect that this is related to is off-main-thread image decoding. Some of the recent work on ImageBitmap has enabled this to be done in JavaScript; see https://developers.google.com/web/updates/2016/03/createimagebitmap-in-chrome-50. But this requires awkward contortions because then you have to draw it onto a canvas, instead of using a simple <img>. I am unsure to what extend this async proposal overlaps---would this allow off-main-thread image decoding? Is it in some sense sugar over the ImageBitmap technique that makes it less weird? /cc @junov since he's our ImageBitmap guy, both for Chrome and for this spec.

My sense is that this area is of definite interest to Chrome, and as such they're good candidates for second-vendor interest. I've tried to pepper @-mentions of people who might know more above. Any thoughts on Mozilla or Edge?

@smfr

This comment has been minimized.

Show comment
Hide comment
@smfr

smfr Oct 18, 2016

For example, a UA could choose to simply make all images have this "async" behavior and draw placeholders. Presumably this might not be great for authors (why, exactly?)

Some authors need a guarantee that if an image is ready, it gets painted. An image popping in later is not acceptable in some kinds of content.

If we were to, in certain circumstances, flip the default to async

Interesting idea, but I don't think it would fly compat-wise.

I am unsure to what extend this async proposal overlaps---would this allow off-main-thread image decoding?

I'm proposing this now specifically because WebKit has been doing work on off-main-thread image decoding, for some subset of images (webkit.org/b/155322 and related). I don't think a UA would be required to do off main thread decoding to implement this (for example, they could trigger decodes in something like a requestIdleCallback), but pages would still suffer from UI stalls.

smfr commented Oct 18, 2016

For example, a UA could choose to simply make all images have this "async" behavior and draw placeholders. Presumably this might not be great for authors (why, exactly?)

Some authors need a guarantee that if an image is ready, it gets painted. An image popping in later is not acceptable in some kinds of content.

If we were to, in certain circumstances, flip the default to async

Interesting idea, but I don't think it would fly compat-wise.

I am unsure to what extend this async proposal overlaps---would this allow off-main-thread image decoding?

I'm proposing this now specifically because WebKit has been doing work on off-main-thread image decoding, for some subset of images (webkit.org/b/155322 and related). I don't think a UA would be required to do off main thread decoding to implement this (for example, they could trigger decodes in something like a requestIdleCallback), but pages would still suffer from UI stalls.

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Oct 18, 2016

Member

Some authors need a guarantee that if an image is ready, it gets painted. An image popping in later is not acceptable in some kinds of content.

I think I understand what you're saying, but can you check if the following expansion is correct?

As a web developer it seems like all of my images just pop in at random times when the browser has finished with them and is ready to show them. The difference between showing a placeholder while it's fetching and then janking while decoding and then painting, vs. showing a placeholder while fetching and decoding, and then painting, is almost unobservable to me.

The difference is in the specific case where I end up listening to the load event. For example, if I create an image out of document, then wait for the load event, I anticipate that when I insert it into the document it will be painted synchronously. (This might even affect from-script-observable things like offsetWidth.)

But as an author, it seems like any time I'm not listening for the load event and assuming the image is fully ready at that time, I'd be able to sprinkle async on the <img> elements, and there would be no observable difference. Right?

Member

domenic commented Oct 18, 2016

Some authors need a guarantee that if an image is ready, it gets painted. An image popping in later is not acceptable in some kinds of content.

I think I understand what you're saying, but can you check if the following expansion is correct?

As a web developer it seems like all of my images just pop in at random times when the browser has finished with them and is ready to show them. The difference between showing a placeholder while it's fetching and then janking while decoding and then painting, vs. showing a placeholder while fetching and decoding, and then painting, is almost unobservable to me.

The difference is in the specific case where I end up listening to the load event. For example, if I create an image out of document, then wait for the load event, I anticipate that when I insert it into the document it will be painted synchronously. (This might even affect from-script-observable things like offsetWidth.)

But as an author, it seems like any time I'm not listening for the load event and assuming the image is fully ready at that time, I'd be able to sprinkle async on the <img> elements, and there would be no observable difference. Right?

@smfr

This comment has been minimized.

Show comment
Hide comment
@smfr

smfr Oct 18, 2016

... anticipate that when I insert it into the document it will be painted synchronously

That's exactly it. This is mostly useful for images created in script.

any time I'm not listening for the load event and assuming the image is fully ready at that time, I'd be able to sprinkle async on the elements, and there would be no observable difference

Yes. The UA may choose to do async decoding for some or all elements. Addition of the "async" attribute to an may be treated as a hint or may do nothing.

smfr commented Oct 18, 2016

... anticipate that when I insert it into the document it will be painted synchronously

That's exactly it. This is mostly useful for images created in script.

any time I'm not listening for the load event and assuming the image is fully ready at that time, I'd be able to sprinkle async on the elements, and there would be no observable difference

Yes. The UA may choose to do async decoding for some or all elements. Addition of the "async" attribute to an may be treated as a hint or may do nothing.

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Oct 18, 2016

Member

OK, cool, glad I understand!

What do you think of an alternate proposal, to delay the load event until decoding is finished, in all cases?

Member

domenic commented Oct 18, 2016

OK, cool, glad I understand!

What do you think of an alternate proposal, to delay the load event until decoding is finished, in all cases?

@smfr

This comment has been minimized.

Show comment
Hide comment
@smfr

smfr Oct 18, 2016

What do you think of an alternate proposal, to delay the load event until decoding is finished, in all cases?

You can't do that, because loading and decoding are two separate steps. Not all images that are loaded are decoded (we never decode those that don't get painted).

Note that fetching metadata from an image, to get its size, for example, doesn't count as decoding.

smfr commented Oct 18, 2016

What do you think of an alternate proposal, to delay the load event until decoding is finished, in all cases?

You can't do that, because loading and decoding are two separate steps. Not all images that are loaded are decoded (we never decode those that don't get painted).

Note that fetching metadata from an image, to get its size, for example, doesn't count as decoding.

@ojanvafai

This comment has been minimized.

Show comment
Hide comment
@ojanvafai

ojanvafai Oct 19, 2016

I'm pretty sure Chrome strongly supports doing something roughly like this. I'm consulting with folks to make sure I say something representative of our opinion since we've had a lot of complicated discussion about this issue. Will report back soon (please ping me if I lose track of it).

ojanvafai commented Oct 19, 2016

I'm pretty sure Chrome strongly supports doing something roughly like this. I'm consulting with folks to make sure I say something representative of our opinion since we've had a lot of complicated discussion about this issue. Will report back soon (please ping me if I lose track of it).

@igrigorik

This comment has been minimized.

Show comment
Hide comment
@igrigorik

igrigorik Oct 19, 2016

Member

Naive question: how would "ready" event work in cases where we have hardware decoding?

Member

igrigorik commented Oct 19, 2016

Naive question: how would "ready" event work in cases where we have hardware decoding?

@smfr

This comment has been minimized.

Show comment
Hide comment
@smfr

smfr Oct 19, 2016

I don't think it matters. Hardware decoding happens under a software call to the decoder, just like software decoding. It just might be faster.

smfr commented Oct 19, 2016

I don't think it matters. Hardware decoding happens under a software call to the decoder, just like software decoding. It just might be faster.

@junov

This comment has been minimized.

Show comment
Hide comment
@junov

junov Oct 21, 2016

Collaborator

Responding to @domenic 's earlier comment that there is overlap with ImageBitmap... Yes and no.

I've experimented with getting jank-free scrolling and canvas draws under intense image loading. I used XHR -> creatImageBitmap -> canvas. It worked like a charm, but it has one non-negligible drawback: it pins all the decoded images in RAM unless the app explicitly discards them, so implementing something like Facebook's infinite scrolling on top of this is a bit involved. It would require the web app to be responsible for evicting and triggering predictive redecodes in order to avoid extreme memory bloat (and OOM crashes). Considering that JS code has no visibility into the system's memory contention, using ImageBitmap as a blanket solution for de-janking image-intensive pages is really not that great.

This proposal lets the UA continue to manage the memory occupied by decoded resources, which is a big win for many use cases.

On the other hand, the ImageBitmap approach guarantees that the image will be drawn and that it will be fast. This is useful for cases that don't jive with this statement of the proposal: "if the UA paints an image after the "load" event has fired, but before the image has been decoded, the UA is allowed to not paint the image (rather than block on decoding it)"

Collaborator

junov commented Oct 21, 2016

Responding to @domenic 's earlier comment that there is overlap with ImageBitmap... Yes and no.

I've experimented with getting jank-free scrolling and canvas draws under intense image loading. I used XHR -> creatImageBitmap -> canvas. It worked like a charm, but it has one non-negligible drawback: it pins all the decoded images in RAM unless the app explicitly discards them, so implementing something like Facebook's infinite scrolling on top of this is a bit involved. It would require the web app to be responsible for evicting and triggering predictive redecodes in order to avoid extreme memory bloat (and OOM crashes). Considering that JS code has no visibility into the system's memory contention, using ImageBitmap as a blanket solution for de-janking image-intensive pages is really not that great.

This proposal lets the UA continue to manage the memory occupied by decoded resources, which is a big win for many use cases.

On the other hand, the ImageBitmap approach guarantees that the image will be drawn and that it will be fast. This is useful for cases that don't jive with this statement of the proposal: "if the UA paints an image after the "load" event has fired, but before the image has been decoded, the UA is allowed to not paint the image (rather than block on decoding it)"

@smfr

This comment has been minimized.

Show comment
Hide comment
@smfr

smfr Oct 21, 2016

If the author really want the image pixels to be available on first paint, they need to either insert the image in script after receiving the "ready" callback, or toggle some CSS so that the image becomes visible.

However, there's a problem with my proposal; there's no way for the author to indicate that the UA should start decoding an image (if that image is unparented), and I don't think UAs will want to start decoding all unparented images with "async".

smfr commented Oct 21, 2016

If the author really want the image pixels to be available on first paint, they need to either insert the image in script after receiving the "ready" callback, or toggle some CSS so that the image becomes visible.

However, there's a problem with my proposal; there's no way for the author to indicate that the UA should start decoding an image (if that image is unparented), and I don't think UAs will want to start decoding all unparented images with "async".

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Oct 21, 2016

Member

However, there's a problem with my proposal; there's no way for the author to indicate that the UA should start decoding an image (if that image is unparented), and I don't think UAs will want to start decoding all unparented images with "async".

Since this is most useful for images created in script anyway, maybe an imperative "decode me now" API? Like

img.ensureDecoded().then(() => {
  // OK, now you can insert it into the document
});

Maybe "ensureReady" if we want to hide the implementation details a little more.

Member

domenic commented Oct 21, 2016

However, there's a problem with my proposal; there's no way for the author to indicate that the UA should start decoding an image (if that image is unparented), and I don't think UAs will want to start decoding all unparented images with "async".

Since this is most useful for images created in script anyway, maybe an imperative "decode me now" API? Like

img.ensureDecoded().then(() => {
  // OK, now you can insert it into the document
});

Maybe "ensureReady" if we want to hide the implementation details a little more.

@smfr

This comment has been minimized.

Show comment
Hide comment
@smfr

smfr Oct 22, 2016

That could work, yes. I think that means you don't need an "async" attribute for images created in JS.

It could still be useful as a hint on content images to say that you're OK with an image temporarily painting blank even after the load event fires, trading off against possible UI stalls. No doubt people will want CSS pseudo classes to style the various states of an image.

smfr commented Oct 22, 2016

That could work, yes. I think that means you don't need an "async" attribute for images created in JS.

It could still be useful as a hint on content images to say that you're OK with an image temporarily painting blank even after the load event fires, trading off against possible UI stalls. No doubt people will want CSS pseudo classes to style the various states of an image.

@vmpstr

This comment has been minimized.

Show comment
Hide comment
@vmpstr

vmpstr Oct 25, 2016

Contributor

As @ojanvafai mentioned, here are our (Chrome) thoughts after having discussed with our team:

We agree that this is an important area to look into and we’re excited to work on solutions in this space!

We see two important use cases with different characteristics.
(1) Deferring: Ability to defer decodes of images in dom order to avoid janking the page.
(2) Predecoding: Ability to know when images have been decoded so they can be inserted into dom without jank

An async attribute on images solves deferring, but not predecoding. An on ready event solves predecoding, but not deferring. It seems like these could potentially be solved separately.

We have some concerns about tricky parts of the ready event:
* Does every image get "on ready" or just ones marked async?
* Would a browser need to predecode all out of dom images so that they would receive this event? Or just ones marked async?
* Is there an "on unready" event?
* If an image is evicted and then re-decoded, is there a second ready event? The original proposal said it’s just the first time, but we’re worried this is too confusing and can lead to jank in subsequent uses.
* If an author has a series of images (e.g. carousel) out of dom, how does an author communicate which ones are important to decode first?

Here's two rough proposals to handle each of these use cases.

(1) Proposal for handling deferring images:
Give the async attribute three values: async, auto, and sync. sync behaves as today’s image elements do without any attribute specified, where once an image has been loaded it will appear immediately (and be decoded synchronously) if inserted into the document. Images marked async can get loaded/decoded best-effort without janking. auto would involve browser heuristics to decide if the image could and should be loaded async or if it needs to be sync. async is therefore just a more aggressive version of auto in practice. sync is mostly just there as a safety valve for developers in case browser heuristics get it wrong.

There’s no ready event here, as it’s not part of this use case. In particular, this helps avoid confusion around what happens when a decoded image is discarded and then later decoded again (e.g. because the image was scrolled out of view and then back in). The browser can treat it the same way regardless of whether it was the first time the image was being decoded or not.

We’re not 100% sure we’ll need the async value without more experience on trying to ship changes here. It might be that we can make all images that we want to async and just have auto and sync in the end.

(2) Proposal for predecoding images (similar to what @domenic suggested):
Provide an explicit decode function for an image with some sort of callback when the image is decoded. This would notify the author that it's ready to be appended to the page without jank. The browser would promise to keep that image decoded for at least that raf frame. It would also allow an author to call this decode function again in the future if needed, and to prioritize images in an image carousel case. Finally, we might want to make this decode function return some success/failure (in case of an overloaded cache).

Possible sample code:
var img = new Image();
img.src = ‘...’;
// The decoded image is guaranteed to be kept in memory until the frame
// after the decode promise resolves.
img.decode().then(function() {
// This is guaranteed to paint the image without flicker or decoding jank.
document.body.appendChild(img);
});

Other side questions:
* What about large images that are scaled and cropped? Do these parameters need to be specified? We found that scaling and gpu uploading can be a significant part of image preparation time. Do we just punt on this?
* How is image data that isn’t image elements handled, e.g. should we add an async keyword for background images? Divorcing async from decode makes it easier to do something consistent between image elements and CSS background images.

Contributor

vmpstr commented Oct 25, 2016

As @ojanvafai mentioned, here are our (Chrome) thoughts after having discussed with our team:

We agree that this is an important area to look into and we’re excited to work on solutions in this space!

We see two important use cases with different characteristics.
(1) Deferring: Ability to defer decodes of images in dom order to avoid janking the page.
(2) Predecoding: Ability to know when images have been decoded so they can be inserted into dom without jank

An async attribute on images solves deferring, but not predecoding. An on ready event solves predecoding, but not deferring. It seems like these could potentially be solved separately.

We have some concerns about tricky parts of the ready event:
* Does every image get "on ready" or just ones marked async?
* Would a browser need to predecode all out of dom images so that they would receive this event? Or just ones marked async?
* Is there an "on unready" event?
* If an image is evicted and then re-decoded, is there a second ready event? The original proposal said it’s just the first time, but we’re worried this is too confusing and can lead to jank in subsequent uses.
* If an author has a series of images (e.g. carousel) out of dom, how does an author communicate which ones are important to decode first?

Here's two rough proposals to handle each of these use cases.

(1) Proposal for handling deferring images:
Give the async attribute three values: async, auto, and sync. sync behaves as today’s image elements do without any attribute specified, where once an image has been loaded it will appear immediately (and be decoded synchronously) if inserted into the document. Images marked async can get loaded/decoded best-effort without janking. auto would involve browser heuristics to decide if the image could and should be loaded async or if it needs to be sync. async is therefore just a more aggressive version of auto in practice. sync is mostly just there as a safety valve for developers in case browser heuristics get it wrong.

There’s no ready event here, as it’s not part of this use case. In particular, this helps avoid confusion around what happens when a decoded image is discarded and then later decoded again (e.g. because the image was scrolled out of view and then back in). The browser can treat it the same way regardless of whether it was the first time the image was being decoded or not.

We’re not 100% sure we’ll need the async value without more experience on trying to ship changes here. It might be that we can make all images that we want to async and just have auto and sync in the end.

(2) Proposal for predecoding images (similar to what @domenic suggested):
Provide an explicit decode function for an image with some sort of callback when the image is decoded. This would notify the author that it's ready to be appended to the page without jank. The browser would promise to keep that image decoded for at least that raf frame. It would also allow an author to call this decode function again in the future if needed, and to prioritize images in an image carousel case. Finally, we might want to make this decode function return some success/failure (in case of an overloaded cache).

Possible sample code:
var img = new Image();
img.src = ‘...’;
// The decoded image is guaranteed to be kept in memory until the frame
// after the decode promise resolves.
img.decode().then(function() {
// This is guaranteed to paint the image without flicker or decoding jank.
document.body.appendChild(img);
});

Other side questions:
* What about large images that are scaled and cropped? Do these parameters need to be specified? We found that scaling and gpu uploading can be a significant part of image preparation time. Do we just punt on this?
* How is image data that isn’t image elements handled, e.g. should we add an async keyword for background images? Divorcing async from decode makes it easier to do something consistent between image elements and CSS background images.

@smfr

This comment has been minimized.

Show comment
Hide comment
@smfr

smfr Oct 27, 2016

I like this analysis.

(1) Proposal for handling deferring images
... Images marked async can get loaded/decoded best-effort without janking

Presumably the effects of "async" on a content image is that it can show blank after its load event has fired, right? Seems fine if so. "auto" is weird as an attribute; I would prefer that we say that "async" is a hint and the UA is free to ignore it; if it becomes a cargo cult, I don't want to have to respect it on thousands of images in a document.

(2) Proposal for predecoding images
I like the decode() Promise form; it makes it easy to figure out the behavior with repeated calls on the same image.

What about large images that are scaled and cropped? Do these parameters need to be specified? We found that scaling and gpu uploading can be a significant part of image preparation time. Do we just punt on this?

I say punt; doing scaling/cropping is usually a paint-time operation, and if you tried to do something for these, you'd end up having to do lots of things in the painting pipeline.

  • How is image data that isn’t image elements handled, e.g. should we add an async keyword for background images?

Since you can't detect load on CSS images directly, it seems less important to support "async" for them.

smfr commented Oct 27, 2016

I like this analysis.

(1) Proposal for handling deferring images
... Images marked async can get loaded/decoded best-effort without janking

Presumably the effects of "async" on a content image is that it can show blank after its load event has fired, right? Seems fine if so. "auto" is weird as an attribute; I would prefer that we say that "async" is a hint and the UA is free to ignore it; if it becomes a cargo cult, I don't want to have to respect it on thousands of images in a document.

(2) Proposal for predecoding images
I like the decode() Promise form; it makes it easy to figure out the behavior with repeated calls on the same image.

What about large images that are scaled and cropped? Do these parameters need to be specified? We found that scaling and gpu uploading can be a significant part of image preparation time. Do we just punt on this?

I say punt; doing scaling/cropping is usually a paint-time operation, and if you tried to do something for these, you'd end up having to do lots of things in the painting pipeline.

  • How is image data that isn’t image elements handled, e.g. should we add an async keyword for background images?

Since you can't detect load on CSS images directly, it seems less important to support "async" for them.

@vmpstr

This comment has been minimized.

Show comment
Hide comment
@vmpstr

vmpstr Oct 27, 2016

Contributor

Presumably the effects of "async" on a content image is that it can show blank after its load event has fired, right? Seems fine if so. "auto" is weird as an attribute; I would prefer that we say that "async" is a hint and the UA is free to ignore it; if it becomes a cargo cult, I don't want to have to respect it on thousands of images in a document.

To clarify a bit, the proposal is to have a single attribute called async, which could have three possible values (async/sync/auto). I agree that these values should still be just hints to the UA and it can ignore them.

The idea behind auto value and how it's different from async is that it allows the UA to adjust its heuristics to either include or exclude more images from being async based on whatever criteria it decides. In other words, async value would mean it's always safe to show a blank image after the load event fired; auto would mean that the UA should decide whether or not it is OK do this.

It's not entirely clear that we need both auto and async, but if auto is the default it would allow the UA to make these decisions for content that didn't explicitly specify what it wants.

Contributor

vmpstr commented Oct 27, 2016

Presumably the effects of "async" on a content image is that it can show blank after its load event has fired, right? Seems fine if so. "auto" is weird as an attribute; I would prefer that we say that "async" is a hint and the UA is free to ignore it; if it becomes a cargo cult, I don't want to have to respect it on thousands of images in a document.

To clarify a bit, the proposal is to have a single attribute called async, which could have three possible values (async/sync/auto). I agree that these values should still be just hints to the UA and it can ignore them.

The idea behind auto value and how it's different from async is that it allows the UA to adjust its heuristics to either include or exclude more images from being async based on whatever criteria it decides. In other words, async value would mean it's always safe to show a blank image after the load event fired; auto would mean that the UA should decide whether or not it is OK do this.

It's not entirely clear that we need both auto and async, but if auto is the default it would allow the UA to make these decisions for content that didn't explicitly specify what it wants.

@smfr

This comment has been minimized.

Show comment
Hide comment
@smfr

smfr Nov 18, 2017

You don't want to have to decode before firing the load event, because decoding involves allocating memory for the image backing store, and at that time you don't know if that image will ever be painted.

smfr commented Nov 18, 2017

You don't want to have to decode before firing the load event, because decoding involves allocating memory for the image backing store, and at that time you don't know if that image will ever be painted.

@othermaciej

This comment has been minimized.

Show comment
Hide comment
@othermaciej

othermaciej Nov 18, 2017

a bunch of the use cases for decoding=sync don't actually require sync decoding that blocks the main thread -- they just require an event that fires when decoding is complete, whether that's the load event or the result of the decode() promise.

Those use cases should use image.decode() instead of decoding=sync. Do we need a second way to do what image.decode() does?

othermaciej commented Nov 18, 2017

a bunch of the use cases for decoding=sync don't actually require sync decoding that blocks the main thread -- they just require an event that fires when decoding is complete, whether that's the load event or the result of the decode() promise.

Those use cases should use image.decode() instead of decoding=sync. Do we need a second way to do what image.decode() does?

chromium-wpt-export-bot added a commit to web-platform-tests/wpt that referenced this issue Nov 20, 2017

Rename image async attribute to decoding attribute.
This patch renames the async attribute to be the decoding attribute
due to discussion on whatwg/html#1920

This also adds wpt test for the decoding attribute

R=chrishtr@chromium.org, domenic@chromium.org

Cq-Include-Trybots: master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2
Change-Id: I0524b7e3849c9a922f358c0592d34c30a7fd8914

chromium-wpt-export-bot added a commit to web-platform-tests/wpt that referenced this issue Nov 22, 2017

Rename image async attribute to decoding attribute.
This patch renames the async attribute to be the decoding attribute
due to discussion on whatwg/html#1920

This also adds wpt test for the decoding attribute

R=chrishtr@chromium.org, domenic@chromium.org

Cq-Include-Trybots: master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2
Change-Id: I0524b7e3849c9a922f358c0592d34c30a7fd8914
@vmpstr

This comment has been minimized.

Show comment
Hide comment
@vmpstr

vmpstr Nov 22, 2017

Contributor

decoding=sync don't actually require sync decoding that blocks the main thread -- they just require an event that fires when decoding is complete

I'm not sure that there is even a requirement for the event to fire when decoding is complete. The way the proposal specs this, there is no requirement for any javascript code to be able to determine the effect of different values of "decoding" attribute (other than inspecting the attribute itself of course).

I agree with @othermaciej that I think the cases where the event is desired are best suited for the img.decode() api.

Contributor

vmpstr commented Nov 22, 2017

decoding=sync don't actually require sync decoding that blocks the main thread -- they just require an event that fires when decoding is complete

I'm not sure that there is even a requirement for the event to fire when decoding is complete. The way the proposal specs this, there is no requirement for any javascript code to be able to determine the effect of different values of "decoding" attribute (other than inspecting the attribute itself of course).

I agree with @othermaciej that I think the cases where the event is desired are best suited for the img.decode() api.

@aghassemi

This comment has been minimized.

Show comment
Hide comment
@aghassemi

aghassemi Nov 27, 2017

@vmpstr @smfr @othermaciej One common use-case is hiding a placeholder (e.g. small, in-lined pixalated version of an image) or loading indicator as soon as image is painted.

If we use decoding=async and use load event to hide the placeholder, we may hide it too early.
If we use image.decode() before appending the image instead of attr, we lose progressive decoding.

An event to know decoding is finished combined with decoding=async seems to be needed.

@smfr, as a workaround, I am curious if we can still use image.decode() purely as a signal to know when decoding is done and combine it with the decoding=async attribute. Can this cause double work?

aghassemi commented Nov 27, 2017

@vmpstr @smfr @othermaciej One common use-case is hiding a placeholder (e.g. small, in-lined pixalated version of an image) or loading indicator as soon as image is painted.

If we use decoding=async and use load event to hide the placeholder, we may hide it too early.
If we use image.decode() before appending the image instead of attr, we lose progressive decoding.

An event to know decoding is finished combined with decoding=async seems to be needed.

@smfr, as a workaround, I am curious if we can still use image.decode() purely as a signal to know when decoding is done and combine it with the decoding=async attribute. Can this cause double work?

@smfr

This comment has been minimized.

Show comment
Hide comment
@smfr

smfr Nov 27, 2017

An event to know decoding is finished combined with decoding=async seems to be needed.

But the browser doesn't want to decode images that it doesn't know are going to get painted. This is extra work and possibly massive memory use.

@smfr, as a workaround, I am curious if we can still use image.decode() purely as a signal to know when decoding is done and combine it with the decoding=async attribute. Can this cause double work?

image.decode() is for forcing an (async) decode, not just for detecting when it's done. I don't think using the two things together makes much sense.

smfr commented Nov 27, 2017

An event to know decoding is finished combined with decoding=async seems to be needed.

But the browser doesn't want to decode images that it doesn't know are going to get painted. This is extra work and possibly massive memory use.

@smfr, as a workaround, I am curious if we can still use image.decode() purely as a signal to know when decoding is done and combine it with the decoding=async attribute. Can this cause double work?

image.decode() is for forcing an (async) decode, not just for detecting when it's done. I don't think using the two things together makes much sense.

@aghassemi

This comment has been minimized.

Show comment
Hide comment
@aghassemi

aghassemi Nov 27, 2017

@smfr I didn't have the expectation that it will fire for every image. Maybe an event name like painted is more suitable than decoded to convey that behavior. This will still be very useful to hide partial loading overlays or background placeholders perfectly in sync with image painting.

aghassemi commented Nov 27, 2017

@smfr I didn't have the expectation that it will fire for every image. Maybe an event name like painted is more suitable than decoded to convey that behavior. This will still be very useful to hide partial loading overlays or background placeholders perfectly in sync with image painting.

chromium-wpt-export-bot added a commit to web-platform-tests/wpt that referenced this issue Nov 27, 2017

Rename image async attribute to decoding attribute.
This patch renames the async attribute to be the decoding attribute
due to discussion on whatwg/html#1920

This also adds wpt test for the decoding attribute

R=chrishtr@chromium.org, domenic@chromium.org

Cq-Include-Trybots: master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2
Change-Id: I0524b7e3849c9a922f358c0592d34c30a7fd8914
Reviewed-on: https://chromium-review.googlesource.com/770106
Commit-Queue: vmpstr <vmpstr@chromium.org>
Reviewed-by: Chris Harrelson <chrishtr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#519453}
@vmpstr

This comment has been minimized.

Show comment
Hide comment
@vmpstr

vmpstr Nov 28, 2017

Contributor

The decision to paint (rasterize) content can happen pretty far down the pipeline and independently of the main thread. I think bubbling up an event whenever we decode an image is a lot of overhead, especially if there are only a few cases where it is useful. It's also a bit unclear if the event should be fired for subsequent decodes that may happen on the same image.

In general, the situation you describe would be best suited for (intended use of) the decode api. That is, show a low res image, call a decode() on a full res image and append/replace the content in the promise resolution. You are right that since this implies the load event, we'd lose the progressive decoding.

Just thinking out loud, but I don't readily see an elegant way to get all three of the following:

  1. async behavior
  2. progressive decoding
  3. a signal that the final image is decoded

1+2 sounds like it could be handled by decoding=async
2+3 is implied by not specifying anything (ie decoding=auto or maybe even decoding=sync)
1+3 is the decode() api

Contributor

vmpstr commented Nov 28, 2017

The decision to paint (rasterize) content can happen pretty far down the pipeline and independently of the main thread. I think bubbling up an event whenever we decode an image is a lot of overhead, especially if there are only a few cases where it is useful. It's also a bit unclear if the event should be fired for subsequent decodes that may happen on the same image.

In general, the situation you describe would be best suited for (intended use of) the decode api. That is, show a low res image, call a decode() on a full res image and append/replace the content in the promise resolution. You are right that since this implies the load event, we'd lose the progressive decoding.

Just thinking out loud, but I don't readily see an elegant way to get all three of the following:

  1. async behavior
  2. progressive decoding
  3. a signal that the final image is decoded

1+2 sounds like it could be handled by decoding=async
2+3 is implied by not specifying anything (ie decoding=auto or maybe even decoding=sync)
1+3 is the decode() api

chromium-wpt-export-bot added a commit to web-platform-tests/wpt that referenced this issue Nov 28, 2017

Rename image async attribute to decoding attribute.
This patch renames the async attribute to be the decoding attribute
due to discussion on whatwg/html#1920

This also adds wpt test for the decoding attribute

R=chrishtr@chromium.org, domenic@chromium.org

Cq-Include-Trybots: master.tryserver.chromium.linux:linux_layout_tests_slimming_paint_v2
Change-Id: I0524b7e3849c9a922f358c0592d34c30a7fd8914
Reviewed-on: https://chromium-review.googlesource.com/770106
Commit-Queue: vmpstr <vmpstr@chromium.org>
Reviewed-by: Chris Harrelson <chrishtr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#519453}
@annevk

This comment has been minimized.

Show comment
Hide comment
@annevk

annevk Nov 28, 2017

Member

Also thinking out load: I guess in the future we could add something to decode() that roughly means "take your time"? So you can invoke it early, but the browser knows you're just using the resolved promise as a signal and is therefore not in a hurry to decode.

Member

annevk commented Nov 28, 2017

Also thinking out load: I guess in the future we could add something to decode() that roughly means "take your time"? So you can invoke it early, but the browser knows you're just using the resolved promise as a signal and is therefore not in a hurry to decode.

@aghassemi

This comment has been minimized.

Show comment
Hide comment
@aghassemi

aghassemi Nov 29, 2017

@vmpstr curious why bubbling up an asyc flag from down the stack is expensive? I assume events like playing for video come from deep parts of the stack as well.

For now we are doing 1+2 and will assess if the transition between placeholder/image would flicker. /cc @cramforce

@annevk that's what I sort of assumed decode() already does since I was picturing it as being just the imperative version of the decoding=async attribute.

aghassemi commented Nov 29, 2017

@vmpstr curious why bubbling up an asyc flag from down the stack is expensive? I assume events like playing for video come from deep parts of the stack as well.

For now we are doing 1+2 and will assess if the transition between placeholder/image would flicker. /cc @cramforce

@annevk that's what I sort of assumed decode() already does since I was picturing it as being just the imperative version of the decoding=async attribute.

@vmpstr

This comment has been minimized.

Show comment
Hide comment
@vmpstr

vmpstr Nov 29, 2017

Contributor

The difficulty with something like a decoded attribute is that its value may change at times that I would say are not interesting. For example, in Chromium images are sometimes decoded outside of the viewport based on some heuristics that suggests that we may soon need to rasterize content that uses these images.

However, due to user interactions we may not actually use the image that we decoded. Furthermore, the decode's memory backing may be discarded meaning the image is no longer decoded, but doing the query for this information is done only if we actually need the image to rasterize. Once we use an image to rasterize content, it may also be evicted since its memory backing is no longer necessary to present content.

If I'm understanding this correctly, to implement a decoded attribute, each event that causes a decode or eviction would have to come with corresponding communication with the main thread so that javascript can observe the change. (and this would have to happen for every image?)

So perhaps saying that this is a lot of overhead was premature. I wouldn't necessarily claim that this is either cheap or expensive; I'm simply skeptical that the value this attribute brings warrants this complexity. If there's an interest in this type of attribute, maybe we can split off the discussion into a separate issue?

Contributor

vmpstr commented Nov 29, 2017

The difficulty with something like a decoded attribute is that its value may change at times that I would say are not interesting. For example, in Chromium images are sometimes decoded outside of the viewport based on some heuristics that suggests that we may soon need to rasterize content that uses these images.

However, due to user interactions we may not actually use the image that we decoded. Furthermore, the decode's memory backing may be discarded meaning the image is no longer decoded, but doing the query for this information is done only if we actually need the image to rasterize. Once we use an image to rasterize content, it may also be evicted since its memory backing is no longer necessary to present content.

If I'm understanding this correctly, to implement a decoded attribute, each event that causes a decode or eviction would have to come with corresponding communication with the main thread so that javascript can observe the change. (and this would have to happen for every image?)

So perhaps saying that this is a lot of overhead was premature. I wouldn't necessarily claim that this is either cheap or expensive; I'm simply skeptical that the value this attribute brings warrants this complexity. If there's an interest in this type of attribute, maybe we can split off the discussion into a separate issue?

@cramforce

This comment has been minimized.

Show comment
Hide comment
@cramforce

cramforce Nov 29, 2017

I think a decode event is too coupled to the implementation. What would be interesting is an event for "I'm about to or I could without extra decoding work paint pixels for this image. Either the full image or the initial progressive render."

cramforce commented Nov 29, 2017

I think a decode event is too coupled to the implementation. What would be interesting is an event for "I'm about to or I could without extra decoding work paint pixels for this image. Either the full image or the initial progressive render."

@othermaciej

This comment has been minimized.

Show comment
Hide comment
@othermaciej

othermaciej Nov 30, 2017

If the requirement is to have a signal when the image is decoded enough to paint, even if not fully decoded, maybe that could be a parameter to image.decode(), which would then start decoding as usual, but trigger the promise at minimally ready to paint time instead of fully decoded time.

othermaciej commented Nov 30, 2017

If the requirement is to have a signal when the image is decoded enough to paint, even if not fully decoded, maybe that could be a parameter to image.decode(), which would then start decoding as usual, but trigger the promise at minimally ready to paint time instead of fully decoded time.

@othermaciej

This comment has been minimized.

Show comment
Hide comment
@othermaciej

othermaciej Nov 30, 2017

(I don't think you want to trigger based on the image actually being painted, since it might not even be inserted into the DOM until ready to paint, and since layout engines might be smart enough not to paint it at all if it's fully covered by an opaque placeholder.)

othermaciej commented Nov 30, 2017

(I don't think you want to trigger based on the image actually being painted, since it might not even be inserted into the DOM until ready to paint, and since layout engines might be smart enough not to paint it at all if it's fully covered by an opaque placeholder.)

@chrishtr

This comment has been minimized.

Show comment
Hide comment
@chrishtr

chrishtr Dec 1, 2017

Hi all,

I propose to consider this issue complete for the purpose of committing the spec update in
#3221. The discussion happening now is good, but probably
best in a new issue to separate concerns.

Any objections?

chrishtr commented Dec 1, 2017

Hi all,

I propose to consider this issue complete for the purpose of committing the spec update in
#3221. The discussion happening now is good, but probably
best in a new issue to separate concerns.

Any objections?

@vmpstr vmpstr referenced this issue Dec 1, 2017

Closed

Img decoding attribute #220

1 of 3 tasks complete
@annevk

This comment has been minimized.

Show comment
Hide comment
@annevk

annevk Dec 2, 2017

Member

I also lean in that direction. We can have a new issue to discuss a signal of sorts for the scenarios discussed above. I suggest we merge #3221 (and close this issue) somewhere Tuesday to give folks a few more days to raise concerns with this line of action.

Member

annevk commented Dec 2, 2017

I also lean in that direction. We can have a new issue to discuss a signal of sorts for the scenarios discussed above. I suggest we merge #3221 (and close this issue) somewhere Tuesday to give folks a few more days to raise concerns with this line of action.

@annevk annevk closed this in #3221 Dec 5, 2017

annevk added a commit that referenced this issue Dec 5, 2017

Add <img decoding>
The decoding attribute indicates a decoding hint to the user agent. This hint aids the user agent in deciding how to process and decode the image before rasterizing it. Possible values are as follows:

* "sync": prefer to decode the image synchronously for atomic presentation with other content
* "async": prefer to decode the image asynchronously to reduce delay in presenting other content
* "auto": default mode, which indicates no preference for the decoding mode.

Fixes #1920.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment