When and how should we load media metadata artwork?

[annevk]

It seems highly insufficient UI-wise to not have preloaded the image and a bit of the music file. Since you basically cannot release a player that does not have that, we should make sure that the examples cover it I think, as it might illustrate some API holes as well. (E.g. we might need to have more state on when artwork is loaded and such.)

[richtr]

Right now the spec does not say we must wait for the fetch steps to complete before showing media controls.

I agree it could be good to have a preloaded image. Perhaps we should define 'display media controls' steps similar to notification's display steps that wait until any related fetch steps have completed.

Alternatively, maybe it is ok to 'lazy load' artwork URLs so the fetch steps do not block any display steps.

WDYT?

[annevk]

I can imagine being okay with showing the artwork at a later point (if you're bandwidth constrained, or the user skips very quickly, or some such), but I think by default you want to show the correct artwork and the API needs to accommodate for that.

An alternative could be that you pass a Blob rather than a URL, but maybe that's too low-level... If you stick with URLs though I think you might want either events or some kind of property that returns a promise when everything is loaded.

And the examples should indicate that while you play the current song, you're already caching bits for the next song. And that once you start playing the next song, enough is cached to show the image and play it. (And then you start caching for the song after that of course, or some such.)

[foolip]

setMetadata itself could return a promise that resolves when the artwork is loaded. Not sure if we'll regret that if we end up with more than one kind of artwork, though.

[annevk]

It seems setMetadata() is the wrong API, since you might end up changing title/etc. while still showing the old artwork?

[annevk]

Perhaps as with <canvas> we can require one of HTMLImageElement & friends?

[foolip]

Is ImageBitmap a good primitive for an already-loaded-and-decoded image? We've talked about that, but it seems annoying if all you have is a URL, and what about cross-origin artwork?

[annevk]

Passing a URL might still be okay, but in that case you need a distinct object I think that holds all the metadata and is responsible for loading assets. And then separately you can decide when that object is going to take over from what is displayed now.

[Constructor(MediaSessionDataInit init)]
interface MediaSessionData {
  ...
  readonly attribute Promise<boolean> done;
};
dictionary MediaSessionDataInit {
  ...
};
partial interface MediaSession {
  void setSessionData(MediaSessionData data);
};

[foolip]

Yeah, something like that could work. We had talked about having a mutable MediaSessionData object on the MediaSession object, but that would amount to a tear-off and for any mutation one would have to enqueue a microtask to actually use the new state to group all changes in a script run. Having a separate constructible object with readonly members doesn't have those problems.

[richtr]

This issue was originally discussed in #44.

One way a web developer could ensure media artwork loads immediately is to pass a pre-fetched Blob or data URI as the 'artwork URL'.

Media artwork URL fetching is (currently) unobservable from a web page so perhaps we could defer to the implementation stage to figure out exactly when this fetch should happen. It feels like we can have interoperability in the JS platform without initially specifying this and there are different fetch strategies that could work quite well here.

I could be wrong :/

[annevk]

We should not rely on fetching blob or data URLs (especially the former) to happen synchronously.

[richtr]

We should not rely on fetching blob or data URLs (especially the former) to happen synchronously.

Right. I also plan to remove invocation of the fetch steps from the setMetadata(metadata) algorithm.

At this point it's unclear if we need to change the API. If we want to discuss changing setMetadata to something else perhaps we should re-open #44?

[annevk]

If you remove the fetch steps, when would the image be fetched?

[foolip]

I've been trying to prototype a solution for this. First, a URL and Promise pair:

partial interface MediaMetadata {
  attribute USVString artwork;
  readonly attribute Promise<Response> artworkLoaded;
}

Setting artwork starts the fetch, and artworkLoaded is the promise returned by the internal fetch(). I'm not a huge fan, it's kind of duplicating fetch() and doing metadata.artworkLoaded.then(...) before setting metadata.artwork wouldn't work.

Another idea, not implemented, and possibly crazy:

partial interface MediaMetadata {
  attribute Promise<Response> artwork;
}

The API would then be metadata.artwork = fetch(artworkURL), which looks OK, but no existing API works like this. I think making the type simply Response would not work, because then one cannot start the fetch at the latest opportunity, which a simple USVString does allow.

@annevk, advice on how to integrate with Fetch much appreciated!

[foolip]

We need to resolve this in order to put artwork back in the spec. Currently, I'm thinking that the MediaMetadata constructor shouldn't fetch, but rather the fetch steps (currently never invoked) should be run when the browser knows it's going to use the artwork.

But, how do we make prefetching the artwork load? The best I can come up with is a fetchArtwork() method, but there's something not-great about integrating with Fetch by adding a new method that's a specialized variant of fetch().

@annevk, thoughts on Fetch integration would be much appreciated. At one point I was thinking that the artwork attribute should be of type Promise<Response>, but that would be a first for the platform I think.

[foolip]

@annevk replied (I asked on the wrong issue at first):

For notifications we just fetch as a side effect of the constructor/factory method. That's not super clean. A promise for a Response object does not seem entirely unreasonable. I guess it depends a bit as to what you're aiming for. Looking at the latest in <canvas> might also be an idea, by just feeding artwork one of the various ways to represent an image.

[foolip]

The simple options when going with artwork as a URL are:

1. Fetch in the MediaMetadata constructor
2. Fetch when setting the session.metadata attribute
3. Don't define exactly when to Fetch

[foolip]

Looking at the latest in <canvas> might also be an idea, by just feeding artwork one of the various ways to represent an image.

Do you mean CanvasImageSource? I'm thinking that just for notification icons, it's likely that media session artwork will evolve to allow some kind of manifest for different screen densities and similar, could that be done by pointing to an img element that's the child of a picture element?

[annevk]

img elements are probably not a good fit as the artwork will be rendered outside the normal layout tree. I guess the simplest here would be to do something similar to notifications, which is also a little vague as to when fetching happens.

Did you check what iOS and Android do?

[foolip]

Yes, @doomdavve took a close look at the iOS and Android APIs when we were starting this. IIRC, both allow you to either give a URL or a handle of sorts to a decoded image. We'll be implementing it by passing a decoded image, so that gives us rather a lot of freedom in how the web-exposed bits look.

I'm leaning towards just going with an URL for now and not defining when the fetch steps are invoked, just what they do. Do you think it would be preposterous to later add fetchArtwork() or similar for preloading, or is there just no master plan for how to do preloading using Fetch for the gazillion APIs that currently use URLs only?

[annevk]

I think for preloading you'd use https://w3c.github.io/resource-hints/.

[foolip]

Thanks, I knew that was a thing but it still didn't come to mind! There's also https://w3c.github.io/preload/ and it looks like something like this might work:

var metadata = new MediaMetadata({ artwork: "artwork.jpg" });
var res = document.createElement("link");
res.rel = "preload";
res.as = "image";
res.href = metadata.artwork;
document.head.appendChild(res);

I'm unsure if getting the referrer right might ruin it, @richtr's spec text that I've revived says "no referrer" but I assume that the above would send a referrer. Is there a way to inhibit that? The referrerpolicy isn't on HTMLLinkElement in https://w3c.github.io/webappsec-referrer-policy/ and I guess there's a reason. @mikewest @jeisinger?

[annevk]

Hmm, per w3c/webappsec-referrer-policy#15 that should have been fixed.

[jeisinger]

right, however, it's not yet implemented nor shipped in chrome

[foolip]

OK, sounds like the right primitives are in the pipeline, so for #126 I think I'll keep the "fetch steps" as late as possible, and lean on preloading to handle this. Thanks all!

[mounirlamouri]

FWIW, Android doesn't quite work well with URLs for notifications.

IMO, it would be better to let the UA decide when to fetch. Ideally, if the web application cares about the artwork being available quickly, it could download it and cache it in a SW so it would work offline and be quick to grab on demand.

[foolip]

FWIW, Android doesn't quite work well with URLs for notifications.

Is this only because the resource is needed in many different sizes, or is the timing of the fetch also a problem?

[mounirlamouri]

Hmm, I should have said "Android APIs". There are these METADATA_KEY_* fields that one can use to set up some info. I was told by some Android folks that the METADA_KEY_*_URI ones are not used by some components like Wear. As a result, we must use the Bitmap ones. It's also better for the UA to control the fetch instead of hand it off to the system because it allows SW to intercept it.

[foolip]

@doomdavve has a WIP prototype adding artwork to MediaMetadata where the URL is fetched and decoded as an image similar to the implementation for Notifications. So then one hands a Bitmap to the Android APIs and that seems to work well.