Feature proposal: declarative way to do art direction for <video> (maybe <source media>)

[shrpne]

https://html.spec.whatwg.org/multipage/embedded-content.html#the-source-element:the-source-element-10
Spec says:

The srcset, sizes, and media attributes must not be present.

I don't understand why and I think it's a big mistake. These attributes can save much more bytes for video content than for images.

Someone says, that video media query was supported before Google Chrome 34+: https://walterebert.com/playground/video/media-queries/
I assume they removed it because of spec.

I think spec should be changed to allow these attributes for media elements.

[joachimesque]

The media attribute should be allowed in <source> for videos. I have a case where a different video is served depending on the orientation media property—the classic "art direction" motivation. Having a media condition for <source> would be much more practical than having yet another JS listener ; after all we already enjoy one for <picture> elements, for that exact same case.

[zcorpan]

See https://twitter.com/foolip/status/1179031601998516224 (cc @foolip )

https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/5sWUMC_d8Tg/ZZ0Z7rfeCqUJ is where I removed in from Chromium. Basic issue is that the "resource selection algorithm" runs only once, so it couldn't be adaptive.

I wrote:

So the "layout adaption" or "art direction" use case seems like it's valid for wanting a declarative way to do this, but then the loading algorithm should restart when the MQ matching changes. Which may come down to: how common is art direction videos?

[zcorpan]

More complications:

• if a new video is loaded as a result of environment changes, should the UA try to seek to the same time in the new video? What if the duration is different?
• How important is a "seamless swap"? img loads the new image in the background, and swaps when it's fully loaded. What should video do?

It would be useful to review pages/apps that serve different videos based on MQ (using script), to understand what is needed.

[shrpne]

@zcorpan thank you for attracting attention to this issue.

In my opinion, this attribute has the most impact on page loading. Changing MQ and swapping video sources during website lifecycle has a secondary priority.

I think all these complications are solvable and implementing almost any behavior is better than not implementing it at all. But I agree that it should be investigated and selected the best available implementation.

My thoughts on the above complications:

"resource selection algorithm" runs only once, so it couldn't be adaptive.

It should not runs only once, instead it should run when it needed

if a new video is loaded as a result of environment changes, should the UA try to seek to the same time in the new video? What if the duration is different?

It should try to seek to the same time in the new video.
If the duration is significantly different, e.g. 10% of duration change, we can assume that it is not the same video, so in this case, we can try to seek to the same time for consistency or start from the beginning, I guess it does not really matter here.

How important is a "seamless swap"? img loads the new image in the background, and swaps when it's fully loaded. What should video do?

Seamless swap for videos is desirable

[AndresInSpace]

Edited badly: To try and clarify use cases and why I've mentioned aspect-ratio.

Problem seems to be:
There is no Art Direction for <video>

Viable Use-Cases:

1. video on mobile is a different aspect-ratio than on desktop or vice-versa
2. video on mobile is shorter in time-length than the desktop, or vice-versa
3. video on mobile is different aspect-ratio AND time-length than on desktop, or vice-versa.

An example that people may remember is that PayPal used to have a huge banner video on desktop, with a different aspect-ratio video for mobile (or maybe they just trimmer the excess width off with css, to the same effect to a degree).

@zcorpan @shrpne I agree that these seem like issues that can be fixed without breaking backwards compatibility or existing implementations of <video>. Perhaps additional attributes on <video> or <source> to achieve Art Direction on <video>'s.

I'd also like to see aspect-ratios #4968 introduced if MQ are indeed going to be added to <source> of <video>.
It seems beyond silly to discuss MQ and Art Direction for <video> without discussing aspect-ratio at all considering that is a direct use-case factor for WHY media-queries / art direction is warranted.

Reasoning:
I think the basis we can agree on here is that the developer will know whether each source is identical in video-length, aspect-ratio, or is an entirely different video.

if a new video is loaded as a result of environment changes, should the UA try to seek to the same time in the new video? What if the duration is different?
How important is a "seamless swap"? img loads the new image in the background, and swaps when it's fully loaded. What should video do?

I think an attribute on <video> like seamless="true" would solve this for when the developer knows they have same-length videos on each source and only dimensions/aspect-ratio is changing with the MQ. Default will be false to maintain compatibility.

Addition Problem - extending the base problem
However, my concern with Video Art Direction is when there are multiple <source>'s with different file formats..
Per google

Not all browsers support the same video formats. The element lets you specify multiple formats as a fallback in case the user's browser doesn't support one of them.

In this case should there be an attribute introduced to 'link' them so the browser knows they are identical in video length? <source group="1">?

This seems a bit convoluted when you start to add MQ (and width/height attributes for aspect-ratio) to each source.. the code gets a bit redundant and bloated.

Perhaps a new element, say <sgroup> should be used to specify the MQ (and aspect-ratio for the grouped sources), with multiple <source> under <sgroup> simply being a different format available.

[zcorpan]

It's impossible to evaluate proposals without first having a common understanding about use cases and requirements. See https://whatwg.org/faq#adding-new-features

I think this issue is about the art direction for video use case; please file new issues to discuss other use cases.

If you want to experiment with different syntaxes or solutions, I suggest doing that by creating a custom element that polyfills the thing you want. If this identifies missing primitives to be able to do what you want, that's useful to know. If you come up with a pattern that is common on the web and it would be better to have it natively supported, that's also useful to know.

And again, if you have examples of live web sites that do this kind of thing today, please share.

[brimtown]

Our current hero element on Harry's is an example of this use case: https://www.harrys.com/en/us?sticky=true.

We also heavily rely on art direction using the <picture> tag on https://www.shopflamingo.com for our heroes, and would love to extend that same functionality to <video> elements using the media attribute on <source>s as well.

Being able to differentially load a single video <source> based on the client's viewport width (which may not be known on the server / at build time), and keeping potential URLs in the HTML as opposed to managing them via JavaScript also seem like great performance wins.

I agree with others that aspect ratio feels important to this discussion, that a seamless swap would be a nice to have, and that questions around seeking behavior are less important than enabling this in the first place.

[johannesodland]

We rely heavily on art direction for videos. Many of our articles contain cinemagraphs or short video loops. To give the users the best experience we deliver a different aspect based on the users screen. Often that means delivering a 16:9 video to desktop users and a 1:1 video for mobile devices, as in the example below.

We currently use css and a custom element to swap between two or more video elements.

https://www.nrk.no/viten/xl/hvordan-dannes-en-snokrystall_-folg-reisen-fra-molekyl-til-snokrystall.-1.14787374

[zcorpan]

Thank you both for sharing, and my apologies for the long delay in analyzing them.

@brimtown

Our current hero element on Harry's is an example of this use case: https://www.harrys.com/en/us?sticky=true.

I don't see a video there anymore, but, WayBackMachine to the rescue: https://web.archive.org/web/20191010102520/https://www.harrys.com/en/us

Switching between a big viewport and a small viewport in devtools, a script seems to replace the div that contains the video. (No need to reload.) Here are the two video sources (HP Video V10 Mobile.mp4 at 1.7 MB, HP video V10-Desktop.mp4 at 3.5 MB). The video won't load due to web.archive.org's Content-Security-Policy, and I didn't find the script that does the replacing, so I don't know if it started the video from currentTime = 0 when switching, or tried to do a more seamless switchover. It's a relatively short video, so it wouldn't be terrible to start over, but not starting over seems like it would probably be a better UX.

@johannesodland

https://www.nrk.no/viten/xl/hvordan-dannes-en-snokrystall_-folg-reisen-fra-molekyl-til-snokrystall.-1.14787374

Thanks! This (right now) uses both art direction between two different aspect ratios, and small/medium/large versions for each aspect ratio. So total of 6 videos, all in different video elements in the DOM at the same time, with preload="none" until a new one is selected. At a viewport width of around 400px, the page shows the 360p square video, even though my devicePixelRatio is 2. Resizing the viewport to be larger switches to loading, playing and displaying another video (e.g. 720p square), no need to reload. The video starts over from the beginning when switching, which is OK here because the video is short (10s) with only two scenes. A seamless switchover seems theoretically better, but probably very few would even notice that the video started over when e.g. rotating their phone or resizing their desktop browser.

---

So, from these two examples I gather this:

• Want to use art direction with different aspect ratios between mobile and desktop.
• Want to serve different-size videos depending on viewport size, to save bandwidth.
• Don't want to load a larger video because the user has a high-density pixel screen: 1x is good enough.
• These videos are short enough that starting over when switching isn't a huge problem, but a seamless switchover could be nice to have.
• There videos are looping and without audio.
• No need for DASH-like adaptive switching based on network conditions.

Are these correct?

If so, from what I can tell, if img were able to play video files as if they were animated images (which Safari supports already), picture with media="(min-width: ...)" can already serve this use case acceptably well.

[johannesodland]

@zcorpan
Your conclusions are pretty spot on for the use case of playing a looping short video/animation without audio (lets call it the GIF-usecase).

Broad support / standardisation for video in picture/img that would solve of this use case.

I have a few comments though.

"1x is good enough".
It would be better if the user agent could pick the best resolution from a given source set based on bandwidth, performance and the size of the replaced element.

looping
In the use case above we assume that the authors want the video to loop. This is not always the case. Sometimes authors want the video to play only once. I don't think the img/picture solution provides control over looping.

playback control
It's not necessarily given that the video should play as long as it is visible. Authors should have control over when the playback starts and stops. The img/picture solution would not solve this.

[nhoizey]

I would add that this should IMHO not be limited to videos without audio.

I guess support for soundless videos in <img> was mostly the easiest way to replace animated GIFs with another file without changing HTML much. But I don't believe it to be the best solution, because videos (with or without sound) should always be pausable based on user preferences (reduced motion) and actions.

[johannesodland]

Here's an example that would not be solved with <img>:

https://www.nrk.no/vestland/xl/slik-var-det-ekstreme-sommarregnet-i-jolster-1.15030175

An animation provided from the graphics team was split into short videos and played back (without looping) as the reader scrolls through the story. Art direction is used to provide the user a video based on the aspect-ratio of their device.

This requires control over whether the video is looping and when to start and stop the playback.

[zcorpan]

@johannesodland

Your conclusions are pretty spot on for the use case of playing a looping short video/animation without audio (lets call it the GIF-usecase).

Broad support / standardisation for video in picture/img that would solve of this use case.

I have a few comments though.

"1x is good enough".
It would be better if the user agent could pick the best resolution from a given source set based on bandwidth, performance and the size of the replaced element.

Are there examples of "2x" videos used on the web? Can users tell the difference between 1x and 2x when the picture is in motion? (cc @eeeps)

looping
In the use case above we assume that the authors want the video to loop. This is not always the case. Sometimes authors want the video to play only once. I don't think the img/picture solution provides control over looping.

Indeed it doesn't. Author control over looping could be added to img, but I'm not sure it's a good idea to do.

playback control
It's not necessarily given that the video should play as long as it is visible. Authors should have control over when the playback starts and stops. The img/picture solution would not solve this.

Indeed.

https://www.nrk.no/vestland/xl/slik-var-det-ekstreme-sommarregnet-i-jolster-1.15030175

Thank you! Indeed this example starts to stretch what is appropriate for img.

Question: If the user rotates their device or resize the browser window, should the new video be switched in seamlessly (that is, at least retain currentTime), or is it OK to start over?

[johannesodland]

@zcorpan

Are there examples of "2x" videos used on the web? Can users tell the difference between 1x and 2x when the picture is in motion? (cc @eeeps)

Technically we serve 640x380 video to the smallest screens, which would be equivalent to 2x.
Example here: https://www.nrk.no/nordland/ny-bbc-serie-_a-perfect-planet_-med-unike-opptak-fra-saltstraumen-i-bodo-1.15340084

If support was on par with <picture> we would provide a list of sources with increasing resolution and leave it to the UA to decide. The user can tell the difference, especially when only part of the picture is in motion. Today there is often a stark difference in sharpness and fidelity between images and video.

Question: If the user rotates their device or resize the browser window, should the new video be switched in seamlessly (that is, at least retain currentTime), or is it OK to start over?

I don't have numbers on how many users rotate their devices while consuming the content, but my gut feeling is that it while it would be preferable to retain currentTime, it would be OK to start over.

We do in some cases listen to currentTime and adjust the playbackRate and or currentTime to sync video to text as the user reads. (This is used in articles similar to https://www.nytimes.com/interactive/2019/03/29/business/boeing-737-max-8-flaws.html). If the UA doesn't seamlessly switch over it should be possible to build support in js. Emitting an event when switching source would be helpful.

[shrpne]

Are there examples of "2x" videos used on the web? Can users tell the difference between 1x and 2x when the picture is in motion?

It is the technique I use sometimes to reduce video file size

2x video with bad quality may have better looking and smaller file size than 1x video with good quality

[zcorpan]

@shrpne interesting, thanks. Do you serve the 2x video to everyone?

[nhoizey]

@shrpne this is what the Compressive Images hack provided back when we didn't have native responsive images yet.

I guess it means the browsers consumes much more CPU and RAM (and battery), though.

[zcorpan]

Forward-duping this issue to #6363