HTML Video Element: Proposal For Reintroduction of Media in Source Elements

[scottjehl]

At the kind request of @foolip I'm raising this issue here to ask about feasibility.

In summary, the HTML video element lacks markup-based control over media source sizes. According to @HTTPArchive, avg video weight is heavier on mobile (1897kb) than desktop (1592kb)! It's too hard for developers to deliver appropriate video sizes.

Here is an article and explainer proposal repo arguing for reintroduction of media (and srcset/sizes) in source elements, particularly after considering the success of the picture element.

Article

https://www.filamentgroup.com/lab/video-with-sizes/

Explainer Repo

https://github.com/filamentgroup/html-video-responsive

Thank you!
Scott Jehl
Filament Group, Inc.

cc. @zcorpan @slightlyoff @yoavweiss @stubbornella @jernoble @jensimmons @hober

Duplicate issues with relevant discussion

• Feature proposal: declarative way to do art direction for <video> (maybe <source media>) #4544

[annevk]

cc @whatwg/media

[chrisn]

Much of this is solved with MSE and DASH or HLS. The main usage I see in this proposal would be with short video clips, where file based delivery is good enough, and adaptive bitrate or video resolution switching isn't needed. For longer videos you'd need a way to switch resolution dynamically, e.g., with responsive layout changes or entering fullscreen, as well as controlling bitrate adaptation - which would take us down the path of specifying a fully adaptive DASH/HLS-like implementation.

[scottjehl]

Thanks @chrisn! I agree those address bandwidth concerns well, but they do require more technical coordination that isn't always available to authors dropping a video into a page, and they don't offer control from HTML alone. A simple HTML-driven syntax like picture offers can make video delivery more accessible to simple common use cases. Additionally, media attributes allow an author to specify media, layout, and aspect ratio for situations the server is not aware of.

[zcorpan]

Previous similar issues: #4544, #4968.

I think solving art direction with markup for video would make sense. It's not yet clear to me the use cases and requirements, which makes it hard to know which solution would be the best one or even an acceptable one.

As stated in the explainer already, this question is not yet answered:

"Loading a new resource is "destructive" (resets currentTime etc)." "How would you change video loading to make media="" useful? Is it ok to destructively load a different resource when you rotate your phone? Or do you want a seamless switch-over à la DASH?

It would be good to align on use cases and requirements before talking about how to solve them, I think. This is why I asked about real-world examples in #4544. Two were mentioned so far, which I analyzed today, and concluded that picture/img should be sufficient if browsers would play audio-less videos in img like animated images.

[scottjehl]

Thanks @zcorpan

I agree those use cases with audio-less video could be addressed with picture if picture happened to support video across browsers. That said, I do think the most common use case that media in video would address is loading typical, audio-included video appropriately across viewport sizes. I agree that syncing a media swap is a challenge to address, and maybe it'd be reasonable to compromise on that by only evaluating sources when the HTML is initially parsed. Still, the feature developers are lacking is a simple (picture-like) HTML solution for offering multiple resolutions of typical content in the page, not just audio-free banner image style videos. These are videos that may include audio and captions, and may be shown at different sizes depending on a layout.

A use case with an audio-included, captioned video

This site has an example of a video that actually uses the media attribute to qualify the first source to larger viewport sizes. It of course doesn't work everywhere, but in Safari (and iOS), users on devices with smaller viewports will get the 480 video instead of the 800 one. https://scottjehl.com/lfwp/

[shrpne]

"Loading a new resource is "destructive" (resets currentTime etc)." "How would you change video loading to make media="" useful? Is it ok to destructively load a different resource when you rotate your phone? Or do you want a seamless switch-over à la DASH?

I think the best solution would be not to reset currentTime and continue to play new video from the same time. If browser will need some time to continue it will do the same when network connection is weak and it needs some time to load next video frames.

Example

• Play video A up to 10s
• Resize
• Swap video source to B
• Continue play video B from 10s

At this moment browser does not have next frames from video B so it starts buffering them and play when buffer is enough.
For user, it would look the same as if browser still plays the same video but browser did not load frames from 10s and like browser needs some more time to load it.

I mean that browser doesn't need to load cross frames from both videos and wait for video B to buffer enough before swapping streams. It will be too complicated and don't worth it

[zcorpan]

Thanks @zcorpan

I agree those use cases with audio-less video could be addressed with picture if picture happened to support video across browsers. That said, I do think the most common use case that media in video would address is loading typical, audio-included video appropriately across viewport sizes.

Are there examples I can look at? Or popular JS libraries that enable this, so I can find examples in HTTP Archive?

I agree that syncing a media swap is a challenge to address, and maybe it'd be reasonable to compromise on that by only evaluating sources when the HTML is initially parsed.

I'm not so worried about it being challenging for browser implementers or spec writers; the Priority of Constituencies says to consider the needs of authors over browser implementers. I'm interested in knowing what y'all need, so that we're not ending up with a behavior in HTML that still isn't good enough and needs workarounds or hacks.

Still, the feature developers are lacking is a simple (picture-like) HTML solution for offering multiple resolutions of typical content in the page, not just audio-free banner image style videos. These are videos that may include audio and captions, and may be shown at different sizes depending on a layout.

A use case with an audio-included, captioned video

This site has an example of a video that actually uses the media attribute to qualify the first source to larger viewport sizes. It of course doesn't work everywhere, but in Safari (and iOS), users on devices with smaller viewports will get the 480 video instead of the 800 one. https://scottjehl.com/lfwp/

Thanks. The smaller video is lower resolution but the same aspect ratio, so it's "save bandwidth if the viewport is narrow" use case; not art direction. If the user were to click the full-screen button, would it be good to switch to the larger video source?

[zcorpan]

@shrpne thank you, those are the kinds of tradeoffs I'm looking for. That makes sense to me, and would probably be good enough for the vast majority of use cases. If a more special use case requires that playback doesn't pause at all while buffering a new source, that's still possible with MSE.

[scottjehl]

Thanks @zcorpan

I don't know of many examples that use this currently, but I would think the use case applies for any video dropped into a blog post or marketing page. Being able to specify a few sizes right there from the HTML lowers the barrier for integration of a feature that could help a lot with performance.

For the swapping of sources as media changes, I expect users and developers would like to see the video resume at its current time, rather than restarting, if possible. I could see this coming up when rotating a device orientation to get a larger size of a video that's currently playing.

For the final example, I agree with your point that the example is about bandwidth and not so much about art direction. The idea that authors could use this to easily save bandwidth in addition to any art direction they might want to do is, I think, a primary benefit.

[yoavweiss]

^^ @danyao @stubbornella

[johannesodland]

If media and srcset are added to source elements we should also consider the poster attribute. Posters should be able to match the video both in aspect ratio and in resolution. It's not optimal to serve huge poster images to mobile devices, or 16:9 posters when a 2:3 source is selected.

[PicchiKevin]

On top of the media source size, I think the current network speed should also be taken into account.

[Th3S4mur41]

One more use case for this which falls into art direction as well, would be to load different videos for light and dark mode.
The current workaround require two separate video element.
A similar solution than pictures using media would be much better.

[tigt]

I believe adapting for playback performance is an important use-case, at least for the srcset-like proposal where the browser can opt to downgrade.

Why I think so: YouTube auto-upgrades the video source to be as high-quality as my connection allows, but my computer struggles with 1080p@60 FPS (or the higher resolutions), so I override the automatic choice to something lower-quality. If I don’t, my laptop heats up, its fans get loud, and it’s even crashed twice.

This could be exposed as a new descriptor for the frequency of the video:

<source srcset="
    4k-vsync.mp4 3840w 2160h 60f,
    4k.mp4       3840w 2160h 30f,
    hd.mp4       1920w 1080h 30f,
    …etc.
  ">

• This could also work for source selection on devices with lower refresh rates
• The existing CSS media query for (update-frequency) is probably relevant, which might “just work” with source[media] on such devices
• The f descriptor may also be helpful for animated <img> sources
• Bikeshedding: 60f, 60fps, 60hz?
• Exposing sources the browser thinks it can play in the <video> UI may be a good recommendation: my computer can play back 1080p@60 without dropping frames, but it clearly doesn’t know it’s going to crash by doing so. This would also let users conserve data on metered connections. I guess that also argues for <source media="(prefers-reduced-data)">?

---

Side note: that aforelinked YouTube documentation seems like it may help with discussion around behavior:

The standard aspect ratio for YouTube on a computer is 16:9. If your video has a different aspect ratio, the player will automatically change to the ideal size to match your video and the viewer’s device.

For some video and device aspects ratios like 9:16 vertical videos on computer browsers, YouTube may add more padding for optimal viewing. The padding is white by default, and dark gray when Dark theme is turned on.

And then for the mobile apps (and mobile site on m.youtube.com, I’ve seen this behavior):

The YouTube app for [Android mobile/iPhone and iPad] automatically adapts the player to the size of the video. Whether it’s vertical, square, or horizontal, the video will fill the screen. The vertical (portrait) video player matches the aspect ratio of the video - getting taller for square and vertical videos, and shorter for widescreen videos.

This seems like it should be doable with <source media="(orientation: …)">, right? It also seems to make a strong case for height descriptor support, which I’m not sure exists for <picture> yet.

[jernoble]

WebKit's position: as WebKit has never removed support for media queries, we are interested in their re-introduction to the specification. Additionally, improvements like srcset and sized introduced to the img and picture elements would be welcome additions to video.

[yoavweiss]

^^ @progers

[zcorpan]

@jernoble are you OK with the current implemented behavior of the media query only being considered at the time of the media being first loaded, and subsequent changes to the environment has no impact?

[jernoble]

I think specifying that the media, srcset, and sizes attributes are to be consulted during the resource selection algorithm is straightforward and easy to standardize and would solve the majority of the use cases.

The experience of standardizing the changeType() function in MSE spec has shown that many web-enabled devices have a difficult time seamlessly switching between decoders on the fly. So adding a normative requirement to re-evaluate the list of sources during playback, e.g. on device rotation, would likely require playback to entirely stop while decoders are torn down and recreated. So in addition to being difficult to standardize and difficult to implement, it would also lead to less-than-ideal user-visible behavior.

I could imagine allowing dynamic source switching as a best-effort/quality-of-implementation feature. But given that the straightforward option of just performing reevaluation on load() would solve so many use cases, I think we could wait to tackle the dynamic feature till after we have spec and implementation experience with the load() option.

[zcorpan]

OK. That seems reasonable!

media was removed because of the lack of reaction to environment changes, but given that this has been requested many times it's evidently considered useful enough anyway.

[padenot]

We've removed our implementation in Firefox in December 2016, but it wouldn't be hard to revive it.

Not reacting to environment changes is how it used to work, and it was useful (but not used). I think it could be even more useful today in the current web landscape where videos are more frequently used in place of images. This allows not using script for a bunch of common scenarios, as pointed out above.

Web applications that do more involved video playback can keep using MSE and all the APIs that allow fine-tuning the user experience (MediaCapabilities, VideoQuality, measuring the bandwidth, etc.), and this is fine. Keeping complexity out of <source media="..."> would be nice.

srcset with new ways to select resource (bitrate, frame rate) is rather complex. The correlation between resource usage, video dimensions, bit-rate and frame rate isn't particularly easy to know ahead of decoding and can change in the same media, it's a lot easier for images.

Mozilla's position: positive for the feature @jernoble describes above:

I think specifying that the media, srcset, and sizes attributes are to be consulted during the resource selection algorithm is straightforward and easy to standardize and would solve the majority of the use cases.

[scottjehl]

Renewed attention on this is very exciting to see! I still feel the same about the validity of the use cases and would love to see this move ahead!

[zcorpan]

Let's limit the scope of this issue to the media attribute, so we can separately evaluate srcset/sizes for media elements. (For example, should they be supported for audio?)

@scottjehl do you want to file a new issue for srcset/sizes?

[dangayle]

I don't know of many examples that use this currently, but I would think the use case applies for any video dropped into a blog post or marketing page. Being able to specify a few sizes right there from the HTML lowers the barrier for integration of a feature that could help a lot with performance.

This is 100% the use case at Crate and Barrel, my company. For product videos etc that are loaded as part of a product display page, we have 100% control over the video presentation from the server to the client. But on pages such as the home page, top level category pages, landing pages, etc., that are controlled by web producers using a CMS, videos are literally just mp4s that are thrown onto an image server and the source manually pasted into their markup.

For some videos, they specify vertically cropped videos for mobile and regular horizontal videos for desktop. 99% of the time they just throw the desktop video onto mobile because there's no dead simple way to do the switching without needing to create a Jira ticket to have a frontend dev on a different team help test the html. Currently, we still have to rely on browser sniffing or JS to do the art direction.

I've really wanted this feature (and a similar feature for the poster image, which is often used as the main image, but has zero art direction control or media query support) for a long time.

[dangayle]

As a follow up to my last comment, the idea of supporting img features such as lazy loading for video posters in videos has been brought up here, which includes discussion relevant to this topic

#6636

[johannesodland]

Let's limit the scope of this issue to the media attribute, so we can separately evaluate srcset/sizes for media elements. (For example, should they be supported for audio?)

@scottjehl do you want to file a new issue for srcset/sizes?

Following up on this, has a new issue for srcset/sizes been filed yet? 🙏

[scottjehl]

Thanks for the nudge, @johannesodland and @zcorpan. Here's the srcset and sizes issue: #9446