[selectors-4] additional resource state pseudo-classes for media elements

[hober]

In §11 Resource State Pseudos we currently have the :playing and :paused pseudo-classes, which collectively address the use case of a custom "play/pause" media control which should appear as a play button or as a pause button depending on the current play state of the associated media element. A bunch of us (@beccahughes @mounirlamouri @padenot @jyavenard @eric-carlson @jernoble et al.) would like to address several related use cases with additional resource state pseudo-classes:

• Whether or not the media element is muted. Use case: styling a custom muted control. Proposal: mint a :muted pseudo-class.
• Whether or not the media element is currently stalled (see several paragraphs in the HTML spec starting at "The stall timeout is a"). Use case: changing the appearance of a loading progress indicator to indicate that loading is currently stalled. Proposal: mint a :stalled pseudo-class.
• Whether or not the media element is currently seeking. Use case: a custom seek control can reflect the case where a user is seeking using some other, UA-provided control. Proposal: mint a :seeking pseudo-class.

Currently, authors of custom media controls have to do some combination of UA-sniffing and other logic from script in order to handle these cases. Ideally, the appearance of such controls would be expressible in CSS, just as play state currently is.

[FremyCompany]

Looks good to me

Shameless plug: http://fremycompany.com/TR/2012/ED-css-content-state/#interactive

[tabatkins]

All these sound good.

For the volume one, is the issue that the UA can let the user dictate the volume, ignoring the volume attribute?

[jernoble]

@tabatkins, essentially yes.

As a concrete example, some sites currently try to detect whether changing volume is supported by setting video.volume to some arbitrary value, and verifying whether that volume value "sticks". (Effectively, let v = video.volume; video.volume *= 0.5; return v !== video.volume;.) That doesn't work if the UA implements step 1 of the effective media volume algorithm. We'd like to give those sites a way of determining this explicitly, without relying upon side effects or heuristics.

[tabatkins]

Cool, sounds good. Been trying to think of decent names for the pseudo, tho, and am coming up blank. ^_^

[hober]

Been trying to think of decent names for the pseudo, tho, and am coming up blank. ^_^

Yeah, same.

Would it help if I broke that one out into a separate issue?

[tabatkins]

Eh, sure. Might as well.

[AmeliaBR]

For starting the bikeshed debate on that separate issue (please copy over once there's somewhere to copy to):

• :adjustable-volume
• :not(:user-volume)
• :not(:volume-locked)
• :volume-unlocked
• Or maybe a set of :volume() variations, like :volume(muted), :volume(locked), :volume(max)…

[hober]

Okay, I moved the volume case over to its own issue, #3933. The three remaining cases in this issue should be really straightforward.

[css-meeting-bot]

The CSS Working Group just discussed additional resource state pseudo-classes for media elements, and agreed to the following:

• RESOLVED: Add :muted, :seeking, and :stalled pseudos.

The full IRC log of that discussion

<heycam> Topic: additional resource state pseudo-classes for media elements
<heycam> github: https://github.com//issues/3821
<bkardell_> s/JS Foundation/Open JS Foundation
<iank_> Ian Kilpatrick, Google
<heycam> hober: we currently have playing and pause pseudo classes, which media elements match when they are playing or paused
<heycam> ... the use case is, things like custom media controls, with a unified play/pause button
<heycam> ... there are a handful of other common media element states that have a similar rationale for exposing to CSS
<heycam> ... stuff that people are currently using script for
<heycam> ... this issue is the 3 easy ones
<heycam> ... 1 is whether or not the element is muted
<heycam> ... you may have a mute button, styled volume slider
<heycam> ... very similar to playing/paused
<heycam> ... hope it's uncontroversial
<heycam> ... other 2 are a bit weird
<heycam> ... the HTMl spec has a concept of media being stalled
<heycam> ... and you can see examples of custom UI when media is stalled on popular sites like YouTube, netflix
<heycam> ... the spinner looks different
<heycam> ... if we could provide styling for that it would be nice
<tantek> q+ to ask about "buffering" vs. "streaming connection broken" (have seen different UI)
<heycam> ... the 3rd one is: media elements have a seeking state, which is useful for cases like when you are displaying custom controls, but seeking is happening due to other controls
<heycam> ... maybe seeking with a remote control, but you want you custom control to do a different thing
<tantek> FYI: https://html.spec.whatwg.org/multipage/media.html#htmlmediaelement
<heycam> ... so the HTML spec has the right hooks here
<heycam> ... just need something in CSS to expose them
<heycam> ... so 3 proposed pseudo classes: muted, stalled, seeking
<heycam> florian: what is seeking as a state
<heycam> hober: while you're seeking. if you using the scrubber
<heycam> AmeliaBR: like an active state
<fremy> q+ to talk about additional states
<TabAtkins> q+
<Rossen_> ack dbaron
<Zakim> dbaron, you wanted to ask whether :muted would cover tab-level, app-level, or system-level muting (I suspect not, but want to ask)
<heycam> dbaron: there's ofte na lot of ways to mute something
<heycam> ... my guess here this ignores your tab is muted, hardware control is muted, is that right?
<Rossen_> q?
<heycam> hober: I think that's right. but I'm not sure. I think this shouldb e tied to the host lang's definition
<heycam> dbaron: ok so the HTML definition
<tantek> ack tantek
<Zakim> tantek, you wanted to ask about "buffering" vs. "streaming connection broken" (have seen different UI)
<Rossen_> ack tantek
<heycam> tantek: I like the general direction of the proposal, +1 on them. even the less common features
<heycam> ... this shook loose memories of things I worked on right before Mozilla, doing HTML5 consulting, setting up video UIs
<heycam> ... took a bunch of notes which I forgot about that
<heycam> ... I needed this in 2010
<heycam> ... the :stalled pseudo class, I've seen different UI between buffering to show something, vs the network connection is gone
<heycam> ... even if you're in this "I want to play" state
<heycam> ... wondering if it's worth distinguishing
<heycam> hober: I agree there's a use case there
<florian> +1 to tantek
<heycam> ... I think network state is a little more general. that's going to affect hte page, not just the media elements
<heycam> tantek: but that's a differnt can of worms
<heycam> hober: I'd rather tacklet that as a different thing
<Rossen_> q?
<heycam> tantek: I'm saying I'd rather not, since it is a can of worms
<heycam> hober: could you propose another pseudo for this?
<heycam> tantek: do you want one pseudo that means either of that? two for the specific states?
<heycam> hober: I'd be happy to delegate this to HTML
<heycam> ... some HTML spec refactoring is needed anyway, for stalled
<heycam> ... from our perspective, we'd defer to the host language
<jensimmons> do these apply to <audio> as well as <video>? I’d assume so…
<heycam> tantek: specifially the HTML media element interface?
<heycam> hober: yes
<jensimmons> q+
<heycam> emilio: images have differnet set of states. broken, loaded. Gecko hsa internal pseudos for these
<Rossen_> q?
<heycam> ... I wonder if it is useful to try to expose a common set of pseudos
<heycam> fremy: I was going ot suggest the same
<heycam> hober: in the past when this has come up, we've agreed we should have more general subresource state exposing
<heycam> ... was hoping to not get bogged down on that too much
<heycam> ... there's specific use cases for media in custom media controls
<tantek> note that the HTMLMediaElement does distinguish network state a bit: NETWORK_EMPTY, NETWORK_IDLE, NETWORK_LOADING, NETWORK_NO_SOURCE
<heycam> ... I agree it should be coherent with a more general thing
<heycam> ... so we should keep that in mind
<heycam> ... but the last few times it's come up over 10 years, we end up not doing it
<dbaron> https://github.com//issues/3134 is about image state pseudo-classes
<Rossen_> ack fremy
<Zakim> fremy, you wanted to talk about additional states
<heycam> fremy: we don't need to standardize pseudos for all possible cases
<jensimmons> anything to make doing this kind of stlying possible, without replacing everything with JS, would be ideal: https://thewebahead.net/110
<heycam> ... you say you're sticking to the HTML spec's state. ther's already an error state in there
<heycam> hober: I'm happy for someone to propose those
<heycam> fremy: seems reasonable to want to style them differently if you're on the network, etc.
<heycam> ... in the specific cases for media controls you do want to know about the network state
<heycam> ... in my initial proposal it was with video in mind. I think it's a good idea to see how we can map them
<heycam> ... if you end up in an error state, you want to show that in the UI
<tantek> q+ to ask if :seeking is *while* the user is dragging the seeker, only upon release, or both?
<heycam> TabAtkins: hober's list of things are video specific
<Rossen_> ack TabAtkins
<heycam> ... two specific questions
<jensimmons> q-
<heycam> ... 1st, the current definition of playing says it subsumes seeking and stalled
<heycam> ... keeping that?
<heycam> hober: yes
<heycam> TabAtkins: ok cool. 2nd, :muted is good, do we want a general volume state? low-volume high-volume?
<heycam> hober: I think it's fairly common for there to be a separate mute butotn. and it doesn't seem harmful to have this
<heycam> TabAtkins: not saying as a replacement
<heycam> hober: think it totally makese snse. the other proposal is related to volume, but it's a bit weirder than what you are proposing
<Rossen_> q?
<heycam> TabAtkins: I'll file a separate issue
<heycam> jensimmons: would this apply to audio?
<heycam> hober: it will
<Rossen_> q?
<heycam> TabAtkins: but not all resources in general
<Rossen_> qqqqqqq???
<tantek> I'd like to consider splitting :stalled into :buffering and :stalled distinct states, where :buffering means there is still some data coming from the network, just not enough to play, and :stalled is actually stalled including not getting data from the network
<heycam> jensimmons: +1
<tantek> q?
<heycam> tantek: nitpick, but is :seekin while they're dragging? not when they let go?
<heycam> hober: yes
<heycam> tantek: the specific request then is to split stalled into buffering and stalled
<florian> q?
<heycam> ... as a webdev, that's the two states I want to hook into
<heycam> hober: can't remmeber off hand if they are distinct states in HTML
<heycam> ... anything you can reasonably map it makes sense
<AmeliaBR> q+ for the unofficial q
<heycam> florian: I think the stalled state, when it means buffering, is a variant of playing
<heycam> ... it's a "trying to play" state
<heycam> tantek: it's not playing
<heycam> florian: the play/pause button is in the play state
<heycam> hober: there's all sound like issues to file in the HTML spec
<heycam> AmeliaBR: overall my main concern is there needs to be clear definitions of what all these mean
<heycam> ... shouldn't be exposing things that aren't exposed other ways
<heycam> ... I assume most of these there are events you can listen to when these states change
<heycam> ... if that's not true, don't want to make people poll for pseudo state matching
<heycam> hober: I tried to limit to the state that is already exposed
<chris> rrsagent, here
<tantek> HTMLMediaElement does have both networkState and readyState which are finer grained
<heycam> ... people are already doing this with script, but perhaps a few frame sbehind
<heycam> florian: not saying we should differ from HTML. but just not naively expose every state
<heycam> AmeliaBR: tantek brought up a point, if you want to get more granualr, a general stalled class, then one a parenthesized one later
<heycam> tantek: since we only have 2 states would like to keep them orthogonal
<heycam> ... and looking at the HTML spec now, it's more granular than we need
<heycam> ... per your request, I believe this information is avialable to script right onw, but you need to combine readyState and networkState to figure out what's oging on
<heycam> ... I'm saying let's expose a higher level state concept to UI
<Rossen_> q?
<heycam> ... HTML has these states already, but as a developer you need to check those combinations
<florian> +1 to tantek
<heycam> ... I'm saying CSS should simplify that
<heycam> bkardell_: why only in CSS?
<heycam> tantek: we want to solve use cases, not plumbing
<heycam> AmeliaBR: can you write up a table with the combinations?
<heycam> hober: can you file an issue for buffering?
<heycam> florian: I'm confused about resolving on stalled before knowing what it maps to
<heycam> hober: it's just the HTML stalled state
<heycam> florian: my understanding there are low level bits, and the one called stalled covers more than one use case
<heycam> ... if there are two bits maybe we don't want to expose these two things separately?
<heycam> bkardell_: we shouldn't overload them with the same name...
<heycam> hober: the name stalled in CSS should be the same as stalled in HTML
<AmeliaBR> `:stalled` vs `:buffering` and `:broken-stream`
<heycam> florian: then maybe we should have two different ones, not called stalled
<heycam> tantek: are you asking for new HTML specs rathe rthan new property values?
<heycam> hober: I'm not asking for anything in HTML
<heycam> ... the 3 things I've proposed map to existing concepts in HTML
<heycam> tantek: buffering is expoesd, but there's no event for it
<heycam> hober: I think it would be weird to have a straightforward way ot look for buffering in CSS and not have corresponding APIs
<heycam> tantek: there is
<heycam> hober: but you have to be clever by combining state
<heycam> tantek: DOM APIs are meant to be minimal and fine grained. but not for CSS
<heycam> hober: you should propose the :buffering pseudo, it sounds great
<heycam> florian: I'm also hearing that the stalled concept, if we're going ot have :buffering vs not having it
<heycam> hober: I don't think it impacts that at all. stalled has a clear definition in HTML, that's what it should be
<heycam> AmeliaBR: you're still arguing it for it to be the superclass
<heycam> florian: I am arging to go from use cases
<heycam> ... and if it maps to several bits in HTML, that should be ok, but not name them confusingly
<tantek> HTMLMediaElement summarizes stalled event as "The user agent is trying to fetch media data, but data is unexpectedly not forthcoming. "
<tantek> which reflects the meaning that I'm proposing
<heycam> ... but whether it's useful to expose a single bit just because, we shouldn't
<tantek> and is not inclusive of :buffering
<heycam> hober: I'm not proposing that
<heycam> ... I'm saying netflix has stalled UI, it's a common UI pattern, we have a definition in the spec
<tantek> Netflix has both stalled and buffering UIs - they're different
<heycam> florian: what is the use case? does HTML solve that use case already?
<tantek> Netflix puts up an error when stalled (sorry no data or something), while buffering it shows a spinner and %
<heycam> florian: from the use case PoV, the buffering/stuck things are different
<heycam> hober: so propose a buffering thing
<heycam> ... I'm trying to do the opposite of stop energy here. if you propose it, I'll +1 it
<heycam> florian: if the stalled class means either buffering or stuck ...
<heycam> Rossen_: the stalled event in HTML has a clear definition
<tantek> To be clear I am supportive of :stalled as defined in HTML because it is NOT inclusive of :buffering
<heycam> ... the proposal here is to map a pseudo to this
<heycam> ... that is all that is being proposed, the same way we're proposing for :muted, and :seeking
<AmeliaBR> From HTML:
<AmeliaBR> The stall timeout is a user-agent defined length of time, which should be about three seconds. When a media element that is actively attempting to obtain media data has failed to receive any data for a duration equal to the stall timeout, the user agent must queue a task to fire an event named stalled at the element.
<emilio> q+
<heycam> ... if there are other states in the media element that need to be exposed through events that will eventually map to pseudos, that's great. file HTML issues to add events so script and CSS can benefit from them
<heycam> ... what we have right now satisfies the use cases brought forward
<heycam> fremy: your summary is why I'm confused. you're saying what's in the spec solves the use cases, but I haven't seen the proof of that
<heycam> ... but i will trust hober on this
<heycam> hober: I apologize for the arugmnet from authority, I'm not the one working on this, I took this from media engineers from Mozilla, Google and Apple, and agreed to write it up for them
<heycam> emilio: :muted has a clear definition, since there's is HTMLMediaElement.muted
<heycam> ... :stalled only says when to fire the event, not when to remove it
<heycam> hober: yes I need to follow up with an HTML issue to expose a spec hook to latch on to
<heycam> ... it would be an editorial change to HTML
<heycam> RESOLVED: Add :muted, :seeking, and :stalled pseudos.

[tantek]

Please also add a :buffering (or :waiting) pseudo-class distinct from :stalled, at the same time as adding :stalled.

Use-case: showing loading/buffering spinner, often with a %, e.g. on Netflix.

I’d like to see :buffering defined similar to :stalled https://html.spec.whatwg.org/multipage/media.html#event-media-stalled, however with the key difference that data is forthcoming. Similar to how the waiting event is defined: https://html.spec.whatwg.org/multipage/media.html#event-media-waiting

Alternatively I'd be ok with a :waiting pseudo-class that maps to the waiting event/definition (might be simpler for the platform / web-developers if we "just" do/keep a direct mapping of the pseudo-class to an existing HTML event name).

I do think it is important to add :buffering (or :waiting) at the same time as adding :stalled, in order to avoid having :stalled be errantly overloaded by implementers and/or web developers to handle the buffering use-case.

(Originally published at: https://tantek.com/2019/155/t2/)