[css-contain-2]: Pause media playback in iframes hidden by content-visibility:hidden

[jridgewell]

https://drafts.csswg.org/css-contain/#content-visibility

We have a use case where we would like the ability to pause playback of video when we mark a cross-origin <iframe> as being content-visibility: hidden.

Eg, we include an iframe to youtube video, which the user then plays. After playing, the user becomes disinterested and scrolls on to the rest of the document. After the users is far enough away, we manually mark youtube iframe with content-visibility: hidden. Ideally, this would pause the playback so it's no longer taking resources, audio would stop playing, and the state would be paused. If the user were to scroll back to the youtube iframe, they'd find it in the paused state at the exact point they scrolled away.

In order to achieve this currently, we actually remove the entire iframe from the document. This guarantees the media will stop playback, but it also loses the current track position. If the user scrolls back to the the youtube iframe, they'll find it reset to the 0s mark, which isn't a great experience for them.

/cc @chrishtr

[chrishtr]

I agree that it makes sense to tie this to content-visibility, because content-visibility already pauses similar work such as CSS animations of content, as well as the update-the-rendering loop of embedded iframes (see step 7 here).

Therefore it also makes sense to pause videos, in order to improve performance in that area related to content the user can't see.

Automatically pausing the video and resuming it when it becomes visible again is another benefit of this change. Note also the parallel with pausing CSS animations mentioned above.

Finally, I'll note that, as I understand it, web compatibility requires hidden videos in other circumstances such as being detached from the DOM, or possibly under visibility: hidden or display:none ancestors, in particular ones that play audio, to continue playing. I don't think this compatibility problem will be an impediment to making the change for content-visibility because content-visibility is a relatively new CSS property.

@dalecurtis

[dalecurtis]

Since this is limited to a relatively new tag, I don't have any objections to this. We might even consider not loading media elements that start invisible, but pausing / preventing autoplay is sufficient to allow a significant amount of resource reclamation.

While Safari and Firefox don't (yet?) ship content-visibility, @jernoble @alastor0325 who may have thoughts.

[alastor0325]

The scenario described in the comment0 sounds reasonable to me. But shouldn't we apply that to all iframes (not only for cross-origin) and all playback (also for audio)?

[jridgewell]

But shouldn't we apply that to all iframes (not only for cross-origin)

Yes, please. My use-case was just to highlight that I can't directly access the <video> inside the iframe to force a pause (since it's cross-origin). I think likely apply to any descendant trees (eg, hiding the <div> in <div><video/></div>), and all iframes.

all playback (also for audio)?

Possibly yes (and this would help us). I think it's a very clear example that content-visibility should affect a <video> because it's visual. It's a little iffy if it should affect an <audio>, but I don't think it would be bad to do so.

[emilio]

You could argue the same about plain visibility: hidden or what not, right? It feels a bit sketchy that CSS affects media playback.

[chrishtr]

You could argue the same about plain visibility: hidden or what not, right?

Yes, but two points:

• There is I think a web compatibility reason that videos need to keep playing under visibility:hidden
• Visibility:hidden does not pause animations or iframe rAF or similar, because for better or worse, it's not defined as a performance feature

It feels a bit sketchy that CSS affects media playback.

I think you could argue both ways - it also seems odd, or at least unexpected, that an invisible video keeps "playing".

[css-meeting-bot]

The CSS Working Group just discussed Pause media playback in iframes hidden by content-visibility:hidden, and agreed to the following:

• RESOLVED: conceptually play (invisibly and muted) any media playback for things that are skipped due to content-visibility, skipping any events unless queried (as for animations)

The full IRC log of that discussion

<astearns> topic: Pause media playback in iframes hidden by content-visibility:hidden
<astearns> github: https://github.com//issues/6468
<fantasai> scribenick: fantasai
<JakeA> Thanks again for inviting me!
<fantasai> chrishtr: Web developer raised use case of wanting to automatically pause videos, media
<fantasai> chrishtr: in situation where media happens to be in iframe, page can't really access it, because usually cross-origin
<fantasai> chrishtr: atm can only do that by coordinating with each provider of video
<fantasai> chrishtr: Perhaps platform can make that possible
<fantasai> chrishtr: Suggestion was content-visibility to control that
<fantasai> chrishtr: content-visibility is meant as a perf tool, and this is about reducing unnecessary use of CPU
<fantasai> chrishtr: so ...
<fantasai> chrishtr: so seems that media under content-visibility be automatically paused
<fantasai> chrishtr: final point is, as I understand, for web-compat a lot of sites expect video, esp with audio, when not visibible for other reasons
<fantasai> chrishtr: e.g. off screen or detached from the DOM
<fantasai> chrishtr: But since content-visibility is new, probably not compat worry
<fantasai> TabAtkins: I'm surprised at how many things still allow video to play
<fantasai> TabAtkins: This seems like a good idea
<emilio> q+
<emilio> q+
<fantasai> astearns: I'm not sure I'd look for content-visibility as a way to make cross-origin video stop playing
<fantasai> florian: In that direction not obvious
<fantasai> florian: but in other direction, would you expect a video with content-visibility to play?
<fantasai> flackr: display:none might have similar effect
<fantasai> flackr: might also unload frame...
<astearns> ack emilio
<fantasai> Rossen_: Display:none is a big hammer
<TabAtkins> It does unload the frame
<fantasai> emilio: Idea is to also do this for ?? stuff?
<fantasai> emilio: ...
<astearns> s/??/same-origin/
<fantasai> emilio: Is there a ? on if you remove element from the DOM, then it doesn't stop playing, then you add content-visibility and then remove from DOM it's paused, that's weird
<fantasai> emilio: Doesn't pausing media change styles?
<fantasai> emilio: causes :pause to apply ...
<fantasai> emilio: maybe not a huge deal if under subtree, but also weird
<fantasai> emilio: overall, I think removing video from DOM should pause
<fantasai> emilio: seems weird that content-visibility is the only thing that does this
<fantasai> chrishtr: Yes, it would be all media
<fantasai> chrishtr: issue says iframes, but would also apply to same-origin video within the document
<fantasai> chrishtr: wrt relative timing, seems similar to detaching video after user presses pause button
<flackr> q+
<Rossen_> q
<fantasai> chrishtr: same thing, rendered as visibible under content-visibility
<fantasai> chrishtr: thrid point was about how styling changes when it's paused
<fantasai> chrishtr: I think yes, those styles would conceptually apply, just like all boxes within content-visibility subtree
<fantasai> chrishtr: but UA is allowed to not render within that, for perf reasons
<fantasai> emilio: Does content-visibility currently change styles under its sub-tree?
<fantasai> emilio: I guess it pauses animations, which is something
<fantasai> flackr: It pauses animations in a way that is not very developer-visibible
<fantasai> flackr: the timer keeps going
<fantasai> flackr: For this video, could maybe not change state to pause, but would not continue playing
<fantasai> Rossen_: Is that how we have it specced?
<fantasai> Rossen_: content-visibility is currently defined as behaving like 'display: none' on contents
<fantasai> Rossen_: [quotes spec]
<fantasai> Rossen_: display:none does not exhibit the behavior you just described for animations
<fantasai> flackr: That's a misleading statement, interaction should not continue ..
<fantasai> astearns: That note is about accessibility, but it's not clear what that note is hooked to
<Rossen_> https://drafts.csswg.org/css-contain/#content-visibility
<dholbert> https://drafts.csswg.org/css-contain-2/#valdef-content-visibility-hidden
<fantasai> emilio: But this isn't what the OP filed the issue about
<fantasai> Rossen_: you get to some part of the video, or would you start from the beginning?
<fantasai> astearns: The idea is it will pause, and then resume after visible
<fantasai> astearns: detaching would restart the video which is not good
<fantasai> chrishtr: Pausing video seems simpler to implement
<flackr> content-visibility animation pausing was spec'd by https://github.com/w3c/csswg-drafts/commit/82fc7ddb1abf9ce0c3f616cbd91a0f1d2069a215
<fantasai> chrishtr: already way to pause videos in UAs
<fantasai> chrishtr: Wrt styles, can leave styles uncomputed
<fantasai> emilio: Why not for animations?
<fantasai> Rossen_: display:none will stop animations entirely
<fantasai> Rossen_: but wrt animations, it's different than what I'm hearing about video that's described by ??
<fantasai> Rossen_: in case of animations, supposed to pretend they're running, but not actually run and use up CPU
<fantasai> Rossen_: when you return, get to more advanced stage
<fantasai> Rossen_: video ...
<fantasai> Rossen_: saying 2 are the same is not right
<flackr> q-
<fantasai> Rossen_: If we want this behavior for video or media
<fantasai> Rossen_: differentiate clearly so people would know what to expect
<fantasai> chrishtr: Leaving video playing but not using CPU to do texture stuff is problem, because still have audio
<fantasai> chrishtr: otherwise UA would have to keep audio stream running, but not the video stream
<fantasai> Rossen_: you can pause, and then take them forward, by the delta
<fantasai> Rossen_: as if the video was playing
<emilio> q+
<astearns> ack fantasai
<smfr> q+
<TabAtkins> fantasai: What Rossen said - if you wanted animations and videos to be consistent, you'd need to advacne the timer on it even when it's not playing
<TabAtkins> fantasai: I think it would be good to treat them the same
<fantasai> astearns: and if you did that, then you would not set the pause style on the video
<TabAtkins> astearns: And if you did that, you would *not* set pause on the video
<fantasai> fantasai: right
<fantasai> flackr: because content-visibility:auto hides things as scrolled out of view
<fantasai> flackr: ...
<smfr> q-
<astearns> ack emilio
<fantasai> flackr: this is why we had this behavior for animations
<fantasai> emilio: pausing a video not only changes styles, but also triggers JS events
<fantasai> emilio: really a problem when style change triggers a bunch of JS handlers
<fantasai> emilio: ...
<florian> q?
<fantasai> chrishtr: for animations, we did decide that although video is conceptually advancing in time, it doesn't fire events
<florian> q+
<fantasai> chrishtr: since that goes against perf goals
<fantasai> chrishtr: could potentially just not fire the events
<fantasai> flackr: concern was audio, could also conceptually mute the audio
<fantasai> astearns: that would meet the perf goal, but I wonder
<Rossen_> q?
<fantasai> astearns: I get annoyed when I scroll Twitter just slightly enough, but want to keep hearing it
<fantasai> TabAtkins: I have exactly opposite problem with Youtube videos when I scroll away
<fantasai> chrishtr: Could conceptually allow the video to continue, but mute it, and get all the perf advantages
<fantasai> chrishtr: solve use case of pausing the videos some other way
<TabAtkins> s/Youtube videos open in a tweet where they *don't* stop playing/
<astearns> ack florian
<Rossen_> q+
<fantasai> florian: conceptually playing doesn't mean actually playing, can just move the timer and not do work
<fantasai> florian: but for use cases where authors do want to pause
<fantasai> florian: could send an event
<fantasai> florian: and then video can react to that state
<fantasai> florian: same thing could apply to animations, things applying to canvas, etc.
<fantasai> florian: all things trying to be on the same timer
<fantasai> chrishtr: I think you're referring to an event where you can identify when you are skipped?
<fantasai> chrishtr: Discussed on and off. There are use cases for it, like avoiding work in scripts
<fantasai> chrishtr: A video, for example, inside cross-origin iframe could pause when skipped
<fantasai> chrishtr: Similar use case from gmail, cross-origin iframes, would like to be able to stop themselves when they are noticed they are skipped
<fantasai> chrishtr: don't want to poll server to ask for more when not visible
<astearns> ack Rossen_
<fantasai> Rossen_: appears we're drifting off the main task of pausing the video
<fantasai> Rossen_: rather than aligning behavior with scrolling and animations
<fantasai> Rossen_: user request we have as well, people want to pause video when scrolling away
<fantasai> Rossen_: At least the current described behavior for content-visibility:hidden doesn't quite align with that
<fantasai> Rossen_: so behavior that advances animations and advances stuff, are we talking about a different behavior here
<fantasai> Rossen_: a pause-media? could be helpful
<fantasai> Rossen_: allows two behaviors co-exist
<fantasai> Rossen_: or have users control one or the other
<fantasai> Rossen_: also what is the event model going to look like?
<fantasai> Rossen_: are you going to get all the events on the DOM side
<fantasai> Rossen_: if trying to synchronize animations with your video
<fantasai> Rossen_: one moved ahead and other paused, etc.
<fantasai> Rossen_: 2 different behaviors, let's discuss them separately
<fantasai> Rossen_: define what pausing looks like and then figure out how they can co-existy
<fantasai> chrishtr: I think 2 options are pause the video, or conceptually continue video but muted
<florian> ?
<fantasai> Rossen_: They weren't asking for that
<florian> q+
<dholbert> q+
<fantasai> chrishtr: that's what person was asking for, however ...
<fantasai> astearns: Fits more with existing behavior of content-visibility, to conceptually run it
<fantasai> astearns: So should do that
<fantasai> astearns: and then discuss use case of pausing
<fantasai> chrishtr: Talked to developer
<fantasai> chrishtr: didn't want to keep playing video, audio is annoying
<fantasai> chrishtr: and also didn't want to take up bandwidth
<fantasai> chrishtr: easiest way was to kill it off
<fantasai> chrishtr: I do think that perf and muting things won't solve everything but will help
<Rossen_> q?
<fantasai> emilio: Other question is pausing, does it fire events, does it not fire events
<fantasai> emilio: Media has a bunch of crazy things going on
<astearns> ack fantasai
<fantasai> fantasai: Should we resolve on the timer continuing while it's actually paused?
<dholbert> q-
<fantasai> astearns: Proposed to treat video similar to animations in content-visibility, is conceptually muted, will return with intervening time skipped
<fantasai> astearns: any objections?
<fantasai> flackr: don't object, but that's not identical to animations
<Rossen_> q+
<Rossen_> ack florian
<fantasai> flackr: not ticking animations is what user would have seen if animation continued to tick
<dholbert> q+
<fantasai> flackr: but muting audio is a bit different from ...
<smfr> q+
<fantasai> flackr: but given that content-visibility is a perf optimization I"m not objecting
<fantasai> Rossen_: with this behavior, are there any events?
<fantasai> chrishtr: no events will fire, it's not observable
<fantasai> chrishtr: querying the element might trigger events or something, kinda like animations
<fantasai> Rossen_: So observability will be aligned with that of CSS animations as well
<fantasai> chrishtr: yes
<dholbert> my question (in case my audio won't work) is: does this mean Twitter couldn't use content-visibility & also match Alan's expectations of letting videos continuing to play even when they're scrolled offscreen?
<fantasai> chrishtr: observability to script same, observability to user a bit different because muted
<fantasai> astearns: note Twitter right now stops the video when scrolling out of view
<fantasai> dholbert: Twitter version that matched Alan's expectation mean can't use content-visibility?
<fantasai> dholbert: if want to still have audio play while off-screen
<Rossen_> q-
<fantasai> astearns: You just don't use content-visibility in that case
<fantasai> dholbert: yes, but Twitter might want to use content-visibility ...
<fantasai> TabAtkins:...
<Rossen_> conten-visibility: hidden pause;
<astearns> q?
<fantasai> dholbert: I feel like I"ve heard from our media team that we already stop streaming video in background tabs, there's optimizations that can be made there
<astearns> zakim, close queue
<Zakim> ok, astearns, the speaker queue is closed
<fantasai> dholbert: maybe can do that here too
<dholbert> q-
<astearns> ack dholbert
<astearns> ack smfr
<fantasai> smfr: websites like youtube and ??
<fantasai> smfr: manipulating through script
<TabAtkins> s/.../the benefits of c-v are precisely that the browser can stop committing resources to things, rather than keeping videos around multiple times/
<fantasai> smfr: if no notification to video that it's being paused, will still run all that stuff
<fantasai> smfr: which is a waste
<fantasai> smfr: are we resolving for skipped content (also affects content-visibility:auto) or only content-visibility:hidden
<fantasai> chrishtr: proposal is to apply to all skipped content
<fantasai> smfr: are we trying to avoid detectability of when content becomes skipped or unskipped?
<fantasai> chrishtr: the events that determine whether skipped, can also consider
<fantasai> chrishtr: raised issue of ? video
<fantasai> chrishtr: or youtube video automatically pause themselves when they notice that they are skipped
<fantasai> chrishtr: in order to raise the other use case in the discussion
<astearns> ack fantasai
<Zakim> fantasai, you wanted to respond to flackr
<TabAtkins> fantasai: wrt the idea that muting the audio isn't entirely consistent
<astearns> zakim, open queue
<Zakim> ok, astearns, the speaker queue is open
<TabAtkins> fantasai: it kinda is if the audio is part of the visibility of the element
<TabAtkins> fantasai: which is true in an audio enviro
<TabAtkins> fantasai: so don't think it's entirely inconsistent
<TabAtkins> fantasai: not sure if this is the mechanism we want to use to allow videos to be paused
<TabAtkins> fantasai: might want to do it more explicitly, rather than building it into this
<smfr> +1 for more explicit control. you don’t want Command-F to start videos making noise
<TabAtkins> fantasai: bc depending on the use-case you might want it to still play
<fantasai> florian: idea reasonable, especially with event that allows you to customize behavior
<fantasai> florian: maybe can then combine with API to pause an entire subtree
<fantasai> florian: but comment from smfr got me wondering, if you just think about things being skipped or not being off-screen, whole thing we discussed is reasonable
<fantasai> florian: but if things are unskipped
<fantasai> florian: suppose you search for word 'play', then they all start playing with video
<fantasai> florian: not a great user experience
<fantasai> florian: can no longer be skipped due to things other than being off-screen, maybe isn't great
<fantasai> flackr: if you scrolled something into view, would start playing
<fantasai> flackr: but because most search results off-screen
<fantasai> flackr: we'd resolve their style to figure out whether they match the search, but they're still off-screen and will continue to be skipped
<fantasai> chrishtr: would be able to see the track time, but not playing
<fantasai> smfr: what if UA makes all content unskipped when finding in content-visibility:auto
<fantasai> chrishtr: spec says that [...]
<fantasai> chrishtr: it will only become unskipped when scrolled into view
<fantasai> smfr: per-element granularity?
<fantasai> chrishtr: every element that becomes unskipped
<fantasai> smfr: confused about the granularity of skipping
<fantasai> smfr: suppose container with auto
<fantasai> smfr: inside it each element becomes skipped or not individually?
<fantasai> chrishtr: granularity is content-visibility:auto element
<fantasai> astearns: This seems to be more generally, could you read through spec and open an issue if it's a problem?
<fantasai> smfr: ok
<fantasai> astearns: back to media playback, any other comments before we try for a resolution?
<fantasai> [silence]
<TabAtkins> (There's text in <https://drafts.csswg.org/css-contain-2/#using-cv-auto> with rough guidance about this exact issue.)
<fantasai> RESOLVED: conceptually play (invisibly and muted) any media playback for things that are skipped due to content-visibility, skipping any events unless queried (as for animations)
<fantasai> <br duration=10m>