Firefox picture in picture mode is not capturable with navigator.mediaDevices.getDisplayMedia()

[guest271314]

URL: https://blog.mozilla.org/en/products/firefox/firefox-picture-in-picture-for-videos/

Browser/Version: Firefox 104

Operating System: Linux

What seems to be the trouble?(Required)
The article Try Firefox Picture-in-Picture for multi-tasking with videos fails to state that Firefox picture in picture mode is not specification compliant.

Steps to Reproduce

Launch picture in picture mode on Firefox 104

Expected Behavior:

Picture in picture window to be recognized, listed, capturable with navigator.mediaDevices.getDisplayMedia({video: true})

Actual Behavior:

Picture in picture window is not recognized, listed, capturable with navigator.mediaDevices.getDisplayMedia({video: true}).

Screenshot

[denschub]

The article Try Firefox Picture-in-Picture for multi-tasking with videos fails to state that Firefox picture in picture mode is not specification compliant.

The MediaCapture Screen Share API does not actually define what a window is, and neither is there any specification saying which browser UI elements have to be "a window" and which do not need to be a window. macOS, for example, has a distinction between windows and panels. The claim that PiP is "not spec compliant" and the expectation that this needs to be mentioned in a marketing blog post is... questionable.

Either way, this is a browser feature request (or maybe a bug, I don't actually know), so I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1792496 for someone working on PiP to have a look.

[karlcow]

Firefox picture in picture mode is not specification compliant

Could you clarify? Which specs are you talking about?

Note that PiP have been developed by browser implementers as a UI feature independent of any specs. The same way there are reader views features in some browsers. But these are not part of the web platform. So there is no expectations for them to be working in that environment or to have parity of behaviors and/or features.

Also is there a system of PiP in WebKit and Blink which can be capturable with navigator.mediaDevices.getDisplayMedia({video: true}).

[guest271314]

This is the official W3C Picture-in-Picture specification https://www.w3.org/TR/picture-in-picture/.

And yes, a window is defined in the section you linked to.

It is literally a PictureInPictureWindow per the specification. And yes, there is an expectation that the window should be listed when navigator.mediaDevices.getDisplayMedia({video: true}) is called, and that the window is capturable.

[guest271314]

I already filed an issue there https://bugzilla.mozilla.org/show_bug.cgi?id=1792297.

I think this is a regression. At some point in the past I'm relatively certain the window was listed and capturable.

The problem is Firefox is using the terminology Picture-in-Picture mode while not actually implementing the specifiction.

[karlcow]

I had completely forgotten about this spec almost 3 years already
W3C First Public Working Draft, 30 January 2020

Thanks for giving more context. So it was triaged appropriately by @denschub

[denschub]

This is the official W3C Picture-in-Picture specification https://www.w3.org/TR/picture-in-picture/.

As you correctly noticed, Firefox does not implement this specification, and Mozilla currently has no intentions of doing so. What you are seeing is a browser feature that remains a browser feature, and nothing that's covered by any specification. Even if it would be, that specification still doesn't say how the PiP "window" needs to be implemented on a technical level and leaves that up to the implementation.

Neither Chrome nor Safari, who both implement this specification, expose their "PiP window" in a way that would allow getDisplayMedia() to capture it. Interestingly, on macOS at least, you can alt-tab to Chrome's PiP window, while you can't alt-tab to Safari's PiP window. Neither behavior is violating the spec because, again, in pretty much all cases, web standards do not actually describe how the user agent's UI has to behave.

I already filed an issue there https://bugzilla.mozilla.org/show_bug.cgi?id=1792297.

Well, that would have been good to know. I moved that bug into the right component to make sure the PiP team sees this.

[guest271314]

Neither Chrome nor Safari, who both implement this specification, expose their "PiP window" in a way that would allow getDisplayMedia() to capture it.

That is simply not true and correct.

I have been capturing PiP windows on Chromium and Chrome for a few years now.

Then Firefox needs to cease and desist from using the term "Picture-in-Picture" immediately, and edit existing articles to make what you said clear.

[guest271314]

PiP is literally a window.

Why Firefox is using the term "Picture in Picture" is something you Mozilla folks need to address and correct in-house, and stop writing articles as if you are actually implementing Picture-in-Picture specification.

This is the core of web compat: stopping confusion and ambiguity.

[guest271314]

Neither Chrome nor Safari, who both implement this specification, expose their "PiP window" in a way that would allow getDisplayMedia() to capture it.

To demonstrate that claim is not true and correct, run this code (written 3 years ago) https://github.com/guest271314/MediaFragmentRecorder/tree/getdisplaymedia-pictureinpicture on Chrome or Chromium.

[denschub]

To demonstrate that claim is not true and correct

Thanks -- there's no need to run your code, because I actually checked what other browsers do on different platforms before writing my comment. Here's a screenshot showing Chrome's PiP, and also Chrome's window selector, which clearly is not including the PiP "window". Because it's not a window on macOS, it's a panel. What you're saying might be true on Linux - or, given it's Linux: it might be true for a subset of WMs on Linux - but it's certainly not the universal truth you act like it is.

This is the reason why specs include phrases like "left to the implementation", or why specs don't even include some details in the first place. Platform-specific stuff is messy.

I'm really not sure what your goals are, @guest271314. Everyone did understand your "expectations", and nobody claimed you're wrong in having those expectations. I explained you that the spec is not covering the technical implementation of browser UIs - which, no matter how long you want to discuss this, is a simple fact - and I also elaborated on how other vendors implement the behavior. I filed a bug in the Firefox' bug tracker so that the people working in PiP can decide if they want to expose that window to screen capture APIs or not. Neither decision would be in violation of any spec, but it's a valuable discussion to have - which is why I opened that bug. Opening that bug and moving it into the responsibility of the people who are actively working on the feature we're talking about marks the end of this bug.

[guest271314]

Actually, you're going to need Chrome to playback MP4 files; Chromium does not support playback of MP4 - which is how I arrived at filing this issue after not experimenting with capturing and recording PiP windows in a year or so.

I was on Chromium 108, which does not support MP4 playback, so I played the video on Firefox 104.

You made an absolute claim here

Neither Chrome nor Safari, who both implement this specification, expose their "PiP window" in a way that would allow getDisplayMedia() to capture it.

that is not true.

Yes, I'm on *nix.

I'm really not sure what your goals are

My expectation is for Firefox folks to either

1. Stop using the term "Picture-in-Picture" as if you are implementing the W3C Picture-in-Picture specification;
2. Implement your Picture-in-Picture mode so that is is capturable by getDisplayMedia(). It is literally a window that should be capturable;
3. Edit existing Mozilla/Firefox articles to make it clear Firefox is not implementing Picture-In-Picture W3C specification, and Firefox does not capture the PiP window.

[guest271314]

I think developers who user Mac and Windows think those are the only OS's that matter, tend to neglect the fact that there are a lot of Linux users; and if there are, they are marginal and negligible; can be set aside as mere *nix users, while the real devs are all on MacOS.

That is not the case.

You need to test your gear on all OS'es. That you can make qualified claims without having to retract and rearrange claims to fit into a *indows or *acOS restrivtive environment.

In this case, Mozilla folks need to make it clear in publications that PiP mode on Firefox is not specification compliant; perhaps call it MozPiP like you do for mozCaptureStream(), for disambiguity.

[guest271314]

Just so there is no question what happens on Crrome or Chromium on *nix with Picture-in-Picture and Screen Capture specification implementation I just captured and recorded this PiP window using getDisplayMedia(), HTMLVideoElement.requestPictureInPicture() with the code I posted above
5cfbfa78-2396-4d71-a353-78691720deed.webm.

Firefox should be doing this. I defend Firefox when users at large try to smut up Mozilla/Firefox - because Chrome sends user PII biometric data to remote servers https://www.reddit.com/r/browsers/comments/xnq3p4/comment/ipuoub1/.