mediaSession and stream srcObject?

[PepperAddict]

Hello all!

I discovered mediaSession and media keys yesterday for the first time when researching keypress event listeners outside of an app. So I apologize if this issue ticket isn't done correctly. A wonderful person suggested I create a ticket in my stackOverflow post so now I'm here.

I can't seem to hook up mediaSession with streams such as from MediaDevices. More specificially getDisplayMeida
I'm assuming because these streams are using videoElement.srcObject rather than videoElement.src for their source is the main issue? I'm puzzled on how youtube live and twitch live streams handle theirs or they're not using mediadevices most likely.
I've tested these streams with audio, and I still don't get access to the mediaSession keys.

Here is an example I created to show how I have it going that doesn't work:
https://codesandbox.io/s/optimistic-thunder-zyoz4?file=/src/App.js
First, a custom (randomly picked from the internet) video plays immediately with sound to have access to mediaSession and media keys. I only have pause and play media keys programmed to say something.
Initiate stream button to start recording from your desktop. You can enable your microphone to test stream with sound.
You'll notice you'll lose access to mediaSession once the stream starts. So I'm pretty bummed about that.

Anyway, what I have working is a very sloppy and hacky method which is use the custom (randomly picked from the internet) video to have access to mediaSession but control the stream:
https://codesandbox.io/s/stoppable-stream-5bwvi?file=/src/App.js
Which I know isn't the best idea in the world, so my question is: is there a way to get mediaSession working with stream or is there a reason it doesn't allow it?

[HolgerJeromin]

I have the same problem.
We have a constant audio stream (WebRTC) from a remote device. (audioElement.srcObject = streams[0])
I would be happy to inform the user what is causing the audio output.
Play, Pause, Stop interaction are not really interesting for me.

I did not find anything regarding srcObject in the spec, so I opened
https://bugs.chromium.org/p/chromium/issues/detail?id=1181214

[PepperAddict]

Thank you so much for submitting that bug in chromium, @HolgerJeromin ! I didn't think of that. It looks like it's gaining attention!

[HolgerJeromin]

It would be nice to have some thoughts from @mounirlamouri or @beccahughes on this topic.
The chrome ticket was closed as WontFix with a comment which I do not understand.
Thanks a lot.

[youennf]

FWIW, https://jsfiddle.net/wmtjb2f7/ is working fine in Safari, though not in Chrome and Firefox.
My understanding is that the spec does not preclude the way you want to use it, the metadata part may indeed be useful.
I am not sure what else should be done here, except to file bugs on Chrome and Firefox then close this particular issue.
@jan-ivar, @steimelchrome, any insight?

[HolgerJeromin]

fwiw:
https://webkit.org/blog/13591/webkit-features-in-safari-16-2/

Fixed updating MediaSessionInfo for a media element when a srcObject is used.

[youennf]

We could clarify that all media element types are in scope of media session.
Then, it is up to implementors to catch up.
Marking as P2, Ready for PR based on this.

[steimelchrome]

It feels like this is less about the Media Session spec and more about Chrome's decisions on when to display the media session in its global media controls and/or expose it to system media controls (which allows for media key usage). Chrome doesn't expose all media sessions in those UIs even when they do exist (e.g. we support video conferencing actions through media session even though we don't show those media sessions in those UIs).

I think the proposed Audio Session API could be the solution here as it would allow a website to specify what kind of media is being played in these streams, and then Chrome can make more informed decisions about when to expose the media session to these UIs.