Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebRTC spec should explicitly specify all causes of a PeerConnection-sourced track receiving mute/unmute #2915

Closed
jan-ivar opened this issue Dec 18, 2023 · 9 comments

Comments

@jan-ivar
Copy link
Member

In w3c/mediacapture-main#982 (comment) @guidou wrote:

For example: crbug 941740 implements mute on remote tracks reactively based on (lack of) input, violating the WebRTC spec and causing web compat issues.

In this case, Chromium just is applying the model defined in the main spec to remote tracks. The WebRTC spec indicates some cases in which the muted attribute should be set/unset, but AFAICT it does not say anywhere that this overrides the model defined in the original MediaStreamTrack specification. It also does not state a new definition of muted specific for WebRTC tracks and does not even list the muted/unmuted events in its [Event Summary section].

Crbug 941740 has been open for almost 5 years without mention of this, and I'd argue § 9.3 MediaStreamTrack does "state a new definition of muted specific for WebRTC tracks":

image

I would argue this describes all causes of mute for remote tracks, based on #139. But @guidou is right that it:

  1. it does not say anywhere that this overrides the model defined in the original MediaStreamTrack specification.
  2. does not list the muted/unmuted events in its Event Summary section

But this seems editorial as evidenced by #139. Both specs needs to clarify that while this spec follows the model in mediacapture-main, each source is responsible for specifying its own mute/unmute behavior, if any.

We did this for constraints, and it only makes sense to do the same here.

@jan-ivar
Copy link
Member Author

Crbug 1295295 is another bug about non-standard unmute that might be relevant.

@alvestrand
Copy link
Contributor

Crbug 1295295 is about unmuting without waiting for the first RTP packet to arrive, so it's not the behavior we're discussing here (rather the opposite, in fact).

@henbos
Copy link
Contributor

henbos commented Jan 11, 2024

My understanding is that when we define new types of sources (e.g. WebRTC remote tracks as opposed to local capture tracks) then the spec that defines the new type of source should also define when the muting/unmuting happens or not.

In the case of webrtc, if mute/unmute should be based on RTP or based on SDP is another discussion worth having, but that's a separate issue from sources having its own definitions

@fippo
Copy link
Contributor

fippo commented Jan 16, 2024

Given the long-standing issue of libWebRTC sending RTCP BYE at inappropriate times (and then reusing a SSRC) but not on stopping a transceiver (which would be observable through a new negotiation) I would remove BYE as mute reason.

@youennf
Copy link
Contributor

youennf commented Jan 16, 2024

the spec that defines the new type of source should also define when the muting/unmuting happens or not.

+1 to that.

+1 also for the spec to describe what implementations are doing, given the spec status.
We need those to get interop.

FWIW, this has a real impact as, for instance, in case of connection drop, a video element will either display the last received video frame (track not muted UA) or will display black (track muted UA).

@jan-ivar
Copy link
Member Author

From yesterday's meeting I heard agreement that webrtc-pc should explicitly specify all causes of PeerConnection-sourced tracks being muted or unmuted, but some disagreement over what those causes are.

Since we have agreement we need to list causes here for event firing to be interoperable, the next steps I see are:

  1. Open separate issue/PR on mediacapture-main to clarify each source is responsible for specifying its mute behavior.
  2. Work out here the complete list of when mute and unmute events fire on remote tracks.

@jan-ivar
Copy link
Member Author

jan-ivar commented Jan 17, 2024

Here's https://jsfiddle.net/jib1/p6bzhme0/37/ showing what existing implementations fire on remote PC tracks. It opens a connection, and sender-side calls sender.track.stop() at the 3 second mark, and transceiver.stop() at 5.

Firefox:

0.011: audio track event fired
0.011: video track event fired
0.011: sRD done.
0.016: checking
0.016: connected
0.043: video track unmute event fired
0.045: audio track unmute event fired
3.003: Stopped audio track
3.003: Stopped video track
5.006: Stopped audio transceiver
5.006: Stopped video transceiver
5.010: sRD done.
5.010: audio track ended event fired
5.010: video track ended event fired

Safari:

0.016: audio track event fired
0.016: video track event fired
0.016: sRD done.
0.017: audio track unmute event fired
0.017: video track unmute event fired
0.023: checking
0.023: connected
3.001: Stopped audio track
3.001: Stopped video track
5.008: Stopped audio transceiver
5.008: Stopped video transceiver
5.016: audio track ended event fired
5.016: video track ended event fired
5.017: audio track mute event fired
5.017: video track mute event fired
5.017: sRD done.

Chrome:

0.007: audio track event fired
0.007: video track event fired
0.007: audio track unmute event fired
0.007: video track unmute event fired
0.007: sRD done.
0.013: checking
0.017: connected
3.005: Stopped audio track
3.006: Stopped video track
4.196: video track mute event fired
5.003: Stopped audio transceiver
5.003: Stopped video transceiver
5.010: new
5.013: audio track ended event fired
5.013: video track ended event fired
5.014: sRD done.

Takeaways:

  • Firefox appears to follow webrtc-pc's § 9.3 MediaStreamTrack
  • Safari deviates by
    • unmuting after 1 ms, likely ahead of RTP reception, compared to ~32 ms in Firefox (webkit 209242)
    • muting tracks on stopped transceiver
  • Chrome deviates by

I did not try severing the network connection, but based on the above I would guess Chrome would mute the video track only?

Are there other causes of mute/unmute we should consider?

@jan-ivar jan-ivar changed the title WebRTC spec should explicitly specify all causes of a PeerConnection-sourced track being muted WebRTC spec should explicitly specify all causes of a PeerConnection-sourced track being muted/unmuted Jan 17, 2024
@jan-ivar jan-ivar changed the title WebRTC spec should explicitly specify all causes of a PeerConnection-sourced track being muted/unmuted WebRTC spec should explicitly specify all causes of a PeerConnection-sourced track receiving mute/unmute Jan 17, 2024
@dontcallmedom-bot
Copy link

This issue had an associated resolution in WebRTC March 26 2024 meeting – 26 March 2024 (Clarify each source is responsible for specifying mute/unmute/ended and constraints behavior):

RESOLUTION: close webrtc-pc#2942 as reviewed and webrtc-pc#2915, open an issue on timing of unmuting on RTC reception

@jan-ivar
Copy link
Member Author

Closing per above and #2942 (comment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants