Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Constrainable properties on remote tracks are under-specified #2121

Open
jan-ivar opened this issue Mar 7, 2019 · 11 comments · May be fixed by #2188

Comments

@jan-ivar
Copy link
Contributor

commented Mar 7, 2019

What behaviors specific to remote peer connection tracks are available through the constraints APIs? There's a single paragraph on the topic, and it doesn't say.

From @guidou in #2109 (comment), Chrome appears to have implemented something, so we should probably specify it, like we did in w3c/mediacapture-screen-share#31. Otherwise, the next question might be why don't we support whiteBalanceMode?

Discuss.

@henbos

This comment has been minimized.

Copy link
Contributor

commented Mar 8, 2019

Observations or starting point-opinions:

  • Constraints are usually applied to tracks, not sources. As such, "width" can make sense whether the source is a camera or a network source. However sometimes they only make sense for a particular source... (see below).
  • Starting point: Unless otherwise specified or not applicable, constraints should mean the same thing independent of what the source is.
  • We can either be "defensive", and say "only these constraints are supported for remote tracks" and be explicit about it, or "offensive" and allow all constraints that are still defined in a way that can be applied to remote tracks to be so. Example: "width" means the same thing independent of what source produced the frames.

Otherwise, the next question might be why don't we support whiteBalanceMode?

Definition: "White balance mode is a setting that cameras use to adjust for different color temperatures."
This is a configuration that is clearly not applicable to a remote source. A remote source gives you whatever is sent to you, whereas a camera can be configured to produce different outputs. RTCPeerConnection is not a color temperature sensing camera.


I think a lot of this boils down to if you view "applyConstraints()" as means to change configurations of a source or filter/transform what the source provides you with. The former is not applicable to remote tracks, the latter is.

We may save a lot of time arguing if we can pinpoint what "applyConstraints()" should mean with regards to this!


Side-discussion, though perhaps we should leave this one until later, is what to do if a track gets overconstrained? Given that mute is a possible side-effect of setRemoteDescription(), I am opposed to allowing remote tracks to become muted for other reasons. Two possibilties I see:

  1. Don't support "min" and "max", only support "ideal". As such we never get overconstrained; as such we never get muted for constraints-reasons.
  2. Don't mute tracks on overconstrained or remove overconstrained altogether (discussion separate from the local vs remote discussions).
@henbos

This comment has been minimized.

Copy link
Contributor

commented Mar 8, 2019

@armax00

This comment has been minimized.

Copy link

commented Mar 11, 2019

My 2 cents.

With respect to the behavior of applyConstraints() as well as gUM constraints, my understanding is that they should behave as filters, not as settings. This means that for local tracks it makes sense to filter which device should be used based on some criteria but with remote tracks this seems blurry, mostly because such parameters are anyway negotiated (hence, I am not sure why, after negotiation one should further filter for the right track).

@jan-ivar

This comment has been minimized.

Copy link
Contributor Author

commented Mar 11, 2019

I think a lot of this boils down to if you view "applyConstraints()" as means to change configurations of a source or filter/transform what the source provides you with.

It's clearly both. Constraints outsource (no pun intended) user requirements from a source to its sink(s); a classic "put the control surface on the result object" thing. A width might reconfigure the camera, or software downscale, doesn't matter. Tracks are inseparable from their source; multiple heads of it.

It's an API. We're in an ugly spot of asking: "What behavior should this API produce?", when we need to ask: "What remote track use-cases must be solved (and what's the best API for them)?"

In the case of w3c/mediacapture-screen-share#31 (comment) we had those use-cases.

We can either be "defensive", and say "only these constraints are supported for remote tracks" and be explicit about it, or "offensive" and allow all constraints that are still defined in a way that can be applied to remote tracks to be so. Example: "width" means the same thing independent of what source produced the frames.

It doesn't. A camera track is inherently and observably different from a screen-share track here:

  • width on a screen-share track preserves aspect, to not lose information around the edges.
  • width on a camera track might crop the output, compromising aspectRatio, and/or frameRate.

There are other differences as well: screenshare tracks never fire overconstrained and never mute. advanced and min are effectively useless (and provoke TypeError in getDisplayMedia()). resizeMode can be used to detect retina. Many of these behaviors weren't obvious until we specified them.

If "defensive" means specifying how width should work on a peer connection track, if at all, and not leaving it to implementor whim, then count me in.

@henbos henbos self-assigned this Mar 13, 2019
@henbos

This comment has been minimized.

Copy link
Contributor

commented Mar 13, 2019

I've created a slide for this.

@guidou

This comment has been minimized.

Copy link

commented Mar 13, 2019

I agree with @jan-ivar's comment.
I propose we update mediacapture-main to say that the constrainable properties defined there apply exclusively to tracks backed by getUserMedia() calls and that other specs using MediaStreams and related objects should explicitly define the properties that apply to them while respecting the constrainable pattern in general.

@henbos

This comment has been minimized.

Copy link
Contributor

commented Mar 15, 2019

Count me in

@henbos

This comment has been minimized.

Copy link
Contributor

commented Apr 25, 2019

Related: I filed w3c/mediacapture-main#578 on the gUM spec.

@dontcallmedom

This comment has been minimized.

Copy link
Member

commented Sep 25, 2019

RESOLUTION: define retrieval of constraint properties for getSettings but make them not modifiable

@jan-ivar

This comment has been minimized.

Copy link
Contributor Author

commented Sep 26, 2019

@henbos to provide PR?

@henbos

This comment has been minimized.

Copy link
Contributor

commented Sep 26, 2019

Sure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
7 participants
You can’t perform that action at this time.