Ignore PTZ request if no PTZ camera

[beaufortfrancois]

As raised in #243 (comment) by @youennf, the spec should explicitly say that the PTZ request will be ignored and it will be treated like a regular camera request if there are no cameras with PTZ capabilities on the user machine during getUserMedia.

In the second query, if there are no cameras with PTZ capabilities, the PTZ request will be ignored and it will be treated like a regular camera request.

It seems you say that, if there is no PTZ camera, the permission request will be { name: "camera", panTiltZoom: false }.
Where is this algorithm defined?

[jan-ivar]

This risk of a prompt should be enough to deter trackers from calling getUserMedia.

It's unclear to me though whether we need to do the same for applyConstraints. Required constraints and OverconstrainedError are still arguably useful there.

[eehakkin]

What exactly should be ignored?

If a web site calls navigator.mediaDevices.getUserMedia({video: {pan: {}}}) and there are no PTZ cameras, the permission request will be {name: "camera", panTiltZoom: false} (instead of {name: "camera", panTiltZoom: true}) and what should happen then? Should the pan: {} part be completely ignored (an empty ConstrainDoubleRange value implies no constraints per the spec)?
At the moment Chrome implements the permission ignorance part but throws an OvercontstrainedError which does not make sense.

What if a web site calls navigator.mediaDevices.getUserMedia({video: {pan: {ideal: 0}}}) and there are no PTZ cameras? Should be pan: {ideal: 0} part be completely ignored or should the fitness distance be 0 because a pan constraint does not apply to non-PTZ cameras? This choice has no effect to the selected device but affects how this should be specified in the spec. Or should the fitness distance be positive infinity because there is no actual pan setting? This would cause all non-PTZ cameras to be ignored and an OverconstrainedError to be thrown.

What if a web site calls navigator.mediaDevices.getUserMedia({video: {pan: {exact: 0}}}), navigator.mediaDevices.getUserMedia({video: {pan: {max: 0}}}) or navigator.mediaDevices.getUserMedia({video: {pan: {min: 0}}}) and there are no PTZ cameras? Should be pan: {exact: 0} et al parts be completely ignored or should the fitness distance be positive infinity because the settings dictionary's (missing) value for the constraint does not satisfy the constraint?

[youennf]

If a web site calls navigator.mediaDevices.getUserMedia({video: {pan: {}}}) and there are no PTZ cameras, the permission request will be {name: "camera", panTiltZoom: false}

From my reading of the spec, the request will be {name: "camera", panTiltZoom: true} not {name: "camera", panTiltZoom: false}. I think this is the issue to solve here.

What if a web site calls navigator.mediaDevices.getUserMedia({video: {pan: {ideal: 0}}}) and there are no PTZ cameras?

The request should probably be {name: "camera", panTiltZoom: false}.

What if a web site calls navigator.mediaDevices.getUserMedia({video: {pan: {exact: 0}}})

If there is no PTZ camera, there will be no permission request as the algorithm will bail out early.
Fur simplicity, the request should probably be {name: "camera", panTiltZoom: false}.

[eehakkin]

If a web site calls navigator.mediaDevices.getUserMedia({video: {pan: {}}}) and there are no PTZ cameras, the permission request will be {name: "camera", panTiltZoom: false}

From my reading of the spec, the request will be {name: "camera", panTiltZoom: true} not {name: "camera", panTiltZoom: false}. I think this is the issue to solve here.

Yes, exactly.

But what should happen then?

Should the pan: {} part be completely ignored (an empty ConstrainDoubleRange value implies no constraints per the spec)?
At the moment Chrome implements the permission ignorance part but throws an OvercontstrainedError which does not make sense.

What if a web site calls navigator.mediaDevices.getUserMedia({video: {pan: {ideal: 0}}}) and there are no PTZ cameras?

The request should probably be {name: "camera", panTiltZoom: false}.

Yes, exactly.

But what should happen then?

Should be pan: {ideal: 0} part be completely ignored or should the fitness distance be 0 because a pan constraint does not apply to non-PTZ cameras? This choice has no effect to the selected device but affects how this should be specified in the spec.

Or should the fitness distance be positive infinity because there is no actual pan setting? This would cause all non-PTZ cameras to be ignored and an OverconstrainedError to be thrown.

What if a web site calls navigator.mediaDevices.getUserMedia({video: {pan: {exact: 0}}})

If there is no PTZ camera, there will be no permission request as the algorithm will bail out early.
Fur simplicity, the request should probably be {name: "camera", panTiltZoom: false}.

Yes, probably.

So the pan: {exact: 0} et al parts must not be completely ignored in this case. The fitness distance is positive infinity because the settings dictionary's (missing) value for the constraint does not satisfy the constraint. This causes all non-PTZ cameras to be ignored and an OverconstrainedError to be thrown as there are not other cameras.

I am asking there questions because it does not make sense to ignore PTZ permission request without specifying what do with PTZ constraints (or non-constraints) in whose cases.

[youennf]

But what should happen then?

Since there is no PTZ camera in that case, all devices will react to pan : { ideal : 0 } the same way and it will be as if it is ignored.

The problem you seem to point out is when there is a mix of cameras with and without PTZ.
In that case, the request will be {name: "camera", panTiltZoom: true}.
Chrome implementation will most probably show the prompt asking user to allow 'use & move', though the camera that will be selected might not be a PTZ camera.

We could try fiddling with distances to maximise the ability to select a PTZ.
But there is no guarantee that a PTZ camera will actually be the one that will be used by the page.

[jan-ivar]

The problem you seem to point out is when there is a mix of cameras with and without PTZ.

I've opened #256 for that, so we can focus on how to "ignore" constraints to solve OverconstrainedError fingerprinting.

[jan-ivar]

I fear we may have hit the limits of normative prose in field descriptions, and may need to modify algorithms. So for clarity, here's what I think we need:

• At the start of getUserMedia's in parallel steps, add something like:

  1. If the user has no PTZ cameras attached, remove all pan, tilt, and zoom members from constraints.video (this'll be reflected in track.getConstraints() on success only).
     * Note: This removes a site's ability to use OverconstrainedError on pan, tilt, or zoom to detect PTZ abilities without risking a prompt if no PTZ devices exist.

  2. (Move the ptz-specific permission request modification steps here (or further down where gUM requests permission).

This order should clarify that:

• PTZ constraints are completely ignored unless the user has at least one PTZ camera attached.
• Sites cannot get OverconstrainedError on ptz constraints unless the user has a ptz camera attached.
• Users can't be prompted for PTZ permission ("use & move") unless they have a ptz camera attached.
• PTZ constraints still work in track.applyConstraints where they may trigger a PTZ permission prompt.

[youennf]

• PTZ constraints are completely ignored unless the user has at least one PTZ camera attached.
• Sites cannot get OverconstrainedError on ptz constraints unless the user has a ptz camera attached.

I think we want to go further and make it so that PTZ constraints can never trigger an OverconstrainedError, whatever the underlying setup.

It seems the issue is that getUserMedia algorithm is asking permission for a set of devices.
Until PTZ, all devices were interchangeable in terms of permission but PTZ current model is breaking this.

We could change getUserMedia algorithm so that it asks permission for one specific device of a given kind.
If the device happens to be PTZ and PTZ is requested, a specific PTZ prompt would be used.
If the device happens to be PTZ and PTZ is not requested, a regular prompt would be used.
If the device is not PTZ, a regular prompt would be used.

With this algorithm, the downside would be if opening device fails after user granted access.
In that case, the initial prompt might not be in sync with the actually selected device.
I think this case is rare enough though.

[jan-ivar]

@youennf Sorry how does the permission level granted relate to OverconstrainedError?

We could change getUserMedia algorithm so that it asks permission for one specific device of a given kind.

That's the case in Firefox by default where one-time permissions are per device.

For {pan: true} I imagine Firefox might ask "Will you allow site x to use and move your camera?" IF the user has at least one ptz camera (otherwise leave out "and move" and don't grant that privilege).

If the user picks a different one then obviously the site won't be able to move it.
If they check ☑ Remember this decision then it makes sense to persist both camera and ptz permission I think.

I gather the latter mirrors other browsers more, where even per-session permissions are for all cams or all mics.

[beaufortfrancois]

@youennf @jan-ivar What are the next steps for this issue?

[youennf]

Given there is no required constraints anymore, the problem is simplified a bit.
Whenever PTZ constraints are in the query, the PTZ permission gets requested in addition to the camera permission.
It is then up to the User Agent to show a prompt that will allow to grant PTZ additional permission or not.
I would guess the User Agent will show a regular prompt if there is no PTZ camera, and will not grant the PTZ additional permission in that case.

I still believe we should try to mandate that, should the PTZ permission is granted, the User-Agent tries hard to pick a PTZ device, but this is an orthogonal issue.

[beaufortfrancois]

Thanks @youennf. It makes sense.
@eehakkin Can you update spec accordingly?

[eehakkin]

@eehakkin Can you update spec accordingly?

#261 states that [t]he [getUserMedia()] algorithm MUST request permission to use a PermissionDescriptor with its name member set to PermissionName/camera and its panTiltZoom member set to true, and, optionally, consider its deviceId member set to any appropriate device's deviceId unless there are no pan capable cameras (and similarly for tilt and zoom).

[jan-ivar]

Given there is no required constraints anymore, the problem is simplified a bit.

@youennf It does. Though just to clarify: there are no required basic imageCapture constraints anymore in getUserMedia. They still exist in advanced: [], in track.applyConstraints, and other specs.

I still believe we should try to mandate that, should the PTZ permission is granted, the User-Agent tries hard to pick a PTZ device, but this is an orthogonal issue.

This should fall out naturally once we fix #256 to have the spec match current design intent (right now pan: true has zero impact on fitness distance, which was clearly no-one's intent. The intent was clearly for it to have weight like all other constraints have, like e.g. facingMode: "environment".

I personally don't think we need to go further than that.

[youennf]

The intent was clearly for it to have weight like all other constraints have, like e.g. facingMode: "environment".

I disagree. If user denied PTZ permission, PTZ constraints should have zero effect.
If user granted PTZ permission, PTZ devices should be selected first as only these devices can use this privileged permission.
The fitness distance is not a solution here since it is adding/comparing apples with oranges.

As mediacapture-main is not constraining in any way how selection should happen, mediacapture-image can easily prescribes some selection rules without changing anything with the fitness distance.

[riju]

Thanks for all the discussions here. We have put up #280 to update the spec.

[eehakkin]

#261 states that [t]he [getUserMedia()] algorithm MUST request permission to use a PermissionDescriptor with its name member set to PermissionName/camera and its panTiltZoom member set to true, and, optionally, consider its deviceId member set to any appropriate device's deviceId unless there are no pan capable cameras (and similarly for tilt and zoom).

That did not land and was replaced by w3c/mediacapture-main#707 which is now merged.

---

Regarding the title of this issue (Ignore PTZ request if no PTZ camera), I think it is in essence resolved by w3c/mediacapture-main#707, w3c/mediacapture-main#766 and #271.

According to the current main spec (after w3c/mediacapture-main#707), all required image capture constraints (including the PTZ constraints) are always outright rejected with a TypeError (regardless of whether or not there are any PTZ cameras) and if there are no PTZ cameras,

• optional basic PTZ constraints with value false (like {pan: false}) increase fitness-distance by 0 for all non-PTZ cameras,
• optional basic PTZ constraints with value other than false increase fitness-distance by 1 for all non-PTZ cameras,
• advanced constraint sets with PTZ constraints with values other than false are ignored, and
• PTZ constraints with value false in advanced constraint sets have no effect.

So while the PTZ constraints are not fully ignored if there are no PTZ cameras, the PTZ constraints either increase the fitness distance by the same amount for all non-PTZ cameras, cause some advanced constraint sets to be ignored for all non-PTZ cameras or have no effect all non-PTZ cameras. Thus the PTZ constraints have basically no effect if there are no PTZ cameras.

Therefore, the constraint part is resolved.

According to the current spec (after #271), there are no requirements for a user-agent ever to request a PTZ permission as long as the user-agent does not does not expose pan, tilt nor zoom settings. If there are no PTZ cameras, the user-agent even cannot expose pan, tilt or zoom settings (because there is nothing to expose), so that is a perfect case for a user-agent not to request a PTZ permission.

Therefore, the permission part is resolved, too.

So, in my opinion, the issue at hand is resolved and should be closed.

---

If user denied PTZ permission, PTZ constraints should have zero effect.

@youennf: Since your comment, the spec has be changed (by #271). The intention (there is a bug, see #279 and #280) is that if a user denied the PTZ permission, the user-agent MUST decide not to expose pan, tilt and zoom settings, in which case none of the settings dictionaries contains pan, tilt or zoom settings (regardless of whether the actual camera hardware supports PTZ) thus all algorithms behave like all cameras were non-PTZ cameras thus PTZ constraints have basically no effect like in the case of no-PTZ cameras above.

So, in my option, this issue is also resolved. If you disagree, let's create another issue for this case.

---

If user granted PTZ permission, PTZ devices should be selected first as only these devices can use this privileged permission.
The fitness distance is not a solution here since it is adding/comparing apples with oranges.

Please note that while that is not currently stated in the spec, the user-agent is perfectly allowed to do the reverse and request the PTZ permission only if a PTZ device is about to be selected.

On the other hand, if a page does

navigator.mediaDevices.getUserMedia({video: {deviceId, pan: true}})

to check if the device is pan-capable, another user-agent may want to request a PTZ permission even if the device is pan-incapable, and that is allowed, too.

However, if you want to discuss more on this case, lets create another issue for this.

[youennf]

Therefore, the permission part is resolved, too.

So, in my opinion, the issue at hand is resolved and should be closed.

Sounds good to me.

Please note that while that is not currently stated in the spec, the user-agent is perfectly allowed to do the reverse and request the PTZ permission only if a PTZ device is about to be selected.

That is fine too. While we might not want to go too deep in device selection territory here, which is ultimately UA specific, it would be good to add some guidelines to implementors, something like:

• Only request PTZ permission if PTZ devices are selectable
• If user grants PTZ permission, make sure to use it, so select PTZ devices.

another user-agent may want to request a PTZ permission even if the device is pan-incapable, and that is allowed, too.

Right, but it does not really make sense to do so.
I am ok not forbidding it but it would be valuable to provide guidelines so as to discourage that kind of behavior.
Requesting permissions is expensive and it should only be done when really necessary.