Add semantics choice to getUserMedia() for in-browser picker.

[jan-ivar]

Fixes https://github.com/w3c/mediacapture-main/issues/652

[aboba]

A few comments:

• The PR does not provide a way to retrieve the value of GetUserMediaSemantics. Presumably, the transition plan is for GetUserMediaSemantics to default to browser-chooses initially, then change default to user-chooses, but how does the application determine what the value has been set to? feature-detect only helps to determine whether GetUserMediaSemantics is available at all. It would be good for the application to be able to retrieve the value of GetUserMediaSemantics.
• Reading the PR, I wasn't sure whether user-chooses always results in a prompt. If this is the intent, it should be more explicit.
• "Narrow" might be a better word choice than "widdle".

[henbos]

and if it doesn't? Is prompting in that case a MUST NOT, a SHOULD NOT or MAY?

[jan-ivar]

"Otherwise," below kicks in. There wasn't a MUST there before, but I can add one.

[henbos]

It would be nice if we got as close as possible to not letting implementation-specific logic decide whether or not to prompt. Like with getDisplayMedia()... it always prompts. Can getUserMedia() always prompt no matter what?

When you lost audio during the meeting @youennf made a point of what if the user wants to share only audio and not video or vice versa? That may require further discussion but I like the idea of "user-chooses" to be "audio and/or video" so that you never have to reject and then re-prompt.

[henbos]

Idea in the future "user-chooses" world.

• getUserMedia() always prompts. It gives audio and/or video, depending on user choice. "kind" is optional.
• If you want to change device mid-call, track.changeUserMedia() (or track.applyConstraints("user-chooses") if that makes transition easier) re-prompts the user for audio and/or video, depending on user choice.

[jan-ivar]

Can getUserMedia() always prompt no matter what?

No-one wants that. Users don't want to be prompted for their camera every time.

Many have only one camera, thus no need for a picker at all (ditto microphone).

Even with "user-chooses" (a semantic we aspire to default to as sole modus operandi), it remains up to the app to request what it wants: a specific device, or any device fitting its constraints. A user agent picker would kick in (regardless of permission) only when that request encounters a user with multiple valid devices meeting those constraints.

I think that's the right division of concerns. App needs meet user needs.

The following then guarantees a picker, e.g. letting us replace in-content selection:

if (devices.filter(({kind}) => kind == "videoinput").length > 1) {
  await navigator.mediaDevices.getUserMedia({
    video: true, semantics: "user-chooses"
  });
}

what if the user wants to share only audio and not video or vice versa?

Firefox used to do this, but we had to stop because web sites were expecting two tracks. I'd argue it's an inherent problem with the API, not solvable in the spec.

In fact, the spec algorithm is already written as concurrent steps per kind that "User agents are encouraged to bundle ... into a single user-facing permission prompt."

In fact, Hangouts already asks for camera and microphone separately, and I have a bug filed on Firefox somewhere to combine these requests into one prompt. That problem goes away then.

[jan-ivar]

• Presumably, the transition plan is for GetUserMediaSemantics to default to user-chooses initially, then eventually to not permit browser-chooses

As I recall, we did not reach consensus on a transition plan last virtual interim, just option B for now.

So there are no transition impacts from this PR, since it effectively defaults to "browser-chooses" for backwards compatibility (yet allows Firefox to prompt like it does today).

This is the minimal consensus needed to replace labels and in-content selection.

How does the application determine what choices are available?

Like it does today, count devices per kind using enumerateDevices. Although if constraints are used, this might narrow choices, and it could be hard to detect. Right now I think the API would succeed right away without a prompt in that case (assuming you already have access to the current device). Would you like it to fail? That would be a bigger semantic difference.

The feature detection is there to let us start using this before labels go away.

Reading the PR, I wasn't sure whether "user-chooses" always results in a prompt.

It does not. Only when there's a choice. Otherwise, only a permission prompt is mandated.

[jan-ivar]

A user agent picker would kick in (regardless of permission) only when that request encounters a user with multiple valid devices meeting those constraints.

...otherwise users would instead see a permission prompt if permission is needed, or no prompt at all if the app already has permission.

[jan-ivar]

Summary of PR: Based on consensus in February call (Option A/B in slides). New semantics:

navigator.mediaDevices.getUserMedia({video: constraints, semantics: "user-chooses"});

...to work as follows:

• Users with 1 device satisfying the constraints: permission prompt only if needed (like today).
• Users with >1 device satisfying the constraints: browser opens a device picker always.

Sites avoid picker on revisits by remembering w/deviceId constraint to narrow constraints to 1.

Use this to replace today's in-content device pickers. May also be used on initial call.

[a2sheppy]

@jan-ivar Please be sure any bug around this gets DDN, or file a bug against the MDN content (https://github.com/mdn/sprints/issues/new/choose) so we can be sure this change is documented.

[getify]

I just wanted to chime in that I have used a number of web-based video chatting apps over the last few weeks that don't offer me any choice of my camera (my setup has multiple cameras). Of course, those sites could have used APIs to enumerate my cameras and let me pick, but they didn't. And since that's left to the code, I the user am given no choice in the matter.

I really wish a change could be made that would, without any changes to existing code, give me the option to pick or authorize a different camera. In other words, what I wish is that this "user-chooses" semantics was the default.

Even if the browser UI still only let me pick/authorize a single camera (and not giving me the option to select/authorize multiple cameras and mics), at least I as a user would have the option in the browser to choose to force-expose a specific camera to the site rather than them getting to pick my default camera.

[guest271314]

@getify Is the browser tested Chrome or Chromium?

[getify]

Chrome (windows).

It happens to me every other day or so, I have to connect to webchat service (like telehealth, whatever) and their UI doesn't let me choose what mic or camera I want... I assume JS wise they just grab the first camera and mic, but from my perspective it feels random... sometimes it's the right one of each, sometimes it's the right camera wrong mic, sometimes right mic wrong camera, sometimes both are wrong.

Even though I go out of my way to set the default mic in windows, that's not always the mic that such a webchat app tries to pick up. That boggles my mind. But worse: my windows laptop doesn't let me set a "default camera" -- it just assumes my built-in camera is what I want to use -- though it almost never is!

Chrome doesn't offer me any UI option to select which camera or mic are exposed to a web page. All I can do is just turn on or off camera/mic, but not select.

If this "user-chooses" semantic isn't made the default, then this feature won't do anything to help me, because all those poorly designed webchat apps will never update their code to turn it on -- either they don't know about it or don't care.

[guest271314]

@getify Have not used Windows is some time. FWIW a workaround for Linux #693 (comment) to select audio devices other than default microphone. If Native File System is not enabled by default you can launch Chrome with --enable-experimental-web-platform-features then use the same approach at a Windows shell script with a port of inotify-tools e.g., https://github.com/thekid/inotify-win to launch a native application to change the video device after getUserMedia() has permissions, if Windows has such a display or device selection native application.

[guest271314]

@getify Per https://answers.microsoft.com/en-us/windows/forum/windows_10-hardware/how-to-change-default-webcam-in-windows-10/1e8377c8-4fd5-4e9c-8ab6-67913c5c8e0c you should be able to launch Control Panel to change the camera device and per https://docs.microsoft.com/en-us/windows/uwp/audio-video-camera/camera-profiles it should be possible to change the device progammatically. Either program can be launched in the browser session using the linked approach. Have not found any other means to change the default audio or camera at Chromium other than using a fake media stream and supplying the data thereto by way of a file. There are ways, as demonstrated at below post, see also https://stackoverflow.com/questions/43480867/linux-pipe-audio-file-to-microphone-input/43553706#43553706. The audio device change at Linux using PulseAudio (pavucontrol in the proof of concept) is in stream; the devices changes during the capture. Have not experimented yet with changing the video device being captured during the stream.

[guest271314]

@getify Another alternative is to file an issue with the service to resolve

and their UI doesn't let me choose what mic or camera I want... I

which should be possible utilizing the current iteration (using labels of enumerateDevices()) and selecting device by deviceId. To test hot-plugged a USB camera labeled "Digital_Camera (<UUID>)"

navigator.mediaDevices.getUserMedia({video: true})
.then(async stream => {
  const devices = await navigator.mediaDevices.enumerateDevices();
  // At Chromium _site_ provides UI for device selection after permissions granted
  // why delegate to _site_ and not trust user to decide what devices they want to expose and capture?
  // does the specification really need changing here?
  // circa 2020 C.E. user media can be in various forms outside of what a specification
  // is capable of keeping up with - _unless_ specification authors actuall test _all_ variants constantly
  // just let user decide what to do with the application: Big conspcious note: USE AT YOUR OWN RISK
  const {deviceId} = devices.find(({label}) => label === 'Digital_Camera (093a:2700)' /* selection UI e.g., HTML */ );
  // if (deviceId) {/* do stuff */}
  let [track] = stream.getVideoTracks();
  // await track.applyConstraints({deviceId: {exact: deviceId}}) throws OverConstrained error at Chromium
  console.log(await track.getConstraints());
  // stop the MediaStreamTrack
  track.stop();
  // get exact deviceId
  stream = await navigator.mediaDevices.getUserMedia({video: {deviceId: {exact: deviceId}}});
  [track] = stream.getVideoTracks();
  // check deviceId
  console.log(await track.getConstraints());
  const v = document.createElement('video');
  v.controls = v.autoplay = true;
  document.body.appendChild(v);
  v.srcObject = stream;           
});

[guest271314]

Chrome doesn't offer me any UI option to select which camera or mic are exposed to a web page. All I can do is just turn on or off camera/mic, but not select.

If this "user-chooses" semantic isn't made the default, then this feature won't do anything to help me, because all those poorly designed webchat apps will never update their code to turn it on -- either they don't know about it or don't care.

Agree.

Not an issue at Firefox or Nightly.

Perhaps Chromium authors will listen to you.

Have been asking for this feature for some time in various ways, too many to list here, e.g., https://bugs.chromium.org/p/chromium/issues/detail?id=1032815; https://bugs.chromium.org/p/chromium/issues/detail?id=865799; for reasoning at Chrome, Chromium see https://bugs.chromium.org/p/chromium/issues/detail?id=1013881#c9.

Eventually decided to stop asking at specifications or implementers and create code that achieves the expected result.

[jan-ivar]

Moved to w3c/mediacapture-extensions#9.