Is muting track the right call for empty cropped tracks?

[youennf]

The spec is not really clear about what muting a track means.
It probably means doing https://w3c.github.io/mediacapture-main/#set-track-muted.

Using muted is potentially not a great idea since the application might not know whether the track is muted because of cropping or because of some other reasons. For instance, Safari does allow a user to mute/unmute.

Also, muted is usually tied to the source: if source is muted, all related tracks are muted.
In this case, muted becomes a property of the track.
It is assumed that muted is "outside the control of web applications" while in that case, applications could potentially mute/unmute it.

I think it would be better to first discuss whether we need this (this information can be gathered from other sources, or region producer could instruct the capturer that its region is no longer useful).
If we need this mechanism, maybe it deserves its own mechanism.

[eladalon1983]

I think it would be better to first discuss whether we need this (this information can be gathered from other sources, or region producer could instruct the capturer that its region is no longer useful).
If we need this mechanism, maybe it deserves its own mechanism.

I'd be happy to drop everything mute-related from the spec, and reopen this topic if+when it's relevant in the future. I can envision several possible mechanisms to address this, but I don't think it's a crucial part to an MVP. Wdyt @jan-ivar and @youennf?

[youennf]

I'd be happy to drop everything mute-related from the spec

Sounds good.

and reopen this topic if+when it's relevant in the future.

It seems worth opening a new issue to keep track of this area (crop target is no longer valid, say HTMLElement destroyed say; crop target is valid but will trigger zero pixel frames), should UA take default actions? Should web pages have an easy way to detect these cases?...

[eladalon1983]

I'd be happy to drop everything mute-related from the spec

Sounds good.

To clarify, I'll be removing references to mute, but otherwise the behavior will remain the same - no new frames will be delivered if the application specified a crop and either (i) it's not possible to crop to that element, e.g. if it's gone, or (ii) cropping to that element results in zero pixel frames, e.g. if the crop-target has been scrolled out of the viewport.

It seems worth opening a new issue to keep track of this area

I think we can keep using this issue and continue the discussion here? I've applied the label Improvement to make it clear that we've agreed this is a non-blocking issue.

Continuing the discussion - there are multiple cases where the application either cannot recognize what's going on, or it's not ergonomic for it to do so, or it would introduce a delay due to cross-origin communication to achieve. That's why I used the mute signal. If mute is problematic because it's hard to tell apart from existing uses, we can introduce a similar signal (maybe interrupted) if we find out developers are asking for it. IMHO it's fine to start by using the same signal for all interruption reasons.

Another option, and one which I like better, is to keep using muted, onmute and onunmute, but to add a mute_causes vector:

enum MuteCause {
  "user",
  "target_lost",
  "zero_pixel_frame"
};

partial interface MediaStreamTrack {
  readonly sequence<MuteCause> mute_causes;
}

[eladalon1983]

@jan-ivar, could you please take a look at the current language around the mute events? As per agreement with @youennf, I'll remove mentions of muting before we move to FPWD, but I want to give you some time to have a look and form an opinion about what we'd want to do afterwards.

[alvestrand]

Mute has traditionally been "this track is not producing data, this was intentional, and we're not telling you why".
https://w3c.github.io/mediacapture-main/getusermedia.html#track-muted

Do we have an operational need for the application to know why the track is muted, that can't be solved by the capturer inspecting the situation in other ways, for instance by inspecting the HTMLElement it did produceCropTarget() on?

If adding MuteCause, I'd add this to CropTarget, not to MediaStreamTrack.

[eladalon1983]

...that can't be solved by the capturer inspecting the situation in other ways...

If the capturing frame is cross-origin from the frame hosting the crop-target, this inspection requires asynchronous messaging, and there's a cost associated. Applications that need to react[*] won't be able to do so in a timely fashion. (Also hard to handle robustly, e.g. when the user repeatedly scrolls the crop-target in and out of view.)

[*] That said, the applications I currently have in mind, would probably not allow crop-targets to be scrolled away. I cannot presently think of a realistic use-case where an application would really need this signal. That's why I'm happy removing the discussion of muting from the spec, and continue this discussion as a lower-priority issue.

Mute has traditionally been "this track is not producing data, this was intentional, and we're not telling you why".

Non-rhetorical questions: Is the "not telling you why" part important? Is it important to hide the fact the user pressed "mute" in the browser's UX? And if so, do we have enough possible causes in the mute bin that "user probably muted" could not be inferred? Was it a conscious decision to hide the cause, and if so, is it worth "relitigating" it?

[youennf]

Should we close this issue now that mute language has been removed from the spec?

[eladalon1983]

@youennf, it's your issue. Could you please close it?

[eladalon1983]

Hi @thomas-gg. I am having some trouble following your message. It seems to assume that the cropped video must be played back through a video tag, which is not true. Cropping also works with simply saving the video to disk, transmitting it remotely, etc. The current issue deals with the crop target being empty.

But I may have misunderstood you. If so, please try explaining the issue more briefly and simply.

[eladalon1983]

Thank you for the clarifications. Please note that this GitHub repo is appropriate for spec issues. If you find any Chrome-specific issues (i.e. issues with our implementation of the spec), then I encourage you to file a bug here and assign Blink>GetDisplayMedia>RegionCapture under Components. That should be enough to ensure I get CCed (eladalon@chromium.org).