Undefined how video dimension constraints are met

[stefhak]

As far as I read the spec, it is not defined how constraints on (video track) width, height and aspectRatio are met. Will the video be scaled down, cropped, a combination? Will it be letterboxed/pillarboxed?

A concrete use case that has been discussed is one where square 100x100 thumbnails are to be shown (at a remote end point). Say the camera produces a 16:9 high resolution video feed, applying constraints width:100, height:100 should ideally scale down to height 100 and then crop the width to 100 in this use case. Applying (only) aspectRatio:1 should crop the width to 1080 (at least for this use case).

We should discuss this at TPAC.

[jan-ivar]

width:100, height:100 should ideally scale down to height 100 and then crop the width to 100

More than "how", the spec doesn't mandate downscaling, so this won't work across browsers.

The sole mention of downscaling in the entire spec appears in this paragraph about concurrent use:

"Multiple tracks can be attached to the same source. User Agent processing, such as downsampling, MAY be used to ensure that all tracks have appropriate media."

We're implementing that paragraph in Firefox, to abstract away knowledge of other tabs, which is good for privacy. However, this is separate from boosting the fitness distance pool.

We're open to figuring out how to interpret rescale intent from constraints, as you can see in #472.

thumbnails are to be shown (at a remote end point)

The only spec way to do that today is to downscale in the encoder, using scaleResolutionDownBy.

An implementation would have to be careful to avoid rescaling twice. How sure are we we want the track to be the control surface for the encoder?

[stefhak]

@jan-ivar thanks for your input, I agree this is related to #472

An implementation would have to be careful to avoid rescaling twice. How sure are we we want the track to be the control surface for the encoder?

I think the important thing here is that we have contraints that enable constraining width, height and aspectRatio for video tracks, and therefore we need to specify how an implementation must behave.

[jan-ivar]

@stefhak I think you're thinking of "constraining" = rescaling, which is not the spec definition.

For instance, If getUserMedia worked that way, max and exact would be redundant.

The spec defines applyConstraints() as a filtering function applied to a UA-specific pool of available settings dictionaries (instances of MediaTrackSettings) for a device. A discovery API essentially.

Firefox and Edge implement discrete settings dictionaries. Chrome implements a continuous space.

[stefhak]

@jan-ivar I may very well have gotten things wrong, that happens all the time :)

Could you explain what you think should happen if

• getUserMedia is called with video as only argument
• doing getSettings() on the resulting video track returns a width of 1920 and a height of 1080
• the app then applies mandatory constraints saying that width and height should both be 100?
• or, if the app instead of width and height puts aspectRatio constraint (mandatory) to 1?

[jan-ivar]

The sole mention of downscaling in the entire spec appears in this paragraph about concurrent use

Correction: I found one more, in this note. One has to search for "downsampling" and "resampling".

[alvestrand]

#502 seems to be solving this question.