Clarify how thumbnailing is supposed to work

[turt2live]

Particularly the methods: crop/scale

[ara4n]

The intention is that crop removes data to make the image approximately fit into the requested aspect ratio, and scale shrinks the image down (not adding background colour) to make the image approximately fit instead. It's the same difference as between pan & scan (crop) versus letterboxing (scale), for those familiar with the terms.

The server treats the dimensions provided as an aspect ratio, and then should provide an image at least as large as the requested dimensions. This approximation is to avoid DoS and let servers precalculate thumbnails for commonly requested dimension sizes.

It's worth noting that synapse currently gets this wrong; if you request a bigger thumbnail than the largest one it's precalculated, you get the largest thumbnail (matrix-org/synapse#5302) which may undershoot. Cropping apparently can also undershoot (matrix-org/synapse#1222). When you undershoot, the client has to scale up the resulting image, resulting in horrid blurriness artefacts.

[turt2live]

How should the server handle being requested to give a thumbnail larger than the source material?

[ara4n]

it should return the source material. if in cropped mode, it could also crop it to the requested aspect ratio.

ftr, i personally detest that all of width, height, crop & scale end up being hints at best, and so the client always has to assume that the server may have entirely ignored them. it makes it incredibly hard to produce an optimal performant and sharp UI.

one approach might be to formally spec the recommended pre-computed dimensions that a server should support, so that client developers at least know what they have to work with. In fact, this might have been better than the whole hint API in the first place - you'd just pass an enum for the shape you want, and you then know what you'll get, rather than the whole "make it look like you can ask for anything, but then disappoint" experience we have today.

however, i think we're stuck with this shape for now, and at least it gives some level of flexibility for people to experiment with other dimensions in future.