MSC2700: Thumbnail requirements for the media repo #2700
Conversation
turt2live
added
kind:maintenance
turt2live
changed the title
| * `video/H264` | ||
| * `video/mp4` | ||
|
|
||
| Servers SHOULD support other mimetypes where possible, such as `application/pdf` and `text/plain`. |
There was a problem hiding this comment.
Just to clarify: this means that a media repository can still thumbnail any other formats that they want to?
There was a problem hiding this comment.
Yup, the minimum is provided above though
There was a problem hiding this comment.
The open-ended list means that clients still cannot reliably predict which MIME types a given server can thumbnail, beyond a very limited set.
Can't we just rely on what comes from /sync or /event and move the decision on whether a file has thumbnail or not to the moment of uploading? Basically, the uploading client could instruct the server to automatically generate thumbnail (on top of supplying the mxc: link to the thumbnail, as it's done now) and get a respective message on whether or not this is possible in the response. Downloading clients would just use the event payloads as they can do now to find thumbnail information - except that if there's none, clients SHOULD NOT ask the server for a thumbnail.
There was a problem hiding this comment.
The thumbnail endpoint exists so clients can populate that information in the event. It's not always possible to do the thumbnailing client side (encrypted media shows the artifacts nicely,), and the client rendering the event might have different resolution requirements anyways.
tbh, another msc should remove the thumbnail info from the event as it provides much less value to clients.
The open-ended support is a requirement, however there's zero reason why we can't specify a limited set for clients to always try for.
There was a problem hiding this comment.
The point about different resolutions is quite convincing, thanks for the reminder. But I still struggle to work out a coherent guidance for myself to implement this in Quaternion. Right now I just try to get a thumbnail for any message event with m.image or m.video. This MSC is supposed to change that but how exactly - this escapes me. I certainly don't want to restrain myself to the three required MIME types, but since the list is open-ended, I should - keep doing what I do now in the hope that the server supports more than just basics?
There was a problem hiding this comment.
well, as mentioned: the MSC needs rework.
There was a problem hiding this comment.
Perhaps an endpoint could be added where the server reports which mimetypes it can thumbnail? The client could query that on the first time it wants to thumbnail something and then make a decission based on on that
There was a problem hiding this comment.
There was a problem hiding this comment.
We introduced https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-media-r0-config to be a general key value map of configuration to inform the client before they try a media operation. It feels like including a list of supported mimetypes would be trivial from a spec and impl perspective?
| Servers SHOULD be able to generate thumbnails for the following mimetypes: | ||
| * `image/svg+xml` | ||
| * `video/H264` | ||
| * `video/mp4` |
There was a problem hiding this comment.
maybe also add image/webp to SHOULD as it is becoming more and more standard?
There was a problem hiding this comment.
There doesn't really appear to be a lot of traction around it, at least not enough to warrant an explicit suggestion in the spec.
| Servers SHOULD support other mimetypes where possible, such as `application/pdf` and `text/plain`. | ||
|
|
||
| In order to better support the server's mission in thumbnailing, clients SHOULD NOT upload | ||
| encrypted with a content type that matches the source material. Instead, clients SHOULD use |
There was a problem hiding this comment.
| encrypted with a content type that matches the source material. Instead, clients SHOULD use | |
| encrypted files with a content type that matches the source material. Instead, clients SHOULD use |
| * `video/H264` | ||
| * `video/mp4` | ||
|
|
||
| Servers SHOULD support other mimetypes where possible, such as `application/pdf` and `text/plain`. |
There was a problem hiding this comment.
The open-ended list means that clients still cannot reliably predict which MIME types a given server can thumbnail, beyond a very limited set.
Can't we just rely on what comes from /sync or /event and move the decision on whether a file has thumbnail or not to the moment of uploading? Basically, the uploading client could instruct the server to automatically generate thumbnail (on top of supplying the mxc: link to the thumbnail, as it's done now) and get a respective message on whether or not this is possible in the response. Downloading clients would just use the event payloads as they can do now to find thumbnail information - except that if there's none, clients SHOULD NOT ask the server for a thumbnail.
|
It's possible like the other thumbnail MSC I wrote that this one is far too split apart to see the actual goal. |
| Currently what can be thumbnailed is left as an implementation concern and not one that can be | ||
| predicted or anticipated by clients. This relatively small proposal aims to fix that. | ||
|
|
||
| ## Proposal |
There was a problem hiding this comment.
Thoughts about trying to predict supported mimetypes: Can we add a key to /config which includes which mimetypes are supported, much like we do with the max upload size today?
There was a problem hiding this comment.
I'd rather not tbh, as it means we're in capabilities negotiation territory and less in the realm of spec.
turt2live
added
the
needs-implementation
|
|
||
| ## Security considerations | ||
|
|
||
| Some media, such as SVGs, are vulnerable to various well-known attacks. These should be avoided by |
There was a problem hiding this comment.
These should be avoided by server implementations.
Does "These" refer to the media types or to the attacks?
|
MSC4011 is a much better approach than a list of MUSTs and SHOULDs - closing this MSC in favour of that one. |
turt2live
added
the
obsolete
Rendered