MSC3911: Linking media to events #3911

richvdh · 2022-10-16T22:38:17Z

proposals/3911-linking-media-to-events.md

uhoreg · 2022-10-17T19:25:17Z

proposals/3911-linking-media-to-events.md

+
+### Overview
+
+After an item of media is uploaded, it must be linked to an event (via


For MLS, we may need to reference media within a to-device event, but we might be able to link it to an in-room event. I'll have to look into this more, but flagging this as something that will need to be considered.

there's also ephemeral-ish things like user profiles which aren't directly related to events, though are duplicated/copied to membership events in most cases.

proposals/3911-linking-media-to-events.md

Munna231

#

richvdh · 2022-10-23T23:16:18Z

I've moved a bunch of the changes here out to ~~MSC3716~~ MSC3916, in the hope of making them a little more digestible.

deepbluev7 · 2022-10-23T23:19:35Z

I've moved a bunch of the changes here out to MSC3716, in the hope of making them a little more digestible.

You meant MSC3916 (if anyone else has issues following that link :D)

richvdh · 2022-10-23T23:20:58Z

I've moved a bunch of the changes here out to MSC3716, in the hope of making them a little more digestible.

You meant MSC3916 (if anyone else has issues following that link :D)

Thank you, yes I did 🤦. Edited.

reivilibre · 2023-03-10T17:08:07Z

proposals/3911-linking-media-to-events.md

+  In addition, this could cause duplication of media in the remote media cache,
+  if the implementation does not take steps to deduplicate (eg, storing media
+  by content hash rather than media id).


Can we spec a response header for the media download that provides a suitable hash of the object?
e.g. X-Matrix-Media-Hash: blake2,AAAAAAAAAAAAAAAAAAAAAAAAAAA....
That way, a homeserver (or client!) which is downloading a copy of the media can abort the download if it finds the hash already in its media store. This means that copies of large media do not lead to each server downloading it for each copy.
(Further, for bandwidth-sensitive clients, a HEAD could be used to get the hash before even initiating the download?)

De-duplication is already possible at an implementation level, and would probably be handled by a different MSC.

De-duplication is already possible at an implementation level

I think that reivilibre wasn't talking about the way clients/homeservers are deduplicating files after download, but rather before they're downloaded. I agree that former is an implementation detail that doesn't require uniformity across software, and as such doesn't need to be covered in protocol spec. But if the intention here is to avoid the very process of downloading a file multiple times (from a different linked MXC each time), can this be done without protocol declaring one well known way to discover e.g. that hash of underlying file by HEAD request?

I'm specifically thinking of a situation where you've got a user looking at sticker selector and not-so-maliciously spamming them by clicking them 100s of times. Some people do that, it isn't really a bad behavior in some rooms and contexts (slightly distant example would be custom emote spam on Twitch chats). Since the stickers sent in each event are no longer sharing the same MXC, AFAIK client (and homeserver, if sender is remote) will have to download the sticker 100s of times, and thus generate a lot of unnecessary traffic. Only then it will be able to deduplicate file for local storage/cache.

So my (and I assume reivilibre's) concern here is that with the proposal as it is right now, there's no way for client/homeserver to avoid that "traffic duplication", so to speak. Or is there? 😅

Hm, on the other hand, that choice of hash type used for deduplication before download, might eventually have an effect on future implementations of dedupe after download. For example, iirc currently in Synapse there's a mapping of MXC URI to local filename which is a random string. With this MSC, each server would have to store a mapping from the new generated IDs to the local file (in Synapse's case, again, a random name). If protocol adds a way to map MXC IDs to file hash early, is there a reason for servers to not store files by naming them after their hashes (as received over e.g. federation)? This would save you one more lookup: MXC IDs -> file hash -> randomized local name.

Oh, and I haven't thought how/if this "early dedupe" could realistically work with files attached to E2EE events, and whether it would add any additional metadata leak other than what's caused by this MSC as is.

richvdh · 2023-04-25T14:05:07Z

proposals/3911-linking-media-to-events.md

+1. A new "media upload" endpoint is defined, `POST
+   /_matrix/client/v1/media/upload`.  It is based on the existing
+   [`/_matrix/media/v3/upload`](https://spec.matrix.org/v1.4/client-server-api/#post_matrixmediav3upload)
+   endpoint, but media uploaded this way is not initially viewable (except to
+   the user that uploaded it). This is referred to as a "restricted" media item.


TODO: extend this to /_matrix/media/v1/create, as added by MSC2246.

turt2live · 2023-09-18T03:58:54Z

proposals/3911-linking-media-to-events.md

+   The new media item can be attached to a new event, and generally functions
+   in every way the same as uploading a brand new media item.


Can the user copy restricted media they don't have access to? (I assume not, but what is the appropriate error code?)

I'd say no, it's the same as /download and /thumbnail. 403 and M_UNAUTHORIZED.

turt2live · 2023-09-18T04:01:33Z

proposals/3911-linking-media-to-events.md

+removed. Currently, Synapse does not support removing the events (see also
+[synapse#4720](https://github.com/matrix-org/synapse/issues/4720)); but if at


Synapse was fixed a few days ago, apparently :)

turt2live · 2023-09-18T04:08:26Z

proposals/3911-linking-media-to-events.md

+   It is expected that servers will continue to treat such media as unrestricted
+   (at least for local users), but it would be legitimate for them to, for example,
+   return a different `mxc:` URI for each requesting user, and only allow each user
+   access to the corresponding `mxc:` URI.


I'm not sure I see the advantage of a server issuing distinct URIs for previews. Is there a motivation for putting this here?

I think I was trying to reason about how we might apply access restrictions to media that is part of a URL preview (if we decided that was a sensible thing to do). It may well be that the extra text confuses more than it clarifies.

turt2live · 2023-09-18T04:09:10Z

proposals/3911-linking-media-to-events.md

+
+## Unstable prefix
+
+TODO


(this is what MMR will be using)

turt2live · 2023-09-18T04:32:48Z

proposals/3911-linking-media-to-events.md

+     If any of the `attach_media` parameters do not correspond to known
+     restricted media items, or they refer to restricted media items that have
+     already been attached, the server responds with a 400 error with
+     `M_INVALID_PARAM`.


Should there also be a limitation that one can only attach media that they uploaded?

erm... good question.

I think the answer is "yes, there should", to guard against the possibility that someone gets hold of the mxc: URI between it being uploaded and the media being attached to an event. (In which case, they could attach it to their own event and get access to the media.) Particularly in the case of /create rather than /upload, in which the mxc: URI would appear in access logs.

richvdh · 2023-11-14T16:21:32Z

proposals/3911-linking-media-to-events.md

+   This "copy" api is to be used by clients when forwarding events with media
+   attachments.


We'd need to use this for m.sticker events too.

(and custom emoji, when that exists)

turt2live · 2023-11-14T16:27:10Z

proposals/3911-linking-media-to-events.md

+   --gc0p4Jq0M2Yt08jU534c0p
+   ```
+
+5. New "media copy" API


A downside of the media copy API is clients would no longer be able to reliably cache media. For the example of stickers or emoji, popular/busy rooms could have tens of references to the same media object, which causes tens of downloads because the client doesn't know different.

Possible alternatives (for future MSCs, imo) would be:

Allow some media objects to be reference counted/linked to multiple events.

Include a hash of the file in the event, so clients can locally realize that the objects are all the same content-wise.

I'm more preferential to the second.

turt2live · 2023-11-14T16:28:58Z

proposals/3911-linking-media-to-events.md

+   --gc0p4Jq0M2Yt08jU534c0p
+   ```
+
+5. New "media copy" API


going down the rabbit hole of custom emoji, there's an interaction with MSC4027 we will have to figure out.

turt2live · 2023-12-18T07:01:23Z

proposals/3911-linking-media-to-events.md

+
+Fixes [synapse#1263](https://github.com/matrix-org/synapse/issues/1263).
+
+## Potential issues


@AndrewRyanChama says:

How does this interact with m.replace? Will the client be expected to /copy all of the medias or will it be handled automagically?

AndrewRyanChama · 2023-12-23T20:11:21Z

proposals/3911-linking-media-to-events.md

+#### Redacting events
+
+Under this proposal, servers can determine which media is referenced by an
+event that is redacted, and add that media to a list to be cleaned up.


The problem with this is that if the user who sent the media leaves the room, the homeserver may no longer get the reaction events. That means there will be no mechanism for those events to be removed.

AndrewRyanChama · 2023-12-23T20:18:21Z

proposals/3911-linking-media-to-events.md

+   Content-Type: application/json
+
+   { "restrictions": {
+       "event_id": "$Rqnc-F-dvnEYJTyHq_iKxU2bZ1CI92-kuZq3a5lr5Zg"


This means that the association between (encrypted) media and event ids will be plainly accessible to the homeserver, and federated servers. I'd imagine that isn't something we actually care about but something that others may complain about.

Given that /copy presumably can't actually decrypt the media, that also means that the homeserver can associate any copied encrypted media as well

AndrewRyanChama · 2023-12-23T20:49:59Z

proposals/3911-linking-media-to-events.md

+
+After an item of media is uploaded, it must be linked to an event (via
+parameters to the `/send` api). A given piece of media is only visible
+to a user if the user can see the corresponding event.


Since reference counting is required anyways, I feel like it's better if the server does it automatically. I wrote a short doc on that here: #4086

MSC3911: Linking media to events

b1da78c

minor edits

58e6150

richvdh commented Oct 16, 2022

View reviewed changes

proposals/3911-linking-media-to-events.md Outdated Show resolved Hide resolved

ShadowJonathan reviewed Oct 17, 2022

View reviewed changes

proposals/3911-linking-media-to-events.md Show resolved Hide resolved

ShadowJonathan reviewed Oct 17, 2022

View reviewed changes

proposals/3911-linking-media-to-events.md Show resolved Hide resolved

deepbluev7 reviewed Oct 17, 2022

View reviewed changes

proposals/3911-linking-media-to-events.md Show resolved Hide resolved

uhoreg reviewed Oct 17, 2022

View reviewed changes

Half-Shot reviewed Oct 18, 2022

View reviewed changes

proposals/3911-linking-media-to-events.md Outdated Show resolved Hide resolved

richvdh commented Oct 18, 2022

View reviewed changes

proposals/3911-linking-media-to-events.md Outdated Show resolved Hide resolved

turt2live self-requested a review October 18, 2022 17:52

Munna231 suggested changes Oct 19, 2022

View reviewed changes

Move parts of this out to MSC3916

cc853ef

richvdh force-pushed the rav/proposal/linking_media_to_events branch from 464e5a4 to cc853ef Compare October 23, 2022 23:17

richvdh closed this Oct 23, 2022

richvdh reopened this Oct 23, 2022

add some clarifications

4a10395

cvwright mentioned this pull request Nov 10, 2022

MSC1767: Extensible event types & fallback in Matrix (v2) #1767

Merged

reivilibre reviewed Mar 10, 2023

View reviewed changes

spaetz mentioned this pull request Apr 18, 2023

MSC3860: Media Download Redirects #3860

Merged

richvdh commented Apr 25, 2023

View reviewed changes

turt2live mentioned this pull request Jul 17, 2023

Implement new media auth proposals (MSC3916 & MSC3911) t2bot/matrix-media-repo#421

Open

turt2live mentioned this pull request Sep 10, 2023

Media authentication & event linking (MSC3916, MSC3911) t2bot/matrix-media-repo#465

Draft

13 tasks

turt2live reviewed Sep 18, 2023

View reviewed changes

richvdh commented Nov 14, 2023

View reviewed changes

turt2live reviewed Nov 14, 2023

View reviewed changes

turt2live mentioned this pull request Nov 27, 2023

MSC discussion: Custom emoji/stickers and interactions with authenticated & linked media matrix-org/matrix-spec#1679

Open

This was referenced Dec 16, 2023

MSC2704: Explicitly allow alternative origins in MXC URIs and specify deduplication requirements on uploads #2704

Closed

MSC3468: MXC to Hashes #3468

Open

This comment was marked as duplicate.

Sign in to view

turt2live reviewed Dec 18, 2023

View reviewed changes

AndrewRyanChama reviewed Dec 23, 2023

View reviewed changes

This was referenced Jan 3, 2024

Delete media from the server when the event is redacted element-hq/element-meta#398

Open

There is no way to delete content from the media repo matrix-org/matrix-spec#226

Open

Auth for content repo (and enforcing GDPR erasure) #3796

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MSC3911: Linking media to events #3911

MSC3911: Linking media to events #3911

richvdh commented Oct 16, 2022 •

edited

uhoreg Oct 17, 2022

turt2live Mar 10, 2023

Munna231 left a comment

richvdh commented Oct 23, 2022 •

edited

deepbluev7 commented Oct 23, 2022 •

edited

richvdh commented Oct 23, 2022

reivilibre Mar 10, 2023 •

edited

turt2live Mar 10, 2023

rom4nik Jan 28, 2024

rom4nik Jan 28, 2024 •

edited

richvdh Apr 25, 2023

turt2live Sep 18, 2023 •

edited

richvdh Sep 19, 2023

turt2live Sep 18, 2023

turt2live Sep 18, 2023

richvdh Sep 19, 2023

turt2live Sep 18, 2023 •

edited

turt2live Sep 18, 2023

turt2live Sep 18, 2023

richvdh Sep 19, 2023

richvdh Nov 14, 2023

turt2live Nov 14, 2023

turt2live Nov 14, 2023

turt2live Nov 14, 2023

This comment was marked as duplicate.

turt2live Dec 18, 2023

AndrewRyanChama Dec 23, 2023 •

edited

AndrewRyanChama Dec 23, 2023 •

edited

AndrewRyanChama Dec 23, 2023


		### Overview

		After an item of media is uploaded, it must be linked to an event (via

		The new media item can be attached to a new event, and generally functions
		in every way the same as uploading a brand new media item.

		removed. Currently, Synapse does not support removing the events (see also
		[synapse#4720](https://github.com/matrix-org/synapse/issues/4720)); but if at

-TODO
+While this MSC is not considered stable, implementations should use the following mapped values.
+| Stable | Unstable |
+|-|-|
+| `/_matrix/client/v1/media/upload` | `/_matrix/client/unstable/org.matrix.msc3911/media/upload` |
+| `/_matrix/client/v1/media/create` | `/_matrix/client/unstable/org.matrix.msc3911/media/create` |
+| `attach_media` | `org.matrix.msc3911.attach_media` |
+| `restrictions` | `org.matrix.msc3911.restrictions` (`event_id` and `profile_user_id` are not prefixed) |
+| `/_matrix/client/v1/media/copy/:serverName/:mediaId` | `/_matrix/client/unstable/org.matrix.msc3911/media/copy/:serverName/:mediaId` |

		This "copy" api is to be used by clients when forwarding events with media
		attachments.


		Fixes [synapse#1263](https://github.com/matrix-org/synapse/issues/1263).

		## Potential issues

MSC3911: Linking media to events #3911

Are you sure you want to change the base?

MSC3911: Linking media to events #3911

Conversation

richvdh commented Oct 16, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Munna231 left a comment

Choose a reason for hiding this comment

richvdh commented Oct 23, 2022 • edited

deepbluev7 commented Oct 23, 2022 • edited

richvdh commented Oct 23, 2022

reivilibre Mar 10, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rom4nik Jan 28, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turt2live Sep 18, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turt2live Sep 18, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as duplicate.

Choose a reason for hiding this comment

AndrewRyanChama Dec 23, 2023 • edited

Choose a reason for hiding this comment

AndrewRyanChama Dec 23, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richvdh commented Oct 16, 2022 •

edited

richvdh commented Oct 23, 2022 •

edited

deepbluev7 commented Oct 23, 2022 •

edited

reivilibre Mar 10, 2023 •

edited

rom4nik Jan 28, 2024 •

edited

turt2live Sep 18, 2023 •

edited

turt2live Sep 18, 2023 •

edited

AndrewRyanChama Dec 23, 2023 •

edited

AndrewRyanChama Dec 23, 2023 •

edited