MSC3819: Allowing widgets to send/receive to-device messages

[turt2live]

Rendered

Implementations

• Implement MSC3819: Allowing widgets to send/receive to-device messages matrix-widget-api#57
• Indicate whether to-device events are / should be encrypted matrix-widget-api#63
• Implement MSC3819: Allowing widgets to send/receive to-device messages matrix-react-sdk#8885
• feat(widget): Send custom to-device messages in widgets in e2ee rooms matrix-rust-sdk#5252

Shepherd: @toger5

[robintown]

We need to include the event type somewhere in here. I'd suggest an interface of

{
  type: string;
  messages: { [userId: string]: { [deviceId: string]: unknown } };
}

[robintown]

We also need to control whether the event is encrypted or not

{
  type: string;
  encrypted: boolean;
  messages: { [userId: string]: { [deviceId: string]: unknown } };
}

[Fox32]

I noticed an issue with the current implementation. The payload (unknown above) varies between encrypted: true and encrypted: false in Element Web right now. For unencrypted messages it is directly the content of the message, like {"key": "value"}, but for encrypted events it has to be wrapped like {"type": "event type…", "content": {"key": "value"} }. If that is really desired and not just a mistake, we should make sure to document the behavior.

I created an issue for it: element-hq/element-web#24470

[BillCarsonFr]

I am not sure about this API. Currently if you want to send a message to several devices, the content will be repeated n times as part of the map<user_id, map<device_id, content>

maybe something like

{
  type: string;
  encrypted: boolean;
  recipient_devices: { [userId: string]: [deviceId: string] };
  content: any
}

[Fox32]

While working with the current support of MSC3819 in Element we noticed a problem:

The send endpoint of the toDevice message API allows to target specific device
ids, but there is currently no way to know a specific device id for a widget.
Neither is it possible to embed a device id into the URL,
nor does a received toDevice message indicate the sender device to reply to it.
Sure, it is possible to target all devices via *, but using a specific device
id makes sending much more efficient.

Element Call, which this MSC initially was created for, is cheating a little bit.
As Element Call is a special widget anyway, it is created with
custom code instead of the normal widget API setup code.
This allows to embed additional parameters into the widget URL, including the device id.
From the perspective of bringing the product to the market quickly, I think it
is fine, but we should focus on getting this MSC into a complete state.

Therefore I suggest to amend small section here:

**Passing device id to widgets**

A new template variable is added to the available options for a widget URL:

* `device_id` - The device id of the client instance which is rendering the widget.

And extend the unstable prefix section with:

* `org.matrix.msc3819.device_id` in place of the proposed variable.

If this extension is desired, I'm open to add the implementation to the matrix-react-sdk and matrix-widget-api.

[Fox32]

I started working on implementing this addition over at matrix-org/matrix-widget-api#78

[dhenneke]

In matrix-org/matrix-widget-api#78, we decided to slightly rename the parameter. Thus the amendment should be as follows:

**Passing device id to widgets**

A new template variable is added to the available options for a widget URL:

* `matrix_device_id` - The device id of the client instance which is rendering the widget.

And extend the unstable prefix section with:

* `org.matrix.msc3819.matrix_device_id` in place of the proposed variable.

[BillCarsonFr]

This sounds like a bad idea? If a clear text event is sent with an encrypted field in the event then it could be miss-interpreted has beeing a decrypted plain text message

[toger5]

I am just noticing, that this is still outside of the content is there a way for the sender to modify this part of the json?

The homeserver however could right?

The spec could either enforce a widget driver to overwrite this. (but custom events that want to use a field named like this cannot exist anymore then. We would basically need to enforce in the event spec that encrypted is a reserved field.)
So naming it sth like m.encrypted would make more sense if we keep the structure as is

Another solution one can imagine is changing the format to:

"data":{
  "content" : {...what_was_previous_stored_in_data}
  "room_id": string
  "encrypted": boolean
}

So that the content from the HS is never altered and that the fields the widget driver adds can never be injected by the sender.
This would very much a breaking change.

Last we could put this info next to data. But that would be odd since it would then be on the same level of:
action,response,api... which are clearly specified widget api fields.
Maybe introducing another toplevel property called request_info or data_meta or just widget_info that is exclusive to the widget and since it is on the top level only the widget driver can define it. It would then include all the data that gets added by the widget driver and is always the trusted source.

[BillCarsonFr]

At this point I thing I am against letting the widget decide to encrypt when sending, and also letting the widget if it should accept a clear to-device message.

My proposal would be to remove this encrypted field.

And when sending a to-device:

• If it is a room widget and the room is encrypted then encrypt the to-device, if the room is not encrypted then do not encrypt.
• If it is a "global" widget, then always encrypt.

On receiving a to-device:

• If it is a room widget: If the room is encrypted only forward encrypted to-device (sucessfully decrypted). This should respect the room encryption settings; That is, if the room only allow verified devices, then to-device sent by un-verified devices should not be forwarded. It should also respect the exclusion of insecure device as per the user setting
• For global widget, use the default user preferences for insecure devices

[toger5]

The only feature we would loose is unencrypted to-device in encrypted rooms.
This has quiet a little use-case. The only reason i could find for this is if to-device is used as an ephemeral message to a lot of participants. Then encryption is expensive bandwidth and compute wise.
But this is somewhat artificial since it very much sounds like the exact scenario room events were designed for.
But in all other scenarios encrypted to-device just make much more sense (compared to unencrypted).
I will draft a change and see how this feels.

[andybalaam]

Note that if a widget were ever wanting to verify devices or users, it is possible they would want or need to send unencrypted messages:

Alternatively, verification messages may be sent unencrypted, though this is not encouraged.

https://spec.matrix.org/v1.14/client-server-api/#key-verification-framework

This is quite probably not something we want or need to support here.

[BillCarsonFr]

I think we need to be a lot more restricted.
Widgets should only have access to a sub-set of the to-device traffic, internal/crypto to-device traffic should be excluded by default (m.room.key, m.secret.send, m.verification ...)
And even more there should be a way to scope the to-device traffic to the room.

My proposal would be to add a new scope property in the event content:

{
        "sender": "@alice:example.com",
        "type": "m.custom",
        "content": {
            "scope": {
                "room_id": "!726s6s6q:example.com",
                "id": "org.matrix.msc3401.call"
            }
           "example_content": "put your real message here"
        }
}

This would allow for the widget to only get traffic scoped to the room it sits in. (Global widget would get all traffic -excluding reserved crypto types-?). To device content with no scope will not be passed to widgets.

On the sending side the scope would be added automatically by the widget driver.