Initial formal proposal for the AS API

[ara4n]

I thought more about the requirements for application services over Christmas, and here's a slightly high-level but fairly concrete proposal of what I think they should look like. This largely ignores policy servers and the Storage API idea, in favour of building on our current client-side gateway bots by giving them privileges and making them run as proper server-side services.

This hopefully decouples the AS API problem entirely from the CS v2 API discussion.

https://github.com/matrix-org/matrix-doc/blob/application-services/drafts/application_services.rst

[Kegsay]

Overall this feels well thought out and seems like a great start. Few notes:

If the HS receives an invite to an unknown room which is in the namespace delegated to the AS

and

Events in the namespaces of rooms

Rooms are not currently namespaced at all, how do you propose namespacing them? If you mean room
aliases then this should be made clear throughout. This is also a rather novel use case for room
aliases if so; is the current room directory design sufficient for this use case?

TODO: meaning the CS API needs to support massaged timestamps

The C-S API doesn't rely on timestamps for operational matters. origin_server_ts is purely
informational, so I'm not sure exactly what extra work needs to be done on this. Signing may
be fun however...

or the HS must delegate conversation storage entirely to the AS using a Storage API (not defined here) which allows the existing conversation store to back the HS, complete with all necessary Matrix metadata (e.g. hashes, signatures, federation DAG, etc)

This seems to be a direct contradiction of one of the key selling points of AS:
developers can focus entirely on implementing application logic rather than being concerned with the details of managing Matrix federation.
It would be preferable if say, the content only was accessible via the Storage API rather than counting on random ASes doing signatures
correctly. Of course, you may already be suggesting this, but it doesn't feel clear which side you're coming down on here.

The sending user ID must be explicitly specified, as it cannot be inferred from the access_token, which will be the same for all AS requests.
TODO: or do we maintain a separate access_token mapping? It seems like unnecessary overhead for the AS developer; easier to just use a single privileged access_token and just track which userid is emitting events?

The justification for using C-S API is so that ASes can reuse existing CS SDKs. The
C-S API enforces the use of an access_token so I don't know how you would reuse them
without access tokens.

[ara4n]

Q: Rooms are not currently namespaced at all, how do you propose namespacing them? If you mean room
aliases then this should be made clear throughout. This is also a rather novel use case for room
aliases if so; is the current room directory design sufficient for this use case?

A: I left it open for discussion, but the options i had in mind were registering lists of virtual room aliases (#matrix:matrix.org, #matrix-dev:matrix.org), regexp of aliases (#matrix.:matrix.org), or a whole vhost (.:irc.matrix.org) to the AS (for the example of an IRC gateway). You would also need to support subscribing to room IDs as well as aliases to intercept events for non-aliased rooms (but obviously you wouldn't use these to refer to named rooms on the AS)

TODO: meaning the CS API needs to support massaged timestamps
Q. The C-S API doesn't rely on timestamps for operational matters. origin_server_ts is purely
informational, so I'm not sure exactly what extra work needs to be done on this. Signing may
be fun however...

A. I'm simply asking that we let ASes override origin_server_ts in their sent events (or alternatively we ask all clients to be aware of a new origin_as_ts field, but that wouldn't be backwards compatible and feels unwieldy). It doesn't impact signing.

Q. "or the HS must delegate conversation storage entirely to the AS using a Storage API (not defined here) which allows the existing conversation store to back the HS, complete with all necessary Matrix metadata (e.g. hashes, signatures, federation DAG, etc)"

This seems to be a direct contradiction of one of the key selling points of AS:
developers can focus entirely on implementing application logic rather than being concerned with the details of managing Matrix federation.
It would be preferable if say, the content only was accessible via the Storage API rather than counting on random ASes doing signatures
correctly. Of course, you may already be suggesting this, but it doesn't feel clear which side you're coming down on here.

A. I think we are crosswired. I deliberately give two models of handling history provided by the AS - either replicating it into the HS when the "virtual" room is created (thus avoiding any confusion for the AS implementer, but limiting the amount of history available)... or alternatively go the whole hog and replace sqlite with their existing convo store (unspecced, scary, but necessary if you don't want to end up with two convo stores). So i don't think this is a contradiction; i will try to fix the wording assuming we are aligned.

Q. "The sending user ID must be explicitly specified, as it cannot be inferred from the access_token, which will be the same for all AS requests."
TODO: or do we maintain a separate access_token mapping? It seems like unnecessary overhead for the AS developer; easier to just use a single privileged access_token and just track which userid is emitting events?"

The justification for using C-S API is so that ASes can reuse existing CS SDKs. The
CS API enforces the use of an access_token so I don't know how you would reuse them
without access tokens.

A. I am suggesting extending the CS API slightly so that AS's can say access_token=secret-as-master-key&userid=@matthew:matrix.org in order to inject events on behalf of @matthew:matrix.org. Thus the AS doesn't have to spend its life doing login dances and maintaining userid->accesstoken mappings as we have currently in the various clientside bots and bridges. Obviously this gives ASes god privileges (modulo e2e crypto protection), but this is deliberate - ASes do useful things on behalf of the users in that HS, and are trusted. Unless of course we want users to have to opt-in somehow to them....

[ara4n]

whilst i think of it: this needs to spell out how you handle bridging non-virtual users into the AS domain - eg having @matthew:matrix.org send stuff to IRC via Arathorn rather than m-Arathorn... or in a Lync gateway, how to get traffic to route back and forth correctly to the real @matthew:matrix.org user.

Also need to spell out what happens with group chats which are anchored in the AS domain. We worked through this all with the XMPP gateway use case; just need to incorporate it correctly here.

[NegativeMjark]

• A URL base for receiving requests from the HS (as the AS is a server,
  implementers expect to receive data via inbound requests rather than
  long-poll outbound requests)

• Does the AS expect a linearised event stream (with superseded state events omitted) or would the HTTP URL be hit for every event passing through the HS?
• Do we need a separate URL to hit for modify messages originating at the HS?
• Does it matter if an AS goes down? Do we expect the HS to retransmit the events?

[ara4n]

Can you think of a use case where the AS should do linearisation itself? If not, i think the HS should always do it.

What sort of message modifications are you thinking of that the CS API doesn't already provide?

If the AS goes down without unregistering we'd want the HS to queue and retransmit events.

[NegativeMjark]

What sort of message modifications are you thinking of that the CS API doesn't already provide?

• Censoring/modifying outgoing messages sent by other clients.

If the AS goes down without unregistering we'd want the HS to queue and retransmit events

• The current pull based C+S API handles this without any need for HSes to queue pokes. Is it worth implementing a push based API in the HS with associated queuing?
• How would this work with a clustered HS? Would a single node be nominated as handling the pokes to the given AS?
• Should we support a clustered AS? How would linearisation work in this case?

[Kegsay]

Unless of course we want users to have to opt-in somehow to them....

This is exactly what OAuth2 should be used for. Without that, any AS wields far too much power if they control all the users on the HS. It would be laughable to expect someone like Google to let an XMPP bridge have control over ALL of their users without an opt-in method like OAuth2.

[ara4n]

say i implement an AS that acts as a content indexer and search engine (eg with elastic search) (oh! look! another use case). It needs to subscribe to all rooms to sniff their events. Would we really want users to have to explicitly opt into this? Perhaps there's a difference between submitting traffic on behalf of a real user, and just spying passively on traffic...

[Kegsay]

Indeed there is a difference between passively observing (which hey, if you don't want that, use a different HS or use E2E) and actively masquerading as a user.

[erikjohnston]

How does this all work in the presence of end to end encryption? To send a message as a user would the AS need their private keys? How can the AS sniff the content of events?

[ara4n]

for e2e you either invite the AS into the room and so they get the room key (or however we do it) or all bets are off. masquerading is certainly out of the question

[erikjohnston]

This looks mostly good to me.

I would be interested in the reliability mechanisms of the HTTP push transport, as the I think those will have a bearing on whether we want to support clustering of AS servers or not.

Do we want to expose the HTTP push transport for use by clients?

[ara4n]

Erik: the HTTP push transport would be usable by clients if the HS implemented it (as AS API extensions are not compulsory for an HS). But it'd make little sense to do so.

[ara4n]

I've updated the draft with all the feedback above. The structure of the doc is a bit weird, and the use cases haven't been fully fleshed out due to lack of time, but all the info should be there.

[Kegsay]

is the only option if the implementer wants to avoid duplicating conversation history between the external data source and the HS.

Can we not provide a trade-off and just allow them to implement the event content? The HS would ask for the content which it would then sign/continue as normal.

Namespaces are defined as a list of regexps against which to match room aliases, room IDs, and user IDs.

Use case for regexpping room IDs? They are opaque strings. By all means we may be selecting some existing room IDs to bridge with, but that is a far cry from the regex for room aliases, user IDs, etc. This should be made clearer.

[ara4n]

On 14/01/2015 15:17, Kegsay wrote:

is the only option if the implementer wants to avoid duplicating
conversation history between the external data source and the HS.

Can we not provide a trade-off and just allow them to implement the
event content? The HS would ask for the content which it would then
sign/continue as normal.

Where would the HS store the signatures and all the matrix-specific
metadata? It feels very unwieldy for the HS to store that in its own
DB, but separately from the actual underlying conversation data. Hence
suggesting that in that scenario you push the responsibility of storing
all Matrix's data to your existing conversation store.

Namespaces are defined as a list of regexps against which to match
room aliases, room IDs, and user IDs.

Use case for regexpping room IDs?

Both !.*:matrix.org (for a vhost), and to support aliases as well as IDs.

[Kegsay]

The room ID domains are purely for namespacing the opaque ID; we shouldn't be regexpping that at all.

Where would the HS store the signatures and all the matrix-specific metadata?

This just seems like an HS implementation problem. If the HS caches signatures (and given events are immutatatatable, why wouldn't it?) then this problem disappears.

It feels very unwieldy for the HS to store that in its own DB, but separately from the actual underlying conversation data.

I don't think that is as unwieldly as you are making it out to be. Swap a json blob in a database with a URL to poke for the json blob?

[Kegsay]

The namespace of virtual user accounts should conform to a structure like @.irc.freenode.Arathorn:matrix.org. This lets Matrix users communicate with foreign users who are not yet mapped into Matrix via 3PID mappings or through an existing non-virtual Matrix user by trying to talk to them via a gateway.

I don't think we should do this. If the user doesn't exist on Matrix then we should be 404ing/failing, not silently passing it through in the hopes that One Day they will become mapped. This also fails to account for application service namespacing (e.g. you have two very popular IRC ASes, with two different namespaces for users, which do you attempt to talk to?).

See my HTTP proposal for info on this.

[ara4n]

As per real-life discussion, I strongly think that 3rd party users should be mapped to a unique user-id in Matrix - i.e. Arathorn on irc.freenode.net should always be referred to as @/irc/irc.freenode.net/Arathorn:matrix.org or whatever the convention is. This decouples the virtual user's identity from the AS which is performing the bridging. And you should expect problems if you try to run two ASes against the HS bridging the same chunk of 3rd party namespace into Matrix.

Meanwhile, the actual suggested behaviour is that if I'm on Matrix and want to talk to a 3PID like Arathorn-on-irc.freenode.net, i start off by checking for a 3PID mapping in my identity server. If the ID server has a mapping from Arathorn-on-irc.freenode.net to @arathorn:matrix.org, then we go and use that to talk to them. Otherwise, we can choose to go via a bridge (typically our local HS's AS, but the client could be configured to use another one like matrix.freenode.net, resulting in a virtual user ID of @/irc/irc.freenode.net/Arathorn:matrix.freenode.net) - and we'll end up talking to them via the gateway.

[Kegsay]

As per real-life discussion, I strongly think that 3rd party users should be mapped to a unique user-id in Matrix - i.e. Arathorn on irc.freenode.net should always be referred to as @/irc/irc.freenode.net/Arathorn:matrix.org or whatever the convention is.

Adding a bit more justification for this:

• The goal is to allow the client to attempt to communicate with anyone on any service outside Matrix.
• The AS provides the routing to the Outside World and is specific to a HS.

If you namespace ASes user IDs:

• The client would need to query the HS for a list of ASes running which can communicate X protocol. Multiple ASes may be returned.
• Which AS does the client choose for starters?
• The client will need to map the user's intention (I want to speak to someone on IRC freenode) to some string they get back from the HS (which may even look like .irc.freenode.)
• The client then needs to craft the user ID and send a message.

If you don't namespace ASes user IDs:

• The client can work out the user ID by convention, without needing to hit the HS (e.g. They are URIs, replace / with . or something)
• Only 1 AS should ever be running for a particular namespace of user IDs, and it's the HS server admin's fault if there are conflicts around this. (Note that this isn't always a negative thing, e.g. one AS may provide logging facilities for IRC rather than routing, and so will not conflict with the other AS doing the routing)

We may still want to consider a layer of indirection (e.g. "here is a random URI, please give user ID") rather than regexpping the URI to form an ugly looking user ID, or we may want slightly nicer conversion rules (this will depend on #3 )

[Kegsay]

If the HS receives an invite to an unknown room which is in the namespace delegated to the AS, then the HS queries the AS for the existence of that room

When would this ever happen? If the non-matrix user sends a message, the AS will make a room and invite the matrix user to it. If the matrix user sends a message,
the room must already exist. Can you give an example when this would be required / expected behaviour? I can't design the HTTP API until I know more about how this is supposed to work.

Also, can you justify in the general API doc why the lazy-loading of user IDs is absolutely required? (e.g. so you don't need to create every MSISDN evar). It's briefly touched on but not explained why it is important.

[ara4n]

This is describing the HS receiving an invite (or join) from matrix for a virtual room which needs to be lazy-created by the AS.

So: if your HS is hooked up to an IRC bridge AS which has registered control of the #/irc/irc.freenode.net/ namespace, then if the HS receives an invite/join from a Matrix user on that HS to participate in #/irc/irc.freenode.net/randomchannel, it's possible the HS knows nothing about this room yet, as it's never been accessed before. In this scenario, the HS sees that namespace is delegated to the AS, so queries the AS as to whether that room should exist and be lazy-created. The AS checks if that channel exists on the IRC server, and if so, the AS then goes and creates the room on the HS via the AS/CS API - creating the room with the right alias, name, permissions, history, state, etc and inserting the appropriate virtual users to represent the ones in the IRC channel. Once this is done, the HS continues on with the normal interaction with the new lazy-created room.

The reason why we need lazy-loading of user IDs is because another AS might be PSTN-gatewaying the entire MSISDN space - e.g. supporting @.tel.447700900123:whoever.com style user IDs. The space of possible valid MSISDNs is obviously huge: 10^13 or so. So when you start a conversation with one of these virtual users, they must be lazily provisioned by the AS. (For instance, the AS could reject the invite if it can't route to the given MSISDN. Alternatively, it provisions an account on the HS for that user's name, with appropriate profile info based on what it knows about that MSISDN and metadata about how it's bridging it). It's insane to think that the AS would preprovision accounts for all possible MSISDNs on the HS that it could gateway through to.

[NegativeMjark]

Would it be helpful for the HS to include display name info for users and rooms when pushing events to the AS? It would save the AS from having to track what the current display name is for a user.

[ara4n]

We have an open problem in general on how to specify 3PID originators in
Matrix.

If I've logged into my Matrix app with my MSISDN, and send a message to
a hypothetical Matrix->Email gateway as
@.mailto.someone@matrix.org:mailgateway.com (or whatever the URI is):
what should the From and Reply-To header in the email be? Should it be:
"Display Name" arbitrary-uid@mailgateway.com? Or should it be
matthew:matrix.org@mailgateway.com? Or should it be
MSISDN@mailgateway.com?

In general we don't want to leak Matrix IDs if users are identifying
themselves (i.e. logged in as) with a 3PID. Therefore we need a way to
pass both originator 3PID and Display Names through to the foreign
network (assuming the foreign network supports the concepts of display
names or displaying 3PIDs).

[ara4n]

Also, these virtual user IDs are getting uglier and uglier. Is there
any way we can mux the actual remote URI into the name of the virtual
user? E.g. allocate virtual user IDs as a SHA-1 style hash of the URI?

so: the virtual user id for the foreign matthew@arasphere.net user on a
hypothetical Matrix<->Email AS gateway would be
@{SHA-1('mailto:matthew@arasphere.net'}:emailgateway.com - or
@bec6ec1b384538d78873214cf8cdff4cc4415288:emailgateway.com? (The
displayname and email-address profile field of this virtual user would
be set to "matthew@arasphere.net", and/or the displayname could be set
to the GECOS learned from any inbound mails from this foreign user.)

[Kegsay]

allocate virtual user IDs as a SHA-1 style hash of the URI?

I like this, but what namespace does the AS claim then? An alternative would be to do as I suggested a few comments up:

We may still want to consider a layer of indirection (e.g. "here is a random URI, please give user ID") rather than regexpping the URI to form an ugly looking user ID

though I prefer your hash idea since it avoids adding another API and another round trip.

[Kegsay]

Some application services may want exclusive rights on the namespaces they've claimed in the regex, e.g. to stop humans creating a room alias / user in the AS namespace. Other application services may not want exclusive rights but just want to sniff around (e.g. logging ASes).

This concept isn't covered in the general API.

[Kegsay]

Sometimes application services need to create rooms (e.g. when lazy loading from room aliases). Created rooms need to have a user that created them, so federation works (as it relies on an entry existing in m.room.member). We should be able to add metadata to m.room.member to state that this user is an application service, a virtual user, etc.