[proposal] Add Support for DIDCommv2 Mediation

[nickreynolds]

Background

The DIDComm v2 Specification was approved by DIF a few months ago. While Veramo already has support for much of this version of the spec, it currently lacks support for the "mediation" concept, which can be especially useful for bringing DIDComm to end-user agents, which may not be always online or expose http endpoints

This issue represents the work required to implement and demonstrate the DIDComm Messaging Routing Protocol as well as the Mediator Coordination Protocol.

Requirements

• - create a Veramo plugin that allows Veramo agents to act as DIDComm Mediators

• Plugin should be a MessageHandler that attempts to handle messages after they've been processed by DIDCommMessageHandler

• Plugin should expose some functionality to allow agent to "request mediation", update DIDDocument with Mediator info if mediation granted, and save info (on Mediator-side) required to eventually forward messages to requester (the Mediator Coordination Protocol)

• For this implementation, we can assume that the recipient will eventually send an http request to the mediator to check for messages

• - update didcomm plugin to support mediation via forward messages https://github.com/uport-project/veramo/blob/next/packages/did-comm/src/didcomm.ts#L536

• Must support any number of "forward message" wrappers required. Should demonstrate sending a DIDComm message that passes through at least 2 mediators.

• Likely requires changes to packDIDCommMessage and sendDIDCommMessage.

• - Demonstrate functionality in a browser-based Veramo agent using did:ethr to specify mediator, set up relationship with Mediator, and poll mediator (via http request) to receive messages.

• This can be added in a branch of https://github.com/veramolabs/agent-explorer/

Testing

Extensive testing will be needed to cover both the process of sending a message to a recipient via a mediator, as well as the process of setting up the mediation relationship.

• Agent 1 (DID1) is able to successfully "request mediation" from Mediator Agent, and update their DIDDocument to indicate routing DID
• Agent 2 is able to pack a message for DID1, wrap that message in a forward message, and send it to the Mediator Agent.
• Agent 1 is able to receive their held messages from Mediator Agent. These messages should be handled by Agent 1's DIDCommMessageHandler

As many failure cases as reasonable should also be added.

While not a strict limit, implementers should aim for a code-diff-coverage of ~75%

Constraints

• Minimal new dependencies: Packages in this repository are often installed together so dependencies must be reused between them when possible. No new dependencies should be introduced if this can be avoided.

Resources

https://didcomm.org/book/v2/routing
https://didcomm.org/mediator-coordination/2.0/

[codynhat]

• For this implementation, we can assume that the recipient will eventually send an http request to the mediator to check for messages

I don't see this protocol explicitly mentioned, but I'm assuming the Pickup protocol is what would be used here.

[codynhat]

Has anyone started work on this bounty? I am interested in taking it on, although I see several folks on the Gitcoin issue have also expressed interest.

[nickreynolds]

@codynhat I thought I had updated the description to mention the Pickup protocol, but yes you are correct that is what should be used for the recipient to pick up their messages.

If you were to apply on the Gitcoin bounty, I would approve you to work on it. None of the other applicants so far convinced me that they could do the work correctly/effectively, but since you've already done some work in our codebase (and in DIDComm specifically) I know you could do it.

[codynhat]

create a Veramo plugin that allows Veramo agents to act as DIDComm Mediators

@nickreynolds I have a question about this part. What is the best practice for a plugin that needs to store some state? Should the @veramo/data-store plugin be used and can it store arbitrary data? Or will the @veramo/data-store plugin need to be updated to know about the entities that this new mediation plugin uses?

Looking at DIDStore as an example. The @veramo/did-manager plugin defines AbstractDIDStore. Then the @veramo/data-store plugin implements DIDStore and defines ORM entities, etc. Is this a good pattern to follow?

[nickreynolds]

create a Veramo plugin that allows Veramo agents to act as DIDComm Mediators

@nickreynolds I have a question about this part. What is the best practice for a plugin that needs to store some state? Should the @veramo/data-store plugin be used and can it store arbitrary data? Or will the @veramo/data-store plugin need to be updated to know about the entities that this new mediation plugin uses?

@codynhat Can I ask what kind of new data entities you think you'll need to add? I was thinking that this could possibly be handled by just using the metadata on a Message

[codynhat]

Looking through the Mediator Coordination protocol, it looks like the mediator will need a way to store which DIDs it has granted mediation for, as well as a keylist for each of those DIDs.

For the Pickup protocol, the mediator needs to store a queue of messages that haven't been picked up and a live_delivery flag for each DID it is the mediator of.

I was thinking that this could possibly be handled by just using the metadata on a Message

All of this state can be inferred from messages, so I think it could all be done with metadata. Are there examples of querying metadata on Message?

[codynhat]

@nickreynolds Taking a closer look, it appears that message metadata cannot be queried in DataStoreORM https://veramo.io/docs/api/core.tmessagecolumns

Should a column be added for metadata?

[nickreynolds]

@codynhat

ok, so it seems like both issues could be addressed by extending existing entities in data-store (e.g. adding keyList, live_delivery fields to Identifier and pickedUp flag to Message), right? (although maybe it's slightly more complex than that).

My opinion is that those would likely be acceptable modifications to make in data-store, to avoid having to create a new data source, but I'd want to get @mirceanis or @simonas-notcat opinion. The Identifier entity could probably use a slight update anyways, as there's still no super obvious way of separating identities owned by this agent from those the agent simply knows about.

I think that since this is probably the easiest solution (compared to creating a new database), you could probably move forward this way and you wouldn't be wasting too much time even if we have to refactor later?

You're sure you could manage the queue without creating a new table, as long as you can modify Message, yes?

[nickreynolds]

Also, I was reviewing the Mediator-Coordinator spec while I was trying to answer you, and I'm actually a bit confused on the purpose of the keyList. Do you understand why it's used? Why not just use the keys specified in the recipient's DID Document?

[codynhat]

@nickreynolds

ok, so it seems like both issues could be addressed by extending existing entities in data-store (e.g. adding keyList, live_delivery fields to Identifier and pickedUp flag to Message), right? (although maybe it's slightly more complex than that).

I think this makes sense since most of the state is about an Identifier.

You're sure you could manage the queue without creating a new table, as long as you can modify Message, yes?

I don't think this can be done by just storing a forward message that is received by the mediator. The next field in the body would be the main parameter that is filtered on, which is not in the entity columns. Another possibility could be to save a new Message for each forward received, with the raw field as the raw JWE to send and some of the other fields (id, to, createdAt) saved to be able to produce a Status message.

Also, I was reviewing the Mediator-Coordinator spec while I was trying to answer you, and I'm actually a bit confused on the purpose of the keyList. Do you understand why it's used? Why not just use the keys specified in the recipient's DID Document?

From my understanding, the keylist is a way to store additional hops privately, only telling the mediator about the hops and not having them in the public DID Document.

[nickreynolds]

the keylist is a way to store additional hops privately

ah, I see. Tbh, I don't think this first implementation necessarily needs to support these optional extra hops. Similarly, I don't think we need to support live_delivery at this time (we don't yet have support for a transport that live delivery would require, anyways)

Another possibility could be to save a new Message for each forward received

I think this makes sense, assuming you mean the JWEs that are in attachments of the forward message. What if you saved a new message for each combination of keyAgreement keys in recipient's DIDDoc and JWE in attachments? Then a recipient could pickup once for each encryption key they own? So they could download the message to all their devices? Maybe this is overkill or I'm missing something though.

Either way, I think it's fine for you to add anything to entities (or create new entities) that you need to support these protocols. These are basically core DIDComm protocols, so it makes sense to include them in our core data model.

[codynhat]

Sounds good. And yea, it makes sense to save one for each key. The Pickup protocol takes a recipient_key as an input to allow a user to get all the messages for a specific key they have, like a specific device.