Feature request: Constant-size hash for welcome message proto

[insipx]

Is your feature request related to a problem?

With the current send_welcomes function and associated proto, we quickly reach the gRPC limit which limits the amount of members we can invite at once without chunking first. Even with chunking, very large groups would hit this limit too.

Increasing the gRPC limit is only a temporary solution, and not great anyway since sending large amounts of data over the wire isn't a great thing (this is in the range of 50-100MB a message).

Making this constant size would be a win for performance and scalability of groups.

Describe the solution to the problem

@richardhuaaa had great suggestions for a fix, namely Rich pointed out that these messages are so large because a full copy of the ratchet tree is included in each welcome message. For 800 members this comes out to 436092 bytes.

Rich suggested to store the ratchet tree somewhere else, and only include the hash in the welcome. That would make the welcome message size constant and we would not have to worry about hitting the gRPC limits until much larger group sizes, at which point batching would come into play again.

• there's an MLS configuration to use for this
• Info on storing ratchet tree
• possibly encrypt with something derived from joiner_secret

@franziskuskiefer would love your input on this!

Describe the uses cases for the feature

• adding lots of members to group

Additional details

Here's one of the original error messages:

Benchmarking add_to_empty_group/500: Warming up for 3.0000 sAdding 500 members
2024-05-31T02:23:04.084639Z ERROR xmtp_mls::groups::sync: post commit error Api(xmtp::error::Error(MlsError, Status { code: ResourceExhausted, message: "grpc: received message larger than max (136352000 vs. 52428800)", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None }))
2024-05-31T02:23:04.084662Z ERROR xmtp_mls::groups::sync: error syncing group Sync([Api(xmtp::error::Error(MlsError, Status { code: ResourceExhausted, message: "grpc: received message larger than max (136352000 vs. 52428800)", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None }))])
2024-05-31T02:23:05.355537Z ERROR xmtp_mls::groups::sync: post commit error Api(xmtp::error::Error(MlsError, Status { code: ResourceExhausted, message: "grpc: received message larger than max (136352000 vs. 52428800)", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None }))
2024-05-31T02:23:05.355565Z ERROR xmtp_mls::groups::sync: error syncing group Sync([Api(xmtp::error::Error(MlsError, Status { code: ResourceExhausted, message: "grpc: received message larger than max (136352000 vs. 52428800)", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None }))])
2024-05-31T02:23:06.580907Z ERROR xmtp_mls::groups::sync: post commit error Api(xmtp::error::Error(MlsError, Status { code: ResourceExhausted, message: "grpc: received message larger than max (136352000 vs. 52428800)", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None }))
2024-05-31T02:23:06.580934Z ERROR xmtp_mls::groups::sync: error syncing group Sync([Api(xmtp::error::Error(MlsError, Status { code: ResourceExhausted, message: "grpc: received message larger than max (136352000 vs. 52428800)", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None }))])
thread 'main' panicked at xmtp_mls/benches/group_limit_empty.rs:59:65:
called `Result::unwrap()` on an `Err` value: Sync([Api(xmtp::error::Error(MlsError, Status { code: ResourceExhausted, message: "grpc: received message larger than max (136352000 vs. 52428800)", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None }))])
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[franziskuskiefer]

Not sending the ratchet tree as part of the Welcome is definitely the way to go. I think having the ratchet tree in the Welcome really only works for small groups and allows for a simpler architecture. But as you see here, the trees, and therefore the Welcome messages get big pretty quickly.

The general design for the flow with Welcome messages without the ratchet tree extension is as follows

• The inviter generate the Welcome and Commit messages, and the ratchet tree. In this case, there will be no group info object in the return value of add_members
• The inviter exports the ratchet tree.
• The inviter sends the Welcome and Commit messages to the Deliver Service, and uploads the Ratchet Tree.
• The delivery service sends out the Welcome and Commit messages.
• An invitee, after receiving a Welcome message, downloads the ratchet tree, and creates a new group with new_from_welcome, providing the separately downloaded ratchet_tree.

The open question is how the invitee gets the ratchet tree. This is not defined in MLS and up to the implementation to define. There are different design that I can see

• The delivery service has an API that can be queried with the Group ID and returns the ratchet tree for that group.
• The Welcome contains a GroupInfo with a custom extension that defines how to download the ratchet tree, for example with a digest of the tree that can then be queried at some known endpoint.
• The delivery service sends information along side the Welcome message on how to retrieve the tree for the welcome.

If you consider the tree sensitive information, or don't have another way to protect the API endpoint to download the ratchet tree, encrypting it with a key that is derived from the joiner_secret is a good idea.

Note that the RFC mandates that "the client MUST download the tree before processing the Welcome". This is not entirely possible in practice though, because the client has to retrieve some information about the group in order to download the ratchet tree. This should be read as: process only the necessary parts of the Welcome before downloading the ratchet tree.

Looking at the OpenMLS API I actually think we need to expose an new API to make any option work where the information about the ratchet tree is inside the Welcome message. Right now, OpenMLS is strict on this "MUST" and there's no way to get information from the Welcome message without a ratchet tree. I'll think about potential solutions for this.
What is your preferred solution for retrieving the ratchet tree?

[franziskuskiefer]

I opened openmls/openmls#1586 with an API. Let me know if you think that would work for you here.

[insipx]

Thanks for the super fast response! Plan to dive into this head first tomorrow after finishing up the benchmarks