RFC: XMTP Network Guarantees and Technical Architecture

[shash256]

The vision of the XMTP network is to provide secure, private, and portable communication as a public good. Current messaging platforms are centralized, which presents numerous pain points for both users and developers. These challenges include the absence of identity and conversation self-sovereignty, issues of censorship, spam, lack of interoperability, and the risk of de-platforming for developers. The XMTP network aims to address these challenges by enabling a modern communication experience on decentralized infrastructure that provides the added benefits of interoperability, censorship resistance, and self-sovereign messaging and identity.

Introduction

The purpose of this document is to provide a brief overview of the messaging protocol, outline the guarantees and key properties required of the network, and put forth an initial proposal for the technical architecture of the network.

Messaging Protocol

The messaging protocol defines the rules and procedures for exchanging messages on the XMTP network. It specifies the mechanisms for the private and encrypted transmission of messages. It also specifies the format and structure of the messages in the form of topics and conversations.

Note: The current end-to-end encryption (E2EE) scheme may be updated to the Signal Protocol and the Message Layer Security (MLS) protocol in the future. This section refers to the current version of the messaging protocol.

Topics and Conversations

Topics are a core concept of the XMTP network. They provide the messaging protocol with a mechanism to associate a log or stream of messages intended for specific users and their conversations. While topics at the network-level are a general purpose primitive, the messaging protocol makes use of the following types:

• Conversation invites: Public-writeable topics, owned by individual identities on the network that are used to publish invitations or requests for conversations with those individuals. These topics may be configured such that crypto economics regulate conversation negotiation between two parties on the network. For instance, senders may be required to include a refundable fee for reaching a certain user’s invite topic (i.e., spam control).
• Conversations: Participant-writeable topics where users can publish encrypted messages intended for decryption only by other users in their conversation. These topics are specific to individual conversations, where currently a single conversation is represented by a single topic, but may be expanded in the future such that a conversation is composed of multiple topics over time.
• User Data Storage: User-writeable topics owned by individual identities on the network, where only they can publish encrypted messages representing user metadata and other application data such as their device preferences, allowlists, and blocklists.

Privacy and Encryption

Messages published to topics are publicly viewable, and so should always be end-to-end encrypted by clients before reaching the network. The unlinkability property ensures that an observer can view the topic identifier and the encrypted content of a message, but cannot know participant identity. Unlinkability is maintained for all network interactions, except conversation invite topics and identity registration. The owner intentionally advertises their identity to invite topics mapping so that others can reach them for sending conversation requests. However, these conversation requests are encrypted and observers are unable to derive the resulting negotiated conversation topics.

XMTP Network

There have been ~450 projects built using the XMTP message protocol, and users are actively messaging on some of these platforms. The protocol infrastructure is currently permissioned, and maintained solely by XMTP Labs. This document proposes a gradual transition to a permissionless infrastructure model, and defines the XMTP network core developers as an unbounded set of individual contributors carrying out this pivotal work.

XMTP Network Guarantees and Key Properties

As discussed previously, permissionless networks provide novel features, such as interoperability and self-sovereignty. Designing a sound and resilient permissionless network requires self-sustainable revenue sources, thoughtful mechanism design, and selecting between tradeoffs and competing incentives.

The primary responsibility of the XMTP network is to support the continuous operation of the message service. For this, the network must guarantee a set of fundamental properties.

Security and availability

The network must be available to send and receive messages without censorship failures, and message order should be preserved across the network.

Self-sovereign identity

An important feature provided by open and immutable blockchains, like Ethereum, is self-sovereign identity and data. Self-sovereign in this case refers to a user’s ability to control their identity and messaging data. Self-sovereignty in blockchains exists because the database lacks permissions, which allows the creation of an unlimited set of interfaces for users to access their data, and unique identity is available through public key infrastructure verified by the protocol.

Privacy

Privacy is a unique challenge in open blockchains. Although a user’s real world identity is unknown, a social graph made up of interactions and transactions is publicly associated to a user’s blockchain address. The XMTP network is built on strong encryption standards, and unlinkability feature designs are emphasized to protect identities. Still, privacy remains a spectrum that the network must continually strive to improve upon.

Credibly neutral and decentralized

The rules of the protocol must be neutral and transparent, promoting inclusivity and decentralized governance.

Decentralization is vital to ensure the network's resilience against attacks, with multiple independent node implementations and redundancy to withstand outages. The system should strive for a low barrier to participation, allowing nodes to run on various hardware and software platforms.

Network Architecture

The decentralized infrastructure supporting the aforementioned network guarantees and key properties consists of several pillars, including the Identity Registry, Conversation Data Layer, Consensus & Incentives, and an Accounting Mechanism for tracking resource usage.

The following technical architecture proposal is put forth to the community as a starting place for discussion. The core team will publish technical architecture proposals to the community discussion forum within each pillar. This provides the community a place to contribute to the discussion, or to propose alternate designs.

Proposal

The XMTP network operates as a Byzantine Fault Tolerant network, relying on Delegated Proof of Stake (DPOS) for network security.

Key features of the architecture include:

• An ERC 20 token that coordinates infrastructure providers, message consent rules, and certain aspects of protocol governance.
• Consensus based on DPOS, with inflationary rewards incentivizing consensus and delegation participation. A portion of network fees is burned to maintain token supply.
• Node operation costs covered by messaging fees, tracked through an accounting mechanism. A subsidized free tier ensures seamless and reasonable messaging usage.
• The system comprises an interconnected network of XMTP nodes for replication and smart contracts for coordination and staking. Consensus extends across both smart contracts and the network.

Identity Registry

The concept of wallet-to-wallet communications is core to the XMTP messaging protocol, where users send messages to other users on the network by referencing external blockchain wallet addresses, such as their Ethereum or Polygon address. In order to achieve this, users must first register their identity on the XMTP network by associating their public-discoverable wallet address to their XMTP network identity and a public conversation invite topic where anybody can publish encrypted messages in order to request a conversation. This registration process is facilitated by a Smart Contract Registry, where identities and their public conversation topic identifiers along with any configurations, are stored such that others can discover and retrieve the contact information.

Configurations establish which identities are authorized to publish to the topic, as well as the message retention and eviction policy of the topic. Configurations do not expose the specific identities of participants, but rather allows for validation using anonymous credentials. Additional configurations might consist of optional refundable fees required by senders in order to publish to the topic.

Conversation Data Layer (Data Replication)

An eventually consistent, distributed relay and storage system using Merkle CRDTs is used for representing and synchronizing messages in topics.

Merkle DAG

A Merkle DAG is made up of directed acyclic graphs with content-addressed nodes—meaning their identifiers (hashes) derive from the content they store. This design ensures data integrity and allows for efficient identification of differences between versions of the DAG.

Conflict Free Replicated Data Types (CRDT)

CRDTs are data structures that handle concurrent updates without conflicts, ensuring strong eventual consistency. This means that even when multiple nodes update the data simultaneously, CRDTs can merge those updates into a consistent state without requiring consensus.

Merkle CRDTs combine these two concepts, providing a robust solution for maintaining consistency and synchronizing data.

Checkout the implementation of the conversation data layer at the xmtpd GitHub repository

Consensus and Incentives

The consensus protocol extends the conversation data layer and ensures liveness, consistency and overall reliability in the network through the means of economic and non-economic incentives. Nodes register and stake with the Consensus smart contract, and participate in the process to periodically agree on the heads of the Merkle CRDT DAGs for the topics.

A detailed proposal for the consensus protocol is available here

Resource Accounting (Usage Tracking)

The message protocol requires network read/write endpoints and storage resources to operate. Infrastructure providers (i.e., nodes) volunteer these resources. In permissionless systems, users are free to interact with the protocol as they please. This presents network spam challenges, and as such the cost of network resources must be clearly itemized (i.e., an accounting system).

Bidirectional one-to-one channels facilitated by collateralized accounting nodes may be established to ensure instant and highly scalable tracking of resource usage and address problems like double spending credits. Channels are managed by smart contracts, and allow for seamless swapping between ERC 20 tokens and message credits and posting net state usage at channel closure. Credit usage is associated with a usage proof, which is confirmed and signed by the Accounting Node. In the event of a dispute, users or full nodes can submit signed usage proofs to the smart contract within a specific time-delay during channel closure.

Known Risks and Tradeoffs

Network spam - risk

An assumption, and design goal, is that typical messaging flows should be free for consumers. However, this leaves the network vulnerable to spam. Spam in permissionless networks is a difficult problem, and only effectively addressed (at scale) using auctions. An auction designed to price network resources effectively limits spam transactions, since any user that wants to send many transactions would need to compete with other legitimate users also looking for network resources. Network spam and friction for users will arise if there is not a well-designed mechanism in place to manage these issues effectively

Network economic sustainability - risk

For the network to run autonomously and maintain its functionality, it is essential to establish effective incentives that encourage continuous participation and contribution from network participants. In permissionless networks, such as Ethereum this primarily means block rewards. Unless a sensible burn mechanism can be implemented, token supply spirals out of control and there is diminishing marginal minted value for each additional unit of work.

The natural place for burning tokens is alongside network resource demand. In EIP 1559, as users demand more network resources from validators, more tokens are burned, and thus the more valuable ETH block rewards become. In the XMTP network, demand is comprised of consumer messaging, as well as legitimate businesses looking to reach customers. Unless friction is introduced, consumer may be overwhelmed with marketing messages. In this case, staking and/or burning may be required for widely broadcasting messages.

Compromised security - risk

When it comes to message security, one of the main concerns is the risk of key compromise. If an encryption key used to protect messages is compromised, it could potentially lead to the decryption of all past and future messages. This poses a significant threat to the confidentiality of communication.

Incorporating forward secrecy (FS) and post compromise security (PCS) into the protocol improves message security, such that messages sent today are not decryptable if the keys become compromised in the future and a compromised key does not mean all future keys are compromised, as well. FS/PCS requires deleting key material post message decryption and incorporating fresh key material into new key derivation, as well as treating each device as a separate recipient with its own set of keys.

Interoperable inbox and message security - tradeoff

The prominent user experience of the XMTP network to-date has been interoperable inbox. Any client application utilizing the XMTP open protocol and standard is able to broadcast encrypted user messages to the XMTP network for global replication. All clients on the network listen for new messages for relevant conversation topics.

As outlined above, there are strong user experience benefits from FS/PCS. However, incorporating both FS/PCS and interoperable inbox is problematic to the state replication of nodes for two reasons: (1) FS/PCS assumes already decrypted messages are only readable on a user’s local device, so encrypted messages residing on the network would be useless since the keys have been destroyed; and (2) treating each device as a recipient means network replication and storage scales with the number of installations in a conversation where previously it scaled with the number of participants.

Designing a solution that ensures message security while enabling seamless communication across different platforms, while also minimizing user experience tradeoffs such as higher costs, is a challenging consideration at the network level and requires a creative approach.

Ongoing & Future Work

Following modularity principles, the network can be broken down into the following components:

• Identity registry
• Consent registry
• Conversation provider
• Personal storage provider

Underlying these components is the social layer—a community of contributors that protect and progress the protocol and operate the network.

The development of the network roadmap will follow the modular components, starting with design proposals for the identity and consent registries. Preferably design proposals will be developed with serious engagement from the community, or submitted by community members.