Keystore API

[neekolas]

I wanted to share with you an early preview of some work we are doing for v8 of the xmtp-js SDK.

The first PR around this just landed, but there is still considerable work to be done before this is ready for usage.

Background & Motivation

Version 8 of the xmtp-js SDK is going to tackle a longstanding issue in our codebase. We currently don't have a strong separation of concerns between the cryptographic functions and the core business logic of interacting with the network. Cryptographic operations and crypto classes are sprinkled throughout the SDK, with a baked-in assumption that sensitive private key data is accessible. This makes it awkward and difficult to develop more secure methods for encrypting/decrypting/signing of data and for us to move to a model where key material is compartmented.

The development of our Metamask Snap is not the only place where this is becoming an issue. As we develop a Rust based libxmtp to handle cryptographic operations, there are significant performance implications to passing data between the application context and WebAssembly. Having key material and cryptgraphic operations all live in the same context will make the code much simpler to reason about and lead to greater performance.

Goals/Non Goals

Goals

• Refactor the xmtp-js codebase to establish strict modularization of code that handles business logic/API calls and code that handles private key material/cryptographic operations.
• Establish a well defined interface for interacting with the Keystore
• Do this in a way that minimizes impact to application developers. Should be a non-breaking change for typical apps
• Simplify the Client code by handling all key management and cryptography in one place

Non goals

• Actually move any of the core cryptgraphic operations out of the SDK. This is purely a refactor to allow for future releases that outsource key storage/encryption/decryption elsewhere (Snap, Chrome Extensions, mobile apps, lixbxmtp).
• Change any of our encryption primitives
• Specify how this would work in languages other than JS (that may be desirable, but the immediate use-cases are all around the web SDK)

Proposed Solution

I propose refactoring the codebase to move any code that interacts with private keys, topic keys, and encryption/decryption into a separate module with a strict API boundary and well defined interface. In the initial version, all calls to the Keystore service will be in the same process as the rest of the SDK. The interface should be designed in a way that requests can be easily JSON serializable to allow for future providers that are remote (Snap, Chrome Extension, Wallet, 1P mobile app).

The keystore will need to maintain some amount of state (either persisted between sessions or ephemeral) to access the PrivateKeyBundle and any TopicKeys that have been found from invitations.

Components

In this new model, there are two distinct components of our SDK:

Client

The Client is responsible for all API calls to XMTP nodes, high level business logic (conversations abstraction), and handling of message encoding/decoding.

Keystore

The Keystore is responsible for holding the PrivateKeyBundle of the user (and any future delegated keys), encrypting/decrypting V1 messages, storing TopicKey material from invitations, and encrypting/decrypting V2 messages.

Separation of concerns

classDiagram
    class Keystore
    class Client
    Client: Conversations abstraction
    Client: All calls to XMTP APIs
    Client: ContentType management and decoding of content
    Keystore: Encrypt messages prior to publishing
    Keystore: Decrypt messages
    Keystore: Encrypt invitations
    Keystore: Decrypt invitations

Loading

Types Of Keystore

There are a number of potential keystore types I can see us developing over time. These are listed in rough order of priority and timing.

InMemoryKeystore

The default Keystore, and the first one we will need to build, will simply be a module that implements the Keystore API locally. It will hold the user's PrivateKeyBundle and TopicKeys and execute API requests using those keys.

The InMemoryKeystore can run in the same process as the Client, or it can be used to implement some of the remote Keystores listed below.

Effectively, this is just a refactor of our codebase.

Snaps Keystore

The Snaps Keystore will be a light wrapper around the Metamask Snaps API, where all requests to the Keystore are proxied as JSON-RPC calls to an installed Snap. The Snap will handle RPC requests using something similar to the InMemoryKeystore, but with the additional capability of persisting keys in the Metamask encrypted storage.

libxmtp Keystore

As we develop libxmtp, we can include a Typescript wrapper class that sends Keystore API calls across the WASM bridge to libxmtp to fulfill them. This keystore can be used inside other Keystores (for example, the Snap could proxy calls to libxmtp while using the Snap encrypted storage for persistence). Effectively a replacement for the Base Keystore.

Browser Extension Keystore

Any browser extension that supports XMTP could potentially become a Keystore. The Browser Extension would implement some version of the Base Keystore. We would likely use a runtime.Port to communicate back and forth between the extension and the browser session. This would work similarly to window.ethereum (window.xmtp?) where a small bit of code would be injected into all webpages to handle communication.

Users would have to approve a dialog in the Chrome Extension for each domain they wish to use.

Mobile Wallet Browser

Mobile wallets implementing XMTP can implement the Keystore API in the language of their choice. They can then inject window.xmtp into their mobile browser sessions, functioning in a similar way to the Browser Extension Keystore. That will allow any web page that uses XMTP to access the wallet's keystore (after the user grants permission to the domain)

Service Worker

In browser contexts we could use Service Workers to move the Keystore out of the main thread. This would be a great performance boost, and improve the security of the SDK.

API Design

The Keystore API is specified in protocol buffers. While only some Keystore implementations require serialization, this gives us maximum flexibility. We can pass generated protobuf classes to the Keystore and use them as-is, easily serialize to JSON, or serialize to a binary format. And this can all be done in a type-safe way.

// Message content encoding structures
syntax = "proto3";

package xmtp.keystore_api.v1;

import "message_contents/ciphertext.proto";
import "message_contents/invitation.proto";
import "message_contents/public_key.proto";

// Application-specific error codes for the Keystore API.
enum ErrorCode {
    ERROR_CODE_UNSPECIFIED = 0;
    ERROR_CODE_INVALID_INPUT = 1;
    ERROR_CODE_NO_MATCHING_PREKEY = 2;
    // TODO: More error codes
}

// Wrapper class for errors from the Keystore API
message KeystoreError {
    string message = 1;
    ErrorCode code = 2;
}

// A light pointer for a conversation that contains no decryption keys
message ConversationReference {
    string topic = 1;
    uint64 created_ns = 2;
    xmtp.message_contents.InvitationV1.Context context = 3;
}

// Decrypt a batch of messages using X3DH key agreement
message DecryptV1Request {
    // A single decryption request
    message Request {
        xmtp.message_contents.Ciphertext payload = 1;
        xmtp.message_contents.PublicKeyBundle peer_keys = 2;
        bytes header_bytes = 3;
        bool is_sender = 4;
    }

    repeated Request requests = 1;
}

// Response type for both V1 and V2 decryption requests
message DecryptResponse {
    // A single decryption response
    message Response {
        // Wrapper object for success response
        message Success {
            bytes decrypted = 1;
        }

        oneof response {
            Success result = 1;
            KeystoreError error = 2;
        }
    }

    repeated Response responses = 1;
}

// Decrypt a batch of messages using the appropriate topic keys
message DecryptV2Request {
    // A single decryption request
    message Request {
        xmtp.message_contents.Ciphertext payload = 1;
        bytes header_bytes = 2;
        string content_topic = 3;
    }

    repeated Request requests = 1;
}

// Encrypt a batch of messages using X3DH key agreement
message EncryptV1Request {
    // A single encryption request
    message Request {
        xmtp.message_contents.PublicKeyBundle recipient = 1;
        bytes payload = 2;
        bytes header_bytes = 3;
    }

    repeated Request requests = 1;
}

// Response type for both V1 and V2 encryption requests
message EncryptResponse {
    // A single encryption response
    message Response {
        // Wrapper object for success response
        message Success {
            xmtp.message_contents.Ciphertext encrypted = 1;
        }

        oneof response {
            Success result = 1;
            KeystoreError error = 2;
        }
    }

    repeated Response responses = 1;
}

// Encrypt a batch of messages using the appropriate topic keys
message EncryptV2Request {
    // A single encryption request
    message Request {
        bytes payload = 1;
        bytes header_bytes = 2;
        string content_topic = 3;
    }

    repeated Request requests = 1;
}

// Request to create an invite payload, and store the topic keys in the Keystore
message CreateInviteRequest {
    xmtp.message_contents.InvitationV1.Context context = 1;
    xmtp.message_contents.SignedPublicKeyBundle recipient = 2;
    uint64 created_ns = 3;
}

// Response to a CreateInviteRequest
message CreateInviteResponse {
    ConversationReference conversation = 1;
    bytes payload = 2;
}

// Request to save a batch of invite messages to the Keystore
message SaveInvitesRequest {
    // Mirrors xmtp.envelope schema
    message Request {
        string content_topic = 1;
        uint64 timestamp_ns = 2;
        bytes payload = 3;
    }

    repeated Request requests = 1;
}

// Response to a SaveInvitesRequest
message SaveInvitesResponse {
    // A single response
    message Response {
        // Wrapper object for success response
        message Success {
            ConversationReference conversation = 1;
        }

        oneof response {
            Success result = 1;
            KeystoreError error = 2;
        }
    }

    repeated Response responses = 1;
}

The Protobuf API will then be referenced in the Typescript implementation like this:

import { keystore, publicKey } from '@xmtp/proto'
export interface Keystore {
  // Decrypt a batch of V1 messages
  decryptV1(req: keystore.DecryptV1Request): Promise<keystore.DecryptResponse>
  // Decrypt a batch of V2 messages
  decryptV2(req: keystore.DecryptV2Request): Promise<keystore.DecryptResponse>
  // Encrypt a batch of V1 messages
  encryptV1(req: keystore.EncryptV1Request): Promise<keystore.EncryptResponse>
  // Encrypt a batch of V2 messages
  encryptV2(req: keystore.EncryptV2Request): Promise<keystore.EncryptResponse>
  // Decrypt and save a batch of invite for later use in decrypting messages on the invite topic
  saveInvites(
    req: keystore.SaveInvitesRequest
  ): Promise<keystore.SaveInvitesResponse>
  // Create the sealed invite and store the Topic keys in the Keystore for later use
  createInvite(
    req: keystore.CreateInviteRequest
  ): Promise<keystore.CreateInviteResponse>
  // Get V2 conversations
  getV2Conversations(): Promise<keystore.ConversationReference[]>
  // Used for publishing the contact
  getPublicKeyBundle(): Promise<publicKey.SignedPublicKeyBundle>
  // Technically duplicative of `getPublicKeyBundle`, but nice for ergonomics
  getAccountAddress(): Promise<string>
}

Common workflows

Loading V2 conversations

sequenceDiagram
    Participant C as Client
    Participant N as Node
    Participant K as Keystore
    C-)N: Query for invites
    N-->>C: Receive encrypted invites
    C-)K: saveInvites with encrypted payloads
    C-)K: getV2Conversations
    K-->>C: Receive conversation list

Loading

Listing messages

sequenceDiagram
    Participant C as Client
    Participant N as Node
    Participant K as Keystore
    C-)N: Query for envelopes
    N-->>C: Receive envelopes
    C-)K: decryptV2 with batch of payloads
    K-->>K: decrypt all payloads
    K-->>C: Receive decrypted payloads
    C-->>C: Decode content

Loading

Sending a message

sequenceDiagram
    Participant C as Client
    Participant N as Node
    Participant K as Keystore
    C-)K: getV2Conversations
    K-->>C: Receive conversations
    C-)K: encryptV2 { content, topic }
    K-->>C: Encrypted payload
    C-)N: Publish

Loading

Breaking changes

• The Keystore is the beginning of the end for Client.getKeys() in the main SDK. While we may offer support for this feature for cases where the keys are in the browser context, I do not want to support any mechanism for extracting keys from secure contexts like Snaps or the Chrome Extension. Instead, the ability to export and cache keys would be supported in the Keystore implementation itself. That way, private key exports can stay inside the secure context the Keystore runs in.
• Caching conversations should happen inside the keystore. This should remove the need for user-facing APIs like conversations.export(). I would suggest we create a stateful keystore that uses LocalStorage to support the current use-cases of exporting Conversations. Keystore implementations can have pluggable storage providers to support a range of devices with differing storage capabilities (for example, on Node.js you may want to cache on the filesystem)
• Some fields will disappear from user-facing classes like Conversation or DecodedMessage

[nakajima]

This all looks rad, thanks for the update.

If conversations.export() and Client.getKeys() are going away, we're definitely going to want that stateful keystore, since right now a react native app needs to share that data with the iOS notification extension in order to decode message contents. I think as long as they can both read from the same store, we should be good though 👍 .

[neekolas]

I should probably rephrase that to "going away from the main SDK". A prerequisite to release will be that those functionalities are available from the Keystore itself.

[tg44]

After I "wasted" a week of decoupling the encode-decode part from the conversation, I'm really happy with this idea! If you start this refactor I generally propose to separate the functions and the data for every API (basically stop using classes). Like, why we have a fully functioning conversation in the DecodedMessage? Why the MessageV1 has a toBuffer() function in it? In general we should be able to Thing.from(JSON.parse(JSON.stringify(thingInstance))) anything at least.

[neekolas]

In general we should be able to Thing.from(JSON.parse(JSON.stringify(thingInstance))) anything at least.

I hear you on this pain point. v8 of the SDK is definitely going to better in terms of serializability. For us, however, JSON isn't the target we are primarily focused on. Pretty much every object uses Uint8Arrays, so Protobuf is a better-suited (and typesafe) serialization format. But the Keystore API is at least forcing us to use objects closer to the generated Protobuf types and will rely on less of our wrapper classes for user-facing code.

[tg44]

Yes, most of the protobuff types has a from/to JSON function (or something similar) which is nice, but not really documented, and we have no functions like;

static from(msg: MessageV1 | any): MessageV1 {
    if (msg instanceof MessageV1) {
      return msg
    } else {
      if (typeof msg.bytes === 'object') {
        msg.bytes = parseObjectToUint8Array(msg.bytes)
      }
      const m = { v1: proto.MessageV1.fromJSON(msg), v2: undefined }
      return new MessageV1(
        msg.id,
        msg.bytes,
        m,
        proto.MessageHeaderV1.fromJSON(msg.header),
        msg.senderAddress
      )
    }
  }

(Also, here I couldn't use the MessageV1.fromJSON as is, bcs the serialization on the other side did something strange...)