Complete Matrix SDK v1 plan and merge mainline updates

README.md

@@ -50,7 +50,7 @@ For Node bots, use a durable Chat SDK state adapter and a durable Matrix store.
 ### Raw Matrix core (no Chat SDK)
 
 ```ts
-import { createMatrixClient } from "better-matrix-js/node";
+import { createMatrixClient, onMessage } from "better-matrix-js/node";
 import { createFileMatrixStore } from "@better-matrix-js/state-file";
 
 const client = createMatrixClient({
@@ -60,17 +60,14 @@ const client = createMatrixClient({
   recoveryKey: process.env.MATRIX_RECOVERY_KEY,
 });
 
-client.events.onMessage(async (event) => {
+await onMessage(client, undefined, async (event) => {
   if (event.sender.isMe) return;
   await client.messages.send({
     roomId: event.roomId,
     text: "Got it.",
     replyTo: event.eventId,
   });
 });
-
-await client.connect();
-await client.sync.start();
 ```
 
 ### Cloudflare Worker
@@ -83,6 +80,16 @@ Use `MatrixSyncDurableObject` for live sync and feed each webhook body to `clien
 
 Use `@better-matrix-js/state-memory` for tests, `@better-matrix-js/state-file` or `@better-matrix-js/state-sqlite` in Node, `@better-matrix-js/state-indexeddb` in browsers, and `@better-matrix-js/cloudflare` for Durable Object or KV storage. For anything custom, wrap a simple getter/setter with `@better-matrix-js/state-simple`.
 
+Docker-backed storage smoke tests are available for service-style stores:
+
+```sh
+pnpm test:docker
+pnpm test:docker:down
+```
+
+The Redis smoke uses `@better-matrix-js/state-simple` against a real Redis container
+to prove the minimal Matrix store contract works with external server-side storage.
+
 Browser apps should load `matrix-core.wasm` with `wasmUrl`, `wasmBytes`, or a bundler-provided `wasmModule`, and should persist Matrix state in IndexedDB.
 
 ## Feature support matrix
@@ -92,7 +99,7 @@ Browser apps should load `matrix-core.wasm` with `wasmUrl`, `wasmBytes`, or a bu
 | Node bots | Supported via `better-matrix-js/node` and file, SQLite, or custom stores. |
 | Browser apps | Supported with explicit WASM loading and IndexedDB-backed state. |
 | Cloudflare Workers | Supported with Durable Object state and `MatrixSyncDurableObject`. |
-| Live `/sync` loop | Supported with `client.sync.start()` in long-lived runtimes. |
+| Live `/sync` loop | Supported with `client.subscribe(filter, handler)` in long-lived runtimes. |
 | Serverless sync | Supported by applying webhooked responses with `applyResponse`. |
 | E2EE | Supported when the crypto store, `pickleKey`, and optional `recoveryKey` are durable. |
 | Beeper ephemeral events | Supported only on Beeper homeservers. |

docker-compose.e2e.yml

@@ -0,0 +1,11 @@
+services:
+  redis:
+    image: ${BMJS_E2E_REDIS_IMAGE:-public.ecr.aws/docker/library/redis:7.4-alpine}
+    command: ["redis-server", "--save", "", "--appendonly", "no"]
+    ports:
+      - "${BMJS_E2E_REDIS_PORT:-6380}:6379"
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 1s
+      timeout: 1s
+      retries: 30

docs/API.md

@@ -0,0 +1,197 @@
+# better-matrix-js API Overview
+
+`better-matrix-js` is a Matrix client SDK built around one lifecycle model:
+
+- `createMatrixClient(options)` is synchronous and inert.
+- First awaited Matrix method lazily boots WASM, store, account identity, and crypto.
+- `client.boot()` exists when an app wants startup failures early.
+- Live events only flow through `client.subscribe(filter, handler)`.
+- Serverless sync payloads enter through `client.sync.applyResponse({ response, since })`.
+
+There is no public `connect()`, `events`, `sync.start()`, `sync.once()`, or `sync.stop()`.
+
+## Migration Stance
+
+This SDK has not had a stable public release. The v1 API intentionally deletes old generated shapes instead of preserving aliases. Treat stale examples that mention `connect()`, root `events`, or public `sync.start()` as obsolete.
+
+## Account Objects
+
+Use `MatrixAccount` as the serializable account/session shape:
+
+```ts
+type MatrixAccount = {
+  homeserver: string;
+  userId: string;
+  deviceId: string;
+  accessToken: string;
+  metadata?: Record<string, unknown>;
+};
+```
+
+`deviceId` is immutable identity returned by Matrix login/whoami. Do not generate or edit it as a runtime option for an existing access token.
+
+```ts
+const login = createMatrixLogin({ homeserver: "https://matrix.example.com" });
+const account = await login.token({ token: process.env.MATRIX_LOGIN_TOKEN! });
+
+const client = createMatrixClient({
+  account,
+  pickleKey: process.env.MATRIX_PICKLE_KEY!,
+  store,
+});
+
+await client.whoami();
+```
+
+`metadata` is carried through login results if provided, but it is never used as Matrix identity. Runtime identity is always `userId`, `deviceId`, `homeserver`, and `accessToken`.
+
+## CLI Usage Without Sync
+
+Request-style programs can send/fetch and exit without subscribing:
+
+```ts
+const client = createMatrixClient({ account, store, pickleKey });
+
+await client.messages.send({
+  roomId: "!room:example.com",
+  text: "done",
+});
+
+await client.close();
+```
+
+No `/sync` loop is started by construction, `boot()`, `whoami()`, send, fetch, or pagination.
+
+## Live Subscriptions
+
+Use one root live primitive:
+
+```ts
+const sub = await client.subscribe({ kind: "message", roomId }, async (event) => {
+  if (event.kind !== "message" || event.sender.isMe) return;
+  await client.messages.send({
+    roomId: event.roomId,
+    text: "ack",
+    replyTo: event.eventId,
+  });
+});
+
+await sub.stop();
+await sub.done;
+```
+
+The first subscriber starts the internal sync runner. Stopping the last subscriber stops it. Multiple subscribers share one runner.
+
+Optional sync tuning lives on the subscription call, not under `client.sync`:
+
+```ts
+await client.subscribe(filter, handler, {
+  timeoutMs: 30_000,
+  retryDelayMs: 1_000,
+});
+```
+
+## Catch-Up
+
+Subscriptions are future-only by default. A reused account does not replay stored-cursor backlog unless the caller asks:
+
+```ts
+const sub = await client.subscribe({ kind: "message" }, onMessage);
+await sub.catchUp();
+```
+
+`catchUp()` replays missed events from the stored cursor through that subscription.
+
+## Helper Functions
+
+Pure helpers are thin wrappers over `client.subscribe`:
+
+```ts
+import { onInvite, onMessage, onRawEvent, onReaction } from "better-matrix-js";
+
+await onMessage(client, { roomId }, handler);
+await onReaction(client, { relationEventId: "$event" }, handler);
+await onInvite(client, undefined, handler);
+await onRawEvent(client, { roomId }, handler);
+```
+
+They are not separate event systems.
+
+## Serverless Apply
+
+When `/sync` is owned by another process, pass raw Matrix sync JSON to the client:
+
+```ts
+await client.sync.applyResponse({ response, since });
+```
+
+Only one writer should advance an encrypted Matrix device cursor and crypto store at a time. In serverless deployments, serialize work through a Durable Object, a lock, or another single-writer mechanism.
+
+Live mode owns the cursor inside `client.subscribe(...)`. Webhook mode owns the cursor in the external sync producer and applies the payload to the account client. Cloudflare mode should use one sync Durable Object to poll Matrix and one account Durable Object to apply responses and run bot code.
+
+## Raw Requests
+
+Use `client.raw.request` for advanced Matrix endpoints without adding throwaway wrappers:
+
+```ts
+const result = await client.raw.request({
+  method: "POST",
+  path: "/_matrix/client/v3/rooms/!room:example/send/m.room.message/txn",
+  body: { msgtype: "m.text", body: "hello" },
+});
+```
+
+The path must be relative to the homeserver.
+
+## E2EE Storage
+
+Encrypted bots should always use durable storage and a stable `pickleKey`:
+
+```ts
+const client = createMatrixClient({
+  account,
+  store,
+  pickleKey: process.env.MATRIX_PICKLE_KEY!,
+  recoveryKey: process.env.MATRIX_RECOVERY_KEY,
+});
+```
+
+`recoveryKey` unlocks Matrix key backup for historical encrypted messages. `pickleKey` protects local crypto state and must remain stable for the device/store pair.
+
+## Store Ownership
+
+Each Matrix account/device store is single-writer. Do not run two live clients, webhook consumers, or Durable Objects against the same store prefix at the same time. Multiple logical bots can share a process by creating separate clients with separate account/device stores.
+
+The storage adapters persist fast-boot state only: account/session material supplied by the app, crypto state, sync cursors, pending decryptions, and small summaries/caches. They intentionally do not model a full gomuks-style timeline database.
+
+## Unsupported Chat SDK Features
+
+Matrix has no native portable equivalent for Chat SDK modals, scheduled messages, or interactive cards/actions. The adapter may render plain text only when that does not imply unsupported interactivity; otherwise it should throw clearly.
+
+## Beeper
+
+Beeper is first-class, but non-standard behavior stays explicit. Native stream events and ephemeral sends live under `client.beeper.*`, and the Chat SDK adapter only uses them when the homeserver is Beeper or `beeper: true` is configured. Standard Matrix homeservers use Matrix edit-based streaming and reject Beeper-only ephemeral sends.
+
+Beeper login helpers are stateless request functions:
+
+```ts
+import { createBeeperLogin } from "better-matrix-js/beeper-login";
+
+const beeper = createBeeperLogin();
+const token = await beeper.requestEmailToken({
+  clientSecret,
+  email,
+  sendAttempt: 1,
+});
+
+const registered = await beeper.register({
+  auth: {
+    type: "m.login.email.identity",
+    threepid_creds: { sid: token.sid, client_secret: clientSecret },
+  },
+  password,
+  username,
+});
+```
+
+The helper does not embed QA secrets, fixed OTP values, or environment-specific assumptions.

e2e-scripts/.gitignore

@@ -0,0 +1 @@
+.out/

e2e-scripts/README.md

@@ -0,0 +1,52 @@
+# E2E Scripts
+
+These scripts were moved from the private E2E harness so we can keep the test
+coverage close to the SDK. They intentionally do not include Beeper QA account
+creation, fixed OTP flows, private tokens, or any account provisioning logic.
+
+They are not CI tests by default. They require reusable Matrix/Beeper accounts
+with access tokens and, for encrypted-history coverage, recovery keys.
+
+## Account File
+
+Create `e2e-scripts/.out/accounts.json` yourself:
+
+```json
+{
+  "accounts": [
+    {
+      "homeserverUrl": "https://example.com",
+      "userId": "@user:example.com",
+      "deviceId": "DEVICEID",
+      "accessToken": "ACCESS_TOKEN",
+      "recoveryKey": "OPTIONAL_RECOVERY_KEY",
+      "loginToken": "OPTIONAL_JWT_LOGIN_TOKEN_FOR_FRESH_DEVICE_TESTS",
+      "username": "stable-label"
+    }
+  ]
+}
+```
+
+The test suite reuses accounts and stores by default so old history, old devices,
+and recovery behavior stay exercised. Use `MATRIX_E2E_RESET_STORES=1` when you
+want to force a clean local store. Use `MATRIX_E2E_FRESH_DEVICE=1` only when the
+accounts include reusable Matrix JWT login tokens.
+
+## Running
+
+Build the SDK first:
+
+```sh
+pnpm build
+```
+
+Then run from this directory or from the repo root:
+
+```sh
+cd e2e-scripts
+MATRIX_E2E_SDK_ROOT=.. npm run test:surface
+MATRIX_E2E_SDK_ROOT=.. npm test
+```
+
+The Chat SDK adapter test requires the upstream `chat` package to be resolvable
+in Node, for example by installing/linking it in your local environment.

e2e-scripts/package.json

@@ -0,0 +1,13 @@
+{
+  "name": "better-matrix-js-e2e-scripts",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "test": "node --test --test-concurrency=1 --test-timeout=420000 test/*.test.mjs",
+    "test:browser:serve": "node src/browser-smoke-server.mjs",
+    "test:core": "node --test --test-timeout=420000 test/core-e2e.test.mjs",
+    "test:adapter": "node --test --test-timeout=420000 test/chat-adapter-e2e.test.mjs",
+    "test:cloudflare": "node src/cloudflare-runtime-smoke.mjs",
+    "test:surface": "node --test --test-timeout=420000 test/sdk-surface-happy-path-e2e.test.mjs"
+  }
+}

e2e-scripts/src/accounts.mjs

@@ -0,0 +1,22 @@
+import { readFile } from "node:fs/promises";
+import { ACCOUNTS_PATH, HOMESERVER_URL } from "./config.mjs";
+
+export async function loadAccounts(count) {
+  const data = JSON.parse(await readFile(ACCOUNTS_PATH, "utf8"));
+  const accounts = Array.isArray(data) ? data : data.accounts;
+  if (!Array.isArray(accounts)) {
+    throw new Error(`Expected ${ACCOUNTS_PATH} to contain an accounts array`);
+  }
+  if (accounts.length < count) {
+    throw new Error(`Expected at least ${count} E2E accounts in ${ACCOUNTS_PATH}, found ${accounts.length}`);
+  }
+  return accounts.slice(0, count).map(normalizeAccount);
+}
+
+function normalizeAccount(account) {
+  const homeserverUrl = account.homeserverUrl ?? account.homeserver ?? HOMESERVER_URL;
+  return {
+    ...account,
+    homeserverUrl,
+  };
+}