Store operation content once and enforce per-plan history retention

BUILD_INSTRUCTIONS.md

@@ -177,8 +177,8 @@ running at a loss; it is not the optimization target.
 
 The axes actually used:
 
-- **History retention** — bounds `operation_files.payload`, the only table that grows
-  without limit. This is the real cost governor.
+- **History retention** — bounds `operations`, the only table that grows without limit.
+  This is the real cost governor.
 - **Autonomy tier availability** — auto-always (Phase 9) is the "I trust it now" moment,
   which is the honest point to ask for money.
 - **Org controls** — SSO, audit export, SLA. What organizations actually buy.
@@ -215,14 +215,6 @@ ever stop working.
 **Not yet done:**
 
 - The Stripe account itself, and therefore any real checkout.
-- **Retention is declared but not enforced.** `PLAN_LIMITS[plan].historyRetentionDays` is
-  surfaced to users and is the intended cost governor, but nothing prunes against it yet.
-  This is not a small wiring job: `apps/service/src/prune.ts` deliberately refuses to touch
-  `operations` because cursor-based reconnect lets a long-offline replica download
-  everything after its last-known cursor, and age-pruning would silently break that
-  guarantee. A safe implementation needs either (a) pruning only below the minimum cursor
-  across live replicas, or (b) a defined "cursor too old, resync from Git" protocol
-  response. Until one of those exists, storage is unbounded on every plan.
 - Per-seat pricing mechanics for Team (the plan enforces Unlimited's caps today; the
   per-seat *charge* has no implementation because there is no billing provider).
 
@@ -241,6 +233,18 @@ ever stop working.
   at 10/min: it is unauthenticated, so there is no identity to charge, and that limit is the
   brute-force defense for the 40-bit code space.
 
+**Retention is enforced** (option (b) of the two designs sketched here previously).
+`PgStore.pruneOperationsByRetention()` deletes each workspace's operations outside
+`PLAN_LIMITS[plan].historyRetentionDays` and records how far it reached in
+`workspaces.operations_pruned_through`. `GET /v1/operations` answers a cursor below that
+watermark with an explicit `cursor-too-old` resync status (`410 Gone` for daemons that
+predate it) instead of a truncated page, and the daemon adopts the watermark and reports
+the gap — see docs/protocol.md. The sweep runs on a service-side interval configured with
+`CROSSCODE_RETENTION_DATABASE_URL` (the request-serving role deliberately cannot delete
+operations); `pnpm service:prune` runs it manually. Content is also no longer stored
+twice: `operations.event` is the single home of a transaction's file bodies, and
+`operation_files` is a per-path index into it.
+
 `assertSemanticReviewCallAvailable` is deliberately left unwired rather than pending: semantic review is delegated to the member's own MCP agent and never reaches the service, so there is no per-call cost to meter, and wiring it would mean adding a network round-trip before every local review purely to bill for it (see the comment above `incrementSemanticReviewUsage` in `apps/service/src/billing.ts`). Every plan now carries an unlimited cap for it, so it can never fire.
 
 ### Phase 11 — Pairing a checkout to an account (backend v1 shipped)

README.md

@@ -223,7 +223,7 @@ Set on the host: `DATABASE_URL`, `SUPABASE_URL`, and — when a proxy in front t
 product, which means no browser origin may call the API cross-origin; it exists for anyone
 building their own browser client against the service.
 
-Run `pnpm service:migrate` with a migration-owner connection before starting a new service version. `CROSSCODE_RUNTIME_DB_ROLE` applies the required least-privilege grants, and service startup refuses a role that can update/delete immutable operations or audit rows. The runtime never executes DDL. Non-loopback PostgreSQL URLs must specify exactly one `sslmode=verify-full` and cannot use host/SSL query overrides. For local-only testing against a plain (non-Supabase) Postgres instance, `infra/docker-compose.yml` still starts one on `127.0.0.1:5432`; it is not used in production, where `DATABASE_URL` points at Supabase.
+Run `pnpm service:migrate` with a migration-owner connection before starting a new service version. Set `CROSSCODE_RETENTION_DATABASE_URL` (optionally `CROSSCODE_RETENTION_SWEEP_MINUTES`, default 60) to that same privileged connection to enable the scheduled history-retention sweep; the least-privilege runtime role cannot delete operations, so without it retention only runs when an admin invokes `pnpm service:prune`. The interval needs a persistent process, so on the Vercel function deployment the sweep must be driven externally (a scheduled `pnpm service:prune`) until a platform cron is wired. `CROSSCODE_RUNTIME_DB_ROLE` applies the required least-privilege grants, and service startup refuses a role that can update/delete immutable operations or audit rows. The runtime never executes DDL. Non-loopback PostgreSQL URLs must specify exactly one `sslmode=verify-full` and cannot use host/SSL query overrides. For local-only testing against a plain (non-Supabase) Postgres instance, `infra/docker-compose.yml` still starts one on `127.0.0.1:5432`; it is not used in production, where `DATABASE_URL` points at Supabase.
 
 ## Workspaces, members, and invites (CLI and API only)
 
@@ -433,7 +433,7 @@ For the implementation plan and current milestone ledger, see [BUILD_INSTRUCTION
 
 ## Current limitations
 
-- Production PostgreSQL role grants still need environment-specific deployment hardening. Retention is opt-in and admin-only: `pnpm service:prune -- --older-than-days <n>` deletes old audit events and ended sessions; cursor-reconnect-dependent tables are deliberately never pruned.
+- Production PostgreSQL role grants still need environment-specific deployment hardening. Operation history is pruned to the workspace plan's `historyRetentionDays` — on a service-side schedule when `CROSSCODE_RETENTION_DATABASE_URL` names a role that may delete, and on demand via `pnpm service:prune`, which also deletes audit events and ended sessions older than `--older-than-days <n>`. A replica whose cursor falls outside the retained window is told to resynchronize explicitly; the other cursor-reconnect tables (tasks, claims, handoffs, intents, validations) are still never pruned.
 - Supabase refresh tokens are stored in the OS keychain when available (macOS `security`, Linux `secret-tool`); otherwise, including on Windows, the mode-`0600` Git-directory configuration fallback is used.
 - Binary files are shared base64-encoded with byte-exact materialization; any conflict involving a binary file requires human approval, since deterministic hunk/merge analysis is text-only.
 - Renames are tracked as first-class rename changes (old path, new path, content); a rename conflicting with pending work on either path, moving into or out of a critical path, or whose source has diverged locally always requires approval.

apps/daemon/src/client.ts

@@ -22,7 +22,7 @@ type Status = {
   eventSequence: number;
   remoteCursor: number;
   pendingOutbound: number;
-  service: { configured: boolean; online: boolean; lastSyncAt?: string; lastSyncError?: string };
+  service: { configured: boolean; online: boolean; lastSyncAt?: string; lastSyncError?: string; lastResyncAt?: string; lastResyncMessage?: string };
 };
 
 /**

apps/daemon/src/index.test.ts

@@ -8,12 +8,30 @@ import { afterEach, describe, expect, it } from "vitest";
 import { CoordinationService } from "../../service/src/index.js";
 import { contentHash } from "@crosscode/core";
 import { discoverRepository, unifiedDiff } from "@crosscode/git";
-import { LocalDaemon } from "./index.js";
+import { LocalDaemon, type RemoteSyncTransport } from "./index.js";
 
 const exec = promisify(execFile); const directories: string[] = [];
 async function repo(): Promise<string> { const path = await mkdtemp(join(tmpdir(), "crosscode-daemon-")); directories.push(path); await exec("git", ["init", "-q", "-b", "main", path]); await exec("git", ["-C", path, "config", "user.email", "test@example.com"]); await exec("git", ["-C", path, "config", "user.name", "Test"]); await writeFile(join(path, "a.txt"), "one\n"); await exec("git", ["-C", path, "add", "."]); await exec("git", ["-C", path, "commit", "-qm", "initial"]); return path; }
 afterEach(async () => { await Promise.all(directories.splice(0).map((path) => rm(path, { recursive: true, force: true }))); });
 
+/** A transport with nothing to sync, for tests that care about exactly one of its methods. */
+function emptyTransport(): RemoteSyncTransport {
+  return {
+    upload: async (record) => ({ id: record.transaction.id, workspaceId: "w", senderReplicaId: "replica", transaction: record.transaction, sequence: 1, createdAt: new Date().toISOString() }),
+    list: async (after) => ({ operations: [], nextCursor: after }),
+    uploadTask: async (record) => ({ eventId: record.event.id, workspaceId: "w", senderReplicaId: "replica", task: record.event.payload, updatedAt: new Date().toISOString() }),
+    listTasks: async (after) => ({ tasks: [], nextCursor: after }),
+    uploadClaim: async (record) => ({ eventId: record.event.id, workspaceId: "w", senderReplicaId: "replica", claim: record.event.payload, released: record.event.type === "claim.released", updatedAt: new Date().toISOString() }),
+    listClaims: async (after) => ({ claims: [], nextCursor: after }),
+    uploadHandoff: async (record) => ({ eventId: record.event.id, workspaceId: "w", senderReplicaId: "replica", handoff: record.event.payload, updatedAt: new Date().toISOString() }),
+    listHandoffs: async (after) => ({ handoffs: [], nextCursor: after }),
+    uploadIntent: async (record) => ({ eventId: record.event.id, workspaceId: "w", senderReplicaId: "replica", intent: record.event.payload, updatedAt: new Date().toISOString() }),
+    listIntents: async (after) => ({ intents: [], nextCursor: after }),
+    uploadValidation: async (record) => ({ eventId: record.event.id, workspaceId: "w", senderReplicaId: "replica", validation: record.event.payload, createdAt: new Date().toISOString() }),
+    listValidations: async (after) => ({ validations: [], nextCursor: after })
+  };
+}
+
 describe("local daemon coordination", () => {
   it("shares a proposal only after explicit acceptance", async () => {
     const senderRoot = await repo(); const receiverRoot = await repo(); const service = new CoordinationService();
@@ -579,6 +597,67 @@ describe("local daemon coordination", () => {
     expect(result.uploaded).toBe(1);
   });
 
+  // The failure this guards against is silent: with retention deleting operations, a
+  // replica whose cursor sits below the deleted range would be handed a short (often empty)
+  // list, conclude it was caught up, and never learn the proposals existed. The service now
+  // refuses that cursor outright, and the daemon's job is to adopt the watermark, say so,
+  // and keep syncing rather than stall.
+  it("resynchronizes from the retention watermark when its cursor is too old to serve", async () => {
+    const root = await repo();
+    let daemon = await LocalDaemon.open(root, { workspaceId: "w", replicaId: "replica", actorId: "actor" });
+    const survivor = {
+      id: "operation-after-retention",
+      workspaceId: "w",
+      senderReplicaId: "other",
+      transaction: {
+        id: "operation-after-retention",
+        base: { files: [] },
+        changes: [{ path: "b.txt", kind: "add" as const, afterContent: "kept\n", afterHash: contentHash("kept\n") }],
+        provenance: { source: "filesystem" as const, confidence: "known" as const },
+        safety: { risk: "low" as const, requiresApproval: false }
+      },
+      sequence: 6,
+      createdAt: new Date().toISOString()
+    };
+    const requested: number[] = [];
+    const transport = {
+      ...emptyTransport(),
+      list: async (after: number) => {
+        requested.push(after);
+        // Everything at or below sequence 5 aged out of the plan's window.
+        if (after < 5) return { status: "cursor-too-old" as const, resyncFrom: 5, retentionDays: 7 };
+        return { operations: [survivor], nextCursor: 6 };
+      }
+    };
+
+    const result = await daemon.syncRemote(transport);
+
+    expect(requested).toEqual([0, 5]);
+    expect(result).toEqual({ uploaded: 0, downloaded: 1, cursor: 6 });
+    expect(daemon.operations.get(survivor.id)?.status).toBe("proposed");
+    const service = (await daemon.status()).service as { lastResyncAt?: string; lastResyncMessage?: string };
+    expect(service.lastResyncAt).toEqual(expect.any(String));
+    expect(service.lastResyncMessage).toContain("7 days");
+    expect(service.lastResyncMessage).toContain("resynchronized from sequence 0 to 5");
+
+    // The jump has to be durable, or the next start walks back into the unservable cursor.
+    daemon.close();
+    daemon = await LocalDaemon.open(root, { workspaceId: "w", replicaId: "replica", actorId: "actor" });
+    expect((await daemon.status()).remoteCursor).toBe(6);
+  });
+
+  it("refuses a resync order that would rewind its cursor", async () => {
+    const root = await repo();
+    const daemon = await LocalDaemon.open(root, { workspaceId: "w", replicaId: "replica", actorId: "actor" });
+    // resyncFrom at or below the cursor contradicts the refusal: the service can serve this
+    // cursor. Obeying it would re-download and re-propose operations already resolved here.
+    await expect(daemon.syncRemote({
+      ...emptyTransport(),
+      list: async () => ({ status: "cursor-too-old" as const, resyncFrom: 0, retentionDays: 7 })
+    })).rejects.toThrow("resync to a cursor it can already serve");
+    expect((await daemon.status()).remoteCursor).toBe(0);
+  });
+
   it("recognizes a same-HEAD hard reset as a Git transition", async () => {
     const root = await repo();
     const daemon = await LocalDaemon.open(root, { workspaceId: "w", replicaId: "replica", actorId: "actor" });