Error reports: stable client id, env-aware R2 keys

- Drop server-side `ERR-XXXXX` regeneration. The api server now validates and reuses the client-supplied `meta.id` so the dialog preview, upload toast, and Discord embed all show the same id. The trailing UUID in the R2 key already guarantees object uniqueness; on the astronomically rare collision we retry with a fresh UUID, never a fresh id.
- New R2 key shape: `error-reports/{prod|dev}/{yyyy-mm-dd}/{ERR-XXXXX}-{uuid}.zip`. Env first (release → `prod`, debug → `dev`) so dev-run reports sort separately. Legacy keys without the env segment still aged out via the 90-day R2 lifecycle; `extractDateSegment` reads the date out of both shapes so eviction keeps working oldest-first.
- Discord title prefix is now explicit both ways: `[DEV]` for debug builds, `[PROD]` for release.
- Reject malformed/missing `meta.id` with 400. Added tests for the stable id flow, the `dev/` key shape, and the new validation paths.
- Update `apps/api-server/CLAUDE.md` with the new key shape and id behavior.

Author: vdavid

apps/api-server/CLAUDE.md

@@ -190,9 +190,11 @@ Cron (every 12h): scheduled handler runs three jobs:
 A single `scheduled` handler runs every 12 hours (`0 */12 * * *`). It runs three independent jobs, each in its own
 try-catch so one failure doesn't block the others:
 
-1. **Crash notifications** (every invocation): queries `crash_reports WHERE notified_at IS NULL`, groups by
-   `top_function`, marks rows as notified, then sends a summary email via Resend. Marks before sending to prefer missed
-   notifications over duplicates. Requires `CRASH_NOTIFICATION_EMAIL` and `RESEND_API_KEY`.
+1. **Crash notifications** (every invocation): queries `crash_reports WHERE notified_at IS NULL`, sorted newest-first,
+   marks rows as notified, then sends an email via Resend with one row per crash report (When, Env, ID, Site, Signal,
+   Version). Marks before sending to prefer missed notifications over duplicates. Pre-fix-\* this grouped by
+   `top_function`; the per-row layout is easier to scan and includes the user-visible `CRASH-XXXXX` id. Requires
+   `CRASH_NOTIFICATION_EMAIL` and `RESEND_API_KEY`.
 
 2. **Daily aggregation** (00:00 UTC only): aggregates yesterday's `update_checks` into `daily_active_users` via
    `INSERT OR IGNORE ... GROUP BY`, then prunes raw update checks older than 7 days. Idempotent via existence check.
@@ -253,9 +255,10 @@ SHA-256 + daily salt for deduplication without storing PII. D1 write is fire-and
 
 **Crash report tracking:** Uses D1 (binding: `TELEMETRY_DB`, table: `crash_reports`). Receives crash reports from the
 desktop app via `POST /crash-report`. Columns: hashed_ip, app_version, os_version, arch, signal, top_function,
-backtrace. IP is hashed with SHA-256 + daily salt (same pattern as update checks). Validates payload size (max 64 KB)
-and required fields before writing. D1 write is fire-and-forget via `waitUntil` + `.catch(() => {})`. No authentication
-required.
+backtrace, build_mode (`'release'` / `'debug'`, nullable for legacy rows), short_id (`CRASH-XXXXX`, nullable for legacy
+rows). IP is hashed with SHA-256 + daily salt (same pattern as update checks). Validates payload size (max 64 KB),
+required fields, and the shape of optional fields before writing. D1 write is fire-and-forget via `waitUntil` +
+`.catch(() => {})`. No authentication required.
 
 **Device tracking (fair use):** On each `/validate` call with a `deviceId`, the server tracks the device in KV
 (`devices:{seatTransactionId}`) and logs to Analytics Engine (binding: `DEVICE_COUNTS`, dataset: `cmdr_device_counts`).
@@ -269,12 +272,18 @@ and its own 6-device allowance.
 licensed). Without this, there's no signal for how many people actually run the app — Umami only tracks website visitors
 and download tracking only captures installs.
 
+**Error report R2 key shape:** `error-reports/{prod|dev}/{yyyy-mm-dd}/{ERR-XXXXX}-{uuid}.zip`. The env segment (`prod`
+for release builds, `dev` for debug builds, inferred from `meta.buildMode`) keeps dev-run reports out of the production
+sort order. Legacy keys (`error-reports/{yyyy-mm-dd}/...` — pre-env-prefix) still exist; eviction reads the date segment
+via `extractDateSegment` which handles both shapes. The 90-day R2 lifecycle drains the legacy shape naturally — there's
+no migration.
+
 **Error report eviction (8/6 GB watermarks + lifecycle):** Three layers keep the bucket bounded.
 
 1. **On-upload eviction**: every `POST /error-report` schedules `tryEvict` in `waitUntil(...)`. If `total_bytes` (KV) >
    8 GB and `eviction_in_progress` (KV, 60-s TTL lock) isn't set, lists R2 objects under `error-reports/`, sorts
-   oldest-first by date prefix in the key (`yyyy-mm-dd`) then by `uploaded`, deletes until ≤ 6 GB, then resets the
-   counter to the recomputed ground truth.
+   oldest-first by the embedded `yyyy-mm-dd` segment (via `extractDateSegment`, which handles both new and legacy key
+   shapes) then by `uploaded`, deletes until ≤ 6 GB, then resets the counter to the recomputed ground truth.
 2. **Daily cron sweep**: corrects KV drift by recomputing from R2 and re-running `tryEvict`.
 3. **R2 lifecycle rule**: 90-day expiration applied at provisioning time via `scripts/setup-cf-infra.sh`.
 
@@ -289,7 +298,10 @@ for presigned URLs. Convenience of click-to-download outweighs leak risk because
 
 **Short ID generation:** `generateShortId(prefix, len)` in `license.ts` produces IDs like `ERR-A2345` from the same
 unambiguous alphabet (`23456789ABCDEFGHJKMNPQRSTUVWXYZ`) as license short codes. Rejection sampling avoids modulo bias.
-The error report route retries up to 3 times on R2 HEAD collision.
+The error report route does NOT regenerate the id server-side — it validates the client-supplied `meta.id` against the
+shape `^ERR-[23456789ABCDEFGHJKMNPQRSTUVWXYZ]{5}$` and uses it as-is. On the astronomically rare R2 key collision (same
+id + same date + UUID clash), the route retries with a fresh UUID — never a fresh id — so the user-visible id from the
+preview dialog stays stable through to the toast.
 
 ## Local development
 

apps/api-server/src/discord.test.ts

@@ -73,7 +73,7 @@ describe('buildErrorReportPayload', () => {
                 "value": "[Download bundle](https://example.com/bundle.zip?sig=abc) (link valid 7 days)",
               },
             ],
-            "title": "Error report ERR-A2345",
+            "title": "[PROD] Error report ERR-A2345",
           },
         ],
       }
@@ -106,9 +106,9 @@ describe('buildErrorReportPayload', () => {
     expect(payload.embeds[0].title).toBe('[DEV] Error report ERR-A2345')
   })
 
-  it('does not prefix the title when buildMode is release', () => {
+  it('prefixes the title with [PROD] when buildMode is release', () => {
     const payload = buildErrorReportPayload(baseNotification) as { embeds: { title: string }[] }
-    expect(payload.embeds[0].title).toBe('Error report ERR-A2345')
+    expect(payload.embeds[0].title).toBe('[PROD] Error report ERR-A2345')
   })
 })
 

apps/api-server/src/discord.ts

@@ -10,10 +10,10 @@ export interface ErrorReportNotification {
   id: string
   kind: 'user' | 'auto'
   /**
-   * Forwarded from the manifest. `'debug'` reports come from a dev build of the
-   * desktop app (`cfg!(debug_assertions)`), and we prefix the embed title with
-   * `[DEV]` so triage can tell them apart from production traffic at a glance.
-   * Defaults to `'release'` upstream when older clients don't set it.
+   * Forwarded from the manifest. The embed title gets `[DEV]` for debug builds
+   * (`cfg!(debug_assertions)`) and `[PROD]` for release builds so triage can tell
+   * them apart at a glance regardless of channel. Defaults to `'release'` upstream
+   * when older clients don't set it.
    */
   buildMode: 'release' | 'debug'
   appVersion: string
@@ -67,7 +67,7 @@ export function buildErrorReportPayload(n: ErrorReportNotification): unknown {
     fields.push({ name: 'User note', value: truncatedNote })
   }
 
-  const titlePrefix = n.buildMode === 'debug' ? '[DEV] ' : ''
+  const titlePrefix = n.buildMode === 'debug' ? '[DEV] ' : '[PROD] '
   return {
     embeds: [
       {

apps/api-server/src/error-report-eviction.test.ts

@@ -192,6 +192,43 @@ describe('tryEvict', () => {
     expect(result.newTotal).toBe(3 * GB)
   })
 
+  it('sorts both legacy and new key shapes by the embedded date', async () => {
+    // Mixed shapes: legacy (no env segment) vs new (`prod/`). The legacy 2026-01-01
+    // entry should evict before the newer `prod/2026-03-01` entry, even though the
+    // raw key strings compare differently.
+    const objs: StubObj[] = [
+      {
+        key: `${ERROR_REPORT_PREFIX}prod/2026-03-01/ERR-NEWER-u.zip`,
+        size: 2 * GB,
+        uploaded: new Date('2026-03-01'),
+      },
+      {
+        key: `${ERROR_REPORT_PREFIX}2026-01-01/ERR-OLDER-u.zip`,
+        size: 2 * GB,
+        uploaded: new Date('2026-01-01'),
+      },
+      {
+        key: `${ERROR_REPORT_PREFIX}dev/2026-04-01/ERR-EVEN-NEWER-u.zip`,
+        size: 1 * GB,
+        uploaded: new Date('2026-04-01'),
+      },
+    ]
+    const bucket = createR2(objs)
+    const kv = createKv({ [TOTAL_BYTES_KEY]: String(5 * GB) })
+
+    await tryEvict(
+      { ERROR_REPORTS_BUCKET: bucket, ERROR_REPORT_META: kv },
+      { highWatermark: 4 * GB, lowWatermark: 3 * GB },
+    )
+
+    const remaining = await bucket.list({ prefix: ERROR_REPORT_PREFIX })
+    // Oldest by embedded date (2026-01-01) should be the one gone.
+    expect(remaining.objects.map((o) => o.key).sort()).toEqual([
+      `${ERROR_REPORT_PREFIX}dev/2026-04-01/ERR-EVEN-NEWER-u.zip`,
+      `${ERROR_REPORT_PREFIX}prod/2026-03-01/ERR-NEWER-u.zip`,
+    ])
+  })
+
   it('evicts oldest by key date prefix, then upload time for ties', async () => {
     const objs: StubObj[] = [
       // Same day — two uploads with different upload times

apps/api-server/src/error-report-eviction.ts

@@ -11,7 +11,11 @@ import type { Bindings } from './types'
  * - `total_bytes` — running byte total across all bundles. Approximate, drift-prone.
  * - `eviction_in_progress` — short-lived (60 s TTL) lock to prevent concurrent eviction.
  *
- * The R2 key shape is `error-reports/{yyyy-mm-dd}/{ERR-XXXXX}-{uuid}.zip`.
+ * The R2 key shape is `error-reports/{prod|dev}/{yyyy-mm-dd}/{ERR-XXXXX}-{uuid}.zip`.
+ *
+ * Legacy shape (still present in the bucket; aged out via the 90-day R2 lifecycle):
+ * `error-reports/{yyyy-mm-dd}/{ERR-XXXXX}-{uuid}.zip`. Eviction sorts oldest-first
+ * via {@link extractDateSegment}, which handles both shapes.
  */
 
 /** Eviction starts when total bytes exceed this. */
@@ -70,6 +74,25 @@ interface ListedObject {
   uploaded: Date
 }
 
+/**
+ * Pull the `yyyy-mm-dd` date segment out of an R2 key. Handles both shapes:
+ * - new: `error-reports/{prod|dev}/yyyy-mm-dd/{id}-{uuid}.zip`
+ * - legacy: `error-reports/yyyy-mm-dd/{id}-{uuid}.zip`
+ *
+ * Returns the empty string when the key matches neither shape — that pushes
+ * unrecognized keys to the front of an ascending sort, so they get evicted first.
+ */
+export function extractDateSegment(key: string): string {
+  if (!key.startsWith(ERROR_REPORT_PREFIX)) return ''
+  const rest = key.slice(ERROR_REPORT_PREFIX.length)
+  const segments = rest.split('/')
+  // Pick the first segment that looks like a date. Both shapes have exactly one.
+  for (const segment of segments) {
+    if (/^\d{4}-\d{2}-\d{2}$/.test(segment)) return segment
+  }
+  return ''
+}
+
 /** Fetch every object under the prefix into memory. R2 list page = 1000 objects max. */
 async function listAllObjects(bucket: R2Bucket): Promise<ListedObject[]> {
   const out: ListedObject[] = []
@@ -109,11 +132,15 @@ export async function tryEvict(
   let freedBytes = 0
   try {
     const all = await listAllObjects(env.ERROR_REPORTS_BUCKET)
-    // Sort oldest first. Date prefix in the key is the primary signal (yyyy-mm-dd
-    // sorts lexically); R2 `uploaded` breaks ties for same-day uploads.
+    // Sort oldest first. The date segment inside the key is the primary signal
+    // (yyyy-mm-dd sorts lexically) — extracted via `extractDateSegment` so the
+    // sort works across both the new `{env}/{date}` layout and the legacy
+    // `{date}` layout. R2 `uploaded` breaks ties for same-day uploads.
     all.sort((a, b) => {
-      if (a.key < b.key) return -1
-      if (a.key > b.key) return 1
+      const da = extractDateSegment(a.key)
+      const db = extractDateSegment(b.key)
+      if (da < db) return -1
+      if (da > db) return 1
       return a.uploaded.getTime() - b.uploaded.getTime()
     })
 

apps/api-server/src/error-report.test.ts

@@ -13,6 +13,7 @@ function createR2(): R2Bucket & { _store: Map<string, StoredObj> } {
   const store = new Map<string, StoredObj>()
   return {
     _store: store,
+    head: (key: string) => Promise.resolve(store.has(key) ? ({ key } as unknown as R2Object) : null),
     put: async (
       key: string,
       value: ReadableStream | ArrayBuffer | Uint8Array | string,
@@ -121,6 +122,7 @@ function buildMultipart(bundleBytes: Uint8Array, meta: unknown, bundleName = 'bu
 }
 
 const validMeta = {
+  id: 'ERR-A2345',
   kind: 'user' as const,
   appVersion: '0.13.0',
   osVersion: '15.3.1',
@@ -138,18 +140,18 @@ afterEach(() => {
 })
 
 describe('POST /error-report', () => {
-  it('returns 200 with an ERR-XXXXX id on a valid upload', async () => {
+  it('returns 200 echoing the client-supplied id on a valid upload', async () => {
     const bindings = createBindings()
     const fd = buildMultipart(new Uint8Array([1, 2, 3, 4]), validMeta)
 
     const res = await app.request('/error-report', { method: 'POST', body: fd }, bindings)
 
     expect(res.status).toBe(200)
     const body = await res.json<{ id: string }>()
-    expect(body.id).toMatch(/^ERR-[23456789A-HJ-NP-Z]{5}$/)
+    expect(body.id).toBe(validMeta.id)
   })
 
-  it('writes the bundle to R2 with the expected key shape and metadata', async () => {
+  it('writes the bundle to R2 with the new env/date key shape and metadata', async () => {
     const bucket = createR2()
     const bindings = createBindings({ ERROR_REPORTS_BUCKET: bucket })
     const fd = buildMultipart(new Uint8Array([9, 9, 9]), validMeta)
@@ -158,7 +160,8 @@ describe('POST /error-report', () => {
     const { id } = await res.json<{ id: string }>()
 
     const [[key, obj]] = [...bucket._store.entries()]
-    expect(key).toMatch(new RegExp(`^error-reports/\\d{4}-\\d{2}-\\d{2}/${id}-[0-9a-f-]{36}\\.zip$`))
+    // Default `validMeta` has no `buildMode`, which is treated as `'release'` → `prod`.
+    expect(key).toMatch(new RegExp(`^error-reports/prod/\\d{4}-\\d{2}-\\d{2}/${id}-[0-9a-f-]{36}\\.zip$`))
     expect(obj.customMetadata).toMatchObject({
       id,
       kind: 'user',
@@ -170,6 +173,41 @@ describe('POST /error-report', () => {
     expect(obj.size).toBe(3)
   })
 
+  it('places debug-build uploads under the `dev/` env prefix', async () => {
+    const bucket = createR2()
+    const bindings = createBindings({ ERROR_REPORTS_BUCKET: bucket })
+    const fd = buildMultipart(new Uint8Array([1, 2, 3]), { ...validMeta, buildMode: 'debug' })
+
+    await app.request('/error-report', { method: 'POST', body: fd }, bindings)
+
+    const [[key]] = [...bucket._store.entries()]
+    expect(key).toMatch(new RegExp(`^error-reports/dev/\\d{4}-\\d{2}-\\d{2}/${validMeta.id}-[0-9a-f-]{36}\\.zip$`))
+  })
+
+  it('returns 400 when the meta id is missing', async () => {
+    const bindings = createBindings()
+    const { id: _id, ...rest } = validMeta
+    void _id
+    const fd = buildMultipart(new Uint8Array([1]), rest)
+
+    const res = await app.request('/error-report', { method: 'POST', body: fd }, bindings)
+
+    expect(res.status).toBe(400)
+    const body = await res.json<{ error: string }>()
+    expect(body.error).toBe('Invalid meta shape')
+  })
+
+  it('returns 400 when the meta id is malformed', async () => {
+    const bindings = createBindings()
+    const fd = buildMultipart(new Uint8Array([1]), { ...validMeta, id: 'ERR-LOWER' })
+
+    const res = await app.request('/error-report', { method: 'POST', body: fd }, bindings)
+
+    expect(res.status).toBe(400)
+    const body = await res.json<{ error: string }>()
+    expect(body.error).toBe('Invalid meta shape')
+  })
+
   it('returns 413 for a bundle over 10 MB', async () => {
     const bindings = createBindings()
     // 11 MB of 0s

apps/api-server/src/error-report.ts

@@ -1,7 +1,6 @@
 import { Hono } from 'hono'
 import { AwsClient } from 'aws4fetch'
 import type { Bindings } from './types'
-import { generateShortId } from './license'
 import {
   ERROR_REPORT_PREFIX,
   incrementTotalBytes,
@@ -14,13 +13,23 @@ import { postErrorReportNotification, postEvictionNotification } from './discord
 const errorReport = new Hono<{ Bindings: Bindings }>()
 
 const MAX_BUNDLE_BYTES = 10 * 1024 * 1024 // 10 MB hard cap on the multipart payload
-const SHORT_ID_LEN = 5
-const SHORT_ID_PREFIX = 'ERR'
-const COLLISION_RETRY_LIMIT = 3
 const PRESIGN_TTL_SECONDS = 7 * 24 * 60 * 60 // R2 max for presigned URLs
 const DEFAULT_BUCKET_NAME = 'cmdr-error-reports'
 
+/**
+ * Matches the client-side `ERR-XXXXX` short ID produced by
+ * `error_reporter::generate_short_id` (alphabet kept in sync in
+ * `apps/desktop/src-tauri/src/short_id.rs`).
+ */
+const SHORT_ID_PATTERN = /^ERR-[23456789ABCDEFGHJKMNPQRSTUVWXYZ]{5}$/
+
 export interface ErrorReportMeta {
+  /**
+   * Client-generated `ERR-XXXXX` shown in the UI before upload. The server uses this
+   * id as-is — the trailing UUID in the R2 key guarantees object uniqueness, so we
+   * never regenerate. The server validates the shape and rejects malformed ids.
+   */
+  id: string
   kind: 'user' | 'auto'
   /**
    * Set by the desktop client from `cfg!(debug_assertions)`. `'debug'` reports
@@ -40,6 +49,7 @@ export interface ErrorReportMeta {
 function isValidMeta(value: unknown): value is ErrorReportMeta {
   if (!value || typeof value !== 'object') return false
   const v = value as Record<string, unknown>
+  if (typeof v['id'] !== 'string' || !SHORT_ID_PATTERN.test(v['id'])) return false
   if (v['kind'] !== 'user' && v['kind'] !== 'auto') return false
   for (const k of ['appVersion', 'osVersion', 'arch', 'generatedAt']) {
     const val = v[k]
@@ -54,19 +64,18 @@ function todayDatePrefix(): string {
   return new Date().toISOString().slice(0, 10) // YYYY-MM-DD
 }
 
-/** Pick a fresh `ERR-XXXXX` short ID, retrying on R2 HEAD collision (rare). */
-async function generateUniqueId(bucket: R2Bucket, datePrefix: string): Promise<string | null> {
-  for (let i = 0; i < COLLISION_RETRY_LIMIT; i++) {
-    const id = generateShortId(SHORT_ID_PREFIX, SHORT_ID_LEN)
-    // Check anything already starting with this ID under today's prefix.
-    const list = await bucket.list({ prefix: `${ERROR_REPORT_PREFIX}${datePrefix}/${id}-`, limit: 1 })
-    if (list.objects.length === 0) return id
-  }
-  return null
+/** `'prod'` for release builds, `'dev'` for debug. Friendlier than `release`/`debug` for ops. */
+function envSegment(buildMode: 'release' | 'debug' | undefined): 'prod' | 'dev' {
+  return buildMode === 'debug' ? 'dev' : 'prod'
 }
 
-function buildR2Key(datePrefix: string, id: string, uuid: string): string {
-  return `${ERROR_REPORT_PREFIX}${datePrefix}/${id}-${uuid}.zip`
+/**
+ * R2 key shape: `error-reports/{prod|dev}/yyyy-mm-dd/{ERR-XXXXX}-{uuid}.zip`.
+ * Env first so dev and prod sort into separate sub-prefixes (eviction by oldest still
+ * works within each environment because the date segment sorts lexically).
+ */
+function buildR2Key(env: 'prod' | 'dev', datePrefix: string, id: string, uuid: string): string {
+  return `${ERROR_REPORT_PREFIX}${env}/${datePrefix}/${id}-${uuid}.zip`
 }
 
 /**
@@ -194,14 +203,18 @@ errorReport.post('/error-report', async (c) => {
     return c.json({ error: 'Invalid meta shape' }, 400)
   }
 
+  const id = meta.id
   const datePrefix = todayDatePrefix()
-  const id = await generateUniqueId(c.env.ERROR_REPORTS_BUCKET, datePrefix)
-  if (!id) {
-    return c.json({ error: 'Could not allocate ID' }, 503)
+  const env = envSegment(meta.buildMode)
+  // The trailing UUID guarantees object uniqueness on its own. On the astronomically
+  // rare (id, date, uuid) collision, retry with a fresh UUID — never a fresh id, so
+  // the user-visible id the dialog showed stays stable.
+  let key = buildR2Key(env, datePrefix, id, crypto.randomUUID())
+  for (let attempt = 0; attempt < 3; attempt++) {
+    const existing = await c.env.ERROR_REPORTS_BUCKET.head(key)
+    if (!existing) break
+    key = buildR2Key(env, datePrefix, id, crypto.randomUUID())
   }
-
-  const uuid = crypto.randomUUID()
-  const key = buildR2Key(datePrefix, id, uuid)
   const sizeBytes = bundle.size
   const uploadedUnixSeconds = Math.floor(Date.now() / 1000)