[Feat]: admin-managed LLM API keys with sequential fallback

[FireTable]

Feature: admin-managed LLM API keys with sequential fallback

Today OPENAI_API_KEY is a single env var baked into the container image. There's no way to rotate without redeploy, no way to fall back when a key gets rate-limited, and no way for a non-operator to manage credentials. This issue replaces the env-var model with an admin-managed key pool backed by Postgres, with ordered fallback so a single rate-limited key doesn't break the app.

Goals

1. Role on user — role: 'admin' | 'user', default 'user', NOT NULL. Bootstrap path is a single env var set at deploy time (no manual SQL).
2. Admin dashboard — /admin/keys (or under settings) lists, adds, edits, enables/disables, deletes LLM keys. Per-key provider + apiKey + optional base_url + priority + enabled. UI never sees plaintext key after creation (only last4 for display).
3. Sequential fallback — the backend tries keys in priority ASC order; on a retryable error (rate-limit / 5xx / network) it falls back to the next enabled key. Retryable-error classification lives in one place.
4. Self-host-first — no new service. The key pool lives in our existing Postgres. Keys encrypted at rest with an env-supplied KEK. Single VPS continues to be enough.
5. Health visibility — the dashboard shows last_used_at, last_success_at, last_error_at, last_error_message per key, plus a "test" button that pings with that key. Operators can spot a dead key without reading logs.
6. Graceful migration — when the llm_key table is empty, fall back to OPENAI_API_KEY env (preserves today's single-env-var setup so existing deployments don't break).

Out of scope

• BYOK (bring your own key) per end-user — admins configure org-wide keys only.
• Multi-provider in MVP — schema supports it via the provider column, but only openai is implemented in this issue. Anthropic / Gemini / etc. are follow-ups.
• Usage metering / cost attribution — separate issue.
• Auto-discovery of free-tier rotation / quota refresh timers.
• LiteLLM / Portkey as a sidecar — they're good products but adding another container is overkill for a single-provider pool. Revisit if we go multi-provider.

Data model

user.role

// db/schema.ts (extends lib/auth/schema.ts)
export const userRole = pgEnum("user_role", ["admin", "user"]);
export const user = pgTable("user", {
  // ...existing columns...
  role: userRole("role").notNull().default("user"),
});

Migration adds the enum + the column. The default makes it backward-compatible — every existing user becomes 'user'.

Bootstrap the first admin

INITIAL_ADMIN_EMAIL env var. On every login / session read, if the env is set AND the user has role = 'user' AND their email matches (case-insensitive), promote to 'admin'. Idempotent — safe to leave the env set forever. Document in .env.example + docs/AUTH.md.

After the first admin exists, only admins can promote others (via the dashboard — there's a "promote user to admin" action on the existing user list, gated by requireAdmin()).

llm_key table

New module lib/admin/schema.ts (mirrors lib/threads/, lib/memory/):

export const llmProvider = pgEnum("llm_provider", ["openai"]);  // extend later

export const llmKey = pgTable("llm_key", {
  id: text("id").primaryKey(),                            // nanoid, 12 chars
  label: text("label").notNull(),                         // "Prod key #1"
  provider: llmProvider("provider").notNull().default("openai"),
  // AES-GCM ciphertext; never returned to client in plaintext.
  encryptedKey: text("encrypted_key").notNull(),
  iv: text("iv").notNull(),                               // AES-GCM nonce
  keyLast4: text("key_last4").notNull(),                  // for UI; computed at create
  baseUrl: text("base_url"),                              // optional override of OPENAI_BASE_URL
  enabled: boolean("enabled").notNull().default(true),
  priority: integer("priority").notNull().default(0),     // lower = tried first
  createdByUserId: text("created_by_user_id").references(() => user.id),
  createdAt: timestamp("created_at").defaultNow().notNull(),
  updatedAt: timestamp("updated_at").defaultNow().notNull(),
  lastUsedAt: timestamp("last_used_at"),
  lastSuccessAt: timestamp("last_success_at"),
  lastErrorAt: timestamp("last_error_at"),
  lastErrorMessage: text("last_error_message"),
}, (t) => [
  index("llm_key_priority_idx").on(t.priority),
  index("llm_key_enabled_idx").on(t.enabled),
]);

userId ON DELETE SET NULL on created_by_user_id so removing the creator doesn't cascade-delete the key.

Update docs/DB.md in the same commit (rule #1 spirit).

Encryption at rest

Keys are encrypted with AES-256-GCM using a server-only KEK in env (LLM_KEY_ENCRYPTION_KEY, 32 random bytes hex). Required at startup; the app refuses to boot without it (or refuses to start the admin routes — make the failure explicit, not silent).

• IV: 12 random bytes per row, stored alongside ciphertext.
• Auth tag: included in encryptedKey (GCM packs it).
• Never decrypt on the client. The list / detail endpoints return keyLast4 only.
• Re-encryption: if the KEK ever rotates, a one-shot script re-encrypts all rows. Out of scope for this issue, but the encryption helper should be parameterized by KEK from day one.

API surface (new routes, all withAuth + new requireAdmin)

Route	Method	Auth	Notes
/api/admin/keys	GET	admin	List keys (excludes encryptedKey + iv; returns keyLast4 only).
/api/admin/keys	POST	admin	Create. Body: { label, provider, apiKey, baseUrl?, priority? }.
/api/admin/keys/[id]	PATCH	admin	Edit label / baseUrl / priority / enabled. Cannot edit apiKey here.
/api/admin/keys/[id]	DELETE	admin	Soft delete + remove from pool.
/api/admin/keys/[id]/rotate	POST	admin	Replace encryptedKey with a new apiKey.
/api/admin/keys/[id]/test	POST	admin	Pings the key (one cheap request); updates last_success_at / last_error_*.
/api/admin/users	GET	admin	List users with their role. (Read-only.)
/api/admin/users/[id]	PATCH	admin	Set role. Self-demotion blocked (you can't demote the only admin).

requireAdmin lives at lib/auth/require-admin.ts — wraps withAuth, throws 403 if user.role !== 'admin'.

Rule #9: every route wrapped in withAuth. New requireAdmin is layered on top.

Update docs/APIS.md in the same commit (rule #1).

Backend — backend/model.ts rewrite

Replace the two ChatOpenAI singletons with a chatOpenAIWithFallback(opts) helper:

async function chatOpenAIWithFallback<T>(
  opts: ChatOpenAIFields,
  invoke: (model: ChatOpenAI) => Promise<T>,
): Promise<T> {
  const keys = await getActiveLlmKeys();          // cache 30s, fall back to env when table empty
  let lastErr: unknown;
  for (const k of keys) {
    try {
      const model = makeChatOpenAI({
        ...opts,
        apiKey: decrypt(k.encryptedKey, k.iv),
        configuration: { baseURL: k.baseUrl ?? undefined },
      });
      const out = await invoke(model);
      await markKeySuccess(k.id);
      return out;
    } catch (err) {
      if (!isRetryable(err)) throw err;          // 4xx (non-429), bad-prompt, tool errors → bail
      lastErr = err;
      await markKeyError(k.id, err);
      // continue to next key
    }
  }
  throw lastErr ?? new Error("no LLM keys available");
}

isRetryable(err) returns true for: HTTP 429, 5xx, network errors, ETIMEDOUT, ECONNRESET, DNS failures. false for 400 (bad request), 401 (bad key — non-retryable, key is dead), 422 (validation), etc. Centralized so all sub-agents benefit.

getActiveLlmKeys():

1. If table has any enabled rows: return them ordered by priority ASC.
2. Else: return a synthetic single-key array from process.env.OPENAI_API_KEY (so existing deployments work unchanged). Log a one-line warning the first time the fallback path is hit per process.

Caching: in-process LRU keyed by (process.pid, "llm_keys"), 30s TTL. Avoid DB hammering on every chat turn.

triggerBackgroundAgentNode already uses runs.create — it doesn't construct a ChatOpenAI itself, so no change there. Only the per-sub-agent ChatOpenAI constructions need the wrapper.

The two compiled graphs (agent + background_agent) — the background agent doesn't call LLMs (summarize uses store, not model). So the fallback wrapper only touches the four sub-agents' call-model-node.ts. Verify.

Admin dashboard UI

New route app/admin/keys/page.tsx, server component guarded by requireAdmin (server-side redirect to /login if not signed in, to / with toast if not admin).

UI sections (mirror the existing Memory tab's chrome — same bg-muted/30 shell, same shadcn primitives per rule #11):

1. Stat card — total keys / enabled / last-error-in-last-hour. Reuse the aggregate.ts pattern from observability.
2. Keys table — columns: label, provider, keyLast4, baseUrl, priority, enabled toggle, last used, last success / last error (with hover for full error message), actions (edit / rotate / test / delete).
3. Add key — form: label, provider (dropdown, only "openai" for now), apiKey (password input — never sent back), baseUrl (optional), priority (number, default 0).
4. Empty state — "No keys configured. Add one to start using the admin-managed pool. Until then, the app will use OPENAI_API_KEY from env."

The "Users" sub-tab (under /admin/users) reuses components/settings/memory-view.tsx's table pattern — same skeleton, different data.

XSS / security

• apiKey field on the create form: <input type="password"> + autocomplete="off" + never echoed back via React state (one-shot, dispatched on submit).
• Display only keyLast4 in the list — never the full key.
• Rotate action: replace the existing key with a new one; the old ciphertext is overwritten in the same transaction.
• Delete action: hard delete the row + its ciphertext (no soft delete — there is no recovery path for an API key).

Acceptance criteria

• New user.role column with default 'user'; existing users unchanged.
• INITIAL_ADMIN_EMAIL env var promotes matching users to admin on login.
• Admin can add / edit / enable / disable / delete / rotate keys from /admin/keys.
• Backend uses the key pool in priority ASC order with sequential fallback on retryable errors.
• Empty llm_key table → app falls back to OPENAI_API_KEY env (today's behavior).
• Non-admin users hitting /admin/* get 403 (or redirect to / with toast).
• API keys are AES-256-GCM encrypted at rest; keyLast4 is the only field returned in any list / detail.
• last_used_at, last_success_at, last_error_at, last_error_message updated on every chat invocation.
• pnpm lint, pnpm typecheck, pnpm test green.
• No plaintext apiKey / encryptedKey ever appears in any log, span, or response payload. The existing FORBIDDEN regex in bulkInsertSpans already redacts api[_-]?key — verify with a span assertion test that the new key-handling path doesn't bypass it.
• docs/APIS.md updated (rule feat(001): stage 1 — user auth (Better Auth + email verification + thread ownership) #1).
• docs/AUTH.md updated with the new env var + admin role mechanics.
• docs/DB.md updated with the new tables.

TDD plan (rule #2)

1. tests/api/admin/keys.test.ts — non-admin user → 403 on every route. RED → GREEN via requireAdmin.
2. tests/lib/admin/encryption.test.ts — encrypt-then-decrypt round-trip; tampered ciphertext fails; wrong KEK fails. RED → GREEN.
3. tests/backend/model/fallback.test.ts — mock the S3… sorry, the OpenAI client; force key A to 429 once, assert key B is tried next and the result is returned from B. Force key A to 401 (non-retryable) — assert no fallback, error propagates. RED → GREEN.
4. tests/backend/model/fallback.test.ts — empty llm_key table, process.env.OPENAI_API_KEY set; assert the synthetic single-key fallback works. RED → GREEN.
5. tests/api/admin/bootstrap.test.ts — INITIAL_ADMIN_EMAIL="alice@example.com", user with that email logs in, user.role === 'admin' after. Set the env to a different email — assert no promotion. RED → GREEN.
6. tests/frontend/admin/keys.test.tsx — render the keys page with 2 mocked keys; assert keyLast4 displayed, no full key visible; click "test" → loading state → success / error state. RED → GREEN.
7. tests/observability/redaction.test.ts — assert a chat run that goes through a key-pool path produces spans whose meta.llm_kwargs does NOT contain apiKey, OPENAI_API_KEY, or the decrypted plaintext. (Carry-over from the existing redaction contract — see bulkInsertSpans FORBIDDEN regex.)

Notes for the implementer

• Single key for the duration of a chat run. The fallback happens at the request boundary — a streaming response that started with key A won't swap to key B mid-stream. This avoids partial-state confusion in the chat UI.
• Don't cache the decrypted key in module scope. Always decrypt on demand inside the loop; the encrypted-at-rest invariant is the whole point.
• Don't write your own AES wrapper — Node's built-in crypto.createCipheriv('aes-256-gcm', ...) is fine. No new dep.
• Rate-limit budget: a misbehaving admin who adds 100 keys doesn't make the app 100× slower — the cache makes the DB read cheap. But the chat invocation will try up to N keys on a single bad request. Set maxAttempts = keys.length and don't add a hard cap (the pool is small by design).
• Rule feat(backend): code sub-agent + lazy-register key-needing tools + shared UI primitives #5: comments only where the AES / OpenAI SDK quirks matter (e.g. "OpenAI SDK's apiKey is constructor-only — passing it via bind later won't work").
• Rule [Bug]: Memory tab Socials row missing email per linked provider #10: no, this is not a "lazy-registered" pattern — the admin keys table IS the registration. Document the deviation in the issue.
• Self-promotion guard: an admin can promote others, but at least one admin must remain. Server-side check before allowing self-demotion or demotion of the only admin.
• Audit log: every admin action (create / rotate / delete / role change) appends a row to a new admin_audit_log table for forensics. Out of scope for MVP — flag in follow-up.

Related

• backend/model.ts — the two ChatOpenAI singletons this issue replaces.
• lib/auth/with-auth.ts — the withAuth wrapper that requireAdmin builds on.
• lib/memory/merge.ts + backend/store.ts — existing module pattern to mirror for lib/admin/.
• lib/observability/callback-collector.ts — existing redaction that must continue to cover the new key-pool path (see FORBIDDEN regex in bulkInsertSpans).
• docs/AUTH.md — operator guide; gets the new env var + role mechanics.
• docs/DB.md — schema conventions; new tables follow the same shape.
• docs/APIS.md — new admin routes (rule feat(001): stage 1 — user auth (Better Auth + email verification + thread ownership) #1).
• .env.example — INITIAL_ADMIN_EMAIL + LLM_KEY_ENCRYPTION_KEY go here.

[richardchen874-sys]

I like the move from a single env var to an admin-managed key pool. Once an app depends on LLM calls in production, key rotation and rate-limit fallback become operational features, not just deployment details.

One thing I would add early is a usage receipt per key/provider: model, tokens, latency, error class, retry count, and fallback reason. That makes it easier to see whether the fallback chain is improving reliability or just hiding cost spikes behind another key.

If this is still being designed, is the bigger constraint rate-limit resilience, cost attribution per key, or making non-operators manage credentials safely? I am testing an OpenAI-compatible multi-model API layer focused on official Chinese models, and this fallback/key-pool pattern is a good fit for comparing providers under real limits.

[FireTable]

Status update — partially shipped under #15 (PR #35).

Most of the management plane landed as a side-effect of the per-LLM-call credit quota work; the runtime fallback chain is what's still missing. Tracking here so #14 doesn't get lost.

Already done (via #35):

• provider table with baseUrl + encrypted apiKeys[] (AES-256-GCM, KEK from env).
• Admin UI (/admin): provider / model / key CRUD. GET responses expose only the key name tail, not plaintext.
• getChatModelFromDB() resolves (providerId, modelName) from the DB, decrypts, builds a ChatOpenAI. LRU + invalidateModelCache() for admin writes.
• Dev / pre-seed fallback to OPENAI_API_KEY env so the backend boots before the seed provider is wired.

Still open (the actual #14 scope):

1. Sequential fallback chain. lib/provider/model-registry.ts:54 calls pickRandomApiKey(...) — random pick, not priority ASC. lib/credit/build-model.ts:24 is worse: it reads row.apiKeys[0] only, with no notion of fallback at all. Neither path catches RateLimitError / 5xx / network errors and walks the list.
2. Retryable-error classification. No single place that decides "this error → try next key". Needs a small wrapper, then call sites swap getChatModel → getChatModelWithFallback.
3. Health visibility (goal 5). provider row has no last_used_at / last_success_at / last_error_at / last_error_message. No "Test key" button in the admin UI.

Suggested split (two cheap PRs):

• #14a — fallback chain only. Add getChatModelWithFallback(opts) next to model-registry.ts; iterate apiKeys by priority ASC; on retryable error advance; swap callers. No schema change, no UI change. Ships the core behavior.
• #14b — health observability. Add the 4 timestamp/error columns to provider; record from the fallback wrapper; admin UI shows status per key + a "Test" button that runs a minimal chat.completions.create({ max_tokens: 1 }).

Happy to pick up #14a whenever — small diff, one file + one swap.

[FireTable]

Scope update — #14b dropped.

Keeping #14 limited to the runtime fallback chain (#14a). Health observability columns + "Test" button are out of scope; if the need surfaces, open a separate issue.

Remaining work: #14a only.

• getChatModelWithFallback(opts) next to model-registry.ts: iterate apiKeys by priority ASC, catch RateLimitError / 5xx / network, advance to next key.
• Swap call sites (7 × createAgent({llm: chatModel}) consumers in backend/model.ts and friends) from getChatModel → getChatModelWithFallback.
• No schema change, no UI change.

[richardchen874-sys]

Scope update — #14b dropped.

Keeping #14 limited to the runtime fallback chain (#14a). Health observability columns + "Test" button are out of scope; if the need surfaces, open a separate issue.

Remaining work: #14a only.

• getChatModelWithFallback(opts) next to model-registry.ts: iterate apiKeys by priority ASC, catch RateLimitError / 5xx / network, advance to next key.
• Swap call sites (7 × createAgent({llm: chatModel}) consumers in backend/model.ts and friends) from getChatModel → getChatModelWithFallback.
• No schema change, no UI change.

That scope cut is good. Keeping #14 to the runtime fallback chain makes the issue much easier to finish and review.

I’d make getChatModelWithFallback(opts) return a small receipt along with the selected model/client path, even if there is no schema/UI change yet: provider id, key name tail, attempt number, retryable error class, and final selected key. That gives you enough debugging data without turning this into health observability.

The main correctness rule is that fallback should advance only on retryable failures like rate limit, 5xx, or network errors. Auth/config errors should probably fail fast so a bad key does not silently hide setup problems.