RFC: Owner / identity seam, and server persistence as the only durable store

[wyuc]

RFC: Owner / identity seam, and server persistence as the only durable store

Status: Draft · Area: storage, auth · Supersedes nothing; builds on #1167, #1191, #1209, #1639, #944, #945, #982

Summary

OpenMAIC answers two questions separately and neither completely:

• Where a course lives: browser IndexedDB by default, or Postgres behind /api/persistence.
• Who owns it: an anonymous anon:<uuid> cookie, a fixed PERSISTENCE_SHARED_OWNER_ID, or, for runtime data, a development bearer token plus a client-supplied x-learner-key.

The storage half is now solid (documents, runtime, assets with reclamation, per-owner quotas). The identity half is three independent resolvers, none of which is a production answer, and every deployment that needs real users has to fork the app to get one (#23, #582, Discussion #454, Discussion #404).

This RFC proposes:

1. One pluggable OwnerAuthenticator seam that every route, Server Action and background runner resolves identity through. Built-ins reproduce today's behavior, plus a trusted-proxy-header authenticator so real accounts (OIDC, SAML, LDAP via oauth2-proxy / Authelia / a gateway) work without code changes.
2. Server persistence becomes the only durable store. When DATABASE_URL is unset, the server falls back to an embedded PGlite database in a local data directory, so pnpm dev and single-container Docker stay zero-config. Browser IndexedDB keeps only device preferences and caches. Deployments on serverless platforms (Vercel) or with more than one replica must set DATABASE_URL.

Motivation

• Demand. The most repeated asks are real accounts ([Feature]: 最好能支持多用户和后台管理 #23, [Bug]: Authentication (CODE/ACCESS_CODE) fails to trigger a password prompt and erroneously overrides the Server-side API Key #287, 请问本地部署后有没有教师账号和学生账号类的功能？/Does the local deployment include features like teacher and student accounts? #325, [Feature]: 现在是本地保存数据的，能否增加全部数据在服务端保存的功能？ #582), cross-device access and no data loss on "clear browsing data" ([Bug] 课程数据未持久化到服务器磁盘 #53, [Feature]: Show disk-persisted courses on home page #214, [Bug]: 重启之后课程不见了 #321, [Feature]: 请问项目里哪里调用了api/stages/[id]和api/media-meta/[id]接口，如何可以像官方一样持久化数据库？ #1013), and sharing inside an institution ([Feature]: Classroom Sharing Functionality #237, [Feature]: 建议优化内网部署版本 #596). Browser storage cannot answer the second; the missing identity seam blocks the first and third.
• Security. GHSA-6r6j-96xr-9jxm (client-controlled x-learner-key) and the open design questions in [Task] Asset conversion follow-ups: ownership alignment, conditional document writes, aggregate probe budgets #1129 and [Task] Learner runtime endpoints are unreachable in production builds without the insecure development authenticator #1413 all come from identity being chosen by the client instead of derived by the server.
• Maintenance. Two persistence paths must behave identically; a build-time switch (NEXT_PUBLIC_PERSISTENCE) must agree with runtime configuration (DATABASE_URL, PERSISTENCE_DEV_TOKEN), and the README spends a section on what happens when they disagree. Most new features (workbench, agent runtime, materials, skills) already require the server path.

Non-goals

• A built-in user account system (sign-up, passwords, email). Accounts stay a host responsibility; OpenMAIC consumes an identity, it does not issue one.
• Organizations / RBAC policy engines. The seam carries roles; deciding which roles exist beyond a small core vocabulary is host policy.
• Teacher / student roles in core ([Feature]: 最好能支持多用户和后台管理 #23, Discussion 请问本地部署后有没有教师账号和学生账号类的功能？/Does the local deployment include features like teacher and student accounts? #325). OpenMAIC is a single surface where the same person creates a course and learns from it; splitting identities into teachers and students would add a permission model the product does not have. Deployments that need classroom roles can map them onto host roles through their authenticator.
• A native OIDC / SAML client in phase 1. Handling redirects, callbacks, token refresh, logout and key rotation inside OpenMAIC is a long-lived security surface. trustedProxyHeader (§3) covers OIDC, SAML and LDAP through standard gateways. Revisit only if running a gateway proves to be a real adoption barrier.
• Changing the capability-by-id read model for courses (fix: restore the reference's stage access model #1209). Shared links keep working without login.

Design

1. The principal

type SubjectKind = 'anonymous' | 'user' | 'device' | 'shared' | 'service';

interface OwnerPrincipal {
  /** Opaque, host-minted, stable. Stored verbatim in every owner_id / learner_key column.
   *  Core never parses it; it only enforces a charset and length guard. */
  ownerId: string;
  kind: SubjectKind;
  /** Core vocabulary: 'course:publish', 'admin'. Hosts may add their own. */
  roles: ReadonlySet<string>;
  /** How much the credential proves. Billing- or trust-sensitive hosts decide what to do with it. */
  assurance: 'verified' | 'unverified-legacy' | 'minted';
  channel?: string;                       // 'web' | 'api-key' | 'proxy' | ...
  /** An anonymous identity presented alongside a stronger one: the claim candidate (§5). */
  pendingClaim?: { fromOwnerId: string; assurance: OwnerPrincipal['assurance'] };
}

Rules:

• Opaque ids. Core must not derive meaning from the id's shape. Every ownerId.startsWith('anon:') check becomes principal.kind === 'anonymous' or a role check (course:publish). Existing anon:<uuid> and un-prefixed shared ids remain valid.
• Kind and roles are resolved per request, not stored. Runners that only have a stored id use principalFromStoredOwner(ownerId) (§4).

2. The seam

type AuthOutcome =
  | { ok: true; principal: OwnerPrincipal; setCookies?: string[] }
  | { ok: false; status: 401; code: 'INVALID_CREDENTIAL' };

interface OwnerAuthenticator {
  authenticate(req: { headers: Headers; method?: string; url?: string }): Promise<AuthOutcome>;
  /** For Server Actions / RSC where no Request exists. */
  authenticateFromContext?(): Promise<AuthOutcome>;
  /** Resolve a possibly merged-away id to its current owner (§5). */
  canonicalize?(tx: Queryable, ownerId: string): Promise<string>;
}

function configureOwnerAuthenticator(a: OwnerAuthenticator): void;   // server bootstrap, once, validated at boot

Contract:

• An invalid credential is a 401, never a fallback to anonymous. A host that presents a bad token must not be silently re-identified as a fresh anonymous owner.
• setCookies ride every response, including 4xx/5xx, so minting is not lost on an error path.
• One resolution per request, memoized. The persistence route (documents, runtime, assets), /api/stages/*, folders, materials, agent sessions and skills, stage-meta, publish/unpublish and Server Actions all go through the seam. Today there are three resolvers (agent-runtime/owner.ts, the Server Action copy in workspace-actions.ts, and persistence/server-auth.ts); they are folded into it.
• Edge-safe minting. Concurrent first-load requests must not each mint a different anonymous owner. The seam exposes a Web-Crypto-only mintAnonymous() that middleware may call on document requests.
• Cookie parsing lives only inside the authenticator. No other module reads the owner cookie. This lets hosts use their own cookie formats without clashing with the default.

3. Built-in authenticators

Authenticator	Use	Behavior
anonymousCookie (default)	Personal and local use	Today's anon:<uuid> httpOnly cookie. kind: 'anonymous', no course:publish.
sharedTeam	One team behind ACCESS_CODE	Today's PERSISTENCE_SHARED_OWNER_ID, still requires ACCESS_CODE. kind: 'shared', may publish.
trustedProxyHeader	Real accounts via a gateway	Reads a user id (and optional roles) from configured headers such as X-Forwarded-User / X-Forwarded-Groups, only from requests whose peer is in a configured trusted-proxy CIDR list, or which carry a configured shared secret header. Anything else is a 401. kind: 'user', assurance: 'verified'.

trustedProxyHeader is the supported way to get real accounts: a gateway (oauth2-proxy, Authelia, Keycloak Gatekeeper, an institutional reverse proxy) performs the sign-in against the organization's identity provider and forwards the verified user in a header. OpenMAIC never handles passwords or tokens. The trust boundary is the pairing "only accept these headers from the gateway": deployments must restrict the app to the gateway's network, or configure the shared secret header, and the authenticator refuses to start without one of the two.

Hosts with their own identity (platform JWTs, API keys, device bindings) implement OwnerAuthenticator directly and register it at boot. That is the supported path instead of forking routes.

4. Runtime data, assets and background work

• Runtime learner key = principal.ownerId in server mode. x-learner-key and PERSISTENCE_DEV_TOKEN are removed from the server path. This closes the GHSA-6r6j class and [Task] Learner runtime endpoints are unreachable in production builds without the insecure development authenticator #1413.
• Assets get a per-owner principal derived from the same resolution, instead of one shared asset principal. Reads stay capability-by-id so shared courses render ([Task] Asset conversion follow-ups: ownership alignment, conditional document writes, aggregate probe budgets #1129).
• Runners work from stored ids. An agent or job runner uses the ownerId recorded on its durable row, calls principalFromStoredOwner(ownerId) to get kind without re-authenticating, and calls canonicalize before creating rows, because a claim (§5) can land mid-run. Runners never build an id by string concatenation.

5. Claiming anonymous work on sign-in

When a request carries a verified identity and a pendingClaim from an anonymous cookie, the host may call:

registerClaimParticipant({ name, rekey(tx, from, to) });  // core registers its own tables
claimOwner(from, to);                                      // one transaction

• One transaction across all participants, serialized by a per-owner advisory lock (lockOwnerIdentity), with a fixed global participant order to avoid lock-order deadlocks.
• Core registers: stage_meta, folders, owner materials, agent sessions and skills, runtime learner rows, asset ownership. Hosts register their own tables.
• Core records the merge in an owner_merges table. canonicalize follows it, and document creation under a merged-away id is refused.
• Runtime mergeLearner becomes a participant of the same transaction. The merge authorization in the storage handler becomes: allowed only for pendingClaim.fromOwnerId → principal.ownerId where the target is not anonymous.
• Whether claim runs automatically or on an explicit action after login is host policy.

6. Tenancy lives in stage_meta only

stage_meta already answers ownership, visibility and tombstones for every access decision (stage-access.ts). document_stages.owner_id duplicates it, and a second tenancy column doubles what a claim has to re-key. Proposal: PgDocumentStore becomes tenant-agnostic; owner scoping is applied through stage_meta. Migration: stop writing document_stages.owner_id, keep the column nullable for one release, backfill stage_meta from it (the backfill already exists), then drop it.

7. Host extension hooks

So hosts can add product behavior without forking the persistence route:

Hook	Purpose
onCreate(tx, principal, stageId) / authorizeCreate(tx, principal, stageId)	Side effects and refusals at document creation, inside the create transaction
LibraryProvider.list(principal)	What GET /api/stages lists. Default: owned courses. A host may list saved or shared courses too.
beforeAssetAllocate(principal, req)	Rate limits / quotas on uploads. Returns a Response to refuse.
configureAssetByteStore(factory)	Replaces the hard-wired ASSET_S3_BUCKET check so any byte store (S3, S3-compatible, other object stores) plugs in for both the route and the collector.

8. Server persistence as the only durable store

• Default backend: no DATABASE_URL → embedded PGlite at ./data/pglite (configurable), behind a pg-compatible adapter that serializes transactions on PGlite's single session. A lock file prevents two processes from opening the same directory; the second process fails at boot.
• OPENMAIC_REQUIRE_DATABASE_URL=1 makes a missing DATABASE_URL a boot error instead of a silent local database. Recommended for every hosted deployment.
• Build-time switch removed. NEXT_PUBLIC_PERSISTENCE and NEXT_PUBLIC_PERSISTENCE_TOKEN go away; the client always uses the HTTP seams.
• Library without the agent runtime. GET /api/stages must work whenever persistence is available, not only when OPENMAIC_AGENT_RUNTIME_ENABLED is on.
• Browser data import. For N releases, the client enumerates IndexedDB courses that are not on the server, shows them in the library, and migrates them with their media and folder membership on open or via "import all". The existing lazy migration (lib/document-store/migration.ts) is the base; media conversion and folder migration are added.
• What stays in the browser: settings, playback position, UI state, caches.

Measured cost of the embedded default (spike, single machine): first open of a fresh data dir 0.6–0.8 s, boot to first persistence response 1.9 s, server RSS 167 MB → 470–560 MB, 27 MB on disk empty, +21.6 MB standalone output. All requests queue behind the single session, so the embedded default is for personal and small-team use.

Default-deployment exposure

With browser storage gone, a default deployment stores everyone's courses on the server under per-browser anonymous owners. Reads remain capability-by-id with unguessable server-issued ids (#1209, #1489); listing and writes are owner-scoped. A deployment reachable from the internet should set ACCESS_CODE or an authenticator; the docs and the first-run banner say so.

Phasing

Phase	Change	Data migration
1	OwnerAuthenticator seam with anonymousCookie + sharedTeam reproducing today; fold the three resolvers; replace anon: prefix checks with kind / roles	None
2	Runtime learner key from the principal; per-owner asset principal; trustedProxyHeader; runtime tombstone guard	None
3	Host hooks (§7)	None
4	PGlite default, OPENMAIC_REQUIRE_DATABASE_URL, remove NEXT_PUBLIC_PERSISTENCE, library without agent runtime, browser import	Browser → server, client-driven
5	Tenancy in stage_meta only	Stop writing document_stages.owner_id, then drop
6	Claim registry, owner_merges, identity lock	New table
7	Versioned, ordered SQL migrations per store alongside ensure*	None for hosts using ensure*
8	Remove the Dexie persistence path	After the import window

Open questions

1. Folders: keep folders as "owned courses only", or move to a membership table keyed (owner_id, stage_id) so a library can hold saved courses too?
2. Import window length before the Dexie path is removed.
3. Claim trigger: automatic when pendingClaim is present, or always an explicit user action?

References

• Identity / ownership: [Feature]: 最好能支持多用户和后台管理 #23, [Feature]: Classroom Sharing Functionality #237, [Feature]: 现在是本地保存数据的，能否增加全部数据在服务端保存的功能？ #582, [Feature]: UI上支持选择课程本地持久化 #585, [Feature]: 建议优化内网部署版本 #596, [Task] Asset conversion follow-ups: ownership alignment, conditional document writes, aggregate probe budgets #1129, [Task] Learner runtime endpoints are unreachable in production builds without the insecure development authenticator #1413, Single-tenant deployments: course list is empty in every other browser (anonymous cookie owner), publish refused #1550, [settings] Token plans and personal keys are mutually exclusive on shared providers (single apiKey slot) #1645; Discussions 请问本地部署后有没有教师账号和学生账号类的功能？/Does the local deployment include features like teacher and student accounts? #325, OpenMAIC Enterprise Edition #404, OpenMaic Multi-user fork for school managment #454, Feature Proposal: FERPA-Compliant Content Handling for U.S. Educational Deployments #537
• Decisions this builds on: feat(runtime): device-anonymous learner identity + RuntimeStore app bootstrap (#869 Part C, step 1) #885, feat(runtime): injectable RuntimeStore backend + learner-key provider (#939 Part A) #944, feat(storage): runtime reference server — auth-derived learnerKey (#939 Part D) #945, feat(storage): server-backed documents — HTTP contract, Postgres backend, one reference server for both stores #979, feat(persistence): one-command server-backed stack (compose profile + embedded API + docs) #982, feat(agent): adapt the agent-session store and runtime foundations #1167, feat(agent): add agent session and owner event streams #1170, feat(storage): add an ownership scope to stage documents #1191, fix: restore the reference's stage access model #1209, feat(api): folder routes, stage-meta viewer surfaces, and the material upload contract #1215, fix(workbench): PG-mode home listing via owner stages; keep the interrupted terminal course card #1218, feat(media): write generated media through the asset pool under server-backed persistence #1392, feat(storage): give the server the asset entry lifecycle (#1007 amendment, part 1) #1472, feat(persistence): turn on the server-owned asset lifecycle and release assets on course deletion (#1007 amendment, part 2) #1473, fix: enable Pro workbench flag in Docker builds; allow non-TLS localhost cookie #1484, fix(classroom): generate classroom ids server-side and create files exclusively #1489, feat(persistence): resolve single-tenant requests to a shared owner id [AI-assisted] #1639
• Browser storage pain: [Bug] 课程数据未持久化到服务器磁盘 #53, [Feature]: Show disk-persisted courses on home page #214, fix: persist classroom to server for cross-device access #218, [Bug]: 重启之后课程不见了 #321, [Bug]: UnknownError: Error preparing Blob/File data to be stored in object store — fails on all WebKit browsers #910, PDF images can disappear when session IDs contain underscores #919, [Feature]: 请问项目里哪里调用了api/stages/[id]和api/media-meta/[id]接口，如何可以像官方一样持久化数据库？ #1013
• Security: GHSA-6r6j-96xr-9jxm, GHSA-m9pr-3435-gmfp

[wyuc]

Phased work is tracked in #1658.

[wyuc]

Progress update on the phased work tracked in #1658. Phases 1–3 have landed on the feat/owner-identity-seam integration branch (#1659, #1660, #1661); the branch reaches main as one PR at the end.

One deviation from the design above, in §3 (trustedProxyHeader):

No CIDR allowlist. A shared secret is required. Next.js does not expose the TCP peer address to route handlers, middleware or Server Actions (NextRequest.ip was removed in Next 15). The only use of the socket address is x-forwarded-for ??= socket.remoteAddress, so a client-sent X-Forwarded-For wins. Trusting "requests from the gateway's network" by forwarded headers would be spoofable. The authenticator therefore requires TRUSTED_PROXY_SECRET (at least 32 printable ASCII characters). The gateway injects it in a configurable header, and it is compared in constant time. Without the secret the server refuses to start. The docs also require the app to be reachable only through the gateway, and the gateway to strip client-supplied copies of the user, groups and secret headers. The README has a verified oauth2-proxy (v7.14+) example.

Other decisions made during implementation, for reviewers of the final PR:

• Runtime learner keys and asset partitions are derived from the resolved owner. The development bearer token and the client-supplied x-learner-key are gone. Existing runtime sessions from the development-grade server persistence mode are not migrated, because a migration would have to trust a client-supplied key.
• Asset reads stay capability-by-id only for committed assets that the owner's own live course references. Pending allocations stay private. A document that names another owner's asset id neither commits it nor exposes it.
• admin from a gateway groups header is trusted on the strength of the secret alone, which the server warns about at boot. No admin surface exists yet; revisit when one is added.

Next phases: host extension hooks, tenancy in stage_meta only, and the claim registry.

[wyuc]

Design update to §3 ("Built-in authenticators") before the integration branch lands on main (#1669).

One identity entry point, composable auth methods, no built-in gateway authenticator (#1676):

• Composable auth methods. A deployment registers an ordered list of auth methods with configureOwnerAuthentication({ methods, anonymousFallback }). It no longer picks a single authenticator. Each method answers one of:

  • authenticated: the first one wins;
  • invalid: a 401 at once, never downgraded to anonymous;
  • not-applicable: move on to the next method.

  If no method applies, the anonymous cookie is the fallback, which can be turned off.

• Built-ins. The anonymous cookie and the shared team owner remain built-in methods. With no configuration, behavior is exactly what deployments have today.

• No gateway authenticator in core. The trustedProxyHeader built-in from phase 3, with its shared-secret header, OWNER_AUTHENTICATOR and TRUSTED_PROXY_*, is removed. Setting any of those variables now fails boot, so no deployment silently runs anonymous.

  • The reason: a header trusted only on the strength of a static secret is weak. Each built-in login mechanism would also be another security-sensitive surface for core to maintain.
  • Instead, the README has a recipe that verifies a gateway-forwarded, IdP-signed JWT against the provider's JWKS. It works with oauth2-proxy --pass-authorization-header, Cloudflare Access and Cloud IAP.
  • Host method files go in lib/server/identity/host/.

• Claims work with any host method. Core attaches pendingClaim whenever a non-anonymous host method authenticates a request that also carries a valid anonymous cookie. Claims are no longer tied to one built-in.

The CIDR-vs-secret deviation described in my previous update no longer applies, since the gateway authenticator is gone.

[wyuc]

Update: the identity, tenancy and claims parts of this RFC landed on main in #1669 and will ship in the next release. The final design differs from the original §3 as described in my previous comment: a single identity entry point with composable auth methods, and no built-in gateway authenticator (a gateway-JWT recipe is documented instead). The storage parts (embedded PGlite default, retiring browser persistence) remain open and are tracked in #1658.