fix: unblock SP calls for UC in production

[atilafassina]

Add on-behalf-of-user (OBO) support to the files plugin so SDK calls run as the end user, unblocking Unity Catalog volume access for OBO apps deployed on Databricks Apps.

What changed

Per-volume opt-in to OBO via a new auth field on each volume config:

files({
  volumes: {
    user_uploads: { auth: "on-behalf-of-user", policy: usersOnly() },
    shared:       { auth: "service-principal" }, // default; existing behavior
  },
})

When auth: "on-behalf-of-user":

• HTTP routes require x-forwarded-user + x-forwarded-access-token (401 in
  production if either is missing; dev mode falls back to SP identity with a
  warning).
• The identity passed to volume policies is the end user (not the SP).
• SDK calls execute via runInUserContext so UC sees the user, not the SP.
• files.auth_mode span attribute distinguishes OBO vs SP traffic in
  telemetry.
• Programmatic appKit.files("vol").asUser(req).read(...) works on any volume
  (forces OBO regardless of the volume's default auth mode).

Note

OBO read tier intentionally disables caching (the cache layer keys by user; a write by user A wouldn't invalidate user B's view of the same path). Documented in the code; future work tracked for per-(volume, path) generation counters.

What's NOT changed

• Default volume behavior is unchanged — existing apps keep auth: "service-principal" semantics with no migration.
• The static manifest still grants WRITE_VOLUME to the SP for every volume. OBO volumes need the user to also have UC permission, which has to be communicated out-of-band until the manifest schema gains a per-volume auth scope. (TODO noted in getResourceRequirements.)

Hardening

• Upload Content-Length parsing (strict regex, rejects negative / partial /
  array values; stream-level byte counter as the real gate).
• Error-message hardening: _handleApiError no longer reflects raw SDK error.message on 4xx (CWE-209); generic body, raw to server log.
• /read route is atomic: drains the body into a memory buffer up to maxReadSize; on overflow responds 413 atomically (no partial 200 body leak).
• List-cache invalidation is symmetric: root-level writes invalidate both "__root__" and connector.resolvePath("/") so ?path=/ listings stay fresh.
• _extractObiUser is strict in production (both headers required), warning + SP fallback only in dev.
• _invalidateListCache is awaited before the HTTP success response so a follow-up GET /list in the same tick can't observe stale data.

Tests

• 113 plugin tests + 28 integration tests + 33 helpers/policy
• New tests cover: OBO read/write happy path + 401 paths, OBO write cross-user freshness (cache disabled), asUser() programmatic path, files.auth_mode span attribute on every code path, root-level cache invalidation key parity.

Test plan

• Unit tests in packages/appkit/src/plugins/files/tests/
• Integration tests against a real Express app (plugin.integration.test.ts)
• pnpm -r typecheck

[Copilot]

Pull request overview

This PR adds per-volume authentication mode support to the Files plugin so Unity Catalog SDK calls can execute either as the service principal (default) or on-behalf-of-user (OBO) using forwarded user headers, unblocking UC volume access for OBO apps in production and improving telemetry + request hardening.

Changes:

• Introduces auth configuration at both plugin (IFilesConfig.auth) and volume (VolumeConfig.auth) levels, wiring HTTP routes and VolumeHandle.asUser(req) through runInUserContext for OBO execution.
• Adds files.auth_mode telemetry tagging across HTTP + programmatic code paths (implemented via AsyncLocalStorage attribute propagation into connector spans).
• Hardens HTTP behavior: stricter header/token requirements in production, safer error responses, upload Content-Length enforcement, and /read response-size limiting via maxReadSize.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
packages/appkit/src/plugins/files/types.ts	Adds new config fields (auth, maxReadSize) and updates public API docs to reflect SP vs OBO execution semantics.
packages/appkit/src/plugins/files/policy.ts	Expands FilePolicyUser docs to clarify id/isServicePrincipal behavior across auth modes and call paths.
packages/appkit/src/plugins/files/plugin.ts	Implements auth resolution, OBO context wiring (runInUserContext), telemetry attributes, query parsing hardening, cache behavior changes, and updated error handling.
packages/appkit/src/connectors/files/client.ts	Adds ALS-based span attribute propagation (runWithFilesSpanAttributes) merged into connector spans.
packages/appkit/src/plugins/files/tests/plugin.test.ts	Adds extensive unit coverage for auth inheritance, OBO identity selection, cache behavior, invalidation timing, and telemetry attributes.
packages/appkit/src/plugins/files/tests/plugin.integration.test.ts	Improves integration test reliability by using ephemeral ports and adds coverage for header-less SP-mode behavior.
docs/docs/plugins/files.md	Documents per-volume auth modes, policy-user matrix, and programmatic asUser(req) semantics.
docs/docs/api/appkit/Interface.FilePolicyUser.md	Updates generated API docs for FilePolicyUser fields and behavior matrix.
CHANGELOG.md	Adds an Unreleased entry describing new auth modes, telemetry attribute, behavior changes, and limitations.
apps/dev-playground/server/index.ts	Adds an OBO demo volume + probe route to validate end-to-end OBO behavior.
apps/dev-playground/client/src/routes/policy-matrix.route.tsx	Adds UI smoke tests/panels for the per-volume OBO demo.
apps/dev-playground/app.yaml	Adds env wiring for the OBO demo volume.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[MarioCadenas]

Agentic review — performance & placement notes (Xavier panel)

This comment summarizes three medium-severity performance / cost-of-operations findings from an automated review pass (Xavier remoras on origin/sp-files vs origin/main). They are not correctness or security blockers; they describe tradeoffs and tuning opportunities.

Where to look: PR “Files changed” → packages/appkit/src/plugins/files/plugin.ts (search _readSettings, _handleRead).

---

1) HTTP /read no longer uses read-tier cache; uses download path

Summary: /read loads bytes via connector.download + _downloadSettings. Inline comments note /read no longer participates in the read-tier cache. For service-principal volumes, hot repeated /read of the same small object may hit the SDK/network every time instead of benefiting from the read cache → higher latency, API usage, and load.

Anchor: _handleRead — block calling connector.download(getWorkspaceClient(), path) with _downloadSettings(mode) (~L821–L833 on sp-files).

---

2) /read accumulates stream chunks then Buffer.concat

Summary: The handler reads via getReader(), chunks.push(value) per read, then Buffer.concat(chunks, bytesRead). Many tiny chunks → extra allocations, a large chunks array, and GC churn, while staying within maxReadSize. Peak memory can spike briefly.

Anchor: _handleRead reader loop + Buffer.concat (~L843–L873 on sp-files).

---

3) OBO volumes: read-tier cache disabled by design

Summary: _readSettings sets cache.enabled: false when authMode === "on-behalf-of-user". Correctness motivation: cache keys tie to getCurrentUserId(), so cross-user staleness would be wrong for OBO without richer keying. Tradeoff: higher latency and backend cost on OBO-heavy workloads until e.g. per-(volume, path) generation counters (as noted in comments).

Anchor: _readSettings (isObo branch) (~L542–L555 on sp-files).

---

Bonus (docs — flagged by correctness + security remoras)

Stale NOTE on _enforcePolicy: The doc block still implies SDK execution stays on the service principal and OBO is a “later phase,” but handlers now use _runWithAuth / runInUserContext for OBO. Updating that NOTE avoids maintainer confusion (search for “still uses the service principal until a later phase”).

---

Panel verdict (context): correctness approve, security approve, performance approve — no request changes / rethink from the automated panel; items above are follow-ups / polish.