fix(platform): make BYO storage switches and blob backfill trustworthy

[larryro]

Summary

Saving a BYO object-storage connection made every blob the org had stored before that moment unreadable, and the blob backfill that was supposed to move them copied without a content type, without verification, without deleting the source, over only two of the four ref-holding tables, and without a status fence. This PR makes the switch and the move trustworthy:

• Mixed-store reads. resolveOrgObjectStoresForRead lists own-then-default (only when the default tree names a different physical bucket); locateOrgObjectStore / locateOrgObject HEAD them in turn (no round-trip for a single-store org; the WebDAV GET reuses that HEAD instead of issuing a second one); deleteOrgObject clears every store that may hold a key. Every serve/read/delete lane (files, sandbox stage, TTS, project text, knowledge, WebDAV, erasure, blob_access) and every reclaim lane (abandoned upload intents, replacement-upload cleanup) now goes through them; mint lanes keep resolving the org's current store.
• A verified move. handleRef HEADs source and target, GETs the source with its stored Content-Type, PUTs it as-is, HEADs the copy to verify the size (a short copy is deleted and counted failed), and only then deletes the source. A target copy finishes a cut-off move only when size AND stored content type agree with the source (a source with no type compares on size); any other target copy — including the application/octet-stream copies the pre-move engine left behind — is re-copied over and the source retired after.
• Same-store refusal, proven not inferred. sameObjectStore compares an endpoint identity (URL-parsed, host case-folded, trailing slash dropped; the same bucket with no endpoint is the same AWS bucket whatever region string names it). On top of that, every run — dry runs too — probes physical identity with sharesPhysicalStore: a marker written through the target under a fresh org-namespaced key must stay invisible through the source; the marker is deleted again on every path. Either check failing the run as SAME_STORE before any source delete.
• Every ref table. Two more phases walk app.tts_audio_chunks and app.video_link_jobs; migration 0078 widens the phase CHECK (renumbered from 0077 — re-check at merge time, 0077 belongs to fix(platform): close conversation send, dedupe and scoping gaps #3225).
• A fenced writer. Every stamp and the completion/failure UPDATE are fenced on status = 'running' with RETURNING; nothing matched stops the engine without touching the terminal row; a stamp is due at least every 60 s inside a batch so the watchdog measures real liveness.
• POST /files/upload reclaims the blob it just minted when the intent write fails; the lazy sweep inside recordUploadIntent never fails the mint.
• The unused /files/upload-handoff door, createUploadHandoff, and the dead file_utils exports are gone; the two empty .catch(() => {}) in audio_preprocess.ts now warn.
• Docs (en/de/fr data-residency) describe the move the engine performs.

Findings fixed

• files-1 — read/serve/delete lanes locate a pre-switch blob in the default store after a BYO switch (locateOrgObjectStore / locateOrgObject / deleteOrgObject).
• files-2 — backfill copies with the stored content type and verifies the landed size before counting a blob migrated; an existing same-size octet-stream copy is re-copied, not trusted.
• files-v1 — backfill deletes the source after verification; a re-run finishes a move cut off between PUT and delete; source and target that are one physical bucket (by config OR by probe) refuse with SAME_STORE.
• files-3 — backfill walks tts_audio_chunks and video_link_jobs too (migration 0078 widens the phase CHECK).
• files-9 — every progress/terminal UPDATE is fenced on status = 'running'; a fenced engine stops; time-based stamps inside a batch.
• files-8 — /files/upload reclaims the minted blob when recordUploadIntent fails; the inline sweep is guarded.
• files-4 — /files/upload-handoff + createUploadHandoff + schema removed; integration-check journeys mint through /blob-upload.
• files-5 — MAX_FILE_SIZE_BYTES + parseObjectStorageSecretsJson removed.
• files-6 — chunk-directory cleanup failures in audio_preprocess.ts are logged.

Review round 1 (adversarial) — addressed

• Blocking (a) — aliased buckets pass the SAME_STORE guard and the move deletes the only copy. sameObjectStore normalises the endpoint (endpointIdentity), and runBackfill proves physical identity with sharesPhysicalStore before anything else happens (dry runs too, so a preview cannot say "nothing to move" for a run that would delete everything). Tests: same bucket spelled with a trailing slash / other host case (config), a DNS alias the config cannot see (probe, dry and real), the marker gone on every path, the failure UPDATE fenced on status = 'running', the seeded source object surviving. Commit 84a801555.
• Blocking (b) — the resume branch retired the typed source behind an octet-stream copy. s3HeadObject returns { size, contentType } (S3ObjectHead); handleRef finishes a cut-off move only when size AND type agree, else re-copies. Tests: application/pdf source beside a same-size application/octet-stream target → target re-copied as application/pdf, source deleted, migrated=1 not skipped=1; a typeless source resumes on size alone. Commit 84a801555.
• Non-blocking — sweepUploadIntents and runReplacementCleanup delete through deleteOrgObject (every store that may hold the key) instead of the mint store (5e74b6f28); getWebdavBlobUrl reuses the HEAD locateOrgObject already issued (57cac5339); stampIfDue has a fake-timer test that advances past STAMP_INTERVAL_MS inside one batch; the SAME_STORE tests assert the failure UPDATE's AND status = 'running' fence.
• Non-blocking, not taken — dry run still counts a target copy that already matches as skipped: an honest "pending source delete" number needs a new counter column + contract + UI (a migration for a status-line nuance), and counting it as a 0-byte candidate would make the dry run's candidates disagree with the real run's migrated. The docs describe the real run as deleting the source after verification, which covers the matching copies too.
• Commit headers: the three headers over 72 chars are in already-pushed history; the PR title (70 chars) is what the squash-merge keeps.

Skipped

• none

Tests & gates observed (final tree, 57cac5339)

• bunx tsc --noEmit (services/platform): TSC_OK.
• bunx oxlint --type-aware (services/platform): exit 0, no findings.
• bunx vitest --run --project server: Test Files 521 passed (521) / Tests 6140 passed (6140).
• bun run check (worktree root): Tasks: 38 successful, 40 total — the one red task is @tale/platform#test with exactly one failure, lib/engine/core/execute/execute.test.ts > subautomation > resolves name@version explicitly (Test Files 1 failed | 528 passed (529) / Tests 1 failed | 73378 passed (73379)); this branch touches nothing under lib/engine, and the file passes 3/3 reruns in isolation (Tests 29 passed (29)). turbo listed Failed: @tale/platform#test as the only failed task (38 successful of 40; the 40th is the task downstream of it).
• bun run --filter @tale/platform test:ui alone (run while the integration proof was also running): Test Files 1 failed | 455 passed (456) / Tests 1 failed | 3511 passed (3512), the one failure a Timed out in waitFor in settings-rail.test.tsx (untouched); rerun of that file alone: Test Files 1 passed (1) / Tests 2 passed (2). The branch's only app-side change is the BackfillPhase union in app/lib/backend/contract/object-storage.ts.
• bun run knip:check: exit 0 (one pre-existing configuration hint, cron-parser in services/platform/knip.config.ts, untouched).
• Real-Postgres integration proof (run-itest.sh, branch contains fix(platform): pass the corpus scope rows as one jsonb parameter #3222 so no cherry-pick): first run (alongside test:ui): RUN TRUNCATED at checkTasksCollabIntegrity — 121/134 lanes ran on a Postgres 40001 serialization failure in audit_logs lockChainHead (tasks/collab lane, 426/428 before it). Rerun alone on a quiet machine: [itest] 474/476 checks passed across 134/134 lanes, no truncation. This theme's lane both times: PASS data residency: object storage connection + blob backfill to BYO — … dry=completed/69c/0m, real=completed/39m/208253B, landed=true, sourceRetired=true, preMoved=true, servedBeforeMove=true, servedAfterMove=true, type=text/markdown→text/markdown (want kept). The two reds are outside this theme: webdav re-home (red on main before the campaign) and messages: concurrent appends each take their own slot — appends=12, failed=1 (want 0) (conversations lane, fix(platform): close conversation send, dedupe and scoping gaps #3225; it PASSED on this same tree in the first run — a concurrency flake, flagged to the coordinator).

Notes for the reviewer

• Migration renumbered 0077 → 0078 (0078_object_storage_backfill_phases.sql), assigned by the campaign coordinator; 0077 is fix(platform): close conversation send, dedupe and scoping gaps #3225's (merged). Please re-check the number at merge time.
• Reads for an org with its own bucket cost one HEAD per served blob until the backfill has moved it (and one HEAD against the own store after). Single-store orgs pay nothing. A completed non-dry backfill short-circuiting the default-store fallback is a possible follow-up, not in this PR.
• The backfill now DELETES the source after a size+type-verified copy, and refuses (SAME_STORE) when source and target are one physical bucket by config or by probe. The probe writes and deletes one marker object under <prefix>/<org>/<uuid> in the target on every run, dry runs included.
• The webdav re-home integration lane is red on main before this campaign (known, not this theme).
• Branch merged with origin/main (f02b00412, no conflicts); git merge-tree --write-tree origin/main HEAD is CLEAN at 3997f6a2a.