feat(worker): sync app source per-file from Hypha instead of Ray GCS

[nilsmechtel]

Summary

Move BioEngine app-source transport off the ephemeral Ray GCS onto durable Hypha per-file download with remote-timestamp incremental sync. Replicas now materialise <app_dir>/source/ by syncing files directly from the Hypha artifact (keyed on each file's last_modified), instead of pulling a gcs://_ray_pkg_<hash>.zip package the introspect task uploaded.

Why

The in-memory Ray GCS intermediary was the root of three problems, all hit in production on 2026-07-22:

• Head-restart outage — a Ray head restart wipes the GCS, so the cached built_app.app_source_uri package dangles (package prematurely deleted from GCS) and redeploys wedge (both prod apps were down until a manual pod kill).
• Head-node OOM — a package accumulates per app×version in the head's memory-backed store (a likely driver of the biweekly head restarts).
• Redeploy stale-code footgun — a redeploy reusing a cached/stale package served old code.

Hypha is already a hard dependency (a BioEngine app can't register its service without it), so pulling source from Hypha at replica start adds no new availability surface — if Hypha is unreachable the deployment fails and auto_redeploy retries.

What changed

• bioengine/_app/replica_init.py — per-file incremental sync: _list_source_files (recursive /files listing with last_modified), _download_file (streamed per-file GET), _sync_source_from_hypha (snapshot diff via .source_snapshot.json → download changed/new, delete removed, reconcile the whole tree so migrated app_dirs shed GCS-era orphans). The sys.meta_path finder is now the sole source path — the app source is no longer shipped as a Ray py_module.
• bioengine/_app/bootstrap.py / bioengine/apps/builder.py — deleted _package_source_to_ray_gcs; introspect returns {spec} only; removed all app_source_uri / BIOENGINE_APP_SOURCE_URI plumbing. Read token TTL → 30 days (replicas pull for their lifetime; public artifacts stay anonymous). The bioengine framework package stays on Ray py_modules — it's the bootstrap that runs the sync (chicken-egg), is re-uploaded on every reconnect, and is content-hash-dedup'd (not the OOM driver).
• bioengine/apps/manager.py / cluster/ray_cluster.py / worker/worker.py — kept the head-restart in-place recovery this uncovered: Serve-client reset on reconnect, controller-loss detect+redeploy with cooldown, is_ready liveness backstop, exc_info diagnostics, and the stop_app pop-before-serve.delete race fix. (An earlier rebuild-on-redeploy approach was reverted — durable Hypha source makes the cached built_app re-submittable, so it's unnecessary.)

Validation

83 unit tests pass. End-to-end on a persistent standalone Ray head (dev image 0.11.31-dev6), both external-cluster and single-machine modes:

Check	Result
Monolithic entry (sibling import before import bioengine), py_modules channel removed	✅ RUNNING, serves sibling code — finder is sole path
Composed app (EntryApp↔RuntimeApp) + child-deployment sibling import	✅ works
Per-file Hypha sync on introspect+build+replica+proxy; no app-source _ray_pkg on head	✅
Redeploy 0.1.0→0.2.0 picks up changed code	✅ new code served
Head-restart recovery	✅ in-place ~74s, correct code, zero "prematurely deleted from GCS"
Single-machine mode	✅ RUNNING + serves

Known limitation / follow-ups

• Cross-version incremental isn't there yet: upload_app re-uploads every file, so all get fresh last_modified on a commit → a version bump re-downloads all files. Same-version re-syncs (replica restart / autoscale / head-restart) are incremental (0 downloads). Full "big weights not re-fetched on a small change" needs an upload-side change (skip unchanged files) — tracked as a follow-up. Moot for model-runner (weights come from HF, not the artifact).
• Reconsider the biweekly Ray-head restarts now that GCS no longer accumulates app packages.

Rollout

Draft until model-runner is validated on a dev image (worker change; even-clam drives the redeploy-cutover check on an isolated dev worker). pyproject.toml/_version.py version bump lands as the final commit before gh pr ready.

🤖 Generated with Claude Code