Spike: bundle workerd into the compiled XMD host

[taras]

Spike

Prove whether Executable.md can ship a single compiled xmd executable that contains a Cloudflare Worker and a platform-specific workerd, launches that Worker locally, and preserves a Durable Object Workspace across xmd process restarts.

This spike produces evidence for the bundled-local-host option in #346. It does not select that topology or implement the provider-neutral <Workspace> contract in #218.

Questions to answer

1. Can deno compile include the Worker bundle, workerd configuration, and the correct workerd executable for each XMD release target?
2. Can the compiled host materialize and supervise workerd without requiring Deno, Node, Wrangler, or an installation step on the user's machine?
3. Can a local Durable Object retain SQLite state when both xmd and workerd stop and restart against the same explicit data directory?
4. Does the pinned @cloudflare/computer release construct its filesystem-only Workspace in standalone workerd using Durable Object storage?
5. Which workspace.runtime backends actually work in standalone workerd?
6. What Cloudflare-hosted behavior is absent or materially different in the bundled runtime?

Experimental slices

1. Packaging and supervision

• Add a minimal Worker with one Durable Object and a deterministic local identity.
• Bundle its JavaScript, workerd configuration, and one host-platform workerd executable into a compiled proof executable.
• Materialize the executable into a content-addressed/versioned cache location, validate its digest, and launch it on loopback.
• Use an explicit persistent state directory and an ephemeral control endpoint.
• Stop the proof executable cleanly and prove its child workerd also stops.
• Restart it without Node, Wrangler, or Deno installed in the test environment.

2. Durable Object persistence

• Write state through the Durable Object.
• Terminate both processes.
• Restart against the same state directory and identity.
• Read the same state.
• Restart against a different state directory or identity and prove isolation.

3. Cloudflare Computer filesystem

• Pin the currently tested @cloudflare/computer version; do not test against a floating tag.
• Construct a filesystem-only Workspace from ctx.storage.
• Exercise mkdir, writeFile, readFile, readdir, and rm through the Worker boundary.
• Restart both processes and prove the resulting filesystem frontier is preserved, including a create/delete/create sequence.

4. Execution backends

Test each shipped backend independently:

• worker shell;
• isolated JavaScript Worker; and
• Cloudflare Container.

For each backend, record one of: works under standalone workerd; unavailable because a named binding/runtime facility is absent; or fails because of a reproducible package/runtime defect. Do not add an XMD-specific substitute merely to make the spike pass.

5. Compatibility and operational evidence

Record:

• proof executable size and extracted runtime size;
• cold-start and restart behavior;
• supported release targets and any libc/OS constraints;
• cache location, state location, locking, port selection, and concurrent invocation behavior;
• upgrade behavior when bundled workerd, compatibility date, Worker bundle, or Computer schema changes;
• how stderr, cancellation, forced termination, corrupt state, and orphaned child processes surface; and
• security consequences of running workerd locally, including its documented lack of a hardened sandbox.

Acceptance

• A committed, reproducible spike builds and runs from repository tasks without changing the production XMD contract.
• The generated proof artifact is a single distributable executable for at least the current host platform.
• The artifact starts a bundled Worker under workerd without an external runtime installation.
• Durable Object state and the Cloudflare Computer filesystem survive a complete stop/restart against the same identity and state directory.
• Identity and state-directory isolation are demonstrated.
• Child-process teardown is demonstrated for success, failure, and cancellation.
• Every Computer execution backend has an evidence-backed support result.
• The spike documents exactly which results are representative of deployed Cloudflare and which are local-only.
• Define the local Workspace host topology #346 is updated from the evidence: select the bundled-local-host topology, reject it, or explicitly limit it to a supported subset.
• Implement retained Workflow Workspace restoration #218 is changed only after Define the local Workspace host topology #346 records that decision.

Non-goals

• Productionizing the Workspace provider.
• Defining the public <Workspace> API.
• Claiming equivalence with Cloudflare's distributed Durable Object service.
• Adding snapshot, rollback, or arbitrary journal-prefix rewind.
• Implementing a replacement execution backend when standalone workerd lacks one.
• Shipping all release platforms before the host-platform proof establishes feasibility.

Dependencies

• Informs and blocks the decision in Define the local Workspace host topology #346.
• Define the local Workspace host topology #346 continues to block Implement retained Workflow Workspace restoration #218.
• End goal: Document adversarial implementation workflow WIP #181.

[taras]

Spike complete — PR #348 (spikes/347-workerd/, runs via deno task spike:347). Answers to the six questions, full detail in evidence/EVIDENCE.md:

1. Packaging: yes. deno compile --include carries the workerd binary + Worker bundle + capnp configs. The five npm platform packages (@cloudflare/workerd-*) map 1:1 onto our five release targets; digests pinned in manifest.ts. Proof executable: 191 MB (109 MB of it workerd; ~30-40 MB per artifact compressed). Linux needs glibc 2.35+, no musl build exists.
2. Materialize + supervise: yes. Version-addressed cache with sha256 validation, ephemeral loopback port + readiness via --control-fd, clean SIGTERM teardown proven by child-pid liveness probes. A test runs the whole cycle with PATH=/usr/bin:/bin and a fresh HOME — no Deno, Node, or Wrangler. Warm cycle 0.27 s; cold materialization 1.46 s.
3. DO persistence: yes. localDisk DO storage (the same mechanism miniflare uses) persists KV + SQL across full stop/restart; state-dir and idFromName identity isolation both asserted. Caveat: the schema marks localDisk experimental, subject to backwards-incompatible change.
4. Computer 0.1.1 filesystem Workspace: yes under standalone workerd (nodejs_compat; only @platformatic/vfs actually loads). Frontier survives restarts incl. create→delete→create. Its schema refuses downgrades (vfs_meta.schema_version) — exact pinning is mandatory.
5. Backends: all three work. worker-shell and worker-javascript run in-tests (need a workerLoader binding, a WorkspaceServiceProxy re-export for the ctx.exports loopback, and waitUntil wiring). The Container backend works through containerEngine = localDocker — computerd + egress-proxy sidecar, bidirectional /workspace sync, frontier restored into a brand-new container after restart — but requires a running Docker daemon, an x64-only computerd image (Rosetta on Apple Silicon), and workerd leaks its containers on SIGTERM (host must reap).
6. Local vs deployed: no hardened sandbox (workerd's own warning); KV/R2/Queues/Cache/D1 are protocol shells we'd have to implement; containers are local-Docker-only; one process owns each DO. Representative: Workers JS surface, DO identity + SQLite semantics, Computer's Workspace schema and backend contracts.

Notable operational findings: SIGKILL of the host orphans workerd (no control-fd-close exit) — a production host needs a kernel-backed tie or startup reaper; @effectionx/process daemon teardown alone never terminated the child on natural exit under Deno (explicit child SIGTERM ensure required — candidate upstream issue); workerd 1.20260804.1 rejects the documented two-Config-constants-per-file selection; no state-directory locking exists — concurrent hosts on one state dir would collide.

Evidence for the #346 decision posted there.

[taras]

Spike complete. PR #348 preserves the reproducible bundled-workerd evidence. The topology decision in #346 / draft PR #358 does not bundle workerd in the initial local host, while retaining this path as evidence for a later provider.