Replace findFreePort with scoped, replay-safe attached services

[taras]

Outcome

Eliminate the race and replay defect behind #137 by removing findFreePort completely and adding a scoped, replay-safe attached-service primitive:

• provider-neutral contextual API.Service and startService();
• authenticated XMD service handshake owned by host adapters;
• service=<binding> for an invocation-local live endpoint;
• public ephemeral eval for general live reconstruction; and
• an explicit, non-delegating workflow service-denial provider.

service=<binding> v1 accepts only commands that implement the authenticated XMD service handshake protocol. It does not adapt arbitrary programs.

The built-in LlamafileProvider is removed, not migrated or deferred for restoration. A future OpenRouter integration is separate work and does not inherit a Llamafile launcher requirement.

Why the current contract is broken

findFreePort() binds port 0, closes the probe socket, and returns the number. A daemon binds later. Another process can acquire the port between those operations. #137 has reproduced both corruptions:

• the intended daemon loses the race and startup receives connection refused; and
• a foreign service owns the port and startup succeeds against the wrong process.

The durability contract is also wrong. The selected port is exported by durable eval while the daemon is a live resource reconstructed during partial replay. Replay can therefore restore an endpoint the new process does not own. Publishing a newly selected port as an ordinary durable binding causes the inverse failure: durable effect descriptions interpolate a different command and diverge from the journal.

An API that returns an unreserved port number, retries EADDRINUSE, polls a port, or guesses handshake completion from process output does not close both defects.

Public contract

Contextual service API

Add API.Service with the stable context-api name runtime.service. Export its ordinary operation as startService() from @executablemd/runtime.

interface ServiceEndpoint {
  readonly hostname: string;
  readonly port: number;
}

interface ServiceStartOptions {
  readonly command: string;
  readonly cwd?: string;
  readonly startupTimeout?: number;
}

interface ServiceAttachment {
  readonly endpoint: Readonly<ServiceEndpoint>;
}

interface ServiceHandler {
  start(options: ServiceStartOptions): Operation<ServiceAttachment>;
}

ServiceEndpoint v1 is exactly a frozen { hostname, port } object. It has no URL, credentials, token, socket path, arbitrary metadata, or provider-specific members.

startService() creates a live Effection service attachment. Its process and endpoint remain active until the attachment scope exits. The shared terminal handler throws ServiceProviderError; it never imports a host API, detects a runtime, or silently delegates to native execution.

The built-in xmd run adapters and any real-process test adapter authorize only 127.0.0.1. Unix sockets, IPv6, and externally reachable interfaces require future providers that explicitly authorize those transports. They are not accepted as options or inferred in v1.

XMD service handshake protocol v1

The host adapter starts the command with its inherited environment plus:

XMD_SERVICE_PROTOCOL=1
XMD_SERVICE_TOKEN=<32 random bytes encoded as lowercase hex>
XMD_SERVICE_HOST=127.0.0.1
XMD_SERVICE_PORT=0

The command must implement this protocol. It binds XMD_SERVICE_PORT on XMD_SERVICE_HOST; port 0 lets the same listener-owning process obtain an OS-assigned port. After the listener is active and the application is ready, it writes exactly one newline-terminated stdout record:

XMD_SERVICE_READY:{"version":1,"token":"<same token>","hostname":"127.0.0.1","port":49152}

Parse the JSON to infer its type. Accept exactly version 1, the generated token, hostname 127.0.0.1, and an integer port from 1 through 65535; reject extra or missing members. Freeze a newly constructed endpoint after validation. A matching-prefix line with malformed or incompatible data fails startup immediately. A second matching-prefix line is always an error, including after the handshake.

The adapter installs lossless stdout/stderr observation before spawn and races:

1. the one valid handshake record;
2. premature process exit;
3. the contextual/default startup timeout; and
4. scope cancellation.

Only the authenticated handshake record publishes an endpoint. Do not add port probes, health polling, stderr parsing, EADDRINUSE retry, command-specific heuristics, adapters for arbitrary commands, or weaker handshake guarantees. A program that is not handshake-compatible needs an explicit future adapter or hosting mechanism.

Use the existing scoped @effectionx/process daemon resource, ProcessApi, and Stdio middleware. Do not add Promises, module-scoped registries, sleeps, or a second process-lifecycle implementation. If lossless observation cannot be installed before spawn with the current Effectionx contract, stop and identify the missing contract rather than introducing a timing workaround. thefrontside/effectionx#235 is a lifecycle precedent, not a dependency to modify in this issue.

The daemon handle remains supervised for the complete service-attachment lifetime. If the process exits after the handshake while the scope is active, fail the owning execution immediately and tear down the remaining service attachments. Do not restart it automatically. A later resume may reconstruct a new service when partial replay reaches the service block.

Cancellation before or after the handshake terminates the process and releases the listener. Ordinary scope exit performs the same structured teardown. A teardown failure is reported without replacing an already-active execution failure.

Service output

Suppress the one valid authenticated handshake record. Forward all other stdout and stderr as live output on their corresponding streams, including output before and after the handshake.

Service output creates no durable journal event and is not buffered into a durable error. Process-exit errors may report sanitized exit status or signal metadata, but must not retain service stdout or stderr. Durable operations invoked through middleware installed around the service continue to produce their ordinary filtered journal events.

Durable, bounded service-log capture is a separate observability contract and is outside this issue.

Every protocol diagnostic is secret-safe before it reaches live output, error rendering, logging, or the journal filter. Malformed, incompatible, forged, duplicate, hostname-mismatch, and token-mismatch errors name only the categorical condition. They never quote or retain the expected token, received token, raw protocol line, parsed record, or a JSON parser message containing input. Protocol parsing failures have a sanitized cause or no cause.

Invocation-local live bindings

Each component invocation owns two environments:

• the existing durable eval environment, used by plain eval, expression evaluation, interpolation, and durable effect descriptions; and
• a fresh internal live binding overlay used only for ephemeral reconstruction in that invocation.

The live overlay is never projected into caller content, serialized, journaled, restored, or exposed through ordinary interpolation.

Consumer	Durable bindings	Live bindings
Plain eval	yes	no
Expressions	yes	no
Text/code interpolation	yes	no
Durable exec and other durable effect descriptions	yes	no
ephemeral eval	yes	yes
service=<binding> publication	no	writes live only

ephemeral eval receives a snapshot formed by merging durable bindings followed by live bindings. A live binding cannot shadow a durable binding. Its successful exports commit atomically only to the live overlay.

Do not add {server.port}, URLs, or any other raw-endpoint interpolation. If a future durable operation must refer to a changing service, it needs a stable logical-service reference resolved only during live execution; that is outside this issue.

service=<binding>

Add service=<binding> as an outermost terminal modifier:

```bash service=server exec
node handshake-compatible-server.js
```

• The parameter is required and must be a JavaScript identifier.
• service ignores exec, as daemon does; exec remains present only for executable-block detection.
• It calls startService() with the interpolated command, contextual working directory, and contextual startup timeout.
• It waits for the authenticated handshake before completing.
• It writes the exact frozen endpoint only to the current invocation's live overlay under the requested binding.
• It refuses to overwrite a durable or live binding.
• It renders nothing and appends no journal event.
• Only a later ephemeral eval in the same component invocation can read the binding.

Validate the binding name and both collision conditions before spawning. Create the service attachment in the invocation's long-lived eval scope, not the modifier block's short-lived scope:

const endpoint = unbox(
  yield* invocationEvalScope.eval(function* () {
    const service = yield* startService(options);
    return service.endpoint;
  }),
);

EvalScope.eval() returns Result<T>; unbox() propagates attachment failure and prevents a Result from entering the overlay. Returning from the modifier does not release the service. It remains active through projected-content expansion and is released during invocation teardown stage 3, after projected content and the component body settle.

Public ephemeral eval

Add ephemeral as a wrapping modifier for eval blocks:

```js ephemeral eval
// reconstruct invocation-local live state
```

```js persist ephemeral eval
// retain a resource or middleware for the invocation
```

This is a public, general reconstruction primitive. It may rebuild invocation-local bindings, resources, or middleware unrelated to services.

ephemeral eval:

• runs whenever its expansion is reached during live execution or partial replay;
• reads durable bindings plus the current invocation's live overlay;
• exports only to the live overlay;
• creates no journal entry;
• is valid only with terminal modifier eval; and
• cannot produce document output.

Completed root replay does not expand the document, so it reaches neither service nor ephemeral eval and starts no service. Plain eval remains durable: it cannot see live bindings and replays from its recorded result without executing.

ephemeral eval always returns an empty block result. Its output() binding throws EphemeralEvalOutputError immediately without retaining the supplied value. A non-nullish block return throws the same dedicated error before any exports commit; undefined and null are accepted. Its observable products are live bindings, retained resources, and middleware, never replay-dependent rendered output.

The intended attached-service provider shape is:

```bash service=server exec
node handshake-compatible-server.js
```

```js persist ephemeral eval
const endpoint = server;
yield* Sample.around({
  *sample([request], next) {
    return yield* callService(endpoint, request);
  },
});
```

<Sample>Use the currently attached service.</Sample>

The changing transport endpoint remains live. Sample and its ordinary secret-filtered journal result remain durable.

Existing daemon

Keep the current daemon modifier and its lifecycle unchanged. It remains the primitive for arbitrary long-running processes whose fixed port or other configuration is explicitly managed by the document or host.

Do not describe or test daemon as a dynamic-port discovery mechanism, a XMD service handshake mechanism, or a replay-safe service-publication mechanism.

CLI and workflow authority

Do not install the native host service adapter around all of runXmd().

• Runtime-named entrypoints pass their matching host-service installer into the runtime-neutral CLI.
• The CLI installs it only within xmd run and existing host-backed test execution scopes that need native service startup.
• Help, inspection, agent-worker, and future workflow branches do not inherit that installation merely by sharing the dispatcher.

Provide and export useWorkflowServiceDenial() from the workflow package. It installs API.Service middleware in the workflow execution scope whose start operation throws WorkflowServiceDeniedError and never calls next(). This is a security boundary: an API.Service provider inherited from the caller or ordinary CLI host scope cannot become fallback native execution. A future authorized workflow service adapter may explicitly replace the denial at the workflow adapter boundary; delegation from that adapter still terminates at the denial rather than reaching the caller's host.

The repository has no xmd workflow CLI branch yet. Do not add a placeholder branch in this issue. Make the denial provider independently testable here. #366 records that its future start and resume command scopes must install this denial before any workflow document import or expansion.

This issue adds no native process, filesystem, service, or other host capability to workflow execution.

Required module ownership

• packages/runtime/service.ts: provider-neutral structural types, API.Service, startService(), strict protocol parsing, and shared errors. It imports no host-specific API.
• packages/runtime/mod.ts: public service exports; remove the findFreePort export.
• packages/runtime/test/: provider-neutral scoped stubs only. They do not substitute for real lifecycle tests.
• Runtime-named host adapter modules adjacent to the CLI entrypoints: token generation, protocol environment, process/output integration, and loopback authorization. Shared helper code remains explicitly inside this host-adapter boundary and performs no runtime detection.
• packages/cli/src/{deno,node,bun,compiled}.ts: pass the matching host-service installer to runXmd().
• packages/cli/src/cli.ts: install that provider only inside authorized host-execution branches.
• packages/workflow/src/service-denial.ts and packages/workflow/mod.ts: provider-neutral, non-delegating workflow denial and its public export.
• packages/core/src/live-env.ts: private invocation-local live overlay with collision validation and atomic commits.
• packages/core/src/modifiers/ephemeral.ts plus eval handler/context modules: ephemeral-eval selection, merged snapshots, no-output enforcement, and live-only commits.
• packages/core/src/modifiers/service.ts: service=<binding> integration only; it uses contextual runtime operations and imports no host process/network API.

Shared runtime, core, and workflow modules cannot import Node networking/process APIs, Deno, Bun, Cloudflare, inspect globals, or detect the runtime.

Complete removal and migration scope

Delete findFreePort rather than deprecating it:

• remove packages/runtime/find-free-port.ts;
• remove runtime and core barrel exports;
• remove generated eval-module standard imports from both compilers;
• remove its unit/integration tests and all executable examples;
• remove it from architecture.md, specs/executable-mdx-spec.md, README material, smoke chapters, feature summaries, and embedded/golden content; and
• replace provider and daemon tests that depended on it instead of retaining a private copy.

Remove packages/core/components/LlamafileProvider.md and every standard-library, registry, specification, example, decision-table, and test claim that XMD currently ships that provider. Do not translate its raw command to service=<binding>, add a wrapper implicitly, retain its health-polling implementation, or create restoration scaffolding.

Replace the generic provider integration and smoke coverage with an attached-service Node fixture that binds port 0, emits the authenticated record, and uses persist ephemeral eval middleware. Preserve unrelated OllamaProvider, AnthropicProvider, and middleware-only provider examples.

Revise the executable-mdx specification as current contract, not a migration log: service API and syntax, live/durable environments, ephemeral eval, attached-service provider shape, daemon's fixed-configuration role, host wiring, replay behavior, tests, and decision table must agree. No published documentation may describe findFreePort as an available or deprecated API.

Error contract

Use distinct errors for:

• service provider absent;
• workflow service denied;
• invalid service binding name or durable/live collision;
• malformed or incompatible handshake record;
• handshake-record token or hostname mismatch;
• duplicate handshake record;
• startup timeout;
• process exit before the handshake;
• unexpected process exit after the handshake;
• teardown failure; and
• forbidden ephemeral eval output or return value.

Preserve original causes except where a cause may retain a secret-bearing protocol record or service output; protocol parse errors and service-exit errors use sanitized structured facts only. Cancellation remains cancellation, not a synthetic startup or exit error.

Implementation order

Implement as one PR against current main, with reviewable green commits in this dependency order:

1. Add provider-neutral API.Service, exact endpoint/options types, strict protocol parser, sanitized errors, and API tests.
2. Add runtime-named production host adapters and real attached-service process tests, including live output and full-lifetime supervision.
3. Add the invocation-local live overlay and public ephemeral eval, with durability, replay, projection, and no-output tests.
4. Add service=<binding> and prove attachment/teardown in the invocation eval scope.
5. Scope host-provider installation to authorized CLI branches; add and test useWorkflowServiceDenial() without creating a workflow CLI branch.
6. Remove findFreePort and LlamafileProvider; migrate generic provider, integration, smoke, cross-runtime, and embedded-content coverage to the attached-service fixture.
7. Update architecture.md, specs/executable-mdx-spec.md, runtime/core READMEs, examples, decision tables, and goldens in the same commits as their observable behavior.

Do not edit Effectionx in this PR.

Required tests

Provider-neutral API and protocol

• Missing provider fails and never returns a default service.
• The endpoint is frozen and has exactly hostname and port.
• The parser accepts the one canonical v1 record and rejects extra/missing members, bad versions, wrong token/host, non-integer/out-of-range ports, malformed JSON, and non-object JSON.
• Protocol errors do not retain either token, raw input, parsed record, or unsafe parser cause.
• Contextual middleware replaces the provider only in its Effection scope and does not leak outward.
• useWorkflowServiceDenial() blocks an inherited host provider, never delegates, and raises the dedicated workflow error.

Production host adapter

• A real attached-service Node fixture binds port 0, reports its actual loopback endpoint, serves a nonce-bearing response, and terminates on scope exit.
• Two concurrent attachments have distinct endpoints and each answers only as its own service.
• A foreign listener cannot satisfy startup.
• A command that is not handshake-compatible, exit before the handshake, malformed/forged/duplicate record, timeout, and startup cancellation all fail and release the child without probes or fallback.
• The valid handshake record is suppressed; all other stdout and stderr before and after the handshake are forwarded live on the correct stream and are absent from the durable journal.
• Diagnostics for every invalid record omit tokens and raw protocol content.
• After the handshake, unexpected process exit fails the owning execution promptly, performs remaining teardown, and does not restart the process.
• Cancellation after the handshake terminates the child and releases the listener.
• Every built-in and host-backed test adapter rejects anything other than 127.0.0.1 before spawn.

Core, durability, and replay

• service=<binding> publishes the exact current endpoint, renders nothing, and writes no journal event.
• Missing, invalid, durable-colliding, and live-colliding binding names fail before spawn.
• The service survives modifier completion and remains alive while projected content expands; invocation success, failure, and cancellation release it only at teardown stage 3.
• Projected content cannot observe the live service binding directly through eval, expressions, interpolation, or durable effects, but it can use middleware installed by persist ephemeral eval.
• ephemeral eval works as a general primitive without a service: it reads durable plus live bindings, exports only to live state, retains resources/middleware under persist, and creates no journal event.
• ephemeral eval cannot shadow durable state; failed evaluation, output(), or a non-nullish return commits no exports and leaks no supplied value in its error.
• Plain eval remains durable and cannot observe live bindings.
• Partial replay reconstructs each reached service and ephemeral eval with a new endpoint; completed root replay starts neither.
• persist ephemeral eval reinstalls middleware around the new endpoint. Durable operations through it append their normal filtered events.
• A durable effect after reconstruction cannot interpolate or serialize the endpoint and does not diverge during replay.
• Forced concurrency reproducing Flaky daemon smoke test: findFreePort→bind race causes intermittent test-deno /health Connection refused #137 cannot contact the wrong service.

Removal, compatibility, and cross-runtime coverage

• Public runtime/core exports and generated eval globals contain no findFreePort.
• Source, tests, docs, examples, smoke material, and embedded content contain no remaining available-API claim or executable use of findFreePort.
• The built-in component registry/package and documentation no longer contain LlamafileProvider.
• Existing daemon tests continue to prove arbitrary fixed-configuration process lifetime, crash propagation, and teardown without using it for port discovery or service publication.
• The attached-service embedded smoke test passes repeatedly under concurrent Deno, Node, and Bun verification.

Use production host adapters for handshake, output, replay reconstruction, concurrency, and teardown tests. In-memory middleware covers only provider-neutral contract and error behavior.

Compatibility and non-goals

• Removing findFreePort from source, exports, eval globals, tests, and documentation is an intentional breaking correction in this issue. There is no compatibility export or deprecation period.
• daemon behavior and syntax remain compatible for explicitly configured arbitrary processes.
• The built-in LlamafileProvider is removed without a restoration path in this issue.
• service=<binding> v1 does not support commands that are not handshake-compatible, health checks, separate readiness polling, port probes, stderr parsing, retries, or command-specific heuristics.
• Service endpoints never enter durable bindings, interpolation, durable effect descriptions, the journal, or the WorkflowRun database.
• Service stdout/stderr are live only. Durable log capture is not implemented.
• Stable logical-service references, automatic restart, Unix sockets, IPv6, external interfaces, and provider-specific endpoint metadata are not implemented.
• Workflow execution receives no native host-service fallback and no placeholder CLI command from this issue.

Verification

Run deno task setup only when the prepared dependency layout is absent or dependencies change; builds install nothing. After source/test changes, run all four repository checks:

1. deno task lint
2. deno task check
3. deno task test
4. deno task check:jsr

Also run the repository's concurrent cross-runtime verification that exposed #137. Do not weaken or quarantine the smoke test.

Related

• Closes Flaky daemon smoke test: findFreePort→bind race causes intermittent test-deno /health Connection refused #137.
• Add the xmd workflow start/resume filesystem vertical slice #366 must install the workflow service denial when it adds xmd workflow start and resume.
• Restore LlamafileProvider with a cooperative race-free launcher #388 records the decision not to restore the Llamafile provider.
• ✨ Add configurable Worker shutdown policies thefrontside/effectionx#235 is the contextual, scope-owned resource-lifecycle precedent.