Running a flow step on one specific machine that can't host a worker (open-source file-queue bridge)

[abhinaykrupa]

Sharing a small open-source thing that came out of a Prefect-shaped problem: a flow step that must run on one specific machine, when the flow itself does not run there.

The setup that produced it: most of my pipeline is happy on ephemeral infrastructure, but a couple of steps genuinely are not portable. They need a licensed toolchain installed on one box, or a local Docker daemon, or credentials that are deliberately not in the cloud. The usual answers are a self-hosted worker on that machine, or opening a hole to it. A worker pool is the right answer when the machine is a server. It is a heavier answer than I wanted when the machine is my laptop, or a box behind NAT that will never accept an inbound connection.

cowork-to-code-bridge (MIT) is the smaller version of that. A caller enqueues a task; a daemon on the target machine picks it up, runs it, and writes the result back. Both sides only read and write files in a shared directory — queue/, results/, to_cowork/. There is no listening socket on either end and no inbound port, so it works from a laptop behind NAT, and because the state is entirely on disk it survives the daemon being restarted, the machine rebooting, or the caller timing out mid-task.

from cowork_to_code_bridge import queue_task, poll_task_result

@task
def build_on_host():
    job = queue_task(
        "scripts/run_claude.sh",
        args=["rebuild the artifact and run the smoke tests"],
        timeout=1800,
        idempotency_key=f"build-{flow_run.id}",
    )
    return job["task_id"]

@task
def collect(task_id):
    return poll_task_result(task_id)   # non-blocking, idempotent

Where it maps onto Prefect concepts, and where it does not. The enqueue/poll split is deliberately the same shape as a task that kicks off external work and a later task that collects it, so retries and a paused flow run both behave. poll_task_result being idempotent is what makes a Prefect retry safe: the retry reads the existing result rather than re-running the work. idempotency_key covers the other direction — a caller that retries the enqueue after a dropped connection gets the cached result instead of a second execution, which matters because these steps have side effects.

What it is not is a scheduler. It does no orchestration, no dependency resolution, no state tracking beyond one task's lifecycle. Prefect is the orchestrator; this is only the transport for the one hop onto a specific machine. I think that is the honest boundary, but it is also the part I would most like to be argued with about — see the question at the end.

Guardrails, because "run this on my machine" deserves scrutiny: the daemon executes only scripts from a fixed allowlist directory (resolved path checked back against that root, so ../ and symlinks out are rejected), runs as the invoking user and never sudo, bounds stdout/stderr at 64 KiB while streaming so a chatty child cannot exhaust memory, and takes a per-task max_budget_usd and permission_scope. Two known gaps I will state rather than let someone discover: task output is not redacted, and queued tasks have no maximum age — both are filed, both came out of a security review someone gave me on another thread.

Working and tested (388 tests), macOS/Linux/WSL2, pure standard library on both sides.

Repo: https://github.com/abhinaykrupa/cowork-to-code-bridge

The question I would actually value from this community: for "this step must execute on that specific host," is a self-hosted worker pool with a targeted work queue simply the correct Prefect answer, and this only earns its keep when the target is a personal machine that cannot host a worker? I would rather hear that the framework already solves this properly than keep maintaining something adjacent to it.

[bdalpe]

... I would rather hear that the framework already solves this properly than keep maintaining something adjacent to it.

Yes. Background Tasks solve this exact use case:

• https://docs.prefect.io/v3/how-to-guides/workflows/run-background-tasks
• https://docs.prefect.io/v3/advanced/background-tasks