feat(furrow): reach a run's live workspace from anywhere - #130
feat(furrow): reach a run's live workspace from anywhere#130AbirAbbas wants to merge 19 commits into
Conversation
Defines the one-way surface orchestration depends on: a Handle that travels in reasoner results, a registry Entry mapping a control-plane run ID to the workspace on disk, and an Attacher whose nil implementation no-ops. Availability is discovered by a caller finding a handle, never by asking. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
furrowd serves one run's encrypted blob store over TLS by spawning `furrow __remote` behind a token check, and furrow-dial stands in as FURROW_SSH_COMMAND on the caller's side so a stock `furrow clone` reaches it with no patches to furrow. This is deliberately not sshd. The cloud image runs the control plane and its agent nodes in one container as root, so an ssh key there would be whole-box access to every run and every secret on the volume. A furrowd token unlocks a single run's data root, and what crosses the wire is ciphertext either way. The namespace is passed through to the child rather than checked against the registry: furrow blinds it (keyed BLAKE3) before it ever leaves the client, so the node cannot recompute it, and comparing the two rejected every real clone. The token is what scopes the connection; the namespace only selects a directory beneath the root it already pins, so it is charset-validated instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A build now attaches its clone to furrow, republishes at every DAG level
boundary and at completion, and returns a workspace_handle in its result. A
caller that started the run can use that handle to clone the workspace and watch
it change while the build is still going — including the uncommitted edits,
untracked files and dev state that a git push would never carry.
Availability is discovered, never configured. The handle is present when the
mirror works and absent when it does not, so there is no flag to set, nothing to
probe, and no new failure a user has to understand. Every unavailable path — no
binary, no .git, attach or publish failing, the feature switched off — degrades
to exactly today's behaviour. A nil Attacher disables all of it, which is the
default.
Two things the mocked tests could not have caught, both found by running the
real binary:
- The capture policy has to be written as `exclude <subtree>` lines. Bare
names are rejected, `watch` then fails, and the mirror is off for every
build with only a debug line to show for it. The exact bytes are pinned in
a unit test now, and integration_test.go drives the real binary end to end
so the whole CLI contract is checked rather than assumed.
- .furrowpolicy lands in the repository the agent is working in, so it joins
.artifacts/ and .worktrees/ in .git/info/exclude. Otherwise it shows up in
`git status` and any acceptance criterion about a clean tree is
unsatisfiable.
The run registry closes a real gap: SWE-AF knew its workspaces only by a locally
generated build ID, with nothing tying them to the control plane's run ID. That
mapping is what makes a workspace addressable from outside at all, and the
hourly sweeper bounds what it costs to keep (72h / 20GB by default) — furrow
never prunes a remote on its own.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
furrowd existed but nothing started it, which left the remote path unreachable in exactly the deployment it was built for: in the cloud the control plane and its agent nodes share one container, so there is no separate service to run it. The node supervises it the same way it already supervises the coding engine — backoff, a give-up threshold, and a process-group kill so nothing outlives the node. It stays inert unless there is something to serve: the mirror has to be enabled, FURROW_PUBLIC_ADDR has to name an address the daemon can advertise, and a furrowd binary has to resolve. Any of those missing is the normal case on a developer's machine, so it is a debug line rather than a warning. Verified against a live node rather than only a fake: furrowd comes up on its port, speaks TLS, refuses an unauthenticated peer, and goes away when the node does. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The manager held a single lock for the whole of Attach and Publish, including the furrow invocations inside them. A node serves several builds at once, and an initial capture of a large repository is slow, so one build attaching could park every other build's publish behind it — in a feature whose entire purpose is to let parallel agents be watched while they work. The lock now guards the registry only, and per-run locks provide the ordering that actually matters: two calls for the same run must not both pair it, but different runs have no reason to wait on each other. Publish also re-checks the run still exists before recording a timestamp, so a sweep that retires a run mid-push cannot resurrect a row whose store is already gone. The test parks two publishes inside a fake furrow until both have arrived, which can only happen if they run concurrently — verified to fail against the previous locking and pass against this one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The hourly sweep held the registry lock while deleting store directories and walking the whole store to measure it — the same mistake the previous commit fixed for attach and publish, and a worse one, because retiring a run whose store is gigabytes can take a while and every attach and publish on the node waits behind it. Deletion and measurement now happen with no lock held. Each run is retired under its own lock, so a publish already in flight finishes rather than pushing into a directory being deleted, and staleness is re-checked there: a run that became active between the scan and the retire is left alone. Files go first and the registry row second, so a failed delete leaves a row that the next sweep retries instead of orphaning a store nothing points at. The size pass stops if the run it picked survives its re-check, since measuring again would keep choosing the same victim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e it The handle advertised the directory that holds every run's store rather than the one belonging to this run. Pairing with it finds no workspace — `sync --pull` fails with "sync remote has no published HEAD" — so the local path, which is the default on a developer's machine whenever no public address is configured, could not actually be used. Over the network the address was always right, because furrowd resolves the store from the token. The integration test now builds every argument from the handle instead of assembling paths of its own. A consumer only ever has the handle, so a test that reaches around it can pass while the handle itself is unusable — which is exactly what happened here. Verified to fail against the old value. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Reviewer note on the transport's threat model, since the client skips certificate verification and that deserves an explicit answer rather than a comment in passing. What crosses the furrowd connection: the per-run token, then furrow's framed protocol carrying encrypted objects. Object contents, object ids and the workspace name are all encrypted or blinded by furrow before they reach the transport. What does not: the recovery key. It travels in the So an attacker who fully MITMs the furrowd connection gets a token and a stream of ciphertext. The token lets them pull that one run's objects, which stay unreadable without the recovery key they do not have. They cannot read the workspace, cannot forge a snapshot the real client would accept (objects are BLAKE3-verified against their ids on restore), and cannot reach any other run — the token pins one data root. That is why the default self-signed certificate is tolerable and why The part I would push back on in review is that the token is sent before the client has authenticated the server, so a MITM harvests tokens even though they cannot use them for much. If we want that closed, the options are pinning furrowd's certificate fingerprint in the handle (the handle is already a confidential channel, so it can carry one) or a challenge-response that never puts the token on the wire. Neither is in this PR; the fingerprint would be a small addition if we want it before this is used across untrusted networks. |
The installer builds only entrypoint.build and ignores dependencies.system for Go nodes. Cloud images therefore ship neither furrow nor furrowd, causing workspace mirroring to silently disable. Vendor the Linux amd64 siblings beside swe-planner so the existing resolver can find them.
Cloud secret injection only passes keys declared by the package. Expose the furrow client, daemon, storage, retention, and public-address settings with their code defaults so operators can configure persistent cloud deployments.
A newly paired run has no remote HEAD until its first snapshot and push, leaving handles unclonable throughout early build work. Publish an attached snapshot through the existing non-fatal path while retaining the per-run lock, and cover immediate real-binary materialization.
The binary and store-dir defaults are resolved at runtime (vendored sibling, /usr/local/bin, workspace root); a literal /workspaces path in the manifest would mislead local installs and risks being saved verbatim from config UIs.
furrow-dial runs on the CALLER's machine, not the node's: it is what FURROW_SSH_COMMAND points at when a caller clones an ssh:// handle. Vendoring only furrow and furrowd left the client half of the feature unshippable, since furrow has no release channel and callers would have to build it from source. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review fixes for the workspace-handle feature: - A public ssh:// handle was minted from FURROW_PUBLIC_ADDR alone, even when furrowd never resolved, failed to bind, or burned its restart budget — a valid-looking address nothing listens on. The supervisor now exposes a cached process+TCP health signal and the manager falls back to the run's dir: handle (with one warning) when it is down. - Store and remotes roots defaulted under the workspace tree, which is ephemeral in cloud containers — a restart deleted every published store and the TLS identity. When AGENTFIELD_HOME is set (cloud sets /data) the defaults now live under it; explicit env vars still win. - The remote-store directory was joined from the raw run ID while only the namespace was sanitized; '..'-shaped IDs could land (and later be swept with RemoveAll) outside the root. The dir now uses the sanitized namespace, and '.'/'..' survivors collapse to the fallback name. - The manifest described bin resolution in the wrong order and never mentioned FURROW_DIAL_TOKEN / FURROW_DIAL_INSECURE / FURROWD_TLS_*, which ssh:// consumers need today. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TestConcurrentBuildsOnSameRepo went red in CI: 'git worktree add -b' can create its branch and then lose the repo lock race, and the retry then dies on 'a branch named ... already exists' — the recovery path was the failure. The branch name embeds a per-call build ID nothing else can own, so -B (create-or-reset) turns the leftover branch into recovery instead of an error. Regression test simulates the leftover-branch state directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…volume Two hazards for deployments that upgrade into default-on mirroring: - Furrow subprocesses ran with no timeout while holding the run's mutex, so a hung binary hung the build — before planning, at every DAG level, and at build return. Every invocation now runs under a five-minute deadline (Options.CmdTimeout) and reports the timeout instead of waiting forever. - SWE_FURROW_MAX_GB only measured the remotes root, but every subprocess also writes a client store (FURROW_DATA_DIR) that was never counted or swept — unbounded growth on the /data volume that also holds the control plane's databases. The budget now means what the manifest says: aggregate. Attach refuses new mirrors when over budget (builds proceed unmirrored), and the sweeper measures both roots, warning once when the client store alone stays over the cap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The earlier -B change was broader than its justification: issue branches with commits are deliverables that outlive their build, and build IDs are only 32 random bits, so an unconditional create-or-reset could silently move a delivered branch on a name collision. Now the first attempt uses -b, a branch that exists before anything went transiently wrong is a hard failure again, and only retries after a non-exists failure — the case where our own dying attempt may have created the branch — escalate to -B. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…raw run ID The manager sanitizes the run ID into the directory it creates and records that path as Entry.StoreDir, but the daemon rebuilt its data root from the raw ID — so any ID sanitization alters was served from a directory that does not exist, and a traversal-shaped ID named a path outside the remotes root entirely. Serve the recorded path, falling back to the sanitized namespace for entries that predate StoreDir. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…istic addWorktree decides between refusing a pre-existing branch and reclaiming our own leftover by matching git's stderr phrasing; a translated message would have degraded the refusal into a reset. Run git under LC_ALL=C so the phrase is the phrase. The leftover-recovery test also stops depending on goroutine timing: the transient failure is scripted through a seam and asserts the -b then -B attempt sequence, while the recovery itself still runs real git against the leftover state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A build's workspace is currently only reachable if you happen to be on the same machine as the node. This makes it reachable from anywhere, so the harness that started a run can read the files while the run is still going — including the uncommitted edits, untracked files and dev state a git push would never carry.
It is built so that nobody has to know it is there. Availability is discovered, never configured: a build's result carries a
workspace_handlewhen the mirror is working and simply does not carry one when it is not. There is no flag to set, nothing to probe, and no new failure for a user to understand. Every unavailable path — no binary, no.git, attach or publish failing, the feature switched off — degrades to exactly today's behaviour, and the default (a nilAttacher) is off.What is here
internal/furrow— resolves a vendored furrow binary the same wayinternal/proresolves the engine, attaches a build's clone, republishes at each DAG-level boundary and at completion, and keeps a registry mapping the control-plane run ID to the workspace on disk. That mapping is the thing that was missing: SWE-AF knew its workspaces only by a locally generated build ID, so nothing outside the process could name one. An hourly sweeper bounds what keeping them costs (72h / 20GB by default) because furrow never prunes a remote on its own.cmd/furrowd+cmd/furrow-dial— the transport.furrowdserves one run's encrypted blob store over TLS by spawningfurrow __remotebehind a token check;furrow-dialstands in asFURROW_SSH_COMMANDon the caller's side so a stockfurrow clonereaches it with no patches to furrow.Deliberately not sshd: the cloud image runs the control plane and its agent nodes in one container as root, so an ssh key there is whole-box access to every run and every secret on the volume. A furrowd token unlocks a single run's data root, and the payload is ciphertext either way.
get_workspace_handle— returns a run's handle on demand, or{"available": false}.Three bugs that only a real binary could find
The unit tests inject a fake exec and assert argv, which pins what we mean to run. These are all things a fake happily accepted:
furrow snaptakes-m; there is no--label.key_hex, notkey..furrowpolicylines must beexclude <subtree>. Bare names make furrow reject the file,watchthen exits non-zero, and the mirror is silently off for every build with one debug line to show for it — the worst failure mode a feature like this can have.internal/furrow/integration_test.gonow drives the real binary end to end (attach → publish → materialize elsewhere → diff) and skips when furrow is not installed, so the CLI contract is checked rather than assumed. The exact policy bytes are pinned in a unit test too, since CI has no furrow binary..furrowpolicyalso joins.artifacts/and.worktrees/in.git/info/exclude— it lands in the repository the agent is working in, and without that it shows up ingit statusand any acceptance criterion about a clean tree becomes unsatisfiable.Verified
gofmt,go build ./...,go vet ./...,go test -race -count=1 ./...— all green, nothing regressed..envand a file written after attach all materialize byte-exact on a separate store, with a usable.git.furrow clone ssh://…throughfurrow-dial→ TLS →furrowd→furrow __remotedelivers the workspace; a wrong token is rejected.get_workspace_handle, and an unknown run answersavailable:falserather than erroring.Not yet exercised: a full LLM-backed build (needs a provider key in the node's environment) and the cloud deployment (its node is still
not_configured).Notes for review
watch --no-daemonis deliberate: no orphan daemons inside a container. A periodic publish while a run is in flight is the obvious next increment.furrowdcannot compare it to the registry's; the token is what scopes a connection, and the namespace is charset-validated and passed through. Comparing them rejected every real clone..gitis a real directory. A git worktree's.gitis a file pointing outside the tree, so mirroring one would produce a clone with no object database.Review hardening (second pass)
A follow-up review pass added one commit:
ssh://handle was minted fromFURROW_PUBLIC_ADDRalone, even when furrowd never resolved, failed to bind, or exhausted its restart budget — a valid-looking address nothing listens on. The supervisor now exposes a cached process+TCP health signal and the manager falls back to the run'sdir:handle (one warning logged) when it is down.AGENTFIELD_HOMEis set (cloud images set/data) the defaults now live under it; explicit env vars still win...-shaped IDs could land — and later be swept withRemoveAll— outside the root. The dir now uses the sanitized namespace, with./..survivors collapsing to the fallback name./usr/local/binfirst, vendored sibling second), and the client-side vars ssh:// consumers actually need (FURROW_DIAL_TOKEN,FURROW_DIAL_INSECURE,FURROWD_TLS_CERT/KEY) are now declared.go build ./...,go vet ./...,go test -race -count=1 ./...— all green.