Skip to content

feat(furrow): reach a run's live workspace from anywhere - #130

Open
AbirAbbas wants to merge 19 commits into
mainfrom
feat/furrow-workspace-handle
Open

feat(furrow): reach a run's live workspace from anywhere#130
AbirAbbas wants to merge 19 commits into
mainfrom
feat/furrow-workspace-handle

Conversation

@AbirAbbas

@AbirAbbas AbirAbbas commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

A build's workspace is currently only reachable if you happen to be on the same machine as the node. This makes it reachable from anywhere, so the harness that started a run can read the files while the run is still going — including the uncommitted edits, untracked files and dev state a git push would never carry.

It is built so that nobody has to know it is there. Availability is discovered, never configured: a build's result carries a workspace_handle when the mirror is working and simply does not carry one when it is not. There is no flag to set, nothing to probe, and no new failure for a user to understand. Every unavailable path — no binary, no .git, attach or publish failing, the feature switched off — degrades to exactly today's behaviour, and the default (a nil Attacher) is off.

What is here

internal/furrow — resolves a vendored furrow binary the same way internal/pro resolves the engine, attaches a build's clone, republishes at each DAG-level boundary and at completion, and keeps a registry mapping the control-plane run ID to the workspace on disk. That mapping is the thing that was missing: SWE-AF knew its workspaces only by a locally generated build ID, so nothing outside the process could name one. An hourly sweeper bounds what keeping them costs (72h / 20GB by default) because furrow never prunes a remote on its own.

cmd/furrowd + cmd/furrow-dial — the transport. furrowd serves one run's encrypted blob store over TLS by spawning furrow __remote behind a token check; furrow-dial stands in as FURROW_SSH_COMMAND on the caller's side so a stock furrow clone reaches it with no patches to furrow.

Deliberately not sshd: the cloud image runs the control plane and its agent nodes in one container as root, so an ssh key there is whole-box access to every run and every secret on the volume. A furrowd token unlocks a single run's data root, and the payload is ciphertext either way.

get_workspace_handle — returns a run's handle on demand, or {"available": false}.

Three bugs that only a real binary could find

The unit tests inject a fake exec and assert argv, which pins what we mean to run. These are all things a fake happily accepted:

  • furrow snap takes -m; there is no --label.
  • The recovery key comes back as key_hex, not key.
  • .furrowpolicy lines must be exclude <subtree>. Bare names make furrow reject the file, watch then exits non-zero, and the mirror is silently off for every build with one debug line to show for it — the worst failure mode a feature like this can have.

internal/furrow/integration_test.go now drives the real binary end to end (attach → publish → materialize elsewhere → diff) and skips when furrow is not installed, so the CLI contract is checked rather than assumed. The exact policy bytes are pinned in a unit test too, since CI has no furrow binary.

.furrowpolicy also joins .artifacts/ and .worktrees/ in .git/info/exclude — it lands in the repository the agent is working in, and without that it shows up in git status and any acceptance criterion about a clean tree becomes unsatisfiable.

Verified

  • gofmt, go build ./..., go vet ./..., go test -race -count=1 ./... — all green, nothing regressed.
  • Real furrow binary, end to end: a workspace with an uncommitted edit, an untracked .env and a file written after attach all materialize byte-exact on a separate store, with a usable .git.
  • Real transport, end to end: furrow clone ssh://… through furrow-dial → TLS → furrowdfurrow __remote delivers the workspace; a wrong token is rejected.
  • Live node against a real control plane: furrow discovered via sibling-binary resolution, 32 reasoners registered including get_workspace_handle, and an unknown run answers available:false rather than erroring.

Not yet exercised: a full LLM-backed build (needs a provider key in the node's environment) and the cloud deployment (its node is still not_configured).

Notes for review

  • Freshness is bounded by publish cadence — level boundaries and completion — not a background watcher. watch --no-daemon is deliberate: no orphan daemons inside a container. A periodic publish while a run is in flight is the obvious next increment.
  • The namespace on the wire is furrow's blinded name, so furrowd cannot compare it to the registry's; the token is what scopes a connection, and the namespace is charset-validated and passed through. Comparing them rejected every real clone.
  • Attaching is restricted to a workspace whose .git is a real directory. A git worktree's .git is a file pointing outside the tree, so mirroring one would produce a clone with no object database.

Review hardening (second pass)

A follow-up review pass added one commit:

  • Honest handles: a public ssh:// handle was minted from FURROW_PUBLIC_ADDR alone, even when furrowd never resolved, failed to bind, or exhausted its restart budget — a valid-looking address nothing listens on. The supervisor now exposes a cached process+TCP health signal and the manager falls back to the run's dir: handle (one warning logged) when it is down.
  • Durable stores: store/remotes roots defaulted under the workspace tree — ephemeral in cloud containers, so a restart deleted every published store and the TLS identity. When AGENTFIELD_HOME is set (cloud images set /data) the defaults now live under it; explicit env vars still win.
  • Sanitized paths: the remote-store dir was joined from the raw run ID while only the namespace was sanitized; ..-shaped IDs could land — and later be swept with RemoveAll — outside the root. The dir now uses the sanitized namespace, with ./.. survivors collapsing to the fallback name.
  • Manifest honesty: bin resolution order corrected (/usr/local/bin first, vendored sibling second), and the client-side vars ssh:// consumers actually need (FURROW_DIAL_TOKEN, FURROW_DIAL_INSECURE, FURROWD_TLS_CERT/KEY) are now declared.

go build ./..., go vet ./..., go test -race -count=1 ./... — all green.

AbirAbbas and others added 5 commits August 5, 2026 15:25
Defines the one-way surface orchestration depends on: a Handle that travels in
reasoner results, a registry Entry mapping a control-plane run ID to the
workspace on disk, and an Attacher whose nil implementation no-ops. Availability
is discovered by a caller finding a handle, never by asking.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
furrowd serves one run's encrypted blob store over TLS by spawning
`furrow __remote` behind a token check, and furrow-dial stands in as
FURROW_SSH_COMMAND on the caller's side so a stock `furrow clone` reaches it
with no patches to furrow.

This is deliberately not sshd. The cloud image runs the control plane and its
agent nodes in one container as root, so an ssh key there would be whole-box
access to every run and every secret on the volume. A furrowd token unlocks a
single run's data root, and what crosses the wire is ciphertext either way.

The namespace is passed through to the child rather than checked against the
registry: furrow blinds it (keyed BLAKE3) before it ever leaves the client, so
the node cannot recompute it, and comparing the two rejected every real clone.
The token is what scopes the connection; the namespace only selects a directory
beneath the root it already pins, so it is charset-validated instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A build now attaches its clone to furrow, republishes at every DAG level
boundary and at completion, and returns a workspace_handle in its result. A
caller that started the run can use that handle to clone the workspace and watch
it change while the build is still going — including the uncommitted edits,
untracked files and dev state that a git push would never carry.

Availability is discovered, never configured. The handle is present when the
mirror works and absent when it does not, so there is no flag to set, nothing to
probe, and no new failure a user has to understand. Every unavailable path — no
binary, no .git, attach or publish failing, the feature switched off — degrades
to exactly today's behaviour. A nil Attacher disables all of it, which is the
default.

Two things the mocked tests could not have caught, both found by running the
real binary:

  - The capture policy has to be written as `exclude <subtree>` lines. Bare
    names are rejected, `watch` then fails, and the mirror is off for every
    build with only a debug line to show for it. The exact bytes are pinned in
    a unit test now, and integration_test.go drives the real binary end to end
    so the whole CLI contract is checked rather than assumed.

  - .furrowpolicy lands in the repository the agent is working in, so it joins
    .artifacts/ and .worktrees/ in .git/info/exclude. Otherwise it shows up in
    `git status` and any acceptance criterion about a clean tree is
    unsatisfiable.

The run registry closes a real gap: SWE-AF knew its workspaces only by a locally
generated build ID, with nothing tying them to the control plane's run ID. That
mapping is what makes a workspace addressable from outside at all, and the
hourly sweeper bounds what it costs to keep (72h / 20GB by default) — furrow
never prunes a remote on its own.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
furrowd existed but nothing started it, which left the remote path unreachable
in exactly the deployment it was built for: in the cloud the control plane and
its agent nodes share one container, so there is no separate service to run it.
The node supervises it the same way it already supervises the coding engine —
backoff, a give-up threshold, and a process-group kill so nothing outlives the
node.

It stays inert unless there is something to serve: the mirror has to be enabled,
FURROW_PUBLIC_ADDR has to name an address the daemon can advertise, and a
furrowd binary has to resolve. Any of those missing is the normal case on a
developer's machine, so it is a debug line rather than a warning.

Verified against a live node rather than only a fake: furrowd comes up on its
port, speaks TLS, refuses an unauthenticated peer, and goes away when the node
does.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AbirAbbas and others added 3 commits August 5, 2026 16:00
The manager held a single lock for the whole of Attach and Publish, including
the furrow invocations inside them. A node serves several builds at once, and
an initial capture of a large repository is slow, so one build attaching could
park every other build's publish behind it — in a feature whose entire purpose
is to let parallel agents be watched while they work.

The lock now guards the registry only, and per-run locks provide the ordering
that actually matters: two calls for the same run must not both pair it, but
different runs have no reason to wait on each other. Publish also re-checks the
run still exists before recording a timestamp, so a sweep that retires a run
mid-push cannot resurrect a row whose store is already gone.

The test parks two publishes inside a fake furrow until both have arrived, which
can only happen if they run concurrently — verified to fail against the previous
locking and pass against this one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The hourly sweep held the registry lock while deleting store directories and
walking the whole store to measure it — the same mistake the previous commit
fixed for attach and publish, and a worse one, because retiring a run whose
store is gigabytes can take a while and every attach and publish on the node
waits behind it.

Deletion and measurement now happen with no lock held. Each run is retired under
its own lock, so a publish already in flight finishes rather than pushing into a
directory being deleted, and staleness is re-checked there: a run that became
active between the scan and the retire is left alone. Files go first and the
registry row second, so a failed delete leaves a row that the next sweep retries
instead of orphaning a store nothing points at.

The size pass stops if the run it picked survives its re-check, since measuring
again would keep choosing the same victim.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e it

The handle advertised the directory that holds every run's store rather than
the one belonging to this run. Pairing with it finds no workspace — `sync
--pull` fails with "sync remote has no published HEAD" — so the local path,
which is the default on a developer's machine whenever no public address is
configured, could not actually be used. Over the network the address was
always right, because furrowd resolves the store from the token.

The integration test now builds every argument from the handle instead of
assembling paths of its own. A consumer only ever has the handle, so a test
that reaches around it can pass while the handle itself is unusable — which is
exactly what happened here. Verified to fail against the old value.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AbirAbbas

Copy link
Copy Markdown
Collaborator Author

Reviewer note on the transport's threat model, since the client skips certificate verification and that deserves an explicit answer rather than a comment in passing.

What crosses the furrowd connection: the per-run token, then furrow's framed protocol carrying encrypted objects. Object contents, object ids and the workspace name are all encrypted or blinded by furrow before they reach the transport.

What does not: the recovery key. It travels in the workspace_handle inside the reasoner result — over the control plane's own TLS — and is never sent to furrowd. furrowd never has it either; it only shuttles bytes to furrow __remote, which stores ciphertext it cannot read.

So an attacker who fully MITMs the furrowd connection gets a token and a stream of ciphertext. The token lets them pull that one run's objects, which stay unreadable without the recovery key they do not have. They cannot read the workspace, cannot forge a snapshot the real client would accept (objects are BLAKE3-verified against their ids on restore), and cannot reach any other run — the token pins one data root.

That is why the default self-signed certificate is tolerable and why FURROW_DIAL_INSECURE exists. It is not a claim that certificate verification is worthless: an operator who sets FURROWD_TLS_CERT/FURROWD_TLS_KEY to a real certificate gets verification by leaving that variable unset, and that is the better posture on a shared network. The default is chosen so the feature works with no setup, not because the stronger option was unavailable.

The part I would push back on in review is that the token is sent before the client has authenticated the server, so a MITM harvests tokens even though they cannot use them for much. If we want that closed, the options are pinning furrowd's certificate fingerprint in the handle (the handle is already a confidential channel, so it can carry one) or a challenge-response that never puts the token on the wire. Neither is in this PR; the fingerprint would be a small addition if we want it before this is used across untrusted networks.

The installer builds only entrypoint.build and ignores dependencies.system for Go nodes. Cloud images therefore ship neither furrow nor furrowd, causing workspace mirroring to silently disable. Vendor the Linux amd64 siblings beside swe-planner so the existing resolver can find them.
Cloud secret injection only passes keys declared by the package. Expose the furrow client, daemon, storage, retention, and public-address settings with their code defaults so operators can configure persistent cloud deployments.
A newly paired run has no remote HEAD until its first snapshot and push, leaving handles unclonable throughout early build work. Publish an attached snapshot through the existing non-fatal path while retaining the per-run lock, and cover immediate real-binary materialization.
The binary and store-dir defaults are resolved at runtime (vendored sibling, /usr/local/bin, workspace root); a literal /workspaces path in the manifest would mislead local installs and risks being saved verbatim from config UIs.
furrow-dial runs on the CALLER's machine, not the node's: it is what
FURROW_SSH_COMMAND points at when a caller clones an ssh:// handle. Vendoring
only furrow and furrowd left the client half of the feature unshippable, since
furrow has no release channel and callers would have to build it from source.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AbirAbbas and others added 6 commits August 6, 2026 15:18
Review fixes for the workspace-handle feature:

- A public ssh:// handle was minted from FURROW_PUBLIC_ADDR alone, even
  when furrowd never resolved, failed to bind, or burned its restart
  budget — a valid-looking address nothing listens on. The supervisor now
  exposes a cached process+TCP health signal and the manager falls back
  to the run's dir: handle (with one warning) when it is down.
- Store and remotes roots defaulted under the workspace tree, which is
  ephemeral in cloud containers — a restart deleted every published
  store and the TLS identity. When AGENTFIELD_HOME is set (cloud sets
  /data) the defaults now live under it; explicit env vars still win.
- The remote-store directory was joined from the raw run ID while only
  the namespace was sanitized; '..'-shaped IDs could land (and later be
  swept with RemoveAll) outside the root. The dir now uses the sanitized
  namespace, and '.'/'..' survivors collapse to the fallback name.
- The manifest described bin resolution in the wrong order and never
  mentioned FURROW_DIAL_TOKEN / FURROW_DIAL_INSECURE / FURROWD_TLS_*,
  which ssh:// consumers need today.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TestConcurrentBuildsOnSameRepo went red in CI: 'git worktree add -b' can
create its branch and then lose the repo lock race, and the retry then dies
on 'a branch named ... already exists' — the recovery path was the failure.
The branch name embeds a per-call build ID nothing else can own, so -B
(create-or-reset) turns the leftover branch into recovery instead of an
error. Regression test simulates the leftover-branch state directly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…volume

Two hazards for deployments that upgrade into default-on mirroring:

- Furrow subprocesses ran with no timeout while holding the run's mutex, so
  a hung binary hung the build — before planning, at every DAG level, and at
  build return. Every invocation now runs under a five-minute deadline
  (Options.CmdTimeout) and reports the timeout instead of waiting forever.

- SWE_FURROW_MAX_GB only measured the remotes root, but every subprocess
  also writes a client store (FURROW_DATA_DIR) that was never counted or
  swept — unbounded growth on the /data volume that also holds the control
  plane's databases. The budget now means what the manifest says: aggregate.
  Attach refuses new mirrors when over budget (builds proceed unmirrored),
  and the sweeper measures both roots, warning once when the client store
  alone stays over the cap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The earlier -B change was broader than its justification: issue branches
with commits are deliverables that outlive their build, and build IDs are
only 32 random bits, so an unconditional create-or-reset could silently
move a delivered branch on a name collision. Now the first attempt uses -b,
a branch that exists before anything went transiently wrong is a hard
failure again, and only retries after a non-exists failure — the case where
our own dying attempt may have created the branch — escalate to -B.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…raw run ID

The manager sanitizes the run ID into the directory it creates and records
that path as Entry.StoreDir, but the daemon rebuilt its data root from the
raw ID — so any ID sanitization alters was served from a directory that
does not exist, and a traversal-shaped ID named a path outside the remotes
root entirely. Serve the recorded path, falling back to the sanitized
namespace for entries that predate StoreDir.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…istic

addWorktree decides between refusing a pre-existing branch and reclaiming
our own leftover by matching git's stderr phrasing; a translated message
would have degraded the refusal into a reset. Run git under LC_ALL=C so
the phrase is the phrase. The leftover-recovery test also stops depending
on goroutine timing: the transient failure is scripted through a seam and
asserts the -b then -B attempt sequence, while the recovery itself still
runs real git against the leftover state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant