docs(internal): forkd snapshot registry truthfulness root-cause + operator runbook (fabro-123) - #32
Open
zenprocess wants to merge 5 commits into
Open
docs(internal): forkd snapshot registry truthfulness root-cause + operator runbook (fabro-123)#32zenprocess wants to merge 5 commits into
zenprocess wants to merge 5 commits into
Conversation
The forkd exec wrapper assembles `sh -lc` bodies with a leading `cd
<working_dir> && ...` so subsequent commands run in the VM workspace.
When `[run.clone] enabled = false` the workspace dir is never created
and every command emits:
sh: 1: cd: can't cd to /home/fabro/workspace
Guard the `cd` with `2>/dev/null || true` so a missing directory is
silently skipped while preserving the chained command's exit code.
The directory path is still passed through shell_quote() so spaces and
metacharacters stay safe.
Co-Authored-By: Claude <noreply@anthropic.com>
GET /api/v1/runs/<id>/sandbox still returned state:'running' for forkd microVMs after the run had reached a terminal outcome. forkd destroys its VM at run completion so the persisted sandbox record cannot be introspected anymore — the persisted record would otherwise project a stale 'running' value. Resolve the reported state from the run's already-loaded status at the point the sandbox object is built for the endpoint. Read the run-state projection before loading the sandbox record, pass its terminal flag down through sandbox_details(), and let forkd_details() choose between SandboxState::Running (active) and SandboxState::Deleted (terminal — the existing variant that maps to a destroyed microVM). No HTTP call to forkd is added and no new SandboxState variant is invented; other providers (docker/daytona/local) still source their state from their own introspection and ignore the new parameter. Co-Authored-By: Claude <noreply@anthropic.com>
…uming Deleted
A terminal forkd run does NOT imply a torn-down microVM: ForkdSandbox::stop is
the trait-default no-op and teardown only happens on the run-DELETION path
(cleanup() / DELETE /v1/sandboxes/{id}). A run finished with --preserve-sandbox
(stop_on_terminal=false) leaves the microVM alive, yet forkd_details reported
SandboxState::Deleted for any terminal run, so the UI hid ssh/terminal for a
box that was still reachable.
Replace the is_run_terminal -> Deleted mapping with a real liveness query:
- Add ForkdSandbox::get_sandbox_status (GET /v1/sandboxes/{id}) reusing the
existing bearer-auth + is_retryable_status + HTTP_RETRY_LIMIT retry policy.
It never errors: 200 -> Alive, 404/410 -> Gone, and any indeterminate outcome
(5xx-after-retries, 405/501 endpoint-unsupported, connect/timeout) -> Unknown.
- forkd_details is now async: non-terminal runs still report Running without a
round-trip; terminal runs map Alive/Unknown -> Running and only a
controller-confirmed Gone -> Deleted. This is correct even if the forkd
controller has no status endpoint yet (it simply never claims Deleted).
- The persisted runtime.id carries the "forkd:" sandbox_info prefix; strip it
to recover the raw controller id for the URL.
Also make the forkd feature build/lint standalone (pre-existing gaps that kept
`--features forkd` from compiling): add dep:fabro-github and widen the
docker/daytona cfg gates on clone_source / RunId / runtime_layout_metadata to
include forkd. Correct two cd-guard test assertions that over-specified shlex
quoting, and rustfmt the crate. Local/Docker/Daytona behavior is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… design The comment claimed forkd tears down the microVM at terminal, which contradicts the fix in 7a008fb (terminal != gone; --preserve keeps it alive). Prevents a future maintainer reintroducing the terminal->Deleted shortcut. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…o-123) Writing-FIRST deliverable for fabro-123. Records the evidence from fabro-sh#121 that surfaced the real blocker (restore_many 400 from firecracker on zen-gate-big), separates what the evidence proves from what only a live controller probe can prove, and gives the operator the exact commands to run inside a zenctl maint window to determine which golden is missing, which is present-but-broken, and which is healthy. The controller fix itself is NOT in this commit: the forkd controller source is not in the fabro repo (the client is; the controller lives on dellsrv and its source is not local). Section 6 captures the design for the operator to implement against the real controller source: boot-time re-registration via a real restore-boot canary (NOT a files-exist check), golden re-commit on missing, and a single 'snapshot re-registered tag=...' log line per canary-passing snapshot. The preflight in fabro-github-gate.sh lands in a separate commit on the ao-company branch. I did not reach dellsrv to verify any of the open questions live: the brokered zen-gates path is not currently available (the zen MCP server is not connected, zen-gates/godkb binaries are not installed, and the codebase-memory index has no forkd projects). I made one attempt to discover a brokered path, failed, and stopped, per the brief's boundary. The doc lists the four questions that only the operator's live probe can answer and that the design depends on.
zenprocess
pushed a commit
that referenced
this pull request
Aug 1, 2026
…p, softer token rules (fabro-123 + fabro-122) Orchestrator review of revision 1 caught three errors in the design note. All three are corrected and the most important is promoted to a new §1 Headline section. Design note changes (docs/internal/forkd-snapshot-registry.md): - §1 NEW: Headline — TWO distinct live infra failures with a ~91:1 ratio. Exec-stage EAGAIN (zenprocess/ao-company#122, 1092 occurrences) dominates restore-stage 400 (this issue, 12 occurrences). Both are real; they are different bugs with different fix paths. - §3 NEW: What this worker got wrong in the previous revision (the orchestrator's three corrections, explicit). (a) gated=1 is a gate-attempt count, NOT a success verdict. The authoritative verdict lives in GitHub + fabro-gate-health.json. (b) The trader head 3ed3b8a4 pairing IS verified — gh api returns state=error at 2026-08-01T08:32:33Z with the restore_many 400 description. (c) The "log does not record 3ed3b8a" finding was misleadingly worded — the poll log's posted lines for this slice are uniformly for uniforme, but the trader/foundry verdicts live on GitHub, not in the file. - All section numbers shifted by one (added §1 headline + §3 corrections). - References updated to point at gh api + fabro-gate-health.json as authoritative. Runbook changes (docs/internal/forkd-snapshot-registry-runbook.md): - New §2 covers the exec-stage EAGAIN diagnostic (zenprocess/ao-company#122). Captures controller baseline (ps, /proc/PID/limits, cgroup) and EAGAIN-triggered samples of fd / thread / memory counters. Asks the operator to characterize which resource is exhausted. - §3 (per-tag bootability, was §2) updated; the new EAGAIN step is §2. - §4 (token handling, was §3) softened. The previous revision asserted the token "never lands on a log line because the bash subshell is the only place it is expanded." That was overconfident — shell tracing, error paths, and command capture can expose it. New rules: reference by file path only, confirm shell tracing is OFF, disable shell history capture for the maintenance window, do not paste commands into chat windows or pastebins, do not use curl -v or --trace. - All section numbers shifted. Doc header notes that this PR supersedes zenprocess/fabro PR #32. PR #32 also touches lib/crates/fabro-sandbox/* (DEAD paths post crate reorg; the sandbox crate now lives under lib/components/fabro-sandbox/) — those file changes are NOT carried forward. Co-Authored-By: Claude <noreply@anthropic.com>
zenprocess
added a commit
that referenced
this pull request
Aug 1, 2026
… (fabro-123) (#33) * docs(internal): forkd snapshot-registry root-cause + operator runbook (fabro-123) Writing-FIRST deliverable for zenprocess/ao-company#123 acceptance criterion #4. Records the verified evidence (restore_many 400 on earlier uniforme heads, exec EAGAIN at 2026-08-01T08:32, blast radius from fabro-gate-health.json) and explicitly marks every UNVERIFIED claim with the reason (forkd controller source not local, dellsrv behind egress boundary from this sandbox). The companion operator runbook hands off items 1-2 of the issue to the operator, with T3 maint-window discipline and a token-path-only reference (no token values). The code half (PR fabro-sh#143, gate-side preflight that emits a distinct snapshot-not-registered infra verdict) is already landed on ao/fabro-123/snapshot-preflight in zenprocess/ao-company and is NOT re-implemented here. Cross-referenced in §5 layer 1 and §7 of the design note. Co-Authored-By: Claude <noreply@anthropic.com> * docs(internal): revision 2 — two-failure headline, EAGAIN runbook step, softer token rules (fabro-123 + fabro-122) Orchestrator review of revision 1 caught three errors in the design note. All three are corrected and the most important is promoted to a new §1 Headline section. Design note changes (docs/internal/forkd-snapshot-registry.md): - §1 NEW: Headline — TWO distinct live infra failures with a ~91:1 ratio. Exec-stage EAGAIN (zenprocess/ao-company#122, 1092 occurrences) dominates restore-stage 400 (this issue, 12 occurrences). Both are real; they are different bugs with different fix paths. - §3 NEW: What this worker got wrong in the previous revision (the orchestrator's three corrections, explicit). (a) gated=1 is a gate-attempt count, NOT a success verdict. The authoritative verdict lives in GitHub + fabro-gate-health.json. (b) The trader head 3ed3b8a4 pairing IS verified — gh api returns state=error at 2026-08-01T08:32:33Z with the restore_many 400 description. (c) The "log does not record 3ed3b8a" finding was misleadingly worded — the poll log's posted lines for this slice are uniformly for uniforme, but the trader/foundry verdicts live on GitHub, not in the file. - All section numbers shifted by one (added §1 headline + §3 corrections). - References updated to point at gh api + fabro-gate-health.json as authoritative. Runbook changes (docs/internal/forkd-snapshot-registry-runbook.md): - New §2 covers the exec-stage EAGAIN diagnostic (zenprocess/ao-company#122). Captures controller baseline (ps, /proc/PID/limits, cgroup) and EAGAIN-triggered samples of fd / thread / memory counters. Asks the operator to characterize which resource is exhausted. - §3 (per-tag bootability, was §2) updated; the new EAGAIN step is §2. - §4 (token handling, was §3) softened. The previous revision asserted the token "never lands on a log line because the bash subshell is the only place it is expanded." That was overconfident — shell tracing, error paths, and command capture can expose it. New rules: reference by file path only, confirm shell tracing is OFF, disable shell history capture for the maintenance window, do not paste commands into chat windows or pastebins, do not use curl -v or --trace. - All section numbers shifted. Doc header notes that this PR supersedes zenprocess/fabro PR #32. PR #32 also touches lib/crates/fabro-sandbox/* (DEAD paths post crate reorg; the sandbox crate now lives under lib/components/fabro-sandbox/) — those file changes are NOT carried forward. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Referee <ralukacostache@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writing-FIRST deliverable for fabro-123.
The forkd controller on dellsrv is the live service that runs the
fabro GitHub gate's hermetic microVM. Issue fabro-sh#123 asks for a fix that
makes
GET /v1/snapshotstruthful AND auto re-registers snapshots onforkd boot. The brief is explicit: root-cause first, in writing,
before any fix.
This PR contains ONLY the root-cause + design note. It does NOT
contain a code change to the controller, because:
The forkd controller source is not in the fabro repo. The repo
contains the client (
lib/crates/fabro-sandbox/src/forkd/mod.rs),the provider (
lib/crates/fabro-sandbox/src/provider/forkd.rs),and the e2e BDD spec (
specs/forkd-e2e/behavior.feature). Thecontroller itself is a separate service running on dellsrv; its
source is not local to this checkout.
dellsrv / *.zp.digital / 10.0.201.x are NOT reachable from this
sandbox. The brokered zen-gates path that the 2026-07-11 anchor
references is not currently available:
which zen-gatesreturnsnothing, the
zenMCP server is registered but not connected,codebase-memory has no forkd projects. I made one attempt to
discover a brokered path, failed, and stopped, per the boundary.
What the doc records
Live evidence from Refactor fabro-model: OOP redesign with Catalog, LanguageModel trait, ModelRef fabro-sh/fabro#121 (PR Silent model fallback when model doesn't support agent mode fabro-sh/fabro#140, the descriptor-driven gate
poller): the failure observed on 2026-08-01T08:32:27Z is a firecracker
PUT /snapshot/load400 onzen-gate-big, NOT the exec EAGAIN ofUpdate tar crate to 0.4.45 (security fix) fabro-sh/fabro#122. This reframes the priority: snapshot registry work, not exec
EAGAIN work, is the blocker.
What the evidence proves (the listing endpoint lies, the boot
path can fail with HTTP 500 on
restore_many, the forkd shim doesnot consume
GET /v1/snapshots).What the evidence does NOT prove (registry storage backing,
whether
zen-gate-bigis missing entirely or present-but-broken,whether
zen-gate-baseis still bootable today, thefault_JSONin the 400). I refused to invent answers to any of these.
Design choice for the controller-side fix (§6): boot-time
re-registration via a REAL restore-boot canary (NOT a files-exist
check), golden re-commit on missing, one
snapshot re-registered tag=...log line per canary-passingsnapshot. The design cites the 2026-07-11 anchor as the standing
reason the canary cannot be skipped.
Operator runbook (§7): the exact commands the operator must run
inside a
zenctl maint onwindow to restart the controller, observethe post-boot listing, and run a canary exec on both
zen-gate-baseand
zen-gate-bigto determine which golden is missing, which ispresent-but-broken, and which is healthy. The data the runbook feeds
back is the design input the controller-side implementation needs.
What this PR does NOT do
not in this repo. The design in §6 is mine to write (this doc); the
implementation is the operator's to do against the real controller
source.
Companion PR
The gate-side preflight lives in
zenprocess/ao-company#143 — it asserts the requested snapshot tag
appears in
GET /v1/snapshotsbefore any in-VM work and posts adistinct
snapshot-not-registeredinfra verdict on mismatch. Untilthe controller is fixed, the preflight is what makes the gate silence
on a missing tag observable as its own class.
Refs zenprocess/ao-company#123