Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ config/secondmate-harness PRIMARY launch `<harness> [<model>] [<effort>]`; LOCA
config/account-routing-mode `off|observe|enforce`; direct account directories for new observe/enforce launches, legacy recovery for existing managed metadata; LOCAL, gitignored, default off, inherited (docs/configuration.md "Agent Fleet account routing")
config/secondmate-account-pool optional Agent Fleet pool the PRIMARY uses for SECONDMATE launches when routing is enabled; LOCAL, gitignored; selection-only and NOT inherited
Direct account-directory launch covers ship/scout crewmates and secondmate launches; a secondmate binds only the selected account, never the ship/scout worktree-identity contract.
config/azure-worker-account-home optional canonical absolute Pi home used only as the Azure worker credential pool; LOCAL, gitignored; absent falls back to the primary Pi coding-agent home for compatibility
config/backlog-backend backlog backend override; LOCAL, gitignored; absent or "tasks-axi" = default tasks-axi backend, "manual" = force routine backlog updates to hand-editing; inherited by secondmate homes (section 10)
config/backend new-task runtime override; LOCAL, gitignored, not inherited; absent auto-detects herdr/cmux then tmux, explicit zellij/Orca only, rejects codex-app; tmux reference, herdr/zellij/cmux experimental, Orca legacy (docs/tmux-backend.md, docs/herdr-backend.md, docs/zellij-backend.md, docs/orca-backend.md, docs/cmux-backend.md, docs/codex-app-backend.md)
config/cmux-socket-password optional cmux control-socket password; LOCAL, gitignored; read fresh on every cmux CLI call and passed through without ever overriding an operator's own ambient CMUX_SOCKET_PASSWORD when absent (docs/cmux-backend.md "Setup")
Expand Down
42 changes: 42 additions & 0 deletions bin/fm-pi-refresh.py
Original file line number Diff line number Diff line change
Expand Up @@ -727,6 +727,13 @@ def command_run_once(args: argparse.Namespace) -> int:
now = time.time()
try:
code = _run_once(args)
azure_source = scheduled_azure_source(args)
if code == 0 and azure_source is not None:
azure_args = argparse.Namespace(**vars(args))
azure_args.source = str(azure_source)
azure_args.destination_root = str(scheduler_state_root() / "azure-account-homes")
azure_args.backup_root = str(scheduler_state_root() / "azure-pool-backups")
code = _run_once(azure_args)
except RefreshError as exc:
record_heartbeat(
kind="attention" if exc.attention else "failed", detail=str(exc), now=now
Expand All @@ -740,6 +747,34 @@ def command_run_once(args: argparse.Namespace) -> int:
return code


def scheduled_azure_source(args: argparse.Namespace) -> Path | None:
config_value = getattr(args, "azure_home_config", None)
if not getattr(args, "scheduled", False) or not config_value:
return None
config = Path(config_value)
try:
metadata = config.lstat()
except FileNotFoundError:
return None
except OSError as exc:
fail(f"Azure Pi pool config is unreadable at {config}: {exc.strerror}")
if not stat.S_ISREG(metadata.st_mode) or config.is_symlink():
fail(f"Azure Pi pool config must be a regular non-symlink file at {config}")
try:
lines = config.read_text(encoding="utf-8").splitlines()
except OSError as exc:
fail(f"Azure Pi pool config cannot be read at {config}: {exc.strerror}")
if len(lines) != 1 or not lines[0]:
fail(f"Azure Pi pool config must contain exactly one path at {config}")
home = Path(lines[0])
if not home.is_absolute() or home != home.resolve():
fail(f"Azure Pi pool config must name a canonical absolute directory at {config}")
source = home / "auth.json"
if source.expanduser().resolve() == Path(args.source).expanduser().resolve():
return None
return source


def _run_once(args: argparse.Namespace) -> int:
if args.all and args.slot:
fail("--all and --slot name different selections; pass one")
Expand Down Expand Up @@ -1146,6 +1181,12 @@ def scheduler_job(interval: int, state_root: Path, nonce: str) -> dict[str, Any]
"run-once",
"--all",
"--scheduled",
"--azure-home-config",
str(
Path(os.environ.get("FM_HOME") or BIN_DIR.parent).resolve()
/ "config"
/ "azure-worker-account-home"
),
],
"EnvironmentVariables": {
"PATH": search,
Expand Down Expand Up @@ -1467,6 +1508,7 @@ def build_parser() -> argparse.ArgumentParser:
action="store_true",
help="record the heartbeat, which only launchd's own invocation can do",
)
run.add_argument("--azure-home-config", help=argparse.SUPPRESS)
run.set_defaults(handler=command_run_once)

install = commands.add_parser(
Expand Down
115 changes: 101 additions & 14 deletions bin/fm-spawn.sh
Original file line number Diff line number Diff line change
Expand Up @@ -39,12 +39,13 @@
# account_home and worktree_git_dir_identity bindings that lifecycle derives
# its request from, and window= stays empty so local endpoint probes fail
# closed. Cloud spawns run ENTIRELY on the pi-codex runtime: harness
# dispatch and claude profile routing are bypassed, account_home is the pi
# coding-agent directory, and pi's extension owns multi-profile selection on
# the worker. With the switch off (default) spawns stay byte-identical to
# the local path. Secondmate and account-recovery spawns always stay local,
# and --backend, raw launch commands, or a non-pi harness cannot be combined
# with cloud placement.
# dispatch and claude profile routing are bypassed. account_home comes from
# config/azure-worker-account-home when that file exists, otherwise from the
# pi coding-agent directory for backward compatibility; pi's extension owns
# multi-profile selection on the worker. With the switch off (default),
# spawns stay byte-identical to the local path. Secondmate and
# account-recovery spawns always stay local, and --backend, raw launch
# commands, or a non-pi harness cannot be combined with cloud placement.
# FM_SPAWN_PARENT_TASK / FM_SPAWN_PARENT_TASK_GENERATION mark a cloud spawn as
# a SECONDMATE COMPARTMENT CHILD: both are forwarded to the lifecycle request
# as --parent-task/--parent-task-generation, where the controller's fan-out,
Expand Down Expand Up @@ -611,6 +612,7 @@ if [ "$SPAWN_CLOUD" = azure ] && [ "$STATE" != "$TASK_HOME/state" ]; then
exit 1
fi
CLOUD_ACCOUNT_HOME=
CLOUD_ACCOUNT_MIN_HEADROOM_SECONDS=43200
CLOUD_PLACEMENT_STATE=
CLOUD_WORKER_LAUNCH=
RESUME_META=
Expand All @@ -627,6 +629,80 @@ SPAWN_PREFLIGHT_ID=${POS[0]:-}
spawn_idpart=${SPAWN_PREFLIGHT_ID%%=*}
SPAWN_PREFLIGHT_BATCH=0

resolve_cloud_account_home() {
local config_file="$CONFIG/azure-worker-account-home" configured canonical lines
if [ -e "$config_file" ] || [ -L "$config_file" ]; then
[ -f "$config_file" ] && [ ! -L "$config_file" ] || {
echo "error: config/azure-worker-account-home must be a regular non-symlink file" >&2
return 1
}
lines=$(awk 'END { print NR }' "$config_file" 2>/dev/null) || return 1
[ "$lines" = 1 ] || {
echo "error: config/azure-worker-account-home must contain exactly one line" >&2
return 1
}
IFS= read -r configured < "$config_file" || {
echo "error: cannot read config/azure-worker-account-home" >&2
return 1
}
case "$configured" in
/*) ;;
*)
echo "error: config/azure-worker-account-home must name an absolute path" >&2
return 1
;;
esac
[ -d "$configured" ] && [ ! -L "$configured" ] || {
echo "error: cloud placement account home '$configured' is not a real directory" >&2
return 1
}
canonical=$(CDPATH='' cd -- "$configured" 2>/dev/null && pwd -P) || return 1
[ "$canonical" = "$configured" ] || {
echo "error: config/azure-worker-account-home must name its canonical physical directory ($canonical)" >&2
return 1
}
printf '%s\n' "$configured"
return 0
fi
printf '%s\n' "${PI_CODING_AGENT_DIR:-$HOME/.pi/agent}"
}

validate_cloud_account_pool() {
python3 - "$SCRIPT_DIR/fm-pi-account-home.py" "$CLOUD_ACCOUNT_HOME/auth.json" <<'PY'
import importlib.util
import sys

module_path, source = sys.argv[1:]
spec = importlib.util.spec_from_file_location("fm_pi_account_home", module_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
pool = module.read_pool(module.Path(source))
if not pool:
module.fail("Azure Pi pool must contain at least one profile")
expected = ["openai-codex"] + [
f"openai-codex-{index}" for index in range(2, len(pool) + 1)
]
if sorted(pool) != sorted(expected):
module.fail("Azure Pi pool profiles must be gap-free: " + ", ".join(expected))
faults = {name: module.entry_faults(pool[name]) for name in expected}
broken = [f"{name}: {', '.join(items)}" for name, items in faults.items() if items]
if broken:
module.fail("Azure Pi pool has unusable profile shapes: " + "; ".join(broken))
accounts = [pool[name]["accountId"].strip() for name in expected]
if len(set(accounts)) != len(accounts):
module.fail("Azure Pi pool profiles must name distinct upstream accounts")
PY
}

if [ "$SPAWN_CLOUD" = azure ]; then
CLOUD_ACCOUNT_HOME=$(resolve_cloud_account_home) || exit 1
[ -d "$CLOUD_ACCOUNT_HOME" ] || {
echo "error: cloud placement account home '$CLOUD_ACCOUNT_HOME' is not a directory" >&2
exit 1
}
validate_cloud_account_pool || exit 1
fi

release_secondmate_home_lifecycle_locks() {
[ -z "${TASK_HOME_LIFECYCLE_LOCK:-}" ] \
|| fm_account_lifecycle_lock_release "$TASK_HOME_LIFECYCLE_LOCK" >/dev/null 2>&1 || true
Expand Down Expand Up @@ -2693,15 +2769,15 @@ launch_template() {
# revisit this if pi ever ships a question tool that can park a secondmate -
# fm-watch.sh skips stale-pane wakes for kind=secondmate, so a parked
# secondmate would not trip stale detection.
printf '%s' '__AGENT__ --approve __MODELFLAG____EFFORTFLAG__-e __PITURNEND__ -e __PIWATCH__ "$(cat __BRIEF__)"'
printf '%s' '__AGENT__ --approve --fast __MODELFLAG____EFFORTFLAG__-e __PITURNEND__ -e __PIWATCH__ "$(cat __BRIEF__)"'
else
# --exclude-tools is a plain denylist over built-in, extension, and custom
# tool names (pi 0.84.0 filters it as a Set, ignoring names that are not
# registered), and ask_question is pi's own documented example for it. A
# crewmate's contract is to run autonomously and report through its status
# file, so a tool that halts the run to ask a question nobody is watching is
# never the right behavior here.
printf '%s' '__AGENT__ --approve --exclude-tools ask_question __MODELFLAG____EFFORTFLAG__-e __PIEXT__ "$(cat __BRIEF__)"'
printf '%s' '__AGENT__ --approve --exclude-tools ask_question --fast __MODELFLAG____EFFORTFLAG__-e __PIEXT__ "$(cat __BRIEF__)"'
fi
;;
# grok (Grok Build TUI): a positional prompt starts the supervised interactive
Expand Down Expand Up @@ -4091,7 +4167,7 @@ if [ "$SPAWN_CLOUD" = azure ]; then
# NOT reused - its paths exist only on this machine. --print keeps the run
# bounded and non-interactive under the supervisor's device-null stdin.
# shellcheck disable=SC2016 # single quotes are deliberate: $(cat ...) expands on the worker, not here
CLOUD_WORKER_LAUNCH="env PI_CODING_AGENT_DIR=/mnt/account/pi-agent pi --print --approve --exclude-tools ask_question ${MODELFLAG}${EFFORTFLAG}"'"$(cat /mnt/task/.fm-task/brief.md)"'
CLOUD_WORKER_LAUNCH="env PI_CODING_AGENT_DIR=/mnt/account/pi-agent pi --print --approve --exclude-tools ask_question --fast ${MODELFLAG}${EFFORTFLAG}"'"$(cat /mnt/task/.fm-task/brief.md)"'
# A secondmate compartment has no single worker entrypoint: its session
# legs are built and dispatched by fm-secondmate-cloud-monitor.sh, so the
# crewmate launch string above is never persisted or executed for it.
Expand Down Expand Up @@ -4468,6 +4544,17 @@ spawn_cloud_bind_leased_account() { # <request-stdout-file>
echo "error: leased provider-account home '$leased' holds no credential for $ID" >&2
return 1
}
# Azure's hard VM shutdown is six hours after creation. Require twice that
# much access-token headroom before staging so the guest cannot reach Pi's
# automatic OAuth refresh path and rotate a refresh token independently of
# the controller-owned pool. The host refresh scheduler is the sole refresh
# authority; a stale pool slot is handed back rather than copied to Azure.
"$SCRIPT_DIR/fm-credential-expiry.py" check --harness pi \
--margin-seconds "$CLOUD_ACCOUNT_MIN_HEADROOM_SECONDS" \
--min-state usable "$leased" >/dev/null || {
echo "error: leased provider-account credential lacks twelve hours of access-token headroom for $ID" >&2
return 1
}
# Exactly one provider slot, checked by shape and never by content: a home
# carrying more than one is the pool, and the guest would pick the first.
python3 - "$leased/auth.json" <<'PY' || return 1
Expand Down Expand Up @@ -4591,7 +4678,7 @@ spawn_cloud_persist_convergence_artifacts() {
cp "$SCRIPT_DIR/fm-secondmate-session.py" "$STATE/$ID.cloud-payload/fm-secondmate-session.py" || exit 1
cp "$SCRIPT_DIR/fm-secondmate-spawn.pi-ext.ts" "$STATE/$ID.cloud-payload/fm-secondmate-spawn.pi-ext.ts" || exit 1
fi
CLOUD_ACCOUNT_SOURCE=${PI_CODING_AGENT_DIR:-$HOME/.pi/agent}
CLOUD_ACCOUNT_SOURCE=$CLOUD_ACCOUNT_HOME
if [ ! -f "$CLOUD_ACCOUNT_SOURCE/auth.json" ]; then
echo "error: cloud account source lacks auth.json at $CLOUD_ACCOUNT_SOURCE" >&2
exit 1
Expand Down Expand Up @@ -4820,9 +4907,10 @@ META_WINDOW=$T
# Cloud placement persists the exact identities the elastic worker lifecycle
# derives its bindings from (docs/azure-workers.md "Queue request"): the
# physical worktree Git-dir identity and the provider account home. The cloud
# lane is pi-codex only, so the account home is always the pi coding-agent
# directory - never a claude/codex profile home, and never a local
# account-directory rotation (that machinery is bypassed entirely for cloud).
# lane is pi-codex only, so the account home is the dedicated Azure worker Pi
# pool when configured, otherwise the legacy pi coding-agent directory - never
# a claude/codex profile home, and never a local account-directory rotation
# (that machinery is bypassed entirely for cloud).
if [ "$SPAWN_CLOUD" = azure ]; then
[ "$DIRECT_ACCOUNT_ROUTING" != 1 ] || {
echo "error: cloud placement must not reach direct account-directory routing; refusing to mix profile machinery into the worker lane" >&2
Expand All @@ -4834,7 +4922,6 @@ if [ "$SPAWN_CLOUD" = azure ]; then
exit 1
}
fi
CLOUD_ACCOUNT_HOME=${PI_CODING_AGENT_DIR:-$HOME/.pi/agent}
[ -d "$CLOUD_ACCOUNT_HOME" ] || {
echo "error: cloud placement account home '$CLOUD_ACCOUNT_HOME' is not a directory" >&2
exit 1
Expand Down
20 changes: 13 additions & 7 deletions docs/azure-requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,8 @@ digest `1f238e42...`, and an outcome bundle of one commit
`r5-accept-readme-v3-20260822` (`asg-00000020`) and `r5-accept-package-v3-20260822`
(`asg-00000021`) each returned exit 0, `timed_out false`, and `outcome_commits 0` for their
read-only briefs. Evidence and paths are in R2/R3 and R5.
Placement across distinct upstream accounts is R5, and is now proven live there.
Placement across distinct upstream accounts is R5, where the former single-profile placement
residual is closed and now proven live.

## R2/R3. Secondmates run in Azure, and can spawn crewmates in Azure

Expand Down Expand Up @@ -344,8 +345,9 @@ reaches `close` with its worktree disk released".

Status: DONE, met live on 2026-08-22.

Crosscheck on the pi fleet is done at the roster level: eight pi profiles across eight distinct
upstream accounts, projected into single-profile account homes by `bin/fm-pi-account-home.py`,
Crosscheck on the pi fleet is done at the roster level. The current operating split is six Azure
worker profiles across six distinct upstream accounts plus one separate local Firstmate profile;
Azure profiles are projected into single-profile account homes by `bin/fm-pi-account-home.py`,
with the roster repointed and read back through the real `bin/fm-crosscheck.py` reader and
policy screen. Under the second 2026-08-19 amendment this roster is now the dormant crosscheck
fallback; the pi fleet's primary duties are authors and no-mistakes.
Expand All @@ -358,6 +360,9 @@ into a controller-owned account home, and `bin/fm-spawn.sh` narrows the staged p
to it, so the worker receives exactly one account rather than the pooled `auth.json`. An exhausted
pool refuses by name, listing every leased profile and the task holding it. Mechanics are owned by
`docs/azure-workers.md` ("Provider-account placement across the Pi fleet").
The host is also the sole OAuth refresh authority: staging requires twelve hours of access-token
headroom against a six-hour worker shutdown deadline, so a guest cannot live long enough to rotate
the copied refresh token independently of the Azure pool.

Proven locally against a fixture provider by `tests/fm-worker-placement.test.sh` (eight concurrent
placements racing the controller lock take eight distinct upstream accounts, read back from
Expand All @@ -367,10 +372,11 @@ account) and end to end through the real `bin/fm-spawn.sh` by `tests/fm-spawn-cl
asserts the staged credential is the leased single profile.

Operational consequence the owner must know: concurrent placements are now bounded by
`min(FM_AZURE_WORKER_MAX, distinct upstream accounts in the pool)`. With eight Pi accounts the
ninth concurrent placement refuses although MAX_WORKERS is 16, and compartments compete in the
same pool. Sixteen crewmates never could run on eight accounts without sharing one; they used
to do it silently. Raising the ceiling means adding profiles on distinct accounts.
`min(FM_AZURE_WORKER_MAX, distinct upstream accounts in the pool)`. With the current six-account
Azure pool the seventh concurrent placement refuses although MAX_WORKERS is 16, and compartments
compete in the same pool. Sixteen crewmates cannot run on six accounts without sharing one.
Raising the ceiling means adding profiles on distinct accounts to the Azure pool; adding profiles
to the separate local Firstmate pool does not change Azure capacity.

Acceptance: concurrent crewmates run on distinct pi profiles with no account collision.

Expand Down
Loading
Loading