Repository navigation
04 Flows
Generated from
docs/at a43f824. Edit the source in the repository, not this page.
- The exact order of events when central starts, when a runner joins, and when a runner connects.
- How an operator registers with the rotating code, and why two operators can use one code.
- Every check an
action.runpasses on its way fromdurga's laptop todb-1's guard, and both audit rows it leaves behind. - What happens to work that is already running when a runner or an operator is revoked, and why the two revokes use opposite orders.
- How reconnecting, stale heartbeats and shutdown behave, and what you see when things fail.
- How a workspace is opened, placed on a runner and kept sticky; how
workspace.execand file transfers work (flows 12 to 14); and how closing, the TTL reaper, reconcile and revoke cascades clean up (flows 15 to 18). - How one MCP
exectool call travels from the harness to a container and back, how the session handoff makes--resumereattach, what happens to a workspace when the harness exits, and how the guardedconfirmprompt works (flows 19 to 23). - How the admin GUI starts (first-start setup with a one-time token), how login and sessions
work (idle and absolute expiry, survival across a restart, the generation check), how a revoke
click reaches the same code as the CLI, how
reset-adminrecovers a lost password and why a join token is shown once (flows 24 to 29). - How a host-mode workspace is opened and reattached, how
bashkeeps its full output and is paged, how a background job runs, how a credential travels fromcredential settogit pushand is rotated, and how a restarted runner re-adopts its sessions and jobs (flows 30 to 36).
Chapter 3 showed the parts. This chapter shows them moving: thirty-six flows, each following one thing that really happens in mirafleet, from the command someone types to the rows that end up in the database. Every flow starts with a one-sentence summary, then a diagram and a numbered explanation that names the functions involved, so you can open the code and follow along.
The diagrams use the names from the code as participant names (Server, service.Service,
registry.Registry, sqlite.DB, and so on). A few terms to keep handy, all in the
glossary: principal (the verified identity behind a connection), channel
(one request/response lane inside an SSH connection), tombstone (an in-memory "this runner id is
dead" mark), step (one 60-second window of the registration code).
| Name | Meaning in this chapter |
|---|---|
central.example.com:2222 |
MIRASTACK LABS' central. Data dir /var/lib/mirafleet/central. |
dev, prod
|
Fleets. prod is created with -confirm (a guarded fleet). |
build-1 (in dev), db-1 (in prod) |
Runners. |
durga, kalyan
|
Operators. |
| aarvee | The admin: has a shell on the central host and runs as the same user as central. |
All the admin commands below (mirafleet central fleet create ... and friends) are clients of the
running central: each one connects to <data>/admin.sock, sends one request, prints the answer and
exits. Only central init works with central stopped.
In one sentence: central init prepares a data directory once, and central run starts the
server that listens for everyone else.
sequenceDiagram
participant Admin as aarvee's shell
participant main as cmd/mirafleet
participant keys as keys
participant sqlite as sqlite.DB
participant Service as service.Service
participant Server as central.Server
Admin->>main: mirafleet central init -data D
main->>main: doCentralInit: prepareDataDir (umask 077, mkdir 0700, flock central.lock)
main->>keys: Generate host_ed25519 if absent
main->>keys: LoadSigner (refuses group/other-readable key)
main->>sqlite: Open: checks, pragmas, migrate (one transaction)
main->>Service: InitRegistrationSecret (principal admin, source cli)
Service->>sqlite: CreateSetting registration_secret + audit row
main-->>Admin: data dir, host key, fingerprint, next step
Admin->>main: mirafleet central run -data D -listen 0.0.0.0:2222
main->>main: startCentral: prepareDataDir (takes the lock again)
main->>Server: central.New(Config), Start (net.Listen tcp)
main->>Server: ServeAdmin (listen on admin.sock 0600)
main-->>Admin: central listening on ..., admin socket ...
Look at the lock and the umask: both come before anything is created on disk.
-
centralInitparses-data(default/var/lib/mirafleet/central) and-listen(only used in the hint it prints), then callsdoCentralInit. It is offline: it works whether or not a central is running, but it takes the same lock, so it cannot run whilecentral runholds it. -
prepareDataDirdoes three things in a fixed order.syscall.Umask(0o077)first: the umask removes permission bits from every file the process creates, so every file created afterwards (the database,-wal,-shm) is born0600and never briefly readable by others. Thenkeys.EnsureDircreates the directory with mode0700, or tightens it if it exists. FinallylockDataDirtakes an exclusive, non-blockingflock(an operating-system lock on a file, released automatically if the process dies) on<data>/central.lock. A second central on the same directory would fight over the database and the admin socket, so it fails with "another central is already using ...". -
Host key. If
<data>/host_ed25519does not exist,keys.Generatecreates an Ed25519 pair: the private key with mode0600(usingO_EXCL, so it can never overwrite an existing key) and the public key. An existing key is never replaced: replacing it would break the pin every runner and operator holds.initis idempotent for the same reason. -
Open the database.
loadCentralloads the signer (keys.LoadSignerrefuses a private key whose mode has any group/other bit), then callssqlite.Open.Openrefuses:memory:and symlinked database files (and symlinked-wal/-shmfiles). It opens the file withO_NOFOLLOW, which makes the operating system refuse to follow a symlink: otherwise an attacker who can write in the directory could plant a link namedmirafleet.dbpointing at some other file and trick central into opening or overwriting that file instead.Openthen sets safety settings (pragmas) on every connection and runs the embedded migrations in one transaction. A database newer than this binary makesOpenfail rather than guess. The pragma list and the transaction details are in chapter 6. -
Build the services.
loadCentralthen builds theregistry.Registry(withStaleAfter= 3 x 15 s = 45 s), theOperatorConnstracker and theservice.Servicewithpolicy.TablePolicy{}. -
Service.InitRegistrationSecretis called as theauth.LocalAdmin(uid)principal with sourcecli. It generates 32 random bytes and callsStore.CreateSetting, which inserts theregistration_secretsetting and its audit row in one transaction. If the setting already exists the store returns a conflict, anddoCentralInitreports "already initialised; registration secret kept". The secret is never printed, logged or audited. Onlycentral operator rotate-secretcan overwrite it (viaPutSetting). -
central run(centralRunthenstartCentral) repeats steps 2 and 4 (lock, host key, database), then buildscentral.New(central.Config{...}).Newrejects missing required fields, fills the defaults (the limits are in section 7.9) and prepares the SSH server config:PublicKeyCallback,PublicKeyAuthAlgorithmslimited to Ed25519,MaxAuthTriesof 3. -
Server.Startcallsnet.Listen("tcp", listen)and starts the accept loop.ServeAdminthen bindsadmin.sock:listenUnixdoes it under umask077, a stale socket from a crashed central is removed only after a failed dial proves nobody listens (and only while holding the lock), and the file ischmod 0600. The order matters: SSH first, admin socket second, because removing a stale socket is only safe under the lock. -
centralRunlogs the address and then blocks on<-ctx.Done()until SIGINT or SIGTERM (flow 11).
What would go wrong without the lock? Two centrals on one directory would each keep their own
in-memory registry, so a revoke sent to one would never kick a runner connected to the other.
In one sentence: aarvee asks the running central, over its local admin socket, to create the
prod fleet (guarded) and a join token, and central records both with an audit row.
First the fleet:
sequenceDiagram
participant adminCLI as adminCLI (cmd/mirafleet)
participant AdminCall as central.AdminCall
participant Server as central.Server (adminConn)
participant methodTable as methodTable
participant Service as service.Service
participant sqlite as sqlite.DB
adminCLI->>AdminCall: fleet.create {name:prod, mode:actions, confirm:true}
AdminCall->>Server: dial admin.sock, write one frame
Server->>Server: checkPeer: peer UID == our UID
Server->>Server: api.ReadRequestTimeout (64 KiB, 10 s)
Server->>methodTable: serve (admin table)
methodTable->>Service: handleFleetCreate -> CreateFleet
Service->>Service: authorize (policy + admin kind)
Service->>sqlite: CreateFleet(fleet, now, audit event)
sqlite-->>Service: fleet row + audit row, one transaction
Service-->>adminCLI: fleet prod created
Look at sqlite.CreateFleet: the fleet row and its audit row commit together.
Then the token, which takes the same path through the socket, the peer check and the method table:
sequenceDiagram
participant adminCLI as adminCLI (cmd/mirafleet)
participant Service as service.Service
participant tokens as tokens
participant sqlite as sqlite.DB
adminCLI->>Service: token.create {fleet:prod, uses:1, ttl_s:0} (via the admin socket)
Service->>Service: authorize (policy + admin kind), check limits
Service->>tokens: NewJoinToken(hostKey)
tokens-->>Service: id, secret, token string
Service->>sqlite: CreateToken(id, SHA-256 of secret, ...) + audit row
Service-->>adminCLI: token mfj1.... (only copy)
Look at the last message: the full token exists in memory only until it is printed. The database gets just a hash.
-
adminCLI.fleetCreateparses the flags (-mode,-confirm) and callscentral.AdminCallwith the socket path<data>/admin.sock. -
AdminCallbuilds anapi.Request(version 1, random 16-hex id, method, params), refuses anything overapi.MaxRequestBytes(64 KiB), dials the Unix socket (5 s timeout), writes one frame (4-byte length + JSON) and reads one frame back. -
ServeAdmin's accept loop callscheckPeerbefore reading a single byte. On Linux it asks the kernel for the peer's credentials (SO_PEERCRED) and requires the peer UID to equalos.Getuid(). A process running as another user is closed and logged. -
adminConnreads the request withapi.ReadRequestTimeout: the frame length is checked against 64 KiB before any allocation, and the whole request must arrive within 10 s (api.RequestTimeout). It builds acallCtxwith principalauth.LocalAdmin(uid)and sourcecli, then callsadminTable().serve. -
methodTable.serveapplies theallowedpredicate (admin table:IsAdminMethodplusfleets.list,runners.list,action.run), finds the handler, appliesapi.ListTimeout(10 s, but not foraction.run) and calls it.handleFleetCreatedecodes params strictly (DisallowUnknownFields). -
Service.CreateFleet:authorizewithadminOnlytrue. That means two gates: the policy (TablePolicyallows the admin kind everything) and a directp.Kind == auth.KindAdmincheck. An empty mode becomesactions; the name must match^[a-z0-9][a-z0-9._-]{0,62}$. -
Store.CreateFleetinserts the fleet and the audit row (fleet.create, sourcecli, actoruid:N) in one transaction. Either both exist or neither does. -
Service.CreateToken(formirafleet central token create -fleet prod): limits areDefaultTokenUses1 (maxMaxTokenUses1000),DefaultTokenTTL24 h (maxMaxTokenTTL30 days). There is no "unlimited". Labels are limited to 32 entries. -
tokens.NewJoinTokendraws 40 random bytes: 8 become the public id (16 hex chars), 32 become the secret. The token ismfj1.<id>.<secret in base64url>.<host-key hash>; the last part is the SHA-256 of central's host key, which is what lets the runner pin central later. -
Store.CreateTokenstoresSHA-256(secret), not the secret, plus fleet, labels,auto_approve,max_uses,expires_atandcreated_by, together with the audit row (token.create; the subject is the token id, params are uses/ttl/approve). The token string is not in the audit row. - The CLI prints the token alone on stdout and the human text ("token ... for fleet prod,
expires ...; shown once") on stderr. So
TOKEN=$(mirafleet central token create -fleet prod)captures only the token.
Why hash the secret? (SHA-256 is a one-way function: easy to compute, practically impossible
to reverse.) If the database file leaks, the attacker gets hashes of 256-bit random
values. They cannot turn a hash back into a token that a runner would accept. This is the same idea as
password hashing, minus the slow hash: the secret is random and long, so a fast SHA-256 is enough.
In one sentence: a runner turns a one-time token into an enrolled but not yet trusted runner, refusing to talk to any central whose host key does not match the hash inside the token.
Here aarvee joins build-1 to the dev fleet.
Look at where the host-key pin is checked: during the SSH handshake, before the token is sent.
- aarvee runs
token createon central. It printsmfj1.ID.SECRET.HASHonce (HASH pins central). - aarvee hands the token to the build-1 host by any channel, never through central.
-
runner joindials:2222as userenroll. A host key not matching HASH is refused (no TOFU). - It sends one
join@mirafleetframe: token, runner name, labels. -
Store.Enroll, one transaction: look up by ID, constant-time compare, check stale, insert. - Status is
pending(activeonly if the token auto-approves). The enroll connection closes. - aarvee runs
runner approve dev/build-1: status becomesactive. - The runner reconnects as user
runnerwith its own key and sends heartbeats.
First the runner side and the handshake:
sequenceDiagram
participant runnerJoin as runnerJoin (cmd/mirafleet)
participant runner as runner.Join
participant apiclient as apiclient.Join
participant Server as central.Server
participant KeyAuth as auth.KeyAuth
runnerJoin->>runnerJoin: check -token -, umask 077, runner.yaml absent
runnerJoin->>runnerJoin: readSecret (stdin, max 4096 bytes)
runnerJoin->>runner: Join(dir, addr, name, token)
runner->>runner: keys.Generate runner_ed25519 if absent, LoadSigner
runner->>apiclient: Join(addr, token, name, signer)
apiclient->>apiclient: tokens.ParseJoinToken, PinnedHash(hash)
apiclient->>Server: TCP + SSH handshake, user enroll
Server->>apiclient: host key (client checks pin, aborts on mismatch)
apiclient->>Server: public key auth
Server->>KeyAuth: Authenticate (user enroll: any ed25519 key)
Server->>Server: serveEnroll: limiter.allow, EnrollDeadline timer
Look at the message where the host key arrives: the pin is checked there, before the token is sent.
Then the enroll request itself:
sequenceDiagram
participant runnerJoin as runnerJoin (cmd/mirafleet)
participant runner as runner.Join
participant apiclient as apiclient.Join
participant Server as central.Server
participant Service as service.Service
participant sqlite as sqlite.DB
apiclient->>Server: open join@mirafleet, send enroll.join frame
Server->>Service: Join(token, name, key)
Service->>sqlite: Enroll(tokenID, SHA-256 secret, runner, audit)
sqlite-->>Service: runner row (pending or active)
Service-->>apiclient: {fleet, runner, status}
Server->>Server: close channel and connection
runner->>runner: write known_hosts, then runner.yaml (atomic)
runner-->>runnerJoin: joined fleet dev as build-1, status pending
Look at the last two messages on the runner: runner.yaml is written last, so a half-finished
join leaves the directory re-joinable.
-
runnerJoinrequires-token -(the literal dash: the token is read from standard input, never from the command line, wherepsand shell history could see it). It sets umask077. It checksrunner.yamlbefore reading stdin or touching the network: an already-joined directory returnsErrAlreadyJoined. ThenreadSecretreads at most 4096 bytes and trims them. -
runner.Joinvalidates the address (net.SplitHostPort) and the name (keys.RunnerNameRE), creates the data dir0700, generatesrunner_ed25519if it is absent and loads it withkeys.LoadSigner. -
apiclient.Joincallstokens.ParseJoinTokenjust to extract the host-key hash from the token, thenenroll, which builds the host-key callbackPinnedHash(hash): during the SSH handshake the client hashes the key central presents and compares it, in constant time, with the hash in the token. If they differ, the handshake aborts withErrHostKeyMismatch, and the token never leaves the runner. There is no "trust on first use" fallback. This is the whole point of putting the hash in the token. -
apiclient.Dialconnects with SSH userenroll, Ed25519 only, and a handshake timeout ofHandshakeTimeout(30 s on the client). -
On central,
Server.handlefirst takes a slot fromhsLimiter.acquire(global capMaxHandshakes, per-IP capMaxHandshakesPerIP), runsssh.NewServerConnwith a deadline ofHandshakeTimeout, and the public-key callback callsKeyAuth.Authenticate. For userenroll, any Ed25519 key is accepted and becomes principalKind: enrollwith no id. The wrapper incentral.Newalso stores the presented key inPermissions.Extensions["mirafleet-pubkey"]so the enrol code can register exactly that key. -
serveEnrollaskslimiter.allow(p.Remote): first the per-IP token bucket (EnrollRatePerMin), then the global bucket (EnrollGlobalRatePerMin). It arms a timer so the connection diesEnrollDeadlineafter it was accepted (not after authentication), andenrollKeycross-checks that the stored key's fingerprint equals the principal's. -
Only one channel is allowed. The first channel must be
join@mirafleetorregister@mirafleet; a second channel is rejected; a wrong first type closes the connection.serveEnrollChannelreads the request withapi.ReadRequestTimeout, andenrollCallchecks that it is an enroll method and matches the channel type (enroll.joinonjoin@mirafleet). -
Service.Join(principal kind must beenroll, again checked directly): key must be Ed25519, the name must be valid,tokens.ParseJoinTokenmust succeed, and the token's host-key hash must equal central's own (hostHashEqual, constant time). It hashes the secret with SHA-256 and calls the store. -
Store.Enroll, part 1: is the token good? The checks run in a fixed order so timing can't reveal which check failed: a prober who measures response times learns nothing about which part of a guessed token was wrong. All ofEnrollis one transaction.- Look up the token by its public id.
- Compare
SHA-256(secret)with the stored hash in constant time, against a dummy hash when the id is unknown, so "unknown id" and "wrong secret" take the same time. - Consume one use with a conditional
UPDATE ... WHERE revoked_at IS NULL AND expires_at > now AND uses < max_uses; if no row changes, the token is stale (expired, revoked or used up).
-
Store.Enroll, part 2: may this key become a runner? 4. Refuse the key if it already belongs to an operator (checked only now, after the token proved valid, so a caller without a token learns nothing about operator keys). 5. Read the fleet name, then insert the runner with the token's labels and statuspending(oractiveif the token was made with-approve), and write the audit rowenroll.join(sourceenroll).Because it is all one transaction, a failure at the last step (say, the name is taken) rolls back the token use too: nothing is consumed on any error.
-
Identical errors. Unknown id, wrong secret, expired, revoked, used up, malformed token, and a token that pins a different host key all return the same
errJoinRefused(unauthenticated, "join refused"). The real reason is logged on central only. Only a name or key conflict returnsconflict. So someone probing with guessed tokens cannot learn which ids exist. Failed joins are logged, not audited, so a flood of bad tokens cannot fill the audit table. -
Central writes the framed answer
{fleet, runner, status}, closes the channel and the connection (one request, one response). -
Back on the runner,
runner.Joinwritesknown_hosts(a linecentral.example.com:2222 ssh-ed25519 AAAA...holding the verified key) withwriteAtomic(temp file, fsync, rename), and only thenrunner.yaml.runnerJoinprints the status; if it is notactive, it says an admin must approve the runner.
Why pending by default? A join token is a bearer secret: anyone holding it can enrol. If it
leaks, the worst outcome is a pending runner nobody approved, which can heartbeat but never receive
work. See flow 5.
In one sentence: mirafleet runner run dials central, proves its key, sends a first heartbeat
(an "I am alive" message), and only then is registered as a live session that can receive work.
sequenceDiagram
participant Runner as runner.Runner
participant Server as central.Server (serveRunner)
participant KeyAuth as auth.KeyAuth
participant sqlite as sqlite.DB
participant Registry as registry.Registry
Runner->>Server: dial, SSH handshake as user runner (host key from known_hosts)
Server->>KeyAuth: Authenticate(user runner, key)
KeyAuth->>sqlite: GetRunnerByKey
KeyAuth-->>Server: principal runner (denied if revoked)
Runner->>Server: global request heartbeat@mirafleet (first one)
Server->>sqlite: GetRunnerByID (current status)
Server->>Registry: Register(id, name, fleet, conn, status, hb)
Registry-->>Server: session (or ErrTombstoned)
Server->>sqlite: TouchRunner(last_seen_at, info) via toucher
Server-->>Runner: reply true
loop every 15 s
Runner->>Server: heartbeat@mirafleet
Server->>Registry: Session.Heartbeat(hb)
end
Look at the GetRunnerByID message: the status is read again from the database at the moment of
registration, not reused from login.
-
runnerRunloadsrunner.yaml(runner.LoadConfig: unknown fields are an error), the key (keys.LoadSigner), the allowlist (actions.Load) and opens<data>/audit.jsonl(audit.Open, mode0600). Thenrunner.New(...).Run(ctx). -
Runner.sessiondials with a 10 s dial timeout and a 30 s deadline for the handshake, using SSH userrunner, its key, andkeys.PinnedHostKeys(known_hosts)as the host-key callback. The known-hosts file is read on every attempt, and a key that is not listed is refused. -
Central authenticates.
KeyAuth.Authenticatefor userrunnerlooks the key up (GetRunnerByKey). Unknown key: denied. A runner whose status isrevoked: denied. Pending and disabled runners are allowed to connect (they are visible, but cannot run actions). The peer sees only a generic denial; real errors are logged on central. -
Server.handledecodes the principal (auth.Decode, which fails closed on missing or garbled values) and, for runners and operators, callshs.markAuthed(remote): from now on that source IP is exempt from the per-IP handshake cap while this connection lives.dispatchsends it toserveRunner. -
serveRunnerstartsrejectChannels(runners may not open channels), arms a guard timer ofStaleAfter(45 s): a connection that never sends a valid heartbeat is closed. Then it loops over global requests; anything other thanheartbeat@mirafleetgets replyfalse. - The runner installs its
exec@mirafleethandler before its first heartbeat, then sends the first heartbeat immediately (Runner.heartbeat: it waits up to 2 xHeartbeatEveryfor the reply and closes the connection on timeout). That is why central registers a runner only on its first heartbeat: by then the runner is ready to accept an exec channel. -
Server.registerreloads the runner by id (GetRunnerByID), maps its status (registryStatus: active, pending, disabled; revoked has none and is refused) and callsRegistry.Register. Under one registry lock it:- refuses an id that has a tombstone (
ErrTombstoned); - lets a status recorded by
SetStatusoverride the one passed in (this closes the race where an admin approves the runner between the database read and the registration); - inserts the session, replacing and closing an older session for the same id ("the new connection wins");
- starts the
watchgoroutine (a lightweight background thread of Go code) that closes a connection silent for longer thanStaleAfter. If registration fails, the runner gets replyfalseand the connection is closed.
- refuses an id that has a tombstone (
-
Recording
last_seen_at.claimEdgeallows at most one connect write perTouchInterval/10(30 s) per runner, so a runner that flaps cannot hammer the database. The write itself goes through atoucher: a one-slot mailbox drained by a background goroutine, so a slow database never delays the heartbeat reply.touchRunnerstoreshostname,os,archandversion(each clipped to 256 bytes) and the time. -
Later heartbeats call
Session.Heartbeat(in-memory, free) and only write to the store whenTouchInterval(5 min) has passed since the last write. A runner that is online for a week costs a few thousand tiny writes, not millions. -
Disconnect. When the connection ends, the deferred function calls
Registry.Unregister, marks the disconnect edge and queues a finallast_seen_atwrite in the background. - The heartbeat's
labelsare informational only. The authoritative labels are the ones from the join token, stored byStore.Enroll.
In one sentence: aarvee changes a runner's stored status, and central updates the live session to match, in that order.
All four lifecycle operations (approve, disable, enable, revoke) go through one function,
Service.lifecycle. This section covers the three that keep the connection; revoke is flow 8.
stateDiagram-v2
[*] --> pending: join (token without -approve)
[*] --> active: join (token with -approve)
pending --> active: approve
active --> disabled: disable
disabled --> active: enable
pending --> revoked: revoke
active --> revoked: revoke
disabled --> revoked: revoke
revoked --> [*]
Look at revoked: nothing leaves it. A revoked runner key can never be enrolled again.
sequenceDiagram
participant adminCLI as adminCLI
participant Server as central.Server (adminConn)
participant Service as service.Service
participant sqlite as sqlite.DB
participant Registry as registry.Registry
adminCLI->>Server: runner.lifecycle {fleet:dev, runner:build-1, op:approve}
Server->>Service: handleRunnerLifecycle -> Lifecycle
Service->>Service: authorize (policy + admin kind)
Service->>sqlite: GetRunner(dev/build-1)
Service->>Service: from-set and store.CanTransition
Service->>sqlite: SetRunnerStatus(id, from, to, now, audit runner.approve)
sqlite-->>Service: runner with new status
Service->>Registry: SetStatus(id, active)
Service-->>adminCLI: runner dev/build-1 is active
Look at the order: the database commit comes first, the live session is updated second.
-
mirafleet central runner approve dev/build-1becomes the admin-socket callrunner.lifecyclewithopset to one ofapi.OpApprove,OpDisable,OpEnable,OpRevoke. It is an admin method, so it exists only on the admin socket, never onapi@mirafleet. A prompt-injected AI agent cannot approve its own runner, because it has no way to ask. -
Service.lifecyclecallsauthorize(..., adminOnly=true), picks the allowed "from" statuses and the target (approve: pending to active;disable: active to disabled;enable: disabled to active), parsesdev/build-1withstore.ParseRefand loads the runner withGetRunner(which hides revoked runners). - It checks the current status is in the allowed "from" set and
store.CanTransitionagrees, otherwise it answersconflict("runner is active: cannot approve"). That is how a double approve is harmless. - It builds the audit event
runner.approvewith paramsfromandto, and callsStore.SetRunnerStatus: one transaction that re-reads the status, updates it (withupdated_at) and inserts the audit row. If something else changed the status in between, the transaction returns a conflict instead of overwriting. - Only after the commit does it call
Registry.SetStatus(id, status). This records the desired status even if the runner is offline (thestatusesmap; the nextRegisterapplies it) and updates the live session if there is one. IfSetStatusfails, the service kicks the session: the runner reconnects and registers with the status in the database. Fail closed. -
What each state means for work.
-
pending: the runner may heartbeat, so it shows as online, butRunActionrefuses withconflict("runner is pending") andRegistry.Execwould returnErrPendinganyway. -
disabled: the connection stays up. Newaction.runcalls are refused (conflict), but an action that already started is not interrupted:RunActionchecked the status once at the start and the registry's status gate is only consulted when an exec begins. -
active: runs actions.
-
In one sentence: durga and kalyan get access with a short-lived code that aarvee reads off
central, without aarvee ever handling their keys.
Look at the clock: the code changes every 60 seconds, and any number of operators can use the current one.
- Central keeps a 32-byte secret in SQLite, never logged. Only aarvee (the admin) can read codes.
- Each 60 s window
s(unix seconds / 60) has its own code: base32(HMAC-SHA256(secret, s)) + HASH. - durga runs
mcp loginwith the code aarvee gave durga. Central recomputes this window's code and compares. - kalyan uses the same code in the same window and is accepted too. Codes are not single-use.
- The window ends. The next window
s+1has a brand-new code. - kalyan tries the old code in the next window: refused, with the same error as any wrong code.
- HASH is central's host-key hash, so the code also pins central: no TOFU at first login.
First aarvee shows the code:
sequenceDiagram
participant adminCLI as adminCLI
participant Service as service.Service
participant tokens as tokens
participant sqlite as sqlite.DB
adminCLI->>Service: operator.code
Service->>sqlite: GetSetting registration_secret
Service->>tokens: Code(secret, Step(now), hostKey)
Service->>sqlite: RecordAudit operator.code (who, when - not the code)
Service-->>adminCLI: code mfr1..., seconds left, host fingerprint
Look at the audit message: it records who showed a code and when, never the code itself.
Then aarvee tells durga the code out of band (chat or voice, within the window), and durga registers:
sequenceDiagram
participant mcpLogin as mcpLogin (durga)
participant apiclient as apiclient.Register
participant Server as central.Server (serveEnroll)
participant Service as service.Service
participant tokens as tokens
participant sqlite as sqlite.DB
mcpLogin->>mcpLogin: readSecret (stdin), keys.Generate operator_ed25519
mcpLogin->>apiclient: Register(addr, code, name, signer)
apiclient->>Server: handshake as user enroll, host key pinned by code hash
apiclient->>Server: register@mirafleet, enroll.register frame
Server->>Service: Register(code, name, key)
Service->>sqlite: GetSetting registration_secret
Service->>tokens: VerifyCode(secret, now, body)
Service->>sqlite: RegisterOperator(name, key, audit)
sqlite-->>Service: operator active (created or idempotent)
Service-->>mcpLogin: {operator, status: active}
mcpLogin->>mcpLogin: pinHost, SaveContexts (current = durga)
Look at the VerifyCode message: the code is derived and checked by central itself, so there is
no clock to keep in sync with the laptop.
-
Showing the code.
mirafleet central operator codecallsoperator.code.Service.OperatorCodeloads the 32-byte secret (GetSetting), computes the current step (tokens.Step(now)= unix seconds / 60) and returnstokens.Code(secret, step, hostKey). The code ismfr1.+ the first 26 characters of base32(HMAC-SHA256(secret,"mirafleet-register-v1"+ step)) +.+ the host-key hash, plusSecondsLeftand central's host-key fingerprint. Showing a code is audited (operator.code: who, when) and the audit write happens first: if it fails, the code is not returned. The code itself is never put in the audit row. - aarvee passes the code to
durgain any way they like (chat, voice). A leaked code is useless once its 60-second window ends. -
mcpLoginon Durga's laptop (mirafleet mcp login central.example.com:2222 -name durga -code -) requires-code -, validates the address and name, refuses on Windows (the key loader fails closed there), reads the code from stdin, creates the config dir (opconf.EnsureDir,0700) and generatesoperator_ed25519if it is missing (key comment = the name). -
apiclient.Registerparses the code only to get its host-key hash, pins central withPinnedHash, connects as userenroll, opensregister@mirafleetand sendsenroll.register{code, name}. The path throughServer.handleandserveEnrollis the same as in flow 3 steps 5 to 7, withChanRegisterinstead ofChanJoin. -
Service.Register: principal must beenroll; the key must be Ed25519; the name must be a valid operator name;tokens.ParseCodemust succeed; the code's host-key hash must equal central's. -
tokens.VerifyCode(secret, now, body)recomputes the code for the current step and compares it in constant time. Previous and future steps are refused, so the window is exactly the 60 seconds of the step. Verification has no side effects, so the same code verifies any number of times in its step. That is deliberate:durgaandkalyancan both register with the code shown at 10:03:12, as long as they both arrive before 10:04:00. Anything wrong (malformed, wrong host, wrong step, secret missing) returns the identicalerrRegisterRefused("registration refused"), with the real reason in central's log. -
Store.RegisterOperatorinserts anactiveoperator and its audit row (enroll.register, sourceenroll, subject = name, params = key fingerprint) in one transaction. It is idempotent per key: an already active key returns the existing operator withcreated=false. In that case the service still writes an audit row (message "already registered") throughRecordAudit, so a repeat registration is never invisible. Conflicts: the key belongs to a revoked operator, or to a runner (a key may be a runner's or an operator's, never both, checked inside the transaction), or the name is held by a different active operator, or the key is registered under another name. - No approval step. Only aarvee, on the central host, can read the code, so having it is the approval. There is no pending operator state.
- Back on the laptop,
pinHostwrites central's verified key intoknown_hosts(replacing any older line for the same address) andSaveContextsrecordsdurga -> central.example.com:2222as the current context. Later calls (mcp api ...) useapiclient.KnownHosts, so the pin from registration protects every future connection. -
central operator rotate-secretreplaces the secret (PutSetting, audited), which invalidates every code anyone was holding. Already-registered operators are unaffected.
stateDiagram-v2
[*] --> active: register with the current code
active --> active: register again, same key and name (idempotent, audited)
active --> revoked: central operator revoke
revoked --> [*]
Look at revoked: the key is dead. Registering the same key again is a conflict, but the name can be
given to a new key.
In one sentence: durga asks db-1 in prod to tail its logs, and the request crosses every
layer and every check, leaving two audit rows on central and two lines on the runner. This is the
most important flow in the system.
Look at the two audit rows: one written before anything runs (start), one after (ok or a
failure outcome), both with the same request_id.
- durga's agent calls
action.run(prod/db-1,tail-logs,lines=200) throughmirafleet mcp. -
mcpsends one frame (4-byte length + JSON) on anapi@channel. The operator method table routes it. -
service.RunActionchecks policy, fleet mode, confirm "prod" (guarded fleet's name), runner active. - The audit row "start" is written first. If that fails, nothing runs (fail closed).
-
registry.Execopens anexec@channel on the runner's own connection and sends the request. - The runner's actions guard checks
tail-logsand regex-checkslines=200into an argv. - The process runs (no shell, own process group, timeout). The result JSON goes back.
- The audit row "outcome" is written with the same
request_id. - The response frame returns the same way: exit code, stdout, stderr.
The command, from durga's laptop. The diagram above shows it as an agent's call; the MCP tool
run_action makes this same action.run request (flows 19 and 23 show the MCP side), and
mcp api is the hand-typed way to send it. The "confirm":"prod" is required because prod is a
guarded fleet:
mirafleet mcp api action.run '{"fleet":"prod","runner":"db-1","action":"tail-logs","params":{"lines":"200"},"confirm":"prod"}'
Part 1: connect and authenticate.
sequenceDiagram
participant mcpAPI as mcpAPI (durga)
participant Client as apiclient.Client
participant Server as central.Server
participant serveOperator as serveOperator
participant sqlite as sqlite.DB
mcpAPI->>Client: Dial(addr, user operator, key, KnownHosts)
Client->>Server: SSH handshake (host key pinned, ed25519 only)
Server->>Server: hsLimiter, HandshakeTimeout, KeyAuth (operator, not revoked)
Server->>serveOperator: dispatch (principal operator durga)
serveOperator->>sqlite: GetOperatorByID, TouchOperator (still active?)
Look at the last message: even after login, central re-checks that durga is still active.
Part 2: from the channel to the "start" audit row.
sequenceDiagram
participant Client as apiclient.Client
participant serveOperator as serveOperator
participant methodTable as methodTable
participant Service as service.Service
participant TablePolicy as policy.TablePolicy
participant sqlite as sqlite.DB
Client->>serveOperator: open api@mirafleet, write request frame
serveOperator->>serveOperator: channel allow-list, semaphore (max 8)
serveOperator->>serveOperator: serveAPIChannel: ReadRequestTimeout, ParseRequest
serveOperator->>methodTable: serve(req)
methodTable->>methodTable: allowed = IsOperatorMethod, handler found
methodTable->>Service: handleActionRun -> RunAction
Service->>TablePolicy: authorize (operator current status active?)
Service->>sqlite: GetFleet, GetRunner
Service->>Service: mode, confirm, status active checks
Service->>sqlite: RecordAudit (outcome start, params, request_id)
Look at the last message: nothing has been sent to the runner yet, but the "start" row already exists.
Part 3: service to the runner's guard and back.
sequenceDiagram
participant Service as service.Service
participant Registry as registry.Registry
participant Runner as runner.Runner (serveExec)
participant Set as actions.Set
participant audit as audit.Writer (audit.jsonl)
participant sqlite as sqlite.DB
participant Client as apiclient.Client
Service->>Registry: Exec(ctx with timeout, runner id, ExecRequest)
Registry->>Registry: Get (online, not kicked, not stale), status gate
Registry->>Runner: OpenChannel exec@mirafleet (ExecRequest as channel data)
Runner->>Runner: json parse, semaphore (max 4), Accept
Runner->>audit: Log action, params, outcome start
Runner->>Set: Run -> Plan (allowlist, params, regex) -> execArgv
Set-->>Runner: Result (stdout, stderr, exit code, duration)
Runner->>audit: Log outcome ok or failed or timeout or refused
Runner-->>Registry: write ExecResult JSON, CloseWrite
Registry-->>Service: ExecResult (max 8 MiB read)
Service->>sqlite: RecordAudit (outcome ok, exit_code, duration, same request_id)
Service-->>Client: ActionRunResult via api@mirafleet response frame
Look at the runner's two lines in audit.jsonl: they are written before the result is sent
back to central.
Every gate, with the code that implements it. There are 23, in four groups. The order is deliberate: cheap checks that need no secrets and cause no side effects come first, and the first write (the start audit row) comes only after every precondition holds.
Nothing here has run any business logic yet. The goal is to refuse floods, impostors and revoked keys as cheaply as possible.
| # | Gate | Where | If it fails |
|---|---|---|---|
| 1 | Request fits in 64 KiB |
apiclient.callOn (client) and api.ReadFrame (server, before allocating) |
too_large |
| 2 | Pre-handshake caps |
hsLimiter.acquire: global MaxHandshakes, per-IP MaxHandshakesPerIP (unless this IP has an authenticated connection) |
Connection closed before any cryptography |
| 3 | Handshake finishes in HandshakeTimeout
|
Server.handle sets a connection deadline |
Dropped; slot released |
| 4 | Host key is the pinned one | client: apiclient.KnownHosts
|
Client refuses to continue |
| 5 | Ed25519 key, user operator, key in operators, not revoked |
KeyAuth.Authenticate; MaxAuthTries 3 |
Opaque auth.ErrDenied
|
| 6 | Operator is still active |
serveOperator: GetOperatorByID after OperatorConns.add (closes the add-after-kick window) |
Connection closed |
The connection is trusted as "durga", but the bytes durga sends are not: are they the right kind of channel, the right size, valid JSON, and a method that kind of caller may use?
| # | Gate | Where | If it fails |
|---|---|---|---|
| 7 | Channel type allowed | channelAllowed(KindOperator, "api@mirafleet") |
Channel rejected |
| 8 | At most 8 channels open |
sem of size api.MaxChannelsPerConn
|
ResourceShortage |
| 9 | Request arrives in 10 s, strict JSON |
api.ReadRequestTimeout, ParseRequest (v must be 1, no unknown fields, id 1-128 bytes) |
timeout, bad_request, unsupported_version
|
| 10 | Method is an operator method | methodTable.allowed = api.IsOperatorMethod |
forbidden (admin/enroll methods are also logged) |
| 11 | Params decode strictly |
api.DecodeParams in handleActionRun
|
bad_request |
Now the service decides whether this particular action on this particular runner is allowed.
| # | Gate | Where | If it fails |
|---|---|---|---|
| 12 | Policy: operator is active now |
Service.authorize loads the operator's current status; TablePolicy allows only hello, fleets.list, runners.list, action.run
|
forbidden, and the denial is audited |
| 13 | Kind is operator or admin | direct check in RunAction (independent of policy) |
forbidden, audited |
| 14 |
prod/db-1 is a valid ref; action name 1-64 bytes; at most 32 params, key <= 64 B, value <= 4 KiB; timeout_s not negative |
store.ParseRef, api.ValidateActionParams, ValidateTimeoutS
|
bad_request |
| 15 | Timeout clamp |
api.ActionTimeout: default 2 min, max 10 min |
(clamped, not refused) |
| 16 | Fleet exists and is in actions mode |
Store.GetFleet |
not_found / conflict
|
| 17 | Guarded fleet: confirm equals the fleet name |
RunAction (f.Confirm && confirm != f.Name) |
bad_request, audited as denied
|
| 18 | Runner exists (not revoked) and its stored status is active
|
Store.GetRunner |
not_found / conflict ("runner is pending") |
The audit row comes first. After that, the last gates are enforced on the runner itself, by code central cannot change.
| # | Gate | Where | If it fails |
|---|---|---|---|
| 19 | Start audit row written | Store.RecordAudit |
If it fails, nothing is executed: unavailable ("audit unavailable") |
| 20 | Runner is online (registered, not kicked, heartbeat within 45 s) and its live status is active
|
Registry.Get, Registry.Exec
|
Outcome error, "runner unavailable" |
| 21 | Runner has a free slot |
Runner.sem of size MaxExec (default 4) |
Channel rejected, runner logs a refused audit line |
| 22 | Action is defined in actions.yaml, no unknown params, no missing params, each value fully matches its regex, no leading - unless allowed |
actions.Set.Plan |
ExecResult.Refused; central answers bad_request "refused by runner: ..." |
| 23 | No shell, minimal environment, own process group, deadline, 1 MiB output cap | actions.execArgv |
Process killed; timed_out
|
Why is a missing confirm bad_request but a policy failure forbidden? forbidden means
"you, as a caller, may not do this at all", which is a statement about who is asking. A missing or
wrong confirm is different: durga is allowed to run actions, but this particular request is
incomplete for a guarded fleet, which is a statement about the request. The caller can fix it by
resending with "confirm":"prod". (Both are audited, so a script that keeps forgetting is visible.)
-
Dial.
mcpAPIreadscontexts.yamlto find the current central, loads the operator key and callsapiclient.Dialwithapiclient.KnownHosts(known_hosts). If the host key is not pinned, the call fails. It never asks "do you trust this host?" (thePrompthelper refuses anything but an interactive terminal). -
Authenticate on central (gates 2 to 5).
Server.handlebuilds thePrincipalfor useroperator,markAuthed, anddispatchcallsserveOperator. -
serveOperatorcreates a context tied to the connection (sconn.Waitcancels it when the connection ends), registers the connection inOperatorConns(so a revoke can find it), then re-reads the operator (gate 6) and callsTouchOperatorto recordlast_seen_at. -
Open a channel.
Client.Callopens anapi@mirafleetchannel, writes one frame (4-byte big-endian length, then JSON{"v":1,"id":"...","method":"action.run","params":{...}}; the smallest example,hello, is worked out byte by byte in section 7.3) and waits for one frame back. Cancelling the call closes the channel; one request per channel. -
serveAPIChannelreads and parses the request (gate 9), then armscontext.AfterFuncso that closing the channel cancels the request context. A half-close (the client says "no more input") does not cancel; only a full close or a dead connection does. -
methodTable.serveruns gates 10 and 11, then callshandleActionRun, which callsService.RunActionwith the principal, the sourceapi, and the request context.action.runhas no table-level timeout (timeoutreturns 0); the service applies its own. -
RunActionchecks (gates 12 to 18), in the order of the table. The order is not an accident: cheap, local, no-side-effect checks first; the first write (the start audit row) only after every precondition holds. -
Start audit row (gate 19).
RunActiongenerates a 128-bit randomrequest_id(newRequestID) and writes anAuditEventwith sourceapi, actor kindoperator, actordurga, remote address, fleetprod, runnerdb-1, subjecttail-logs, actionaction.run, params{lines: 200}and outcomestart. If the database cannot take this row, the action does not run: no row, no exec. Without that rule a database outage would turn into unaudited commands. -
Exec.
RunActionderivesectxfrom the request context with the clamped timeout and callsRegistry.Exec.Execfinds the live session (gate 20), checks its status, and callsexecOn:conn.OpenChannel("exec@mirafleet", payload)where the payload is the JSONExecRequest{action, params}. This is the channel opened back over the connection the runner itself made. Opening a channel cannot be cancelled by Go's SSH library, soexecOnruns it in a goroutine and closes any channel that opens late. -
Runner:
serveExec. Parses the request (a bad payload is rejected), takes a slot from the semaphore (gate 21; if full, it writes arefusedline toaudit.jsonland rejects the channel), accepts the channel, and ties the action's lifetime to the channel: a goroutine copies and discards any input, and when the channel closes (central gave up) it cancels the action. It then logs{action, params, outcome: start}to its JSONL file. -
The guard.
executecallsactions.Set.Run->Plan(gate 22): unknown action, an extra parameter, a missing parameter without a default, a value that does not fully match its regex (\A(?:pattern)\z, so authors cannot accidentally leave it unanchored), or a value starting with-(option injection, unlessallow_leading_dash) all produce aRefusedError. OtherwisePlansubstitutes the values into the argv template in a single pass: a value that itself contains{{x}}is never re-expanded. Sotail-logswithlines=200becomes["/usr/bin/journalctl", "-u", "gitlab-runner", "-n", "200"]. -
execArgv(gate 23) starts the process withexec.CommandContext-- an argv list, never a shell -- with onlyPATH(a fixed path) andLANG=Cin its environment, working directory/, its own process group (Setpgid), stdout and stderr limited to 1 MiB combined, and a deadline (actions.yamltimeout, default 60 s). When the deadline passes or the context is cancelled,killTreesends SIGKILL to the whole group, so a grandchild process cannot survive. -
Runner audit and reply. The runner writes the outcome line to
audit.jsonlbefore it writes the result to the channel:refused,timeout,failed(non-zero exit) orok. ThenencodeResultmarshals anExecResult(if it would exceed 8 MiB it is replaced by a small error result), writes it and callsCloseWrite. -
Central reads the result.
execOnreads at mosttransport.MaxResultBytes(8 MiB) plus one byte; more than that is an error ("result too large"). A result that does not parse is an error. A guard refusal comes back as a result withRefusedset, not as an error. -
Outcome row.
RunActionmaps what happened to an outcome:Situation Outcome Wire answer Runner kicked (revoke) cancelled, message "runner revoked"unavailableOur deadline passed, caller still waiting timeout, "timed out"timeoutCaller's request was cancelled cancelled, "request cancelled"unavailableRunner offline or channel failed error, "runner unavailable"unavailableGuard refused error, "refused by runner: ..." (clipped to 200 bytes, control characters removed)bad_requestAction ran (any exit code) ok, exit code stored;timeoutif the runner killed it for timethe result Note that on central,
okmeans "the action ran and reported an exit code", even when that code is non-zero; the code is inexit_code. On the runner's own file a non-zero exit isfailed. The outcome row copies the start row (samerequest_id), drops the params (already in the start row) and uses the runner's reported duration if there is one. -
The outcome write survives cancellation. It uses
context.WithoutCancel(ctx)with its own 10 s limit (outcomeWriteTimeout), so a cancelled request still gets its row. If that write fails it is only logged: the command already ran, and refusing to say so would not undo it. -
Answer. The handler returns
ActionRunResult{exit_code, stdout, stderr, truncated, duration_ms, request_id}.methodTable.servewraps it withapi.NewResult;writeResponseTimeoutwrites the frame and closes the connection if the write stalls forapi.WriteTimeout(10 s), so a client that stops reading cannot pin a handler forever.Client.callOnchecks the version and that the response id matches the request id, andmcpAPIprints indented JSON.
The two audit rows (central).
| id | action | actor | fleet/runner | subject | params | outcome | exit_code | request_id |
|---|---|---|---|---|---|---|---|---|
| 1041 | action.run |
operator durga (source api) |
prod / db-1
|
tail-logs |
{"lines":"200"} |
start |
none | 9f1c…e07a |
| 1042 | action.run |
operator durga (source api) |
prod / db-1
|
tail-logs |
none | ok |
0 |
9f1c…e07a (same) |
Rows 1041 and 1042 are not adjacent in a busy system, but request_id ties them together. The table is
append-only: triggers abort any update or delete. The runner's audit.jsonl holds the matching pair
(start, then ok) with the same action and params but no request_id.
Why two rows? The start row proves intent even if central crashes mid-command. The outcome row proves result. A start row with no outcome row tells an investigator "something began and we never learned how it ended".
In one sentence: aarvee revokes dev/build-1 at the moment durga's restart-service is running
on it, and the revoke must be immediate and must not leave a running process behind.
Look at the order: tombstone first, kick second, commit third. At no moment can the runner slip back in.
- aarvee runs
central runner revoke dev/build-1. The CLI sendsrunner.lifecycle(oprevoke) over the admin socket. - Tombstone: build-1's id goes into the registry, so no new session can slip in during the commit.
- Kick: the session context is cancelled and the runner's SSH connection is closed.
- In-flight action cancelled: the runner kills the process group; the outcome row says
cancelled. - Commit:
status = revokedand the audit row are written in one transaction. - The runner redials. Auth finds a revoked key and refuses it: it never gets a session.
Part 1: central marks the runner dead and wakes the in-flight call.
sequenceDiagram
participant Service as service.Service
participant Registry as registry.Registry
participant Conn as runner SSH connection
participant RunAction as RunAction (durga's call)
participant sqlite as sqlite.DB
Note over RunAction,Registry: Registry.Exec is blocked reading the exec channel
Service->>Service: lifecycle(revoke): authorize, GetRunner, CanTransition
Service->>Registry: Tombstone(id)
Service->>Registry: Kick(id, "runner revoked")
Registry->>Registry: session ctx cancelled with KickError
Registry->>Conn: Close
Registry-->>RunAction: Exec returns context.Cause = KickError
RunAction->>sqlite: RecordAudit (outcome cancelled, "runner revoked")
Look at the order of the two registry calls: tombstone, then kick.
Part 2: the runner kills the process, central commits, and the runner is refused when it returns.
sequenceDiagram
participant Service as service.Service
participant sqlite as sqlite.DB
participant Conn as runner SSH connection
participant Runner as runner.Runner
participant Proc as action process group
Conn-->>Runner: connection lost, session context cancelled
Runner->>Proc: cmd.Cancel -> killTree (SIGKILL to the group)
Runner->>Runner: audit.jsonl: outcome failed, exit_code -1
Service->>sqlite: SetRunnerStatus(id, from, revoked, audit runner.revoke)
sqlite-->>Service: committed
Runner->>Conn: reconnect after backoff
Conn-->>Runner: auth denied (status revoked)
Look at the three places the process dies from: the cancelled context on central, the closed channel on the runner, and the closed connection. Any one is enough.
-
Service.lifecyclewithop = revoke:authorize(admin only), the "from" set is pending, active or disabled (a second revoke answersnot_found, becauseGetRunnerhides revoked runners), and the audit event gets the messageRevokeReason= "runner revoked". -
Registry.Tombstone(id)adds the runner's database id to an in-memory set. Ids come fromAUTOINCREMENTand are never reused, so a tombstone can never accidentally block a different, later runner. After this line,Registry.Registerrefuses the id (ErrTombstoned), under the same lock that guards the sessions. This closes the window between "the runner authenticated" and "the database says revoked". -
Registry.Kick(id, reason)finds the session and callss.cancel(&KickError{Reason}), thenconn.Close(). The registry lock is not held while closing a connection (the package comment forbids holding it across network I/O). -
The in-flight call wakes up.
Registry.Execlinked the session context to its exec context withcontext.AfterFunc(sess.ctx, cancel), so the cancel closes the exec channel andexecOnreturns.Execnotices that the caller's context is fine but the session's is not, and returnscontext.Cause(sess.ctx), the*KickError. -
RunActionrecognises theKickError(errors.As) and writes the outcome rowcancelledwith the message "runner revoked", then returnsunavailable: runner revokedto Durga. Durga's tool sees why it stopped. -
On the runner, the connection drop ends
session; its deferredcancel()cancels the context that everyserveExecderives from. Also, the closed channel ends the goroutine copyingio.Copy(io.Discard, ch), which cancels the action.exec.Cmd.CancelcallskillTree: SIGKILL to the process group, so the shell-free child and any grandchild die. The runner'saudit.jsonlrecordsfailedwith exit code -1 (the channel is gone, so the result never reaches central; central already has its owncancelledrow). -
Commit.
Store.SetRunnerStatusmoves the runner torevokedand writesrunner.revokein one transaction (synchronous=FULL, so it survives power loss). The tombstone stays after a successful commit:registry.Statushas no "revoked" value, so the tombstone is what keeps the id out of the live registry. Tombstones live only in memory, which is safe because after a restart a revoked runner is refused at authentication by its database status. -
If the commit fails,
lifecycleremoves the tombstone only if a fresh read proves the runner is not revoked (ClearTombstone). If the read also fails, it keeps the tombstone: fail closed. -
The runner reconnects (flow 10). Two outcomes. If the commit has finished,
KeyAuthseesStatusRevokedand denies the key at the handshake. If the commit is still in flight, the key still authenticates, but the first heartbeat reachesRegistry.Register, which refuses the tombstoned id;serveRunnerrepliesfalseand closes the connection. Either way the runner never regains a session. It will keep retrying with growing backoff, because it cannot tell "revoked" from "temporarily refused". - The revoked runner disappears from
runners.list(ListRunnersexcludes revoked) and its name can be reused by a new runner with a new key. The old key can never re-enrol:runners.pubkeystaysUNIQUEeven for revoked rows.
In one sentence: aarvee revokes kalyan while kalyan has a long action running, and central commits the
change first and only then cuts kalyan's connections.
aarvee runs mirafleet central operator revoke kalyan.
sequenceDiagram
participant adminCLI as adminCLI
participant Service as service.Service
participant sqlite as sqlite.DB
participant OperatorConns as central.OperatorConns
participant serveOperator as serveOperator (kalyan's connection)
participant RunAction as RunAction
participant Registry as registry.Registry
adminCLI->>Service: operator.revoke {name: kalyan} (aarvee)
Service->>sqlite: RevokeOperator(name, now, audit operator.revoke)
sqlite-->>Service: operator (status revoked), committed
Service->>OperatorConns: KickOperator(id, "operator revoked")
OperatorConns->>serveOperator: Close each ssh.ServerConn
serveOperator->>serveOperator: connection ctx cancelled (sconn.Wait)
serveOperator->>RunAction: request ctx cancelled
RunAction->>Registry: Exec ctx cancelled -> channel closed, runner kills process
RunAction->>sqlite: RecordAudit (outcome cancelled, "request cancelled")
Note over adminCLI,Registry: Kalyan reconnects: KeyAuth sees status revoked, denied
Look at the order: commit first, kick second. A revoked key can never authenticate again.
-
Service.RevokeOperator(admin only) validates the name and callsStore.RevokeOperator, which flips the operator torevokedand writes the audit row in one transaction. - Only after the commit it calls
OperatorConns.KickOperator(id, reason).OperatorConnskeeps a set of livessh.ServerConnper operator id;KickOperatorcopies the set under the lock and closes the connections after releasing the lock. - Closing the connection ends
sconn.Wait()inserveOperator, which cancels the connection context. EveryserveAPIChanneland everyRunActionderived from it is cancelled. For Kalyan's running action,Registry.Exec's context ends, the exec channel closes, and the runner kills the process group, exactly as in flow 8, step 6. - The outcome row says
cancelled, message "request cancelled" (not "runner revoked": the runner is fine, the caller is gone). -
The race. Kalyan could authenticate just before the commit, and
OperatorConns.addcould run after the kick. Then the kick would find nothing to close. That is whyserveOperatorre-reads the operator by id right afteraddand closes the connection if it is no longer active. Revoke commits before it kicks, so either the kick or this check catches every connection. - After revoke,
KeyAuth.Authenticate(useroperator) returnsErrDeniedfor Kalyan's key. If kalyan registers again with the same key, the store answersconflict(the key belongs to a revoked operator). A new key under the namekalyanis allowed, because the unique index on names covers only non-revoked operators.
Why the orders differ
- Runner revoke: tombstone, then kick, then commit (
Service.lifecycle). Runners reconnect on their own with backoff, and a reconnecting runner can authenticate and register at any moment, including while the commit is still in flight. So central first marks the id dead in memory (the tombstone), then kicks the live session, and only then commits. If the commit fails, the tombstone is cleared again, but only if a fresh read proves the runner is not revoked.- Operator revoke: commit, then kick (
Service.RevokeOperator). An operator's status is re-read from the database on every request, so the moment the commit lands, no new request can succeed. The kick only has to cut the connections that were already open. There is no in-memory mark to maintain because the database is the mark.The shared idea: close every gap through which the revoked party could act. For a runner the gap is its own auto-reconnect; for an operator, the gap is only connections that already exist.
In one sentence: when the network fails, the runner retries with growing, jittered delays and central drops sessions that go quiet, so both ends recover without anyone's help.
sequenceDiagram
participant Runner as runner.Runner (Run loop)
participant Server as central.Server (serveRunner)
participant Registry as registry.Registry
Runner->>Server: session 1: connected, heartbeats acked
Note over Runner,Server: network cable pulled, no TCP reset
Runner->>Server: heartbeat (no reply)
Runner->>Runner: wait 2 x 15 s then client.Close
Registry->>Registry: watch: no heartbeat for 45 s -> conn.Close
Server->>Registry: Unregister(session)
Runner->>Runner: backoff 1 s (+/-20 %), dial fails
Runner->>Runner: backoff 2 s, 4 s, 8 s ... up to 60 s
Runner->>Server: dial succeeds, handshake, first heartbeat
Server->>Registry: Register (replaces any old session for this id)
Runner->>Runner: stays up 30 s with an acked heartbeat -> backoff resets to 1 s
Look at the last step: only a connection that proved itself healthy resets the backoff.
Runner side (Runner.Run and Runner.session):
-
Runloops until its context ends. After eachsessionit waitsjitter(backoff), a random value within +/-20 % of the current backoff, so a hundred runners that lost the same central do not all reconnect in the same second. Thenbackoff = min(backoff*2, BackoffMax). Defaults:BackoffMin1 s,BackoffMax60 s. - The backoff resets to the minimum only if
sessionreportshealthy: a heartbeat was acknowledged and the connection lasted at leastHealthyAfter(30 s). A central that accepts TCP and SSH but fails right after (say, it refuses to register the runner) would otherwise let the runner hammer it once per second. -
Runner.heartbeatwaits at most2 x HeartbeatEvery(30 s) for each reply. On a timeout it closes the client, so the loop reconnects instead of waiting on a half-dead TCP connection forever.client.Wait()returning also ends the session ("connection lost").
Central side (registry.Registry and serveRunner):
- A session records the time of its last heartbeat (
lastSeen).Registry.staleis true when that is older thanStaleAfter= 3 x 15 s = 45 s. Thewatchgoroutine checks everyStaleAfter/6(7.5 s) and closes the connection of a stale session. Closing it makesserveRunner's loop end, which runs the deferredUnregister. - A stale session is offline immediately, even before the watcher notices:
Registry.Getreturns nil for it, andSnapshotreportsOnline: false. Soaction.runto a silent runner fails fast with "runner unavailable" andrunners.listshows it offline. -
A new connection wins. If the runner reconnects before the old session is noticed as dead,
Registry.Registerreplaces the old session, cancels its context and closes its connection. An action that was in flight on the old connection ends aserror("runner unavailable"). - A connection that never sends a valid heartbeat is closed by
serveRunner's guard timer afterStaleAfter.
stateDiagram-v2
[*] --> offline
offline --> online: first heartbeat registered
online --> online: heartbeat within 45 s
online --> offline: disconnect, stale 45 s, kicked, or replaced
Look at how online/offline is separate from pending/active: a runner has both a lifecycle status
(stored) and a presence (live, in the registry).
In one sentence: on SIGINT (Ctrl-C) or SIGTERM, central stops accepting, waits for in-flight requests to finish their audit rows, then closes the database and releases the lock.
sequenceDiagram
participant OS as operating system
participant centralRun as centralRun
participant runningCentral as runningCentral.Close
participant Server as central.Server
participant sqlite as sqlite.DB
OS->>centralRun: SIGTERM, ctx done
centralRun->>runningCentral: deferred Close
runningCentral->>runningCentral: cancel() (admin listener loop stops)
runningCentral->>Server: Close: closed=true, close listener, connections, handshakes, admin
Server->>Server: wg.Wait for every handler to return
Server-->>runningCentral: handlers done (in-flight actions audited as cancelled)
runningCentral->>sqlite: Close
runningCentral->>runningCentral: release data-dir lock
Look at the order: connections drain before the database closes, and the lock is released last.
-
signal.NotifyContextturns SIGINT or SIGTERM intoctx.Done(), socentralRunreturns from<-ctx.Done()and itsdefer rc.Close()runs. -
runningCentral.Closeruns four steps in this order:cancel()(stops the admin accept loop),srv.Close(),d.db.Close()and the lockrelease(). -
Server.Closesetsclosedunder its mutex (so no new connection can be tracked:trackandCloseuse the same lock, which removes a WaitGroup race), closes the SSH listener, every establishedssh.ServerConn, every connection still in handshake and the admin listener and its connections, and then waits (wg.Wait) for all handler goroutines. -
In-flight work. Closing an operator's connection cancels its requests, so an
action.runin progress ends with an outcome rowcancelled("request cancelled"). That row is written withcontext.WithoutCancelinside the handler, and the handler is part of the wait group, so it is written before the database closes. Closing a runner's connection also kills the runner's process group for that action. -
ServeAdmin's accept loop removesadmin.sockon its way out, but only if the file is still the one it created (it compares device and inode), so a new central that started quickly does not lose its socket. - Then the database closes and the lock is released. A new
central runcan start immediately. -
What the runners do: their connection drops, and they follow flow 10: retry with backoff while
central is down, then reconnect and re-register. Operators see errors from
mcp apiuntil central is back. Tombstones are lost on restart, but that is safe: revoked runners are refused by their stored status at the handshake. - One caveat: the background
touchergoroutines that writelast_seen_atare not in the wait group (they exist so a slow database never delays a heartbeat). A write still queued at shutdown can fail against a closed database; it only produces a log line.
Flows 12 to 18 are the workspace-mode story (new in M3). Here the cast is dev (a workspace fleet, not
guarded), its runner build-1 (started with -workspace /etc/mirafleet/workspace.yaml,
concurrent: 2, containerd executor), and durga and kalyan. aarvee created the fleet earlier:
mirafleet central fleet create dev -mode workspace \
-image docker.io/library/golang@sha256:... -images 'docker.io/library/golang@sha256:*' \
-network none -cpus 2 -memory 4GiB -idle-ttl 24h -max-ttl 168h
Four terms recur. A slot is one place for a workspace on a runner (concurrent: 2 means two
slots). A session is a name the client chooses (for example Claude Code's session id); there
is at most one live workspace per (operator, fleet, session), which is what makes it sticky.
A workspace id (ws_id) is 32 hex characters that central makes (128 random bits); the
runner uses it as the container name mf-<ws_id>. A request id ties the start audit row
of a call to its outcome row.
In one sentence: durga's agent asks for a workspace on dev; central checks the rules,
picks the runner with the most free slots, writes a start audit row and a reserved
(opening) database row, asks the runner to create the container, and then marks the row
open.
Look at the order: the slot is reserved in the database before the runner is asked to build anything, so two simultaneous opens can never over-fill a runner.
- durga's agent calls
workspace.open(fleetdev, sessions1) throughmirafleet mcp. - The
api@frame reachesservice.OpenWorkspace: active operator, workspace-mode fleet, image on the allowlist,confirmon a guarded fleet, valid session id. - Sticky reuse: a live workspace for (durga,
dev,s1) is returned withreused:true. If its runner is offline the answer isunavailable; a fresh workspace is never created elsewhere. - Placement: active, online runners whose executor, trusted labels and image patterns fit, with free slots (heartbeat slots minus live rows). The most free slots wins, random tie-break.
-
ReserveWorkspacerecounts the slots in one transaction and inserts theopeningrow with the start audit row. - A full runner (
ErrFull) or a busy one moves to the next candidate (3 tries at most). Otherwise central sends opcreateonworkspace@mirafleet. - The runner re-validates the spec against its own
workspace.yaml, builds the container spec from its fixed template, pulls the image if needed and creates the container. - The row becomes
open, the outcome audit row is written and the reply saysreused:false.
The same flow as a sequence diagram, with the function names:
sequenceDiagram
participant Op as durga (mcp api)
participant Svc as service.OpenWorkspace
participant DB as sqlite.DB
participant Reg as registry.Registry
participant Run as build-1 (serveWorkspace)
participant Ex as ctrd.Executor
Op->>Svc: workspace.open {fleet dev, session s1}
Svc->>DB: GetFleet, GetFleetWorkspace, live rows of durga
Svc->>Svc: sticky? per-operator cap? image allowed? candidates()
Svc->>DB: ReserveWorkspace (row opening + start audit row), one transaction
Svc->>Reg: Workspace(create, spec)
Reg->>Run: workspace@mirafleet channel, op create
Run->>Run: re-validate against workspace.yaml, take a slot
Run->>Ex: Create(spec)
Ex-->>Run: container running
Run-->>Reg: WorkspaceResult ok
Svc->>DB: SetWorkspaceState opening to open (+ outcome audit row)
Svc-->>Op: workspace {id, state open}, reused false
-
Gates.
handleWorkspaceOpen(internal/central/workspace_handlers.go) decodes the parameters strictly.Service.OpenWorkspace(internal/service/workspace.go) checks that the caller is an active operator, the fleet name and the session (^[A-Za-z0-9._:-]{1,128}$) are valid, the labels are well-formed, and the fleet is in workspace mode (otherwiseconflict). On a guarded fleetconfirmmust equal the fleet name, or adeniedrow is written and the call fails. -
Image. An empty
imagemeans the fleet'sdefault_image. The image must match the fleet's allowlist (WorkspaceConfig.ImageAllowed: exact references,repo:*orrepo@sha256:*), or the call fails withbad_request. Without this check an agent could ask for any image from the internet, including one built to attack the host. -
Sticky session. Central lists durga's live workspaces. If one with the same fleet and
session is
open,reusereturns it withreused: trueand writes areusedaudit row. If its runner is offline the answer isunavailable: central never silently makes a fresh workspace, because durga's files are on that runner. If the existing row is stillopening, the answer isconflict("workspace is still opening"). -
Per-operator cap. An operator may hold at most 8 live workspaces (the service setting
DefaultMaxWorkspacesPerOperator), or the call fails withconflict. Without it, one runaway agent could take every slot of a fleet. -
Placement.
Service.candidates(workspace_sched.go) keeps runners that are:activeand online; running the executor the fleet asks for; advertising a secure runtime if the fleet is guarded; carrying every requested label among their trusted labels (the ones that came from the join token, not the self-reported heartbeat); and advertising an image pattern that matches. Free slots are the heartbeat'sslotsminus live rows in the database (opening,openandclosingall count). The runner with the most free slots wins, with a random tie-break fromcrypto/rand. -
Reserve.
Store.ReserveWorkspacecounts and inserts in one transaction, together with thestartaudit row (fail closed: no row, no workspace). If the runner filled up in the meantime it returnsErrFulland central tries the next candidate (at most 3). -
Create.
registry.Workspaceopens aworkspace@mirafleetchannel on the runner's own connection and sends acreaterequest (image, pull policy, limits, network mode, egress rules, labels). The 5-minute open timeout covers a slow image pull. -
The runner re-checks everything.
wsState.preparevalidates the request againstworkspace.yaml: the image against the runner's own allowlist, limits against its caps, the network mode and egress rules against what it permits. If the runner is out of slots it rejects the channel before accepting it (ResourceShortage), which central reports as busy and answers by trying the next candidate. Only then doesctrd.Executor.Createbuild the OCI spec and start the container (flow 13 shows what that spec contains; chapter 5 explains why). -
Outcome. On success the row moves
openingtoopentogether with the outcome audit row. On any failure it moves tofailedwith a reason, and a busy runner makes central try the next one. Image or pull failures are not retried elsewhere.
In one sentence: durga's agent runs go test ./... inside their workspace; the command runs in the
container, the result comes back as one buffered answer, and two audit rows record exactly what
ran.
Look at the two rows again: start is written before the runner is asked, the outcome after,
and both carry the same request_id and the workspace id.
- durga's agent calls
workspace.exec(argvgo test ./...) throughmirafleet mcp. - One
api@frame arrives; the method table routes it toservice.ExecWorkspace. - Operator, ownership (
not_foundfor others), stateopen, argv, cwd, env (denylist), confirm and timeout are checked; then thestartaudit row is written FIRST. No row, no exec. -
registry.Workspaceopens aworkspace@mirafleetchannel on the runner's own SSH connection; the op is JSON in the channel-open data, any stdin follows as raw bytes. - The runner re-validates argv, cwd, env (its
env_allowplus the denylist) and the timeout. Central is not trusted. - The command runs inside the container as an unprivileged user, with no shell. Output is captured head and tail, trimmed to the budget; timeout or cancel sends SIGKILL.
- The runner writes one result frame: exit code, stdout, stderr,
truncated,timed_out. - Central writes the
outcomeaudit row (samerequest_id) and replies to durga.
The same flow as a sequence diagram:
sequenceDiagram
participant Op as durga (mcp api)
participant Svc as service.ExecWorkspace
participant DB as sqlite.DB
participant Run as build-1 (runWS)
participant Ex as ctrd.Executor
participant C as container
Op->>Svc: workspace.exec {workspace, argv [go test ./...]}
Svc->>DB: owner check (not_found otherwise), state open
Svc->>DB: audit row start (argv, cwd, env NAMES), fail closed
Svc->>Run: workspace@mirafleet, op exec
Run->>Run: re-validate, take an exec slot (max 4 per workspace)
Run->>Ex: Exec(argv, cwd, env, timeout)
Ex->>C: task.Exec (no caps, uid 1000)
C-->>Ex: exit code, stdout, stderr
Ex-->>Run: result (head+tail capture)
Run-->>Svc: WorkspaceResult exec
Svc->>DB: audit row outcome (same request_id)
Svc-->>Op: exit_code, stdout, stderr, truncated, timed_out
-
Find the workspace and its owner.
Service.wsTargetloads the row byws_id. It must exist, be owned by the caller, and beopen. For anyone else (kalyan, for instance) the answer is the samenot_foundas for an id that does not exist, so kalyan cannot even learn that the workspace is there. -
Validate the input.
argvis an array, at most 256 elements and 32 KiB, no NUL bytes. There is no shell string on the wire, so the audit log shows exactly what ran.cwdis cleaned withworkspace.CleanPath(a lexical check:..and absolute paths outside/workspaceare refused).envnames must passworkspace.FilterEnv, which applies a hard denylist (*_API_KEY,*_TOKEN,*_SECRET*,ANTHROPIC_*,AWS_*,*PROXY*, ...) that beats any allowlist. A refused name is reported by name, never by value. -
Guarded fleets need
confirmequal to the fleet name here too. -
Timeout. The effective timeout comes from the fleet (
exec_default_s, default 120 s;exec_max_s, default 30 minutes, hard ceiling 24 h) and is further capped by the runner's ownlimits.exec_max. A client value above the maximum is refused. -
Audit
start, fail closed.startRowwrites the row before dispatch. If it cannot be written, the call fails withunavailableand nothing runs. The row holds the argv as JSON (clipped to 4 KiB), the cwd and the env names only, never values, plus the timeout. -
Touch.
last_used_atis updated at dispatch and again at completion, and a workspace with an op in flight is never reaped as idle (flow 16). -
Dispatch.
registry.Workspacesends theexecrequest on a newworkspace@mirafleetchannel. If the runner isdisabledorpending, the registry refuses it (adisabledrunner still allowsread,closeanddestroy, so people can rescue files). -
The runner runs it.
runner.runExecre-validates, takes one of at most 4 exec slots of that workspace, writes its own localstartline, and callsExecutor.Exec. In containerd that istask.Execwith an OCI process: the given argv, cwd under/workspace, filtered environment, uid 1000,NoNewPrivileges, no capabilities. If the fleet usesegress, the executor addsHTTP_PROXYandHTTPS_PROXYpointing at the in-container proxy. A deadline or a cancel sends SIGKILL to the exec. -
Buffered output. The runner captures the head and the tail of stdout and stderr and trims
them so the JSON-encoded result fits 1 MiB (
truncated: truetells the caller). Output is made valid UTF-8. There is no streaming in M3: one request, one response. -
Audit
outcome. Central writes the second row with the samerequest_id, the exit code, duration and an outcome ofok,error,cancelledortimeout, and returns{exit_code, stdout, stderr, truncated, timed_out, duration_ms, request_id}. - Cancel. If durga's channel closes (or central shuts down, or their operator key is revoked), the context is cancelled, the channel to the runner closes, and the executor kills the exec. One caveat: a daemonised child that double-forks can outlive an exec timeout, until the workspace is destroyed. The pids and memory limits bound the damage.
In one sentence: a file travels in its own length-prefixed body frame after the JSON request, and central reads that frame only after the caller has been authorised.
Why not put the bytes in the JSON as base64? Because every method's request would then need a
bigger size limit (or files would travel in tiny 48 KiB chunks). Instead a request frame stays at
64 KiB at most, and the file payload follows in one more frame of at most 1 MiB
(api.MaxFileChunk). Bigger files go in several calls with an offset.
sequenceDiagram
participant Op as durga (mcp api -in FILE)
participant Srv as central api@ channel
participant Svc as service.WriteWorkspaceFile
participant Run as build-1
participant FS as workspace dir (os.Root, container executor)
Op->>Srv: request frame {workspace, path, length, sha256}
Op->>Srv: body frame (length bytes)
Note over Srv: body is NOT read yet
Srv->>Svc: WriteWorkspaceFile(lazy body)
Svc->>Svc: policy, owner, state, confirm, path, mode, length, sha256 syntax
Svc->>Srv: now read the body (30 s timer)
Svc->>Svc: SHA-256 of body must equal sha256 (constant time)
Svc->>Svc: audit start row (path, bytes, sha256)
Svc->>Run: workspace@ channel: request, then length raw bytes, CloseWrite
Run->>FS: write (temp file and rename when truncating), confined by os.Root (container executor)
Run-->>Svc: result {size}
Svc-->>Op: {size}
Write (workspace.write_file):
- The client sends the request frame (
workspace,path,offset?,truncate?,mode?,length,sha256) and right away the body frame. SSH's 2 MiB channel window means that second write never blocks. -
handleWorkspaceWriteFilehands the service a lazy reader.Service.WriteWorkspaceFileruns every cheap check first: policy, ownership (not_foundfor others), workspaceopen,confirm, path cleaning,modeat most0777,lengthbetween 0 and 1 MiB,sha256is 64 lowercase hex characters. -
Only now does the first
Readpull the body from the wire, armed with a 30 s timer (api.BodyTimeout) that closes the channel. This is the point of the design: an unauthorised caller (or a revoked key, or someone else's workspace id) can never make central buffer up to 1 MiB. The cost of an attack is a refused small frame. - Central computes the SHA-256 of the body and compares it with the announced one in constant
time. A mismatch or a short body is audited as one
errorrow and nothing is sent to the runner. - A
startrow (path, byte count, sha256, never the content) is written, then central sends the runner the request followed by exactlylengthraw bytes andCloseWrite. The runner refuses a mismatch. - The container executor writes through
os.Rootopened on the workspace directory, so a symlink planted by the agent cannot lead outside it. (Thehostexecutor has no container: it writes with plain OS calls below the session directory and the OS user's permissions are the boundary.) Withtruncateit writes a temp file and renames it. New files are chowned to the container's mapped uid. A soft disk quota is checked before the write (over quota refuses; section 5.9 explains why it is only "soft").
Read (workspace.read_file):
- The request has
workspace,path,offset?andlimit?. There is no request body, so nothing special is deferred. - After the same gates, central writes a
startrow (path, offset, limit) and asks the runner. - The container executor opens the file through
os.Root(the host executor: plain OS calls from the session directory) withO_NOFOLLOWandO_NONBLOCK, then checks withfstatthat it is a regular file (a FIFO or a device would hang the runner), and replies with a result frame followed by the raw bytes. - Central buffers the bytes (at most 1 MiB), checks that their count matches the runner's
length, and only then sends durga the JSON result (size,offset,length,mode,eof) followed by one body frame. A runner that lies about the length therefore sends nothing at all. - The outcome row records the path and the byte count.
mirafleet mcp api workspace.write_file '{...}' -in main.go fills length and sha256 for you and
sets the body; -out report.txt on workspace.read_file saves the body (atomically, mode 0600).
In one sentence: the owner closes the workspace; the row goes open, closing, closed
and the container and directory are deleted.
sequenceDiagram
participant Op as durga
participant Svc as service.CloseWorkspace
participant DB as sqlite.DB
participant Run as build-1
participant Ex as ctrd.Executor
Op->>Svc: workspace.close {workspace}
Svc->>DB: SetWorkspaceState open to closing (+ start audit row)
Svc->>Run: workspace@ channel, op destroy
Run->>Ex: Destroy(id)
Ex-->>Run: task killed, container and snapshot deleted, directory removed
Run-->>Svc: ok (or not_found: already gone)
Svc->>DB: SetWorkspaceState closing to closed (+ outcome audit row)
Svc-->>Op: workspace {state closed}
- Only the owner may close; anyone else gets
not_found. Closing something already terminal does nothing and returns it. Closing one that is stillopeningis aconflict. - The row moves to
closingfirst. Aclosingrow still holds its slot, so the runner cannot be over-filled while the container is half deleted. -
Executor.Destroykills every process in the container (KillAll), waits, deletes the task and the container with its snapshot, closes the egress listener, then removes the directory withos.RemoveAll(which does not follow symlinks). It is idempotent: a second destroy of an absent container is not an error. - If the runner is offline or the destroy fails, the row stays
closingand the response says so. The reaper (flow 16) or reconcile (flow 17) finishes the job later. The user never has to retry.
In one sentence: every 60 seconds a goroutine in central closes workspaces that nobody uses any more, so forgotten workspaces cannot pile up and use disk forever.
-
Service.RunReaperis started with central (internal/central/central.go), runs in the server's wait group, and is stopped inClosebefore the store closes. It uses the injected clock, which is how the tests move time forward. - Each sweep (
ReapOnce) listsopenrows and moves one toclosingwhen any of these holds: its operator is revoked; its runner is revoked;expires_athas passed (the fleet'smax_ttl, default 7 days); or it has been idle for more than the fleet'sidle_ttl(default 24 h). -
Idle is counted from the later of
last_used_atand central's start. If central was down for two days, no workspace looks idle at the first sweep after restart, although nobody could reach it. A workspace with an operation in flight is never reaped as idle. - For each online runner, in a bounded number of goroutines (so one hung runner cannot stall the
rest), it destroys that runner's
closingrows and moves them toclosed. - Rows on an offline runner stay
closingand keep their slot until reconcile (flow 17). Rows on a runner that is revoked or has vanished move tolost: that runner will never return to be told to delete them, and its host keeps the containers untilmirafleet runner workspace prune. - Every state change writes a
systemaudit row (workspace.reap,workspace.destroy,workspace.lost) with the reason. Nobody has to be logged in. - A runner-local backstop exists too:
max_ageinworkspace.yamllets the runner destroy anything older, in case central is gone for good. Otherwise a runner never deletes on its own.
In one sentence: the database and the runner can disagree after a crash, so each time a runner connects central compares the two lists and fixes the difference.
flowchart TD
A["runner (re)connects,<br/>first heartbeat"] --> B["central: read live rows<br/>for this runner"]
B --> C["central: ask runner<br/>for List()"]
C --> D{"container vs row"}
D -->|"container, no row or row closed"| E["destroy it (orphan)"]
D -->|"container, row closing"| F["destroy, row closed"]
D -->|"container, row opening older than<br/>open timeout + 30 s"| G["destroy, row failed"]
D -->|"no container, row open"| H["row lost, slot freed"]
D -->|"no container, row closing"| I["row closed"]
-
Trigger. On a runner's first heartbeat
serveRunnerstartsService.ReconcileRunnerin a background goroutine (never under the registry lock). A second trigger for the same runner while one runs is ignored. - Rows first, list second. Central reads the rows, then asks the runner for its list. A workspace that opens between the two is not in the rows, so it cannot be mistaken for a missing container.
-
Decisions are in the flowchart. Each one writes a
systemaudit row (workspace.orphan,workspace.destroy,workspace.recover,workspace.lost). -
When the runner restarts: its containers keep running (they belong to containerd), so
openrows survive andexecworks again. The runner re-attaches the egress proxy listener to each container's network namespace, but only ifegressis still allowed inworkspace.yaml. Its slot table is re-seeded from the executor's list. -
When central restarts:
RecoverWorkspacesruns at startup. A row stillopeningmeans central crashed in the middle of a create, so it becomesfailed. Every other row simply persists in SQLite, and the container is still there. The first reconcile with each returning runner destroys any container that the failed row left behind.
In one sentence: revoking a runner or an operator also closes the workspaces they hold, in their own transaction with their own audit row, with the reaper as a backstop.
-
Runner revoke (flow 8): after the commit that sets
revoked,Service.closeWorkspacesOfmoves that runner'sopeningandopenworkspaces toclosingwith the reason "runner revoked", in one transaction with asystemaudit row (workspace.close_all). Nothing can be destroyed on that host: the runner never reconnects. The next reaper sweep moves the rows tolost. The containers stay on the host until an admin there runsmirafleet runner workspace prune. -
Operator revoke (flow 9): the connection kick cancels in-flight execs. Then the same cascade moves
the operator's workspaces to
closing, and the next sweep destroys them on their runners. - If the cascade write fails, it is only logged: the reaper checks for revoked owners and runners on every sweep, so nothing is missed, only delayed by up to 60 seconds.
-
Disabled runner.
disableis gentler: no newopen,execorwrite_file, butread_file,closeand destroy still work, so people can fetch their files first. -
Admin kill.
mirafleet central workspace kill ID(workspace.kill) moves any workspace toclosingand destroys it.workspace listshows every workspace with its operator.
Flows 19 to 23 are the orchestrator story (new in M5): what happens on durga's laptop between
their harness and central. Chapter 3 draws the packages;
chapter 2 explains MCP. The cast: durga runs Claude
Code with the mirafleet plugin in a project that has a .mirafleet.yaml saying fleet: dev,
and durga's contexts.yaml has one context, mirastack, pointing at central.example.com:2222. A
session is still the client-chosen name from flow 12; here the client is Claude Code, so the
session is cc: plus Claude Code's own session id.
In one sentence: Claude Code asks the MCP server to run go test ./...; the server resolves
where it should run, sends one workspace.exec to central, and returns capped, cleaned output
that the model must treat as data.
Look at the orange box: the output is trimmed on the laptop, after central and the runner have done their work, so the model never sees more than the cap.
- Claude Code (the harness) sends
tools/callexecwithcommand: "go test ./..."as one JSON line on the stdin ofmirafleet mcp. -
mcpserverdecodes the arguments strictly: unknown, duplicate or mis-cased names are refused before any handler runs. -
orchresolves context, fleet and session, and finds or opens durga's workspace. Its guard lets only operator methods through. - One
api@frame, methodworkspace.exec, goes to central over the pinned SSH connection. - Central runs its checks, writes the
startaudit row and sendsworkspace@to the runner (flow 13). - The runner runs the command in the container; exit code, stdout and stderr come back (1 MiB at most).
- On the way back
orchstrips ANSI and invisible characters and caps stdout plus stderr at 30 KiB (1/3 head, 2/3 tail, a marker between). -
mcpserverreturnsstructuredContentplus one text copy. The model gets capped, sanitised data that is not instructions.
The same flow as a sequence diagram:
sequenceDiagram
participant H as Claude Code
participant S as mcpserver (exec handler)
participant O as orch
participant C as central (flow 13)
H->>S: tools/call exec {command: "go test ./..."}
S->>S: DecodeArgs (exact-case, no unknown fields), one of command or argv
S->>O: Resolve (context, fleet), SessionFor
O->>C: fleets.list (guarded?), workspace.open if not bound
C-->>O: workspace id
S->>O: WorkspaceExec(argv /bin/sh -c ..., env, timeout)
O->>C: api@ frame workspace.exec
C-->>O: exit_code, stdout, stderr
O-->>S: result
S->>S: CapOutput (sanitise, 30 KiB)
S-->>H: structuredContent + text, isError false
-
Decode.
wrap(ininternal/mcpserver/server.go) refuses arguments that are not a JSON object.DecodeArgsthen checks every key of the tool's argument struct for an exact-case match, refuses duplicates and unknown fields, and refuses trailing data. The go-sdk does no argument checking for raw tools, so this is the only checker. Chapter 5 explains why exact case matters. -
Shape the command.
execneeds exactly one ofcommandandargv. Acommandbecomes["/bin/sh","-c",command]; the shell runs inside the container, which is the boundary.api.ValidateArgv,ValidatePath(forcwd) andValidateEnvrun locally first so an obviously bad request never costs a network round trip.stdinis text, at most 1 MiB. -
Resolve the target.
Orch.Resolvepicks the context and fleet in this order: the tool argument, then the-context/-fleetflags, then.mirafleet.yaml, then the context'sdefault_fleet. For durga that ismirastackanddev(source "project file"). A guarded fleet that came from a default (the project file ordefault_fleet) is refused: it must be named explicitly (flow 23). Every result carries awhereobject naming the context and fleet and where each came from, because a cloned repository's.mirafleet.yamlcan steer unqualified calls, and the user should be able to see that. -
Find the workspace.
SessionForgives the session (flow 20). The handler looks up the binding for (context, fleet, session). With no binding it opens one only on an unguarded fleet (fleets.listfirst; an unknown fleet fails closed), through the sameworkspace.openas flow 12; for a guarded fleet it says to callworkspace_openafter asking the user. -
One guarded path.
Orch.WorkspaceExecgoes through the privatecall, the only code that reachesapiclient.callrefuses any method for whichapi.IsOperatorMethodis false, takes one of 8 per-connection slots (api.MaxChannelsPerConn), applies the client-side time limit (CallLimit:timeout_splus a 30 s margin) and gets the context's one SSH connection from the pool (dialled lazily, with the host key pinned fromknown_hosts; never trust on first use). -
Central and the runner do flow 13, unchanged. Nothing in central knows an MCP server was
involved: it sees an operator key and a
workspace.execrequest. -
Cap and clean.
orch.CapOutputsanitises stdout and stderr (ANSI sequences, control characters, invisible Unicode) and fits both into the output cap (default 30 KiB, set byoutput_cap_bytesincontexts.yaml, 4 KiB to 256 KiB). A stream that does not fit keeps a third head and two thirds tail around[mirafleet: N bytes omitted], and the result saystruncated: true. Why 30 KiB: it stays under Claude Code's warning size for tool output. -
Answer. The result object (
exit_code,stdout,stderr,truncated,omitted_bytes,duration,workspace,where) is JSON-encoded once and sent asstructuredContentplus the same text. If the harness sent aprogressToken, a progress notification went out every 15 s while this waited. If central dropped the connection mid-call,execis never retried (it may have run): the error says "the connection to central was lost; the call may or may not have completed". Lists,read_fileandworkspace_openare retried once on a fresh connection. -
Cancel. If the user presses Esc,
notifications/cancelledcancels the handler's context.callreturns, the channel closes, and central cancels the exec exactly as in flow 13 step 11; the audit outcome iscancelled.
In one sentence: the workspace is tied to a session name, and a tiny file lets the server learn Claude Code's session id, so resuming the conversation brings back the same container.
Look at the middle box: it is the only thing the hook and the server share, and it holds a session id, not a secret.
- Claude Code starts session
abcand runs the SessionStart hook. The hook readssession_idfrom its stdin and makes no network call. - The hook writes
cc:abcto a handoff file in the state directory, keyed by the parent PID (Claude Code) and the project directory. The file is mode 0600 in a 0700 directory. -
mirafleet mcp, another child of the same Claude Code process, re-reads the file at every workspace open. -
workspace.opencarries sessioncc:abc; central opens durga's workspace or returns the one durga already has. - Claude Code exits. Nothing is closed: the workspace stays
openand the 24 h idle TTL reaps it later. -
claude --resume: same session id, new PIDs. The hook runs again and rewrites the file for the new PID. - The server reads
cc:abcagain and central answersreused: true: same container, same files. - After
/clearthe id is new, so the next open binds a new workspace. With no hook there is no file, and the server uses a random per-process sessionp:....
In detail:
- Why a file. The MCP server never sees Claude Code's session id: MCP does not carry it. The hook does (it is in the hook's input), but the hook and the server are separate processes. So the hook leaves the id where the server can find it.
-
The order.
Orch.SessionFortries, every time: (a) the-sessionflag (works with any harness, no hooks); (b) the handoff file for (parent PID, project directory), written by a hook; (c) a randomp:<32 hex>made once per server process (crypto/rand). The result is reported back assession_source(flag,handofforprocess) inworkspace_open. -
The handoff file.
internal/handoffstores{v, session, project, pid, written}assessions/<sha256(pid|project) first 32 hex>.jsonin the state directory (per-OS paths). The directory is0700and the file0600, written atomically (temp file, then rename). A symlinkedsessionsdirectory is refused, files older than 7 days are pruned on write, andReadreturns "not found" for a file whose pid or project differs from the key. -
Why the parent PID. One laptop can run several Claude Code windows in the same project. Each
has its own PID, so each gets its own file and its own workspace. This relies on the hook
process and the MCP server both being direct children of Claude Code (
os.Getppid()is the same in both). That topology is an assumption checked in the real-Claude-Code smoke test; if it is false the server falls back to (c), which is safe and only loses--resumereattach. -
Re-read at every open. After
/clear, Claude Code starts a new session with a new id; the SessionStart hook rewrites the file; the next open sees the new session and binds a different workspace. The server's in-memory binding is keyed by (context, fleet, session), so the old binding is simply not used. -
Validated. Every value passes
api.ValidateSession(letters, digits,.,_,:,-, at most 128 characters). -
A planted file is harmless. If someone plants a handoff file, the worst it can do is point
the server at another session of the same operator: central enforces ownership of workspaces
per operator key (
not_foundfor anyone else's).
In one sentence: when the harness goes away the MCP server shuts down in seconds but leaves the workspace alone; central's reaper deletes it later if nobody comes back.
- The harness closes the server's stdin (or sends SIGTERM or SIGHUP; on Windows
os.Interrupt). -
mcpserver.Servereturns.runServe(cmd/mirafleet/mcp_serve.go) callsOrch.Close, which cancels every in-flight call (closing a channel makes central cancel the work, flow 13 step 11) and closes the SSH connections. It waits at most 5 seconds, then exits 0. -
No workspace is closed. The row stays
openand holds its slot. Closing on every exit would destroy the files of a user who merely restarted Claude Code. - If a session id is stable (the handoff file, or
-session), the next server finds the same workspace (flow 20). With a random per-process session (no hooks), it cannot: the new server opens a new workspace, which takes another slot. The old one waits for the reaper. - Flow 16's reaper moves a workspace to
closingwhen it has been idle for the fleet'sidle_ttl(default 24 h) and destroys it. The integration testTestM5Doneshows the whole chain: close durga's stdin,Servereturns within 5 seconds, the row is stillopen; a new server with a new random session gets the other runner (concurrent: 1); a third getsunavailable(no slot free); advance the injected clock past the idle TTL and the reaper closes the first row. - To free the slot sooner: call
workspace_close, or (in Claude Code) end the session with/clearor logout, which runs flow 22.
In one sentence: the Claude Code plugin cleans up after a session that is truly over, and keeps the workspace when the session may come back.
- Claude Code runs the SessionEnd hook (
mirafleet mcp hook claude-code session-end) and passes a JSON object withsession_idandreason. - The hook removes its handoff file, but only if the file still names this session
(
handoff.RemoveIf). After a/clear, a newer session may already have rewritten it. - If
reasonis notclearorlogout, it stops.resume,prompt_input_exit(the user quits Claude Code) andotherkeep the workspace, because the user may--resumeit; the idle TTL is the safety net. - For
clearandlogoutit builds its ownorch(a SessionEnd hook is a new process: it has no connection), lists the workspaces of every context (workspace.list, own only), picks the ones whosesessioniscc:<this session id>and sendsworkspace.closefor each. - It is bounded and never blocks the session: 10 seconds of its own (the hook's registered timeout is 15), every error goes to stderr, and the exit code is always 0.
Why not close on every exit? A closed workspace loses its files. Quitting Claude Code is far more
often "see you after lunch" than "I am done forever", and a forgotten workspace costs only a
slot for at most idle_ttl.
In one sentence: on a guarded fleet (prod), the first call is refused, the model must ask
the human, and only the repeated call with confirm: "prod" goes through, after one more
question from the harness.
- The model calls
run_actionwithfleet: "prod",runner: "db-1",action: "tail-logs"and noconfirm. (Before that, the PreToolUse hook runs, sees noconfirm, prints nothing, and the call proceeds.) -
Orchsendsaction.run. Central refuses:bad_request, "fleet requires confirm to equal the fleet name", and writes adeniedaudit row (flow 7). -
mcpserver(guardedRewrite) replaces central's text with: "fleetprodin contextmirastack(...) is guarded. Ask the user. Only after the user explicitly agrees, repeat the call withconfirmset to the fleet name." The model is told to stop and ask. - durga reads the model's question and agrees. The model repeats the call with
confirm: "prod". -
PreToolUse hook. Before the harness runs the tool, Claude Code runs the plugin's hook with
the tool call as JSON on stdin. The hook sees
confirmis set and prints a decision ofaskwith a reason built from sanitised, clipped fields:Guarded fleet "prod": run_action tail-logs on db-1. Claude Code shows that to durga as a permission prompt, whatever its permission mode is (this is checked in the smoke test; see the note below). The hook can only ever sayask, neverallow. - durga approves.
mcpserverpassesconfirmverbatim: it is never filled from config, never cached, never defaulted. - Central checks
confirm == "prod", writes thestartaudit row, and flow 7 continues.
Three rules keep the model out of trouble:
-
Central is the enforcer. If the plugin is not installed, the hook crashes, or the harness
is not Claude Code, central still refuses every guarded call without the exact
confirm. The hook only adds a clear question for the human. In other harnesses the human gate is the harness's normal tool-approval prompt. -
No guarded fleet by default. A fleet named only by
.mirafleet.yamlordefault_fleetis refused when it is guarded, and the implicit open inexec,read_fileandwrite_filenever targets a guarded fleet. The model has to nameprodexplicitly, which is the moment to ask. -
Be exact. The hook and the server read the argument names exactly. A call with
Confirm(capital C) is refused by the server, and the hook, which cannot tell what such a call means, asks with a fixed "ambiguous arguments" reason. So a disguisedconfirmcannot slip past either one (section 5.11 tells the story).
Note on verification: whether a PreToolUse ask still prompts in a permissive permission mode is
one of the conventions to check by hand with real Claude Code (M5 step S17). Nothing depends on it,
because central refuses without confirm; it only changes who sees the question.
Flows 24 to 29 are the admin GUI story. The cast: aarvee, the admin, runs central on
central.example.com. The GUI listens on https://central.example.com:8443. Chapter 3 explains
where the GUI fits, chapter 2
the web basics and chapter 5
how it is attacked. Two promises hold in every flow: the
GUI calls the same service methods as the CLI, and a secret (setup token, password, session
token) is never written to a log or to an audit row.
In one sentence: the first time central starts without a GUI admin it makes a one-time setup
token and leaves it in a file only aarvee can read; aarvee pastes it into /setup to create the
admin account and aarvee's first session.
Follow the setup token from central's start to the file, then to aarvee's browser and the first session.
- Central starts (
startGUIincmd/mirafleet/central_gui_linux.gocallsServer.PrepareSetup). There is no active admin, soservice.EnsureGUISetupmakes 32 random bytes, formats them asmfs1.<base64url>, and stores only the SHA-256 plus a 24-hour expiry in tablegui_setup(audit actiongui.setup_token, sourcesystem; the token is not in the row). - The GUI writes the token to
<data>/gui/setup-token(mode0600, created withO_EXCLthroughos.Root). The log names the file and the URL, never the token: journald is often readable by other users. - aarvee, who has a shell on the central host, runs
cat /var/lib/mirafleet/central/gui/setup-token. aarvee openshttps://central.example.com:8443. Every page except/setup, the static files and the error pages answers with a redirect to/setup, so there is nothing else to see. The browser warns about the self-signed certificate; aarvee compares the fingerprint with the one central logged (the setup page repeats it next to the token field) before continuing. - aarvee sends the token, a username (
aarvee) and a password twice. The form carries a CSRF token that matches the short-lived__Host-mf_precookie (there is no session yet)./setuphas its own per-IP rate limit, and Argon2id hashing is limited to two at a time. -
service.CompleteGUISetupruns in one transaction (store.CompleteGUISetup): the setup row exists, the hash matches in constant time, it has not expired, and no active admin exists. Then it inserts the admin (Argon2id hash, generation 1), deletes the setup row and writes thegui.setupaudit row, all together. Two people racing: one wins, the other gets a plain refusal. - The service then opens the first session in a second transaction (row with
SHA-256(token)plus agui.loginaudit row whose message is "after setup"). The GUI deletes the token file, sets the__Host-mf_sessioncookie and redirects to the Overview. From now on/setupis404.
Why this is safe from the LAN. Central listens on all interfaces by default, so anyone on the
network can open /setup before aarvee does. They still cannot do anything: the token has 256 bits,
so guessing is hopeless; no endpoint creates or reveals a token, so they cannot take it or block
aarvee; and before setup the pages show no data. The one real risk is a man in the middle on the
very first visit (self-signed certificate), which is why the fingerprint check matters and why
the guide recommends doing the first visit through ssh -L 8443:127.0.0.1:8443
(section 5.12). If the token expires, central makes a new
one at its next start.
In one sentence: aarvee types a username and password; if they are right, central starts a session and sets a cookie that proves it on every later request.
sequenceDiagram
participant B as aarvee's browser
participant G as gui.Server
participant S as service.Service
participant DB as sqlite.DB
B->>G: GET /login (Host checked, pre-session cookie set)
G-->>B: form with _csrf token
B->>G: POST /login username, password, _csrf
G->>G: CSRF check, rate limit check, hash slot (max 2)
G->>S: GUILogin(remote, user agent, username, password)
S->>DB: read the admin, verify Argon2id hash
S->>DB: create session row (SHA-256 of token) + gui.login audit row, one tx
S-->>G: raw session token
G-->>B: 303 to Overview, Set-Cookie __Host-mf_session
B->>G: GET / with cookie
G->>S: GUIAuthenticate(token)
S->>DB: look up hash, check expiry and generation
S-->>G: Principal admin aarvee, source gui
-
GET /loginpasses the Host allowlist (any otherHostgets421), gets the security headers, and sets a pre-session cookie__Host-mf_prewhose value keys the form's CSRF token. -
POST /login(handleLogin) first checks the CSRF token, then the login limiter: 5 attempts per minute per client address (IPv6 by /64). After 10 failures in a row the address is locked out for 15 minutes (429withRetry-After, and onegui.lockoutaudit row). More than 100 failures in 10 minutes slows login for everyone to one attempt per 5 seconds, but never locks the account: with only one admin, a lockout would let any attacker shut aarvee out. - The handler takes one of two hash slots (a third waiter waits up to 5 s, then gets
503) and callsservice.GUILogin. For an unknown username the service still burns a dummy Argon2id hash, and a wrong password and an unknown user return the same error, so timing and message reveal nothing about which usernames exist. - On success the store creates the session row and the
gui.loginaudit row in one transaction. The row holdsSHA-256(token), never the token. At most 16 sessions per admin exist; a 17th login deletes the oldest. - The browser gets
__Host-mf_session(Secure,HttpOnly,SameSite=Strict,Path=/, noDomain) and a redirect. Failed logins are only logged (address and username length), not audited, so an attacker cannot fill the audit table. - Every later request runs
requireAdmin, which callsservice.GUIAuthenticate: one indexed read joining the session to the active admin. The result is a principalauth.GUIAdmin("aarvee", <peer ip:port>)with kindadmin.
In one sentence: aarvee clicks Revoke on dev/build-1; the GUI checks the request, then runs
exactly the code path that mirafleet central runner revoke dev/build-1 runs.
Count the gates before the service call; the last half is the same as the CLI revoke in flow 8.
- On
/runners, aarvee clicks Revoke.GET /runners/dev/build-1/revokeshows a page explaining the effect (connection closed, in-flight work cancelled, workspaces closed, the key never enrols again) and asking aarvee to typedev/build-1. aarvee does, and submitsPOST /runners/dev/build-1/revoke. - Middleware (
Server.buildHandler): theHostmust be allowed (421), no more than 32 requests are in flight (503), and the body must be an urlencoded form under 64 KiB (415,413). - Session:
requireAdminhashes the cookie and finds a live row of the active admin, else303to/login?next=...(an htmx request getsHX-Redirectinstead). - CSRF (
CSRF.Check): first the stdlibhttp.CrossOriginProtection(refuses cross-siteSec-Fetch-Site/Origin), then the_csrffield must equal the HMAC derived from the cookie's token, compared in constant time. Failure:403. -
runnerOpchecks that the typed text equalsdev/build-1(otherwise422and the form is shown again with the error), then callsBackend.Lifecycle(ctx, principal, SourceGUI, "dev/build-1", "revoke"). - From here it is the same as the CLI:
service.lifecycleauthorises (policy plus the direct admin check), the tombstone goes up,Registry.Kickclosesbuild-1's connection, in-flight work is cancelled, workspaces cascade toclosing, and the status change and the audit row (runner.revoke,source=gui, actoraarvee) commit in one transaction. The GUI answers303back to the page and shows a one-time "revoked" flash message (flow 8 has the details).
If the runner was changed by someone else a moment earlier (conflict), the page says so and
changes nothing. Because the revoke is a POST followed by a 303 (post, redirect, get),
pressing refresh does not send it twice.
In one sentence: a session ends when it is idle for 30 minutes, 12 hours old, replaced by a
password change or a reset-admin, or deleted by hand; a central restart does not end it.
-
Idle and absolute limits.
GUIAuthenticatecompares the row with the clock the service owns: idle timeout 30 minutes fromlast_active_at, absolute 12 hours fromcreated_at. A dead row is deleted and the request is treated as logged out (303to/login). -
Touches are throttled. Each normal request updates
last_active_at, but at most once per 60 seconds per session. Polled fragments (the Overview, Runners and Workspaces tables refresh every 10 s through htmx) are registered as passive and never touch the row, so a tab left open cannot keep a session alive. -
Restart. Sessions are rows in SQLite (
gui_sessions), not memory. When central restarts, aarvee's cookie still works, because the expiry arithmetic uses stored timestamps. -
Generation. Each admin has a
generationnumber, copied into every session at login. A password change bumps it, deletes every other session in the same transaction and re-issues the current session under a new token. Any row whose generation differs from the admin's is dead on its next request. A stolen cookie therefore dies the moment aarvee changes the password. The Account screen also lists sessions (created, last active, address, browser) so aarvee can sign out one or all others (gui.session_revoke). -
Housekeeping.
PurgeGUISessionsdeletes expired and dead rows at GUI start and every 10 minutes. It is not audited: it is housekeeping.
In one sentence: aarvee forgot the GUI password; because aarvee can still log in to the central host as the user that runs central, aarvee asks central for a fresh setup token.
- On the central host aarvee runs
mirafleet central gui reset-admin. The CLI sends the admin methodgui.reset_adminover the admin socket (never the network; the GUI itself cannot do this). -
service.ResetGUIAdmin(admin only, with the double gate) runs one transaction: the active admin's status becomesreset, every GUI session row is deleted, a new setup token's hash is stored, and the audit rowgui.reset_admin(sourcecli) is written. - The CLI prints the token once on standard output and a reminder on standard error. It is in no log.
- aarvee opens the GUI, which now answers like flow 24: every page goes to
/setup. aarvee pastes the token and picks a username and password. A new admin row is created; the old one stays asreset, and old cookies are already dead.
Why this is safe: anyone who can talk to the admin socket already has central's user id, which is as powerful as the database file itself. The step only makes the GUI no weaker than that.
In one sentence: aarvee creates a join token in the GUI; the page shows the secret exactly once, and refreshing or going back cannot mint a second one.
- On
/tokens/newaarvee chooses fleetdev, labels, uses and a lifetime (at most 30 days). The form contains a nonce: a value signed (HMAC) with aarvee's session's CSRF key. -
POSTtokenCreatechecks the session and CSRF as in flow 26, validates the fields (422with field errors if wrong) and then marks the nonce used in an in-memory table (kept 1 h, at most 1024 entries). Only then does it callservice.CreateTokenwith sourcegui. - The service stores
SHA-256(secret)and the audit rowtoken.createin one transaction, as in flow 2. The page renders directly (no redirect) withCache-Control: no-store, shows the token with a copy button and the command to run on the runner (mirafleet runner join central.example.com:2222 -name build-1 -token -), and says it cannot be shown again. - If aarvee presses back and resubmits, or refreshes, the nonce is already used: the page says
"already created; the token is not shown again" and no second token exists. htmx history
caching is off, so the page with the token is not kept in the browser's
localStorage.
The registration code (Operators screen) follows the same rule for a different reason: each
"Show code" press is audited by service.OperatorCode, so the page never refreshes it by itself.
Flows 30 to 36 are the host-mode story (new in M7). The cast: raj owns a spare VM vm1.example.com
with an ordinary user dev and a runner vm1-dev in fleet dev (workspace mode, executor host).
MIRASTACK LABS' private GitLab, gitlab.example.com, is reachable only from the VM. aarvee stored two
credentials on central: gitlab-deploy (kind ssh-key, host gitlab.example.com) and
gitlab-api (kind https-token, host gitlab.example.com), and assigned both to vm1-dev. Durga's
project has .mirafleet.yaml with fleet: dev and remote: true. Chapter 3 draws the
components.
Summary: the same workspace.open as flow 12, with three extra checks (is this runner free,
does it hold the credentials, is the session already held) and a directory instead of a container.
sequenceDiagram
participant H as Claude Code
participant M as mirafleet mcp
participant S as service.Service
participant St as store (sqlite)
participant R as runner vm1-dev
participant X as hostx.Executor
H->>M: workspace_open {session, credentials?}
M->>S: workspace.open (api@)
S->>S: host fleet? credentials narrowed, never widened
S->>St: reserve (one-operator hold check)
S->>R: workspace@ op=create (labels: session, workdir, credentials)
R->>R: every named credential held?
R->>X: Create
X-->>R: session dir (0700)
R->>R: start agent socket and broker session
R-->>S: created
S->>St: row opening to open, audit workspace.open
S-->>M: workspace id, reused=false
-
The tool call. Claude Code calls
workspace_open. Arguments: an optionalcredentialslist (names; a subset of what the runner holds; the default is all of them), and the session name (cc:<id>, flow 20)..mirafleet.yamlwithremote: truealso makes the plugin's skill tell Claude to use only the mirafleet tools for this project. -
Central checks the fleet. It is the same ownership and policy path as flow 12. A host fleet
must be in workspace mode with executor
host. Asking for credentials on any other fleet is abad request("credentials need a host fleet"), and so is aworkdir. -
Placement filters by credentials.
FilterRunnersByCredentialskeeps only runners whose effective set holds every requested credential.ResolveCredentialsturns "all" into the concrete list. A request can narrow the set but never widen it: naming a credential the runner does not hold fails, and so does naming one nobody assigned. -
One operator at a time. Inside the store's reserve transaction, central counts live
workspaces (
opening,open,closing) of other operators on that runner. If there is any, the open is refused with codeconflictandbusy: runner NAME is in use by kalyan, and placement had already skipped such runners if another was free. No table is added: it is a count overworkspaces. The hold ends when the holder's last live workspace is closed, reaped, expired, killed, orlost. -
The create request. Central sends
workspace@mirafleetcreatewith labels. The runner (never central) decides the directory. -
The runner creates a directory, not a container.
hostx.Manager.OpenFormakes<root>/sessions/<slug>(mode0700), or, when the labelmirafleet.workdircarries a name that matches^[a-z0-9][a-z0-9._-]{0,62}$,<root>/work/<name>. Symlinks are refused.<slug>is the session id when it is a short safe name, elseh_plus a hash of it (hostx.Slug), because ids such ascc:<uuid>are not safe directory names. The runner labels the workspace withmirafleet.session(the orchestrator's session id), so reopening the session reattaches. If the same session is already open on that runner the open is refused (ErrRefused). -
Tools get their secrets-without-secrets. The runner starts the session's ssh-agent on
agent.sock, a broker session with a per-session CA, writesssh_configandca.pem, and hands the executor the environment (SSH_AUTH_SOCK,GIT_SSH_COMMAND,HTTPS_PROXY,SSL_CERT_FILE,GIT_SSL_CAINFO,NODE_EXTRA_CA_CERTS, ...) throughSetSessionEnv. Every child process gets these over anything the agent sets. -
Audit and answer. The row goes
openingtoopenandaudit.action = workspace.openis written, as in flow 12. The result carries the workspace id.
What can go wrong: the runner is held by another operator (step 4, busy); no runner holds all
requested credentials (unavailable); workdir is invalid or the runner is a container runner.
Summary: closing keeps the directory, and opening the same session (or workdir) later finds it
again, with its files, even after a runner restart.
- Durga closes their laptop. Claude Code exits,
SessionEndor the idle reaper closes the workspace (flows 21 and 22).workspace_closestops every process group of the session, closes the agent socket and broker session, and keeps the directory. Theworkspacesrow becomesclosed; a record of the directory (<slug>.json: dir, created, last used) stays in the runner's state. - The next day Durga runs
claude --resume. The handoff file gives the same session namecc:<id>(flow 20). -
workspace_openruns flow 30 again. The sticky-session rule from flow 12 finds no live row (the old one isclosed), so a new row and a new workspace id are made, but the runner receives the same session id as the labelmirafleet.session, sohostx.Manager.OpenFormaps it to the same slug and finds the existing directory. Their checkout, build caches and uncommitted edits are there. Credentials are loaded fresh from the runner's current set, not from anything saved. - With
workdir: shared-appin.mirafleet.yaml, two sessions (or two different Claude sessions of the same operator) share<root>/work/shared-app. -
workspace_listshows the directories with their size and last use, including ones with no live workspace. -
Cleanup is the only thing that deletes a directory. A daily runner sweep
(
hostx.Manager.Sweep) deletes directories not used forcleanup.delete_after(default 30 days;0or a negative value means never). Tool results warn three days before. Explicit deletion:workspace_close {delete: true}(owner only),mirafleet runner host prune [-older-than D] [-yes]on the VM (a dry run without-yes), or the adminworkspace.killwith delete on central. A directory with a running background job is never swept.
What can go wrong: two live handles for the same session are refused. The next operator on a runner can read directories the previous operator left, because they are the same OS user; this is documented (section 5.14).
Summary: a foreground command runs as the runner's user in the session's current directory; its full output is saved on the VM; if the reply had to be cut, the reply says so and how to read the rest.
sequenceDiagram
participant H as Claude Code
participant M as mirafleet mcp
participant S as service.Service
participant R as runner vm1-dev
participant X as hostx (bash)
H->>M: bash {command, timeout_s}
M->>S: workspace.exec
S->>S: owner, confirm, audit start row
S->>R: workspace@ op=exec
R->>X: Bash(command, timeout, request id)
X->>X: bash -lc, tee to out/request id
X-->>R: exit code, inline text, bytes, ref, cwd
R-->>S: result
S->>S: audit end row
S-->>M: result
M->>M: cap, neutralise control tags
H->>M: read_output {ref, offset, tail_lines, grep}
M->>S: workspace.output_read
S->>R: op=output_read
R-->>M: exactly the requested bytes
-
Dispatch. Same path as flow 13 (ownership,
confirmon a guarded fleet,startaudit row before dispatch). The tool isbash{command, timeout_s, description, run_in_background}. The default timeout is 120 s and the maximum is 600 s (10 minutes, like Claude Code's Bash tool). -
A fresh shell each time. The runner runs
bash -lc COMMAND(fallbacksh -c) with a clean environment: only the names in the runner'senv_allow, the runner-set tool variables of flow 30,HOME,USER,PATHandLANG. It starts a new session and process group (Setsid) and passes only file descriptors 0 to 2. Environment variables set in one call do not persist. -
The directory does persist. A wrapper reports the final
pwdon an extra descriptor; the runner saves it per session, and the result carries it asCwd. The next call starts there. This mimics Claude Code's own Bash tool. -
Everything is saved. While the command runs, the runner tees stdout and stderr into
out/<request_id>in the session's state directory. The last 32 such files are kept, each at most 64 MiB (hostx.KeepOutputs,hostx.MaxOutputFile). -
Never a silent cut. The reply holds as much text as fits (a budget of about 32 KiB of JSON
text per stream) and always carries
stdout_bytes,stderr_bytes,truncated(with how many bytes were omitted from the head and the tail) and aref. The text itself says how to page. -
Paging.
read_output{ref, offset, limit, tail_lines, grep}reads the saved file: a byteoffsetandlimit(at most 1 MiB), the last N lines, or only lines matching an RE2 regular expression. It returnsnext_offsetto continue. Paging a 5 MiB build log from offset 0 in chunks returns exactly the original bytes. -
The MCP layer. The
mcpprocess applies the output cap from chapter 5.11 and neutralises harness-control tags (such as<system-reminder>) in every tool result and error. The file tools' own notices are plain[Warning: ...]text. See section 5.11. -
Timeout. When
timeout_spasses, the runner kills the whole process group, and the result says so and still carries the output saved so far.
Summary: run_in_background starts a supervised shell that outlives the tool call, writes its
output to files, and is polled, tailed or killed later.
-
Start.
bash {command, run_in_background: true}becomesworkspace.job_start. The runner starts the hiddenmirafleet host-shim job(cmd/mirafleet/host_shim_linux.go), which runs the command in its own process group and supervises it. The reply returns at once with ashell_id. -
Where output goes. The shim writes
stdout,stderrandstatus.jsonunderjobs/<id>/in the session's state directory. The files belong to the agent's own user, so they are informational: the agent could edit them. Each stream is capped at 256 MiB (hostx.MaxJobStream): the shim checks the file sizes about every poll and, when one passes the cap, kills the job and records it as canceled with the message "job killed: output exceeded the per-stream cap". Nothing is rotated or dropped before that. - Limits. At most 8 background shells per session; a maximum duration per fleet (default 12 hours, at most 24). When it passes, the group is killed.
-
Polling.
bash_output{shell_id, filter?}returns the output since the previousbash_outputcall (each call advances the position), plus the state and exit code. Afilterregular expression returns only matching lines.read_output{shell_id, stream, offset, ...}re-reads from any earlier point.job_listlists the session's shells with state and exit code. -
Killing.
kill_shell{shell_id}signals the whole process group. -
A running job keeps the workspace alive. The runner's heartbeat reports
busy, central touches the workspace row at most every 5 minutes, and the idle reaper (flow 16) skips a busy session. The sweep of flow 31 also never deletes a directory with a running job. - Runner restart. See flow 35: a job survives it.
What can go wrong: the shell id is unknown (not_found), the 8-shell limit is reached, or the
command exits at once (state: exited with an exit code and the output so far).
Summary: aarvee enters a secret once; central stores it encrypted; the runner receives it in memory; the agent uses it without reading it; later aarvee replaces it and the new value is live at once.
sequenceDiagram
participant D as aarvee (GUI / CLI)
participant C as service.Service
participant DB as sqlite (ciphertext)
participant R as runner vm1-dev (memory)
participant A as agent socket / broker
participant G as gitlab.example.com
D->>C: credential.set (value in a body frame)
C->>DB: seal (AES-GCM), audit credential.set, no value
D->>C: credential.assign gitlab-deploy to vm1-dev
C->>C: effective set = runner assignments union fleet assignments
C->>R: creds_sync (generation, whole set)
R->>R: swap the in-memory set atomically
Note over R,A: agent runs git push
A->>G: signed login or injected header
R->>C: events: credential.use (aggregated)
D->>C: credential.set (new value)
C->>R: creds_sync (new generation)
Note over R,A: next git push uses the new value
A. Set. On central the value arrives as a body frame (the same mechanism as file
transfer, flow 14), so it is never in a JSON request
that gets logged. The CLI reads it from stdin only: mirafleet central credential set gitlab-deploy -kind ssh-key -hosts gitlab.example.com -known-hosts ./pinned < ./deploy_key. A flag or argument
carrying the value is refused. The GUI form is write-only too, and has no file upload. Other flags:
-user, -auth basic|bearer|header, -header NAME, -host-binding required|off, -expires,
-rotate gitlab_self -rotate-project P. service.SetCredential validates the kind and hosts,
computes a fingerprint (admin-only), seals the value with internal/secrets (AES-256-GCM, fresh
nonce, AAD = credential id and version), stores the row, and writes an audit row naming the
credential but never the value. Replacing the value bumps version and sets rotated_at.
B. Assign. credential.assign NAME -runner fleet/runner or -fleet F stores a row in
credential_assignments (exactly one of runner or fleet; the database enforces it). A runner's
effective set is its own assignments united with those of its fleet.
C. Deliver. Service.DeliverRunner builds the set for a connected host runner and sends it
as a creds_sync operation on the workspace@mirafleet channel, on connect and on every change
(RunnerConnected, deliverHolders, deliverTo). Each sync carries a generation number
and the whole set. The runner's credHolder.apply replaces its set atomically: a removed
credential stops working at once, and a stale (lower-generation) sync is ignored. Values live in runner
memory only. A revoked runner gets nothing again, because its connection is gone.
The heartbeat reports each credential's name, kind, hosts and status (ok, expiring, expired,
invalid); operators see names, kinds, hosts and status in runners.list, while fingerprints and
expiry dates are admin-only.
D. Use.
-
SSH: the agent runs
git clone git@gitlab.example.com:mirastack/app.git.sshreadsSSH_AUTH_SOCKand asks the session's restricted agent to sign. With host binding the agent first checks that the server's host key is ingitlab-deploy's pinnedknown_hosts; it signs only the user-auth request for the usergit. -
HTTPS:
git clone https://gitlab.example.com/mirastack/app.gitgoes to the broker throughHTTPS_PROXY. Becausegitlab.example.comis a credential host, the broker terminates TLS with the per-session CA, strips whateverAuthorizationheader the client sent, adds the stored one, and forwards over verified TLS (system certificates plus the corporate CA). Redirects are never followed. A host without a credential is a rawCONNECTtunnel. -
Registry: on a
401 Bearerchallenge the broker fetches the token itself and caches it per scope; the client never sees a 401 or a JWT. -
Audit: every use is logged in the runner's local JSONL file and counted. Counts are sent to central
as
eventsbatches (per session, credential, host and status, in 60-second windows; at most 128 events or 64 KiB per batch). Central writes audit rowscredential.usewith sourcerunner; a flood becomes anevents.droppedrow. E. Rotate. Three ways, from least to most automatic.
-
Manual: aarvee runs
credential set(or uses the GUI replace form) with the new value, which goes through step C. Every running session uses it from its next connection. -
Expiry warnings: 14 days before
expires, the credential's status becomesexpiring(GUI, heartbeat, and tool results that name the credential). -
GitLab self-rotation (opt-in): with
rotate: gitlab_self, central'sRunRotationloop callsRotateDueand, when less than 7 days remain,rotate: it calls GitLab'sPOST /projects/:id/access_tokens/self/rotatewith the token itself (the token needs theself_rotatescope), stores the new sealed value in the same transaction as its audit row, then pushes it to every assigned runner. It needs central to reach GitLab over HTTPS (system certificates plus the file named byMIRAFLEET_CORPORATE_CA_FILE). If GitLab is unreachable, the failure is audited (acredential.rotaterow with outcomeerror), shown in the GUI, and the old token keeps working until it expires; rotation then is manual.
What can go wrong: the runner is offline when the value changes (it receives the whole set on
reconnect); the wrong or missing MIRAFLEET_SECRETS_KEY (central refuses to start when credentials exist but the key is missing or wrong;
section 6.19); a credential is deleted while sessions use it (the next creds_sync removes
it, and the next use fails).
Summary: a restarted runner picks its sessions and running jobs back up instead of treating them as lost.
-
What survives. The systemd unit uses
KillMode=process, so stopping the runner kills only the runner process and not its children. Background shells run underhost-shimin their own process group, so they keep going. Directories are just files. - Start-up. The new runner process hardens itself first (non-dumpable, no core dumps, refuses uid 0 and capabilities; section 5.14), then connects to central. It holds no secrets yet.
-
Reconnect and sync. Central sees the runner connect (a newer connection replaces an older half-open one and
writes a
runner.replacedaudit row) and sendscreds_sync(flow 34 C). Only now do the secrets exist in the new process's memory. -
Re-adopt sessions. The executor reads its session records (
hostx.Manager.OpenSessions): sessions whoseopenflag was left set are re-attached. The runner then re-wires their credentials (devMgr.adopt, called once with the adopted workspaces at start): for each session it reads thecreds_scopefile that the original open saved in the session state directory (<state>/sessions/<slug>/) and rebuilds the agent socket, broker session and environment with that same, never wider set. Untilcreds_synchas delivered the values the session has no secrets, and a session whosecreds_scopefile is missing or unreadable gets no credentials rather than all of them (the next open fixes it). -
Re-adopt jobs. For each job directory, the executor reads
status.json, which records the supervisor's pid and its process start time, so a reused pid is not mistaken for the job. A job whose supervisor is still the same process staysrunning; one recorded asrunningwhose supervisor is gone is reportedlost, and one that finished while the runner was down keeps the state and exit code the supervisor wrote. -
Reconcile. Central's reconcile (flow 17) compares its
openrows with the runner's list; sessions the runner re-adopted stayopen, so Durga'sbash_outputsimply continues. A session the runner cannot find becomeslost.
What can go wrong: the runner was down so long that central marked the workspaces lost
(flow 17); the runner process was killed together with its children (the unit file did not have
KillMode=process).
Summary: a once-a-day sweep and one command delete directories nobody is using.
- The runner runs
hostx.Manager.Sweepdaily. For each session directory it compareslast_usedwithcleanup.delete_after. Over the limit and not busy: delete (only insideroot; the deletion refuses paths that resolve outside it). Within 3 days of the limit: add a warning to results. -
mirafleet runner host prune [-older-than D] [-yes]does the same by hand. Without-yesit is a dry run that lists what would go. - The timestamp comes from an injectable clock in tests, so "30 days later" is a test without waiting.
| Thing | States | Changed by | Effect on live connections |
|---|---|---|---|
| Runner (stored) |
pending, active, disabled, revoked
|
join; approve, disable, enable, revoke on central |
approve/enable/disable: status updated in place, connection stays. Revoke: kicked. |
| Runner (live) | offline, online (registered, heartbeat < 45 s) | connect, heartbeat, disconnect, stale | n/a |
| Operator |
active, revoked
|
mcp login with a valid code; operator revoke
|
revoke: connections closed, in-flight actions cancelled |
| Join token | usable until used up, expired or revoked |
token create, token revoke, enroll |
n/a |
| Registration code | valid for its 60 s step | time; rotate-secret invalidates all |
n/a |
| Session handoff file | written (SessionStart), rewritten (resume, /clear), removed (SessionEnd), pruned after 7 days |
the Claude Code hooks (flows 20 and 22) | none: a file only; the MCP server re-reads it at every open |
| GUI setup token | made at first start or by reset-admin; valid 24 h; single use |
EnsureGUISetup, gui reset-admin, CompleteGUISetup (flows 24, 28) |
none: a file and a table row |
| GUI session | live until idle 30 min, 12 h old, replaced by a password change, or deleted |
GUILogin, GUILogout, ChangeGUIPassword, ResetGUIAdmin, the purge (flows 25, 27, 28) |
none to runners; the browser is sent to /login on its next request |
| Workspace |
opening, open, closing, closed, failed, lost (see chapter 6) |
workspace.open / close, the reaper, reconcile, revoke cascades, workspace.kill
|
runner revoke: its workspaces go closing then lost; operator revoke: closing then closed; runner disable: no new open, exec or write |
| Host session directory (M7) | present (live or closed), deleted |
workspace_close {delete: true}, the daily sweep, runner host prune, admin kill with delete |
none: closing never deletes (flow 31) |
| Background job (M7) | running, then exited, killed or timed out |
bash {run_in_background}, kill_shell, job limit, runner restart re-adoption |
a running job keeps its workspace busy and is not reaped (flow 33) |
| Credential (M7) | stored, assigned, delivered (per runner: ok, expiring, expired, invalid), deleted |
credential set/assign/unassign/rotate/delete |
delivery on connect and on every change (flow 34) |
| Failure | What you see | What the system does |
|---|---|---|
| Runner's network drops (no TCP reset) |
runners.list shows it offline within 45 s; action.run answers unavailable ("runner unavailable") |
Central closes the stale session; the runner times out its own heartbeat and redials with backoff (flow 10). |
| Central restarts | Operators get errors from mcp api until it is back; runners show offline then online |
In-flight actions end as cancelled; runners reconnect and re-register (flow 11). |
| Wrong or unknown join token |
join refused (identical for every cause) |
Nothing is consumed or stored; the real reason is logged on central only (flow 3). |
| Central presents an unexpected host key at join | Client aborts with a host-key mismatch before sending the token | The token never leaves the runner (flow 3). |
| Registration code from the previous 60 s window | registration refused |
Nothing stored; ask aarvee for a fresh code (flow 6). |
Guarded fleet and no confirm
|
bad_request: fleet requires confirm to equal the fleet name |
A denied audit row; nothing runs (flow 7). |
Runner is pending or disabled
|
conflict: "runner is pending" or "runner is disabled" |
No start row, nothing runs (flows 5 and 7). |
| Audit database cannot take the start row |
unavailable: "audit unavailable" |
Nothing runs: no row, no exec (flow 7). |
Action not in actions.yaml, or a parameter does not match its regex |
bad_request: "refused by runner: ..." |
The runner writes a refused line; the process is never started (flow 7). |
| Action runs too long | timeout |
The runner kills the whole process group (flow 7). |
| Runner revoked mid-action |
unavailable: "runner revoked" |
Process killed, cancelled row, runner refused when it returns (flow 8). |
| Operator revoked mid-action | The connection closes | Request cancelled, runner kills the process, cancelled row (flow 9). |
| Central crashes after the start row | A start row with no outcome row | Investigators read it as "something began and we never learned how it ended" (flow 7). |
workspace.open and every slot is taken |
unavailable: "no runner can take this workspace right now" |
A failed row at most; no container is made (flow 12). |
workspace.open for a session whose runner is offline |
unavailable: "the runner holding this session's workspace is offline" |
No new workspace is created, because the files are on that runner (flow 12). |
workspace.open twice quickly for one session |
The second gets conflict: "workspace is still opening" |
The first create continues (flow 12). |
| Image not in the fleet allowlist |
bad_request: "image is not allowed by the fleet" |
Nothing reserved or created (flow 12). |
| Operator already holds 8 workspaces |
conflict: "workspace limit reached (8)" |
A denied audit row (flow 12). |
| kalyan uses durga's workspace id |
not_found, identical to an unknown id |
Nothing leaks; no runner is contacted (flow 13). |
Forbidden env name (for example ANTHROPIC_API_KEY) |
bad_request naming the variable, never its value |
A refused audit row; nothing runs (flow 13). |
exec runs past its timeout |
The result has timed_out: true and outcome timeout
|
The runner kills the exec; daemonised children survive until close (flow 13). |
| Output bigger than 1 MiB |
truncated: true, head and tail kept |
Nothing fails; the middle is dropped (flow 13). |
write_file with a wrong sha256, or fewer bytes than length
|
bad_request |
One error audit row; nothing reaches the runner (flow 14). |
| Body frame never arrives |
timeout after 30 s |
The channel is closed; central buffered nothing before authorisation (flow 14). |
| Workspace over its disk quota |
bad_request: "refused by runner: ..." (the runner's quota error) |
Reads, exec and close still work (flow 14). |
Runner offline during workspace.close
|
The call returns the workspace in state closing
|
The reaper or reconcile finishes it (flows 15 to 17). |
Workspace idle for idle_ttl
|
Later calls get conflict: "workspace is closed" |
The reaper destroyed it and wrote a system row (flow 16). |
| Runner restarts | Existing workspaces keep working after it reconnects | Reconcile compares lists; the egress proxy is re-attached (flow 17). |
| Central restarts during a create | The row is failed after restart |
RecoverWorkspaces; reconcile destroys the leftover container (flow 17). |
| Container vanished (host reboot, manual delete) | The row becomes lost
|
Reconcile frees the slot and writes workspace.lost (flow 17). |
| Runner or operator revoked | Their workspaces are closed | Cascade, then reaper as backstop (flow 18). |
| Central is down or unreachable | Every tool call answers unavailable: "cannot reach central". tools/list still works |
orch dials lazily; the next call dials again. Lists, read_file and workspace_open are retried once on a fresh connection; exec, write_file and run_action are never retried (flow 19). |
Connection to central lost mid-exec
|
unavailable: "the connection to central was lost; the call may or may not have completed" |
The connection is dropped and redialled on the next call. The model must check the effect before repeating (flow 19). |
| One of several contexts is dead |
list_fleets returns the live contexts plus one errors[] entry for the dead one; a call that names the dead context gets unavailable
|
The other contexts are unaffected: each has its own connection (flow 19). |
| Context name unknown |
not_found: "unknown context ..." with the list of known names |
Nothing is sent (flow 19). |
| No login yet (no contexts, key or pinned host key) |
unavailable: "not set up: run mirafleet mcp login …" (no paths in the text) |
The server still starts and lists its tools; it never dials without a pinned host key, so there is no trust on first use (flow 19). |
| Central rejects the operator key (revoked, or login redone elsewhere) |
unauthenticated: "central rejected the operator key: run mirafleet mcp login …" |
Nothing runs; the model is told the human fix (flow 9). |
| Central's host key changed |
unavailable: "central's host key does not match the pinned key; the server may have been rebuilt or intercepted" |
The connection is refused; no data is sent (chapter 5). |
| Central is at its channel limit |
unavailable: "central is busy: too many concurrent calls; try again" |
orch retries the open three times with a short back-off first, because nothing had been sent (flow 19). |
Guarded fleet, no confirm
|
bad_request with "... is guarded. Ask the user. Only after the user explicitly agrees, repeat the call with confirm ..." |
A denied audit row (flow 23). |
Guarded fleet came from .mirafleet.yaml or default_fleet
|
bad_request: "fleet ... is guarded and was taken from the project file: name it explicitly" |
Nothing is sent to the fleet (flow 23). |
Invalid .mirafleet.yaml
|
bad_request: "the project file .mirafleet.yaml is invalid" (details on stderr only) |
Nothing is sent (flow 19). |
Argument with a wrong case (Confirm) or an unknown name |
bad_request: "unknown or mis-cased argument ..." |
The handler never runs (flow 19, chapter 5). |
| Harness closes stdin | The server exits 0 within 5 s | In-flight calls are cancelled; the workspace stays open until idle_ttl (flow 21). |
SessionEnd for prompt_input_exit, resume or other
|
Nothing visible | The workspace is kept for a possible --resume (flow 22). |
GUI: Host header not on the allowlist |
421 branded error page |
Nothing is routed; the request never reaches a handler (flow 25, section 5.12). |
| GUI: 5 login attempts in a minute, or 10 failures in a row |
429 with Retry-After and a "try again in N minutes" message |
The address is locked out for 15 minutes; one gui.lockout audit row (flow 25). |
| GUI: session idle 30 min, 12 h old, or password changed elsewhere |
303 to /login?next=... (htmx: HX-Redirect) |
The session row is deleted (flow 27). |
GUI: missing or wrong CSRF token, or a cross-site POST
|
403 page |
Nothing changes (flow 26). |
| GUI: revoke form with the wrong typed name |
422, the form again with an error |
Nothing changes (flow 26). |
| GUI: resubmitted token form | "already created; the token is not shown again" | No second token is minted (flow 29). |
| GUI: all hash slots busy |
503 with Retry-After: 1
|
Argon2id runs at most two at a time (flow 25). |
workspace_open on a host runner another operator holds (M7) |
conflict: "busy: runner NAME is in use by kalyan" |
A denied audit row; placement had skipped held runners if another was free (flow 30). |
| Requested credential not held by any runner of the fleet (M7) |
unavailable, naming no value |
Nothing is created (flow 30). |
bash output too long for the reply (M7) |
The text plus stdout_bytes, stderr_bytes, truncated and a ref
|
The full output is on the VM; read_output pages it (flow 32). |
| Runner restarted while a job runs (M7) |
bash_output continues |
The job survives and is re-adopted (flow 35). |
| GitLab unreachable at self-rotation (M7) | A "rotation failed" notice in the GUI | An audit row; the old token keeps working until it expires (flow 34). |
- In flow 3, why does it matter that the host-key pin is checked during the SSH handshake and not after the token is sent?
-
kalyanregisters at 10:03:59 with a code that was shown at 10:03:12. Does it work? What if kalyan tries at 10:04:01? - In
RunAction, why is the start audit row written before the call toRegistry.Exec, and what happens if writing it fails? - An admin disables
db-1while a 5-minute action is running on it. Does the action stop? What about when the admin revokesdb-1instead? - After
central operator revoke kalyan, Kalyan's laptop is still connected for a fraction of a second. Which two mechanisms make sure it does not stay connected? - durga sends
workspace.openfor sessions1twice. What does durga get the second time, and what ifbuild-1went offline in between? - Why does central read the body frame of
write_fileonly after the policy, ownership and confirm checks? What would an attacker gain if it read the body first? - A workspace has been closed by the reaper while
build-1was offline for a day. What state is its row in, and what finally deletes the container? - Why does
workspace.execwrite astartaudit row before dispatch, and what does it record about the environment? - The MCP
exectool is called while the connection to central drops. Is the call retried? Is alist_fleetscall retried? Why the difference? - durga quits Claude Code for lunch and comes back with
claude --resume. What happens to durga's workspace in between, and which file makes the reattach work? What changes with no hooks? - The model calls
run_actiononprodand the hook shows durga a prompt. Which two components would still stop a call that has noconfirm, even with no plugin installed? - Someone on the LAN opens
https://central.example.com:8443/setupbefore aarvee does. What can they do, and what can they not do? - aarvee changes the GUI password on a laptop. What happens to the session still open in a phone's browser, and why on the very next request?
- aarvee revokes
dev/build-1from the GUI. List the checks beforeService.Lifecycleruns, and say what the audit row records that tells it apart from a CLI revoke. - Durga closes a host workspace on Monday and reopens the same session on Wednesday. Which files are still there, and what is the only thing that deletes them?
- A 200 MiB build log scrolls past in a foreground
bashcall. What does Claude receive, and how can it read the error at line 41,203? - aarvee replaces the value of
gitlab-api. Doesvm1-devneed a restart? What happens to agit pushalready in flight? - The runner is restarted during a 3-minute build. Why does the build survive, and why can the new runner process not read the token it needs?
Answers
- The token is a bearer secret. If the pin were checked afterwards, a fake central (a
man-in-the-middle) would already have received the token and could use it.
PinnedHashruns as the SSH host-key callback, so on a mismatch the handshake aborts and the token is never sent. - At 10:03:59 it works: the code is valid for the whole step 10:03:00 to 10:03:59, and the
same code verifies repeatedly. At 10:04:01 it is refused:
VerifyCodeonly accepts the code of the current step, and the step changed. Kalyan gets the identical "registration refused" error and must ask the admin for the new code. - The start row records intent before any side effect. If it cannot be written (database
unavailable),
RunActionreturnsunavailable("audit unavailable") and nothing is executed: no row, no exec. Otherwise an outage of the database would silently create unaudited commands. - Disable does not stop it.
RunActionchecked the status once at the start, and the registry's status gate only runs when an exec begins; the connection stays up and the action finishes. Newaction.runcalls are refused withconflict. Revoke does stop it:TombstoneandKickcancel the session context, the exec channel closes, the runner kills the process group, and the outcome row sayscancelledwith message "runner revoked". - Central closes kalyan's open connections right after committing the revoke, and a re-check catches
any connection that slipped in around the commit. In detail:
Service.RevokeOperatorcommits the revoked status and thenOperatorConns.KickOperatorcloses every tracked connection, which cancels the in-flight requests. For a connection that authenticated just before the commit and was added toOperatorConnsafter the kick,serveOperatorre-reads the operator right afteraddand closes the connection if it is not active. New connections are refused byKeyAuthbecause the stored status isrevoked. - The second time durga gets the same workspace with
reused: true. Ifbuild-1is offline, durga getsunavailable: central never creates a fresh workspace elsewhere, because their files are onbuild-1. - Reading the body costs memory (up to 1 MiB per call) and time. If central read it first, anyone with a valid connection could make it buffer megabytes for requests that will be refused (wrong owner, revoked key, missing confirm). Reading it last bounds the attacker's cost to a small refused frame.
- The row stays
closing(it still holds its slot), because central cannot reach the runner. Whenbuild-1returns, reconcile finds the container, destroys it and sets the rowclosed(or the next reaper sweep does, once the runner is online). - Fail closed: if the audit database is down, no command should run unrecorded. The row stores the argv, the cwd and the env variable names, never the values, so a secret passed in an environment variable does not land in the audit log.
-
execis never retried: it may already have run, so the error says "the call may or may not have completed".list_fleetsis retried once on a fresh connection, because repeating a read changes nothing (orchretries only lists,read_fileandworkspace_open). - Nothing happens to it: the server shuts down without closing workspaces (flow 21), and
SessionEnd with reason
prompt_input_exitkeeps it. On resume the SessionStart hook rewrites the handoff file with the samecc:<id>;SessionForreads it andworkspace.openanswersreused: true. With no hooks there is no file and the server's session is random, so a new server opens a new workspace and the old one waits for the idle TTL (unless-sessionis set). - The MCP server (it relays central's refusal and rewrites it into "ask the user") and, above
all, central:
servicerefuses a guarded fleet's call unlessconfirmequals the fleet name, and writes adeniedaudit row. The hook is only convenience. - They can load the page and try tokens, within the per-IP rate limit. They cannot succeed: the
token has 256 bits, no endpoint reveals or regenerates it, and every other page just redirects
to
/setupwithout showing data. Their one real chance is a man in the middle on aarvee's first visit, which the fingerprint check (or anssh -Ltunnel) prevents. - The phone session is dead.
ChangeGUIPasswordbumps the admin's generation and deletes every other session row in the same transaction. Even without the delete,GUIAuthenticatecompares the session's generation with the admin's on every request and would refuse it. -
Hostallowlist, in-flight cap and body limits; a live session; the CSRF checks (cross-origin headers, then the token); the typed namedev/build-1. Then the same service code as the CLI. The row hassource=guiand actoraarvee; the CLI's row sayscli. - All of them: closing keeps the directory and reopening the same session finds it again. Only the
cleanup sweep (after
delete_after, default 30 days),workspace_close {delete: true},runner host pruneor an admin kill with delete removes it. - The first part of the output (about 32 KiB per stream), exact byte counts, how much was
omitted, and a
ref. It callsread_output {ref, grep: "error"}ortail_linesor anoffset, and gets exactly those bytes from the file on the VM. - No. Central pushes a new
creds_sync; the runner swaps its in-memory set atomically and the next request uses the new value. A push in flight keeps whatever it was already given. - The unit uses
KillMode=processand the job runs underhost-shimin its own process group, so it is not killed. The new runner starts with no secrets in memory and receives them again bycreds_syncafter it reconnects; nothing was ever on disk.
Next: 5. Security.
- mirafleet quickstart
- The mirafleet developer guide
- Glossary
- 1. What is mirafleet?
- 2. Background: the ideas mirafleet builds on
- 3. Architecture
- 4. Flows
- 5. Security
- 6. Data model
- 7. Protocols
- 8. Code tour
- 9. Dev setup and testing
- 10. Operations
- 11. Contributing
- 12. Why mirafleet? A comparison with the alternatives
Generated from docs/ at a43f824; edit there.