Add telemetry for railway ca launches, lifecycle ops, and setup - #1063
Conversation
`railway ca` had no bespoke telemetry: bare `railway ca` runs a TUI loop that swallows internal errors into toast text, so the generic per-command event nearly always reported success regardless of what actually happened. There was also no visibility into which agent got launched, agent lifecycle actions, session reattach/kill, or setup/wizard choices. Adds a small `cloud_agent::telemetry` module, following the CLI's existing conventions (additive CliTrackEvent detail events, same shape as the per-tool MCP and per-stage SSH events) to close these gaps: - Launch outcome (harness, created vs reused, real duration) fired from inside `code::prepare()` itself, so it fires regardless of caller (`railway code`, `railway ca start`, or the TUI) and regardless of the TUI swallowing the error afterward. - Agent lifecycle ops (sleep/wake/delete) from the manage screen. - Session events: reattach, kill, opening a freshly launched session, and the auto-sleep-on-quit that previously discarded its result with `let _ =` (a failure there means an agent is left running and billing compute). - Setup/wizard completions, one flat event per choice (agent/theme/project/ skills), from both `railway ca setup` and the TUI's first-run wizard. No new GraphQL mutations — everything reuses the existing `cliEventTrack` event already gated by DO_NOT_TRACK / RAILWAY_NO_TELEMETRY.
An adversarial code review of the ca-telemetry changes surfaced several
real bugs, fixed here:
- String::truncate(256)/s[..256] on an error message panics whenever byte
256 lands mid-character (a path or GraphQL error with a non-ASCII
character near the boundary) — and this crate builds with
panic = "abort", so that panic takes down the whole process instead of
just failing to log. This exact pattern existed independently in five
places: the generic per-command event (macros.rs), `setup agent`,
the MCP tool event, ssh/tel.rs, and the new cloud_agent telemetry
module. Added one char-boundary-safe crate::telemetry::truncate_message
and pointed all five at it, with a regression test reproducing the panic
on the old logic.
- handle_message()/open_session() had been made async so their telemetry
calls could be awaited inline instead of spawned, closing a rare
quit-before-flush race — but that makes the whole interactive TUI loop
block on a live network call (up to the 3s telemetry timeout, more for
setup's four sequential events) on ordinary actions like killing a
session, not just on quit. Reverted to fire-and-forget tokio::spawn for
these call sites, consistent with how every other background task in
this file already behaves and with the existing Effect::Agent /
Effect::Reattach spawns — a rare, bounded, silent loss on an
immediate quit is a much smaller cost than a guaranteed UI freeze on
everyday use.
- resolve_agent_choice() failures (conflicting --codex/--claude/--grok
flags, no default agent configured non-interactively) returned before
prepare()'s timer/tracking started, so a real, user-visible failure
category was silently absent from cloud_agent_launch entirely rather
than showing up as a failure. Now tracked under harness "unresolved".
- Softened cloud_agent/telemetry.rs's doc comment, which overclaimed that
nothing here logs free text: error_message carries the same
truncated {err:#} text every other failure event in this CLI already
sends, which can include a user-supplied identifier (e.g. an unknown
environment name) when the underlying error formats one in. The
structured fields (harness/theme/skills slug, sub_command) remain
free of names; the doc now says so accurately instead of overpromising.
|
Ran an adversarial code review against this diff and it surfaced real bugs, now fixed in a follow-up commit:
|
Two conflicts, both where #1063's telemetry landed on lines this branch had already changed: - `prepare_inner`'s resolve_target call: keep master's telemetry wrapper, bind `_project_id` since this branch removed `Prepared::project_id`. - The TUI's `close_and_sleep`: keep both. Sleeping goes through the controller so the disk is flushed first, and the failure still fires `quit_sleep_failed` — an agent left billing with nothing attached is exactly what that event is for. The no-environment fallback keeps the bare mutation, since without a relay target there is nothing to flush through. `run_agent_op`'s trailing `let _ = environment_id` goes: the Sleep arm now uses it to reach the agent.
#1063 covered launches, the TUI's manage-screen ops, and setup. The flat CLI verbs this branch adds had only the generic per-dispatch event, which says `command="cloud_agent"` and cannot say which verb ran. Adds `track_lifecycle`, following the module's existing shape: - One event per verb — `cli_list`, `cli_create`, `cli_ssh`, `cli_wake`, `cli_sleep`, `cli_delete`, with `_failed` variants — carrying real duration, which is the interesting part for `create` and `wake` since both wait for RUNNING. - Detail slugs for the paths worth separating: `cli_ssh_attach` vs `cli_ssh_new_session` vs `cli_ssh_command` (reattaching to existing work, provisioning a first session, and one-shot commands are three different things), and `cli_sleep_all` for the fleet-wide cost valve. The `cli_` prefix keeps these distinct from `track_agent_op`'s `agent_sleep` and friends. Same three mutations, two surfaces — merging them would hide which one people actually reach for. A test pins that separation. Five verbs are wrapped at the dispatch, where the shape is identical. `ssh` tracks itself: it ends in `std::process::exit` to propagate the remote command's status, which would skip anything wrapped around it. Its body moved to `ssh_connect` returning the exit code, so the event fires before the exit, and a non-zero remote status is reported as a success — ssh worked, the command it ran did not. Verified: `ca ssh <agent> -- sh -c 'exit 3'` still exits 3. No free text in any of it: fixed slugs only, with `error_message` following the same convention as every other failure event in the CLI.
…elete (#1064) * feat(ca): flat lifecycle commands — list, create, ssh, wake, sleep, delete The lifecycle operations already existed, wearing flag costumes on a command whose job is something else: `--new` creates, `--rm` destroys, `--keep-awake` declines to sleep. `launch()` even short-circuits `--rm` before the launch pipeline with a comment noting it is not a launch. This gives each one a name. The line that matters is creation. `ca create` makes a VM and nothing else; `ca ssh` connects to an agent that exists and errors otherwise; `ca start` and `railway code` stay the create-and-launch path. So a mistyped agent name is an error rather than a second billed VM. `ca ssh` rather than `ca connect`: it is an ssh connection over the same relay sandboxes use, and `railway sandbox ssh` already means "get onto the box, optionally resuming a named session" — durable-session resume included. Bare `ca ssh` attaches to the agent's session, `ca ssh <agent> -- bash` is a plain shell, and `connect` is a visible alias so both spellings work. Notes on the shape: - One resolver for every verb, in src/controllers/cloud_agent.rs: an explicit name or id, then this environment's remembered agent, then your sole live one, then a list of candidates. No interactive prompt anywhere in it — a lifecycle command that stops to ask is unusable in a script, and ambiguity has a better answer than a guess. - `list` defaults to the whole account via myCloudAgents. Scoping it to the linked environment would print nothing in the common case, since agent work is rarely done from a linked directory. - `ensure_running` will not adopt the launcher's habit of treating a crashed agent as a cue to create a fresh one. That is a fine answer to "get me coding" and a bad one to "wake this agent". - `sleep --all` exists and `delete --all` does not: agents have no idle timeout so bulk sleep is the cost valve, but bulk disk destruction behind one flag is not. - Pointer bookkeeping lives in the controller, so a delete cannot leave `railway code` waking a corpse. `--rm` keeps working and now points at `ca delete`, and the launcher's exit hints name the agent instead of printing a raw ssh line and two UUIDs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ca): roll back a wake when the connect fails, and stop claiming "asleep" Both found running the lifecycle end to end. `ca ssh <agent> --session <typo>` woke a sleeping agent, failed the session lookup, and returned — leaving a machine with no idle timeout awake and billing. Measured at 12s, so the wake was real, not a no-op. Now a failed connect puts back what the run changed, and only that: an agent found already running was someone's deliberate state, possibly with a session open in another terminal, and a failed connect here is no reason to suspend it. The sleep mutation also returns before the agent finishes transitioning, so `ca sleep foo && ca list` printed "is asleep" followed by "running". Reworded to describe the action rather than assert a state the next command contradicts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ca): flush the agent's disk before sleeping it Sleeping an agent could silently lose its most recent work. `cloudAgentSleep` snapshots the disk without quiescing the guest, so pages still dirty in its page cache are absent from the image the next wake restores. Measured on scratch agents, three runs isolating the variable: write, no sync + sleep-on-disconnect -> lost write + sync + explicit sleep, 10s quiesce -> survived write + sync + sleep-on-disconnect, immediate -> survived So it is the flush that matters, not the timing — waiting longer before sleeping would only narrow the window. A side ssh running `sync` is enough, because sync(2) flushes the whole filesystem regardless of which process dirtied it, so it covers whatever the durable session was writing. The flush is paired with the mutation inside controllers::cloud_agent::sleep rather than left to callers: there are four paths that suspend an agent (the two `railway ca` ones, the launcher's disconnect, and the TUI's), and any one of them forgetting is silent data loss. The bare mutation is now private. It is best-effort and bounded at 5s. Failing to reach an agent must not stop us sleeping it — agents have no idle timeout, so the alternative to an imperfect sleep is a machine that bills until someone remembers it. Deliberately not ssh_plumbing, whose ~20s retry budget exists for waking agents and would make every disconnect slow whenever the relay is unhealthy. `sleep --all` runs its flushes concurrently so the cost-control command does not become a second per agent. Verified end to end: the write that vanished before now survives the same sleep/wake, and a disconnect costs ~1s more (1.0s -> 2.1s). The real fix belongs server-side, in cloudAgentSleep quiescing the guest before it snapshots — the dashboard and any future API caller have the same problem, and only the platform can make it a guarantee rather than a narrowed window. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(ca): telemetry for the lifecycle commands #1063 covered launches, the TUI's manage-screen ops, and setup. The flat CLI verbs this branch adds had only the generic per-dispatch event, which says `command="cloud_agent"` and cannot say which verb ran. Adds `track_lifecycle`, following the module's existing shape: - One event per verb — `cli_list`, `cli_create`, `cli_ssh`, `cli_wake`, `cli_sleep`, `cli_delete`, with `_failed` variants — carrying real duration, which is the interesting part for `create` and `wake` since both wait for RUNNING. - Detail slugs for the paths worth separating: `cli_ssh_attach` vs `cli_ssh_new_session` vs `cli_ssh_command` (reattaching to existing work, provisioning a first session, and one-shot commands are three different things), and `cli_sleep_all` for the fleet-wide cost valve. The `cli_` prefix keeps these distinct from `track_agent_op`'s `agent_sleep` and friends. Same three mutations, two surfaces — merging them would hide which one people actually reach for. A test pins that separation. Five verbs are wrapped at the dispatch, where the shape is identical. `ssh` tracks itself: it ends in `std::process::exit` to propagate the remote command's status, which would skip anything wrapped around it. Its body moved to `ssh_connect` returning the exit code, so the event fires before the exit, and a non-zero remote status is reported as a success — ssh worked, the command it ran did not. Verified: `ca ssh <agent> -- sh -c 'exit 3'` still exits 3. No free text in any of it: fixed slugs only, with `error_message` following the same convention as every other failure event in the CLI. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Summary
railway cahad essentially no bespoke telemetry. The only signal was the generic per-dispatch event thecommands!macro fires for every top-level command (command="cloud_agent",sub_commandin{None, "setup", "start"},success= the top-levelResult).That's close to useless for the most common path: bare
railway caruns a TUI event loop that swallows internal errors (SSH attach failures, credential-mint failures, launch failures) into toast text and keeps looping, socommand()almost never returnsErr— every barerailway cainvocation reportedsuccess: trueregardless of what actually happened. There was also no visibility into which agent got launched, whether a VM was created vs. reused, agent lifecycle actions (sleep/wake/delete), session reattach/kill, or setup/wizard choices.This adds a small
commands/cloud_agent/telemetry.rsmodule and wires it in at the natural convergence points, following the CLI's existing conventions (additiveCliTrackEventdetail events, same shape as the per-tool MCP and per-stage SSH events incommands/ssh/tel.rsandcommands/service.rs'strack_service_source). No new GraphQL mutations — everything reuses the existingcliEventTrackevent, already gated byDO_NOT_TRACK/RAILWAY_NO_TELEMETRY/ the local telemetry preference.cloud_agent_launch): harness, created-vs-reused, real duration, fired from insidecode::prepare()itself — so it fires forrailway code,railway ca start, and the TUI'sstart_launchalike, and survives the TUI catching the error afterward.let _ =(a failure there means an agent is left running and billing compute unattended).railway ca setupand the TUI's first-run wizard.No user-visible behavior changes; no free text is ever logged (no prompts, no session/agent/project names — only IDs, fixed slugs, and truncated error messages).
Test plan
cargo buildcargo fmtcargo clippy --all-targets— no new warnings in changed filescargo test --bin railway— 945 passed; the 3auth_simfailures are pre-existing onmastertoo, unrelated to this changeca setup) to confirm events post with the expectedcommand/sub_command/success