Skip to content

Add telemetry for railway ca launches, lifecycle ops, and setup - #1063

Merged
codyde merged 2 commits into
masterfrom
railway/ca-telemetry
Aug 9, 2026
Merged

Add telemetry for railway ca launches, lifecycle ops, and setup#1063
codyde merged 2 commits into
masterfrom
railway/ca-telemetry

Conversation

@codyde

@codyde codyde commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

railway ca had essentially no bespoke telemetry. The only signal was the generic per-dispatch event the commands! macro fires for every top-level command (command="cloud_agent", sub_command in {None, "setup", "start"}, success = the top-level Result).

That's close to useless for the most common path: bare railway ca runs a TUI event loop that swallows internal errors (SSH attach failures, credential-mint failures, launch failures) into toast text and keeps looping, so command() almost never returns Err — every bare railway ca invocation reported success: true regardless of what actually happened. There was also no visibility into which agent got launched, whether a VM was created vs. reused, agent lifecycle actions (sleep/wake/delete), session reattach/kill, or setup/wizard choices.

This adds a small commands/cloud_agent/telemetry.rs module and wires it in at the natural convergence points, following the CLI's existing conventions (additive CliTrackEvent detail events, same shape as the per-tool MCP and per-stage SSH events in commands/ssh/tel.rs and commands/service.rs's track_service_source). No new GraphQL mutations — everything reuses the existing cliEventTrack event, already gated by DO_NOT_TRACK / RAILWAY_NO_TELEMETRY / the local telemetry preference.

  • Launch outcome (cloud_agent_launch): harness, created-vs-reused, real duration, fired from inside code::prepare() itself — so it fires for railway code, railway ca start, and the TUI's start_launch alike, and survives the TUI catching the error afterward.
  • Agent lifecycle ops: sleep/wake/delete from the manage screen.
  • Session events: reattach (connect + open), kill, opening a freshly prepared session, and the auto-sleep-on-quit whose result was previously discarded with let _ = (a failure there means an agent is left running and billing compute unattended).
  • Setup/wizard: one flat event per choice (agent/theme/project/skills) on save, from both railway ca setup and the TUI's first-run wizard.

No user-visible behavior changes; no free text is ever logged (no prompts, no session/agent/project names — only IDs, fixed slugs, and truncated error messages).

Test plan

  • cargo build
  • cargo fmt
  • cargo clippy --all-targets — no new warnings in changed files
  • cargo test --bin railway — 945 passed; the 3 auth_sim failures are pre-existing on master too, unrelated to this change
  • Manual smoke test against a real account (launch, reattach, sleep/wake/delete, quit with a session open, ca setup) to confirm events post with the expected command/sub_command/success

codyde added 2 commits August 9, 2026 06:34
`railway ca` had no bespoke telemetry: bare `railway ca` runs a TUI loop
that swallows internal errors into toast text, so the generic per-command
event nearly always reported success regardless of what actually happened.
There was also no visibility into which agent got launched, agent lifecycle
actions, session reattach/kill, or setup/wizard choices.

Adds a small `cloud_agent::telemetry` module, following the CLI's existing
conventions (additive CliTrackEvent detail events, same shape as the
per-tool MCP and per-stage SSH events) to close these gaps:

- Launch outcome (harness, created vs reused, real duration) fired from
  inside `code::prepare()` itself, so it fires regardless of caller
  (`railway code`, `railway ca start`, or the TUI) and regardless of the
  TUI swallowing the error afterward.
- Agent lifecycle ops (sleep/wake/delete) from the manage screen.
- Session events: reattach, kill, opening a freshly launched session, and
  the auto-sleep-on-quit that previously discarded its result with `let _ =`
  (a failure there means an agent is left running and billing compute).
- Setup/wizard completions, one flat event per choice (agent/theme/project/
  skills), from both `railway ca setup` and the TUI's first-run wizard.

No new GraphQL mutations — everything reuses the existing `cliEventTrack`
event already gated by DO_NOT_TRACK / RAILWAY_NO_TELEMETRY.
An adversarial code review of the ca-telemetry changes surfaced several
real bugs, fixed here:

- String::truncate(256)/s[..256] on an error message panics whenever byte
  256 lands mid-character (a path or GraphQL error with a non-ASCII
  character near the boundary) — and this crate builds with
  panic = "abort", so that panic takes down the whole process instead of
  just failing to log. This exact pattern existed independently in five
  places: the generic per-command event (macros.rs), `setup agent`,
  the MCP tool event, ssh/tel.rs, and the new cloud_agent telemetry
  module. Added one char-boundary-safe crate::telemetry::truncate_message
  and pointed all five at it, with a regression test reproducing the panic
  on the old logic.

- handle_message()/open_session() had been made async so their telemetry
  calls could be awaited inline instead of spawned, closing a rare
  quit-before-flush race — but that makes the whole interactive TUI loop
  block on a live network call (up to the 3s telemetry timeout, more for
  setup's four sequential events) on ordinary actions like killing a
  session, not just on quit. Reverted to fire-and-forget tokio::spawn for
  these call sites, consistent with how every other background task in
  this file already behaves and with the existing Effect::Agent /
  Effect::Reattach spawns — a rare, bounded, silent loss on an
  immediate quit is a much smaller cost than a guaranteed UI freeze on
  everyday use.

- resolve_agent_choice() failures (conflicting --codex/--claude/--grok
  flags, no default agent configured non-interactively) returned before
  prepare()'s timer/tracking started, so a real, user-visible failure
  category was silently absent from cloud_agent_launch entirely rather
  than showing up as a failure. Now tracked under harness "unresolved".

- Softened cloud_agent/telemetry.rs's doc comment, which overclaimed that
  nothing here logs free text: error_message carries the same
  truncated {err:#} text every other failure event in this CLI already
  sends, which can include a user-supplied identifier (e.g. an unknown
  environment name) when the underlying error formats one in. The
  structured fields (harness/theme/skills slug, sub_command) remain
  free of names; the doc now says so accurately instead of overpromising.

codyde commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator Author

Ran an adversarial code review against this diff and it surfaced real bugs, now fixed in a follow-up commit:

  1. Truncation panic (high severity): String::truncate(256)/s[..256] on an error message panics whenever byte 256 lands mid-character — any error with a non-ASCII byte near the boundary (an accented/CJK/emoji character in a path, a non-English GraphQL error). This crate builds with panic = "abort", so that panic takes down the whole CLI process, not just the log call. The same pattern existed independently in five places, including two I didn't touch (macros.rs's generic per-command event — fired for literally every command — and the MCP tool event). Added one shared crate::telemetry::truncate_message (char-boundary-safe) and pointed all five call sites at it, with a regression test reproducing the panic on the old logic.

  2. TUI-blocking regression: I'd made handle_message/open_session async so their telemetry could be awaited inline instead of spawned, to close a race where a fire-and-forget tokio::spawn might not flush before the process exits on quit. That "fix" was worse than the bug: it makes the whole interactive TUI block on a live network call on ordinary actions (killing a session, a failed reattach) — up to the telemetry timeout, and up to ~4x that for setup's four sequential events. Reverted to fire-and-forget tokio::spawn for these, consistent with every other background task already in this file (session prefetch, sweep, refresh) — none of them get joined either. A rare, silent loss of one event on an immediate quit is a much smaller cost than a guaranteed multi-second freeze on everyday use.

  3. Silent gap: resolve_agent_choice() failures (conflicting --codex --claude, no default agent configured non-interactively) returned before the launch timer/tracking even started, so a real, user-hit failure mode was invisible to cloud_agent_launch entirely instead of showing up as a failure. Now tracked under harness "unresolved".

  4. Doc accuracy: the new module's comment overclaimed "nothing here logs free text." error_message carries the same truncated {err:#} text every other failure event in this CLI already sends, which can include a user-typed identifier (e.g., an unknown environment name) when the underlying error formats one in — consistent with existing codebase-wide behavior, not a new leak, but the comment now says so accurately instead of overpromising.

cargo test --bin railway: 948 passed (up from 945 — added truncation regression tests), same 3 pre-existing auth_sim failures unrelated to this change. No clippy warnings in any touched file.

@codyde codyde added the release/skip Author no release label Aug 9, 2026 — with Railway App
@codyde
codyde merged commit 9a0b6ae into master Aug 9, 2026
7 of 8 checks passed
@codyde
codyde deleted the railway/ca-telemetry branch August 9, 2026 07:14
codyde added a commit that referenced this pull request Aug 9, 2026
Two conflicts, both where #1063's telemetry landed on lines this branch had
already changed:

- `prepare_inner`'s resolve_target call: keep master's telemetry wrapper, bind
  `_project_id` since this branch removed `Prepared::project_id`.
- The TUI's `close_and_sleep`: keep both. Sleeping goes through the controller
  so the disk is flushed first, and the failure still fires
  `quit_sleep_failed` — an agent left billing with nothing attached is exactly
  what that event is for. The no-environment fallback keeps the bare mutation,
  since without a relay target there is nothing to flush through.

`run_agent_op`'s trailing `let _ = environment_id` goes: the Sleep arm now uses
it to reach the agent.
codyde added a commit that referenced this pull request Aug 9, 2026
#1063 covered launches, the TUI's manage-screen ops, and setup. The flat CLI
verbs this branch adds had only the generic per-dispatch event, which says
`command="cloud_agent"` and cannot say which verb ran.

Adds `track_lifecycle`, following the module's existing shape:

- One event per verb — `cli_list`, `cli_create`, `cli_ssh`, `cli_wake`,
  `cli_sleep`, `cli_delete`, with `_failed` variants — carrying real duration,
  which is the interesting part for `create` and `wake` since both wait for
  RUNNING.
- Detail slugs for the paths worth separating: `cli_ssh_attach` vs
  `cli_ssh_new_session` vs `cli_ssh_command` (reattaching to existing work,
  provisioning a first session, and one-shot commands are three different
  things), and `cli_sleep_all` for the fleet-wide cost valve.

The `cli_` prefix keeps these distinct from `track_agent_op`'s `agent_sleep`
and friends. Same three mutations, two surfaces — merging them would hide which
one people actually reach for. A test pins that separation.

Five verbs are wrapped at the dispatch, where the shape is identical. `ssh`
tracks itself: it ends in `std::process::exit` to propagate the remote command's
status, which would skip anything wrapped around it. Its body moved to
`ssh_connect` returning the exit code, so the event fires before the exit, and a
non-zero remote status is reported as a success — ssh worked, the command it ran
did not. Verified: `ca ssh <agent> -- sh -c 'exit 3'` still exits 3.

No free text in any of it: fixed slugs only, with `error_message` following the
same convention as every other failure event in the CLI.
codyde added a commit that referenced this pull request Aug 9, 2026
…elete (#1064)

* feat(ca): flat lifecycle commands — list, create, ssh, wake, sleep, delete

The lifecycle operations already existed, wearing flag costumes on a command
whose job is something else: `--new` creates, `--rm` destroys, `--keep-awake`
declines to sleep. `launch()` even short-circuits `--rm` before the launch
pipeline with a comment noting it is not a launch. This gives each one a name.

The line that matters is creation. `ca create` makes a VM and nothing else;
`ca ssh` connects to an agent that exists and errors otherwise; `ca start` and
`railway code` stay the create-and-launch path. So a mistyped agent name is an
error rather than a second billed VM.

`ca ssh` rather than `ca connect`: it is an ssh connection over the same relay
sandboxes use, and `railway sandbox ssh` already means "get onto the box,
optionally resuming a named session" — durable-session resume included. Bare
`ca ssh` attaches to the agent's session, `ca ssh <agent> -- bash` is a plain
shell, and `connect` is a visible alias so both spellings work.

Notes on the shape:

- One resolver for every verb, in src/controllers/cloud_agent.rs: an explicit
  name or id, then this environment's remembered agent, then your sole live
  one, then a list of candidates. No interactive prompt anywhere in it — a
  lifecycle command that stops to ask is unusable in a script, and ambiguity
  has a better answer than a guess.
- `list` defaults to the whole account via myCloudAgents. Scoping it to the
  linked environment would print nothing in the common case, since agent work
  is rarely done from a linked directory.
- `ensure_running` will not adopt the launcher's habit of treating a crashed
  agent as a cue to create a fresh one. That is a fine answer to "get me
  coding" and a bad one to "wake this agent".
- `sleep --all` exists and `delete --all` does not: agents have no idle
  timeout so bulk sleep is the cost valve, but bulk disk destruction behind one
  flag is not.
- Pointer bookkeeping lives in the controller, so a delete cannot leave
  `railway code` waking a corpse.

`--rm` keeps working and now points at `ca delete`, and the launcher's exit
hints name the agent instead of printing a raw ssh line and two UUIDs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(ca): roll back a wake when the connect fails, and stop claiming "asleep"

Both found running the lifecycle end to end.

`ca ssh <agent> --session <typo>` woke a sleeping agent, failed the session
lookup, and returned — leaving a machine with no idle timeout awake and
billing. Measured at 12s, so the wake was real, not a no-op. Now a failed
connect puts back what the run changed, and only that: an agent found already
running was someone's deliberate state, possibly with a session open in another
terminal, and a failed connect here is no reason to suspend it.

The sleep mutation also returns before the agent finishes transitioning, so
`ca sleep foo && ca list` printed "is asleep" followed by "running". Reworded
to describe the action rather than assert a state the next command contradicts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(ca): flush the agent's disk before sleeping it

Sleeping an agent could silently lose its most recent work. `cloudAgentSleep`
snapshots the disk without quiescing the guest, so pages still dirty in its
page cache are absent from the image the next wake restores.

Measured on scratch agents, three runs isolating the variable:

  write, no sync   + sleep-on-disconnect            -> lost
  write + sync     + explicit sleep, 10s quiesce    -> survived
  write + sync     + sleep-on-disconnect, immediate -> survived

So it is the flush that matters, not the timing — waiting longer before
sleeping would only narrow the window. A side ssh running `sync` is enough,
because sync(2) flushes the whole filesystem regardless of which process
dirtied it, so it covers whatever the durable session was writing.

The flush is paired with the mutation inside controllers::cloud_agent::sleep
rather than left to callers: there are four paths that suspend an agent (the
two `railway ca` ones, the launcher's disconnect, and the TUI's), and any one
of them forgetting is silent data loss. The bare mutation is now private.

It is best-effort and bounded at 5s. Failing to reach an agent must not stop us
sleeping it — agents have no idle timeout, so the alternative to an imperfect
sleep is a machine that bills until someone remembers it. Deliberately not
ssh_plumbing, whose ~20s retry budget exists for waking agents and would make
every disconnect slow whenever the relay is unhealthy. `sleep --all` runs its
flushes concurrently so the cost-control command does not become a second per
agent.

Verified end to end: the write that vanished before now survives the same
sleep/wake, and a disconnect costs ~1s more (1.0s -> 2.1s).

The real fix belongs server-side, in cloudAgentSleep quiescing the guest before
it snapshots — the dashboard and any future API caller have the same problem,
and only the platform can make it a guarantee rather than a narrowed window.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(ca): telemetry for the lifecycle commands

#1063 covered launches, the TUI's manage-screen ops, and setup. The flat CLI
verbs this branch adds had only the generic per-dispatch event, which says
`command="cloud_agent"` and cannot say which verb ran.

Adds `track_lifecycle`, following the module's existing shape:

- One event per verb — `cli_list`, `cli_create`, `cli_ssh`, `cli_wake`,
  `cli_sleep`, `cli_delete`, with `_failed` variants — carrying real duration,
  which is the interesting part for `create` and `wake` since both wait for
  RUNNING.
- Detail slugs for the paths worth separating: `cli_ssh_attach` vs
  `cli_ssh_new_session` vs `cli_ssh_command` (reattaching to existing work,
  provisioning a first session, and one-shot commands are three different
  things), and `cli_sleep_all` for the fleet-wide cost valve.

The `cli_` prefix keeps these distinct from `track_agent_op`'s `agent_sleep`
and friends. Same three mutations, two surfaces — merging them would hide which
one people actually reach for. A test pins that separation.

Five verbs are wrapped at the dispatch, where the shape is identical. `ssh`
tracks itself: it ends in `std::process::exit` to propagate the remote command's
status, which would skip anything wrapped around it. Its body moved to
`ssh_connect` returning the exit code, so the event fires before the exit, and a
non-zero remote status is reported as a success — ssh worked, the command it ran
did not. Verified: `ca ssh <agent> -- sh -c 'exit 3'` still exits 3.

No free text in any of it: fixed slugs only, with `error_message` following the
same convention as every other failure event in the CLI.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release/skip Author no release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant