Skip to content

Add agentic CLI telemetry#882

Merged
codyde merged 1 commit into
masterfrom
agentic-cli-telemetry
May 7, 2026
Merged

Add agentic CLI telemetry#882
codyde merged 1 commit into
masterfrom
agentic-cli-telemetry

Conversation

@codyde
Copy link
Copy Markdown
Collaborator

@codyde codyde commented May 7, 2026

Summary

  • Enrich every CLI telemetry event with the agentic-loop context added in mono: sessionId, bounded raw caller, agentSessionId, installRequestId, Railway resource IDs, and coarse errorClass.
  • Emit setupAgentEventTrack lifecycle events from railway setup agent with start/finish, success/failure, configured clients, and the same shared telemetry context.
  • Persist user.id after login for future skills attribution and stamp RAILWAY_INSTALL_REQUEST_ID from install.sh --agents so install setup can be correlated to first CLI telemetry.

Why

This is the CLI producer-side follow-up to the merged agentic telemetry contract in mono (#28652). It starts turning CLI usage into analyzable agentic telemetry so we can answer questions like “which agents are using Railway?”, “which setup flows complete?”, and “which project/environment/service did an agent-driven CLI call touch?”

Caller Attribution

caller is populated on all CLI telemetry events. Resolution order is:

  1. Explicit RAILWAY_CALLER when set by a harness or skill.
  2. Known agent environment markers.
  3. Best-effort Unix process ancestry detection for opencode, claude, cursor, codex, and windsurf.
  4. Fallback values: ci, agent_subprocess, or tty.

This means attribution is reliable when RAILWAY_CALLER is set and best-effort otherwise. Strong cross-surface joins still require harnesses/skills to set RAILWAY_AGENT_SESSION; when absent, agent-like CLI invocations fall back to a per-process session ID.

Safety / Compatibility

  • Telemetry continues to respect existing opt-outs: DO_NOT_TRACK, RAILWAY_NO_TELEMETRY, and railway telemetry disable.
  • Telemetry remains best-effort and must not fail user commands.
  • Enriched CLI telemetry automatically retries once with the legacy payload shape if the backend rejects or fails the new payload, protecting deploy skew and older backend environments.
  • setupAgentEventTrack failures are ignored.
  • Persisting user.id after login is best-effort; login still succeeds if writing the ID fails.

Notes

  • install.sh --agents now generates RAILWAY_INSTALL_REQUEST_ID locally and exports it before invoking railway setup agent.
  • This correlates install script execution to CLI setup telemetry. It does not yet correlate /agents.md fetches to install execution; that remains a future frontend/server-side attribution follow-up.
  • The existing generated CliEventTrack GraphQL type is intentionally left in place for compatibility, but the new telemetry calls are sent as raw GraphQL JSON so the PR is not blocked on regenerated CLI GraphQL bindings for nullable telemetry fields.

Validation

  • bash -n install.sh
  • git diff --check
  • nix shell nixpkgs#rustfmt -c rustfmt --edition 2024 src/telemetry.rs src/commands/setup.rs src/commands/login.rs src/config.rs src/consts.rs src/gql/mutations/mod.rs
  • nix shell nixpkgs#cargo nixpkgs#rustc -c cargo check
  • nix shell nixpkgs#cargo nixpkgs#rustc -c cargo test --no-run

@codyde codyde added the release/minor Author minor release label May 7, 2026
@codyde codyde merged commit 350cdfd into master May 7, 2026
6 of 7 checks passed
@codyde codyde deleted the agentic-cli-telemetry branch May 7, 2026 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release/minor Author minor release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant