feat(telemetry): deep CLI instrumentation across all commands

[willwashburn]

Summary

• Every CLI command now emits cli_command_run + cli_command_complete (success, duration, exit code, error class) — previously only 7 interactive commands emitted any telemetry.
• New domain events at high-signal product surfaces: workflow_run, cloud_auth, cloud_workflow_run, provider_auth, setup_init, swarm_run, bridge_spawn, broker_start_failed.
• Global Commander hooks + async parseAsync + shutdown() flush so PostHog's event queue drains before the process exits.

Why

Today the main product surface — agent-relay run, cloud login, auth <provider>, swarm, bridge — emits no telemetry. The interactiveCommands gate in bootstrap only fires events for up/down/status/version. We cannot answer basic product questions (which commands fail? which workflows succeed? how long does onboarding take?) from the data we collect.

What's in the new schema

Event	Fires from	Key properties
cli_command_run (extended)	Every command (preAction hook)	command_name, flags_used, is_tty, is_ci
cli_command_complete (new)	Every command (postAction / exit / error)	success, duration_ms, exit_code, error_class
workflow_run (new)	agent-relay run <file>	file_type, is_script, is_dry_run, is_resume, success, duration_ms
cloud_auth (new)	cloud login/logout/whoami/connect	action, success, duration_ms, provider?
cloud_workflow_run (new)	cloud run	has_explicit_file_type, sync_code, json_output, success, duration_ms
provider_auth (new)	auth <provider>	provider, use_auth_broker, used_token, success, duration_ms
setup_init (new)	init / setup wizard	is_cloud, broker_was_running, user_started_broker, yes_flag, skip_broker
swarm_run (new)	swarm	pattern, teams, cli, is_list, is_dry_run, exit_code, duration_ms
bridge_spawn (new)	bridge	project_count, cli, has_architect
broker_start_failed (new)	Future broker-lifecycle wiring	stage, error_class

Privacy

• Flag names are captured; values never are.
• No run IDs, task text, file paths, URLs, or tokens reach telemetry.
• Errors are reported as error_class (the Error constructor name) — never the message.
• The existing opt-out path (AGENT_RELAY_TELEMETRY_DISABLED=1 and agent-relay telemetry disable) continues to work; the telemetry management commands themselves are excluded from initTelemetry so opt-in/opt-out flips never leak events.

Key mechanics

• src/cli/bootstrap.ts installs preAction / postAction hooks on the root Commander program. Full command path ("cloud login" not "cloud") is derived by walking parent chain; flag names come from getOptionValueSource(...) === 'cli'.
• runCli is now async and awaits both parseAsync and shutdownTelemetry(). Top-level try/catch captures thrown errors and emits a failure cli_command_complete before flushing.
• process.on('exit') + beforeExit are safety nets for commands that call process.exit(code) mid-flight, so a cli_command_complete still fires with the exit code.
• src/cli/index.ts handles the promise returned by runCli with .catch() and a non-zero exit on unhandled top-level failures.

Test plan

• npx tsc --noEmit — clean
• npx vitest run — 732/732 passing, including updated entrypoint.test.ts mocks for the new telemetry import surface and async runCli
• Local manual smoke: run agent-relay status, agent-relay run some.yaml, agent-relay cloud whoami with telemetry enabled against a PostHog dev project and verify the new events arrive with expected properties
• Confirm AGENT_RELAY_TELEMETRY_DISABLED=1 agent-relay ... still produces no network traffic

Files

• packages/telemetry/src/events.ts, index.ts
• src/cli/bootstrap.ts, index.ts, entrypoint.test.ts
• src/cli/commands/{cloud,auth,setup,swarm,core}.ts
• src/cli/lib/telemetry-helpers.ts (new shared errorClassName)

🤖 Generated with Claude Code

---

[Copilot]

Pull request overview

Expands telemetry coverage across the CLI and broker so product-critical commands (run/auth/cloud/swarm/bridge/setup) emit consistent PostHog events, with improved version/context tagging and lifecycle attribution.

Changes:

• Add global Commander hooks to emit cli_command_run / cli_command_complete for all CLI commands, and flush telemetry on orderly CLI completion.
• Introduce new high-signal domain events (workflow_run, cloud_auth, cloud_workflow_run, provider_auth, setup_init, swarm_run, bridge_spawn) at key command call sites.
• Extend broker telemetry schema with richer agent spawn/release attribution (ActionSource, has_task, is_shadow) and common version/os tagging.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
src/wrap.rs	Add spawn/release attribution fields when broker handles agent-originated wrap actions.
src/telemetry.rs	Extend broker telemetry schema (ActionSource + new fields), add common version/os tagging helpers, and update serialization/tests.
src/main.rs	Populate new broker telemetry fields for agent spawn/release events.
src/cli/lib/telemetry-helpers.ts	New shared helper for privacy-safe error_class extraction.
src/cli/index.ts	Handle async runCli() and ensure top-level failures exit non-zero.
src/cli/entrypoint.test.ts	Update mocks/tests for new telemetry shutdown surface and async runCli.
src/cli/commands/swarm.ts	Emit swarm_run telemetry for dry-run and real broker execution.
src/cli/commands/setup.ts	Emit setup_init and workflow_run telemetry for init + workflow runner paths.
src/cli/commands/core.ts	Emit bridge_spawn telemetry when starting the multi-project bridge.
src/cli/commands/cloud.ts	Emit cloud_auth + cloud_workflow_run telemetry with success/duration/error_class.
src/cli/commands/auth.ts	Emit provider_auth telemetry with provider normalization + error_class.
src/cli/bootstrap.ts	Install global command run/complete hooks, propagate version env vars, switch to parseAsync, and add telemetry shutdown.
packages/telemetry/src/index.ts	Export new event types and InitTelemetryOptions.
packages/telemetry/src/events.ts	Extend event schema/types with new CLI/domain events and common version fields.
packages/telemetry/src/client.ts	Allow callers to pass explicit CLI/SDK/broker versions into common properties.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[willwashburn]

Thanks for the reviews — addressed in 487dcafd:

Resolved

• [Devin + Copilot ×4] telemetry loss on process.exit — introduced src/cli/lib/exit.ts with a CliExit error class. All 8 command modules now share defaultExit from there (throws CliExit instead of calling process.exit). runCli() catches it, emits cli_command_complete with the real exit_code, awaits shutdownTelemetry(), and only then calls the real process.exit. Applied the same fix to swarm.ts's direct process.exit calls. The six previously-lossy sites (swarm dry-run, swarm real, 4× setup.ts workflow_run failure paths) now actually ship their events.
• [Copilot] postAction reports success: true regardless of process.exitCode — now reads process.exitCode ?? 0, emits success: false + exit_code when set.
• [Copilot] ReleaseReason enum doesn't match broker — relaxed to string with JSDoc listing known canonical and broker-local values.
• [Copilot] error_class violates "Error constructor name" contract — header docstring updated to allow stable synthetic category tags (e.g. 'WorkflowNotCompleted'), unified under PascalCase.
• [Copilot] os_version required in TS but optional in Rust — made optional in CommonProperties.
• [Copilot] duplicate errorClassName in bootstrap.ts — removed, now imports from the shared lib/telemetry-helpers.ts.

Not fixed

• [GitHub Advanced Security / CodeQL] cloud.ts:199 — File data in outbound network request — this is pre-existing logout behavior: we read the user's own stored refresh token from ~/.agent-relay/auth.json and POST it to the user's own configured apiUrl to revoke it. My PR only wrapped that call in a try/catch for telemetry; the URL/body construction is unchanged and semantically correct. Flagging here so we can triage separately if desired.

Verified: npx tsc --noEmit clean, 190/190 TS tests, 10/10 Rust telemetry tests.

[willwashburn]

Addressed both Devin findings in 785b668a:

1. Unhandled rejection in signal handlers — the four deps.exit(0) / deps.exit(130) sites in broker-lifecycle.ts run inside async signal handlers registered via deps.onSignal(...). After switching defaultExit to throw CliExit, those async handlers' promises would reject unhandled → Node forced exit code 1 on Ctrl+C. Fixed by wrapping the handler invocation in the default onSignal implementation (core.ts): if it catches a CliExit, it flushes telemetry best-effort and calls process.exit(err.code) with the intended code. DI contract unchanged — tests that pass their own onSignal + exit mocks still see their ExitSignal rejection, and the existing up force exits on repeated SIGINT test still asserts { code: 130 } and passes.

2. setup.ts outer catch printed "Error: cli-exit:1" — the run command's three internal deps.exit(1) sites now throw CliExit, which was being caught by the generic catch (err: any) that prints err.message and emits error_class: 'CliExit' on workflow_run. Fixed by re-throwing CliExit at the top of the catch so it bubbles cleanly to runCli().

3. Belt-and-braces: errorClassName(err) now returns undefined for CliExit instances, so any future try/finally block in cloud/auth that unexpectedly sees CliExit propagate won't pollute its event with error_class: 'CliExit'. Bootstrap updated to the same conditional-spread pattern for consistency.

Verified: tsc --noEmit clean, vitest run src/cli 190/190 (including the SIGINT regression test).