Skip to content

feat(telemetry): propagate caller/agent-session as HTTP headers on the up upload#899

Closed
codyde wants to merge 2 commits into
masterfrom
cody/cli-emit-telemetry-headers-on-up
Closed

feat(telemetry): propagate caller/agent-session as HTTP headers on the up upload#899
codyde wants to merge 2 commits into
masterfrom
cody/cli-emit-telemetry-headers-on-up

Conversation

@codyde
Copy link
Copy Markdown
Collaborator

@codyde codyde commented May 13, 2026

Summary

railway up uploads its build tarball to backboard's POST /project/.../environment/.../up REST handler, which emits the CLI - Create Up telemetry event (warehouse: cli_create_up) via trackCLIEvent. About 107k of these events fire in a typical 48h window, but 100% of them land in the warehouse with caller, agent_session_id, and session_id all NULL because the upload request carries no envelope for backboard to attribute against.

This PR adds those fields as HTTP headers on the upload request, mirroring the propagation contract railway-skills already uses (X-Railway-Skill-* / X-Railway-Agent-Session):

X-Railway-CLI-Version
X-Railway-Session         (CLI session_id from telemetry::session_id())
X-Railway-Caller          (resolved caller, when non-empty)
X-Railway-Agent-Session   (agent_session_id, when set)

Headers are gated by is_telemetry_disabled() so disabled clients stay opted out across both surfaces. Caller / agent-session header omission mirrors the cliEventTrack mutation's null-skipping.

Companion PR

This is the producer half of a two-PR fix. The companion mono backboard PR (forthcoming) reads these headers in the up handler and passes them to trackCLIEvent properties + adds the missing deploymentId.

Both can ship independently — backboard reading absent headers will fall back to null (today's behavior); CLI sending headers to a backboard that doesn't read them is harmless.

Test plan

  • cargo build --bin railway clean
  • 21/21 telemetry unit tests pass
  • CI green
  • Manual: once the companion mono backboard PR ships, run railway up from claude_code and verify warehouse cli_create_up events have caller='claude_code' and a non-null agent_session_id

Context

This is gap #2 from the May 12 agentic-adoption telemetry audit. Once it lands, we unlock:

  • Strict "what % of new deploys are agent-driven" attribution (currently blocked because cli_create_up — the largest deploy event — has no caller)
  • Direct deploy↔session linkage when paired with the deploymentId backboard fix

🤖 Generated with Claude Code

@codyde codyde force-pushed the cody/cli-emit-telemetry-headers-on-up branch from f6d5427 to 738052b Compare June 3, 2026 05:47
…e up upload

The CLI's `railway up` uploads its build tarball to backboard's
`POST /project/.../environment/.../up` REST handler. That handler emits
the `CLI - Create Up` telemetry event (warehouse: `cli_create_up`) via
trackCLIEvent — about 107k events in a typical 48h window. But the
request only carries `Content-Type` today, so backboard has nothing to
attribute the event to: 100% of `cli_create_up` events land in the
warehouse with caller / agent_session_id / session_id all NULL.

Add the cliEventTrack envelope as HTTP headers on the upload request,
mirroring the propagation contract railway-skills already uses for
X-Railway-Skill-* / X-Railway-Agent-Session:

  X-Railway-CLI-Version
  X-Railway-Session         (CLI session_id from telemetry::session_id())
  X-Railway-Caller          (resolved caller, when non-empty)
  X-Railway-Agent-Session   (agent_session_id, when set)

Headers are gated by `is_telemetry_disabled()` so disabled clients stay
opted out across both surfaces. Caller / agent-session header omission
mirrors the cliEventTrack mutation's null-skipping for those fields.

This is the producer half of a two-PR fix. Companion mono backboard PR
(forthcoming) reads these headers in the `up` handler and passes them
to `trackCLIEvent` properties + adds the missing `deploymentId`.

Both can ship independently — backboard reading absent headers will
fall back to null (today's behavior); CLI sending headers to a
backboard that doesn't read them is harmless.

Verified:
- `cargo build --bin railway` clean.
- 21/21 telemetry tests pass (unchanged).
- No new clippy warnings on the touched files.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@codyde codyde force-pushed the cody/cli-emit-telemetry-headers-on-up branch from 738052b to 65e9724 Compare June 3, 2026 05:51
@codyde codyde added release/minor Author minor release release/patch Author patch release and removed release/patch Author patch release release/minor Author minor release labels Jun 3, 2026
Reuse backboard's established x-railway-version header (read by
getCliVersionFromRequest) instead of introducing a parallel
X-Railway-CLI-Version, so the up upload and the rest of backboard
agree on one version-header name.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@codyde
Copy link
Copy Markdown
Collaborator Author

codyde commented Jun 3, 2026

Closing as unnecessary after a cross-repo analysis (CLI + mono backboard + dbt-analytics + the live "Railway Agent Adoption" Hex dashboard).

Finding: the premise — that cli_create_up having a null caller blocks agent deploy attribution — doesn't hold given how the warehouse models it:

  • Every railway up already emits a fully-attributed cli_up event via the commands! macro (caller/agent_session_id/session_id all populated).
  • dim_agent_session.did_deploy keys on command in ('up','create_up','deploy','redeploy') — i.e. the attributed cli_up event — not on cli_create_up.
  • Session caller is array_agg(caller_class ignore nulls ...), so cli_create_up's null caller is discarded and can't mis-cohort a session.
  • No dbt model or Hex cell reads cli_create_up's caller/session/agent_session_id. The deploy metrics are session/user-grained off the marts.

So the attribution this PR set out to unlock already works. Closing the producer half; the merged backboard half (#30564) is being addressed separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants