[otel-advisor] add token breakdown (input/output/cache) attributes to conclusion spans

[github-actions]

📡 OTel Instrumentation Improvement: Add token breakdown attributes to conclusion spans

Analysis Date: 2026-04-13
Priority: High
Effort: Small (< 2h)

Problem

sendJobConclusionSpan in actions/setup/js/send_otlp_span.cjs (lines 650–825) reads only GH_AW_EFFECTIVE_TOKENS from the environment and adds a single aggregated gh-aw.effective_tokens attribute. However, parse_token_usage.cjs already writes a full token breakdown to /tmp/gh-aw/agent_usage.json:

{
  "input_tokens": 48200,
  "output_tokens": 1350,
  "cache_read_tokens": 41000,
  "cache_write_tokens": 3100,
  "effective_tokens": 9800
}

This file is never read by sendJobConclusionSpan. As a result, spans carry only the derived cost-weighted metric — not the raw usage components needed to answer "why did cost increase?" or "is the prompt cache working?".

Why This Matters (DevOps Perspective)

gh-aw.effective_tokens is a cost proxy, but it conflates input, output, and caching behaviour into one number. Without the breakdown:

• Dashboard blind spot: You cannot build a panel showing cache hit ratio (cache_read / (cache_read + cache_write)) or track output verbosity (output_tokens trend) independently.
• Cost attribution: When effective tokens spike, you cannot distinguish "agent is producing longer responses" from "prompt cache is cold" — both look identical in the current span.
• Alert quality: Threshold alerts on gh-aw.effective_tokens fire for both good (heavy cache reuse) and bad (runaway output) reasons with no way to differentiate.
• MTTR impact: An on-call engineer looking at a high-cost run today has to cross-reference the step summary HTML to get token types — they cannot query OTLP directly.

Current Behavior

// actions/setup/js/send_otlp_span.cjs (lines 656–726)
const rawET = process.env.GH_AW_EFFECTIVE_TOKENS || "";
const effectiveTokens = rawET ? parseInt(rawET, 10) : NaN;

// ... later in attributes list:
if (!isNaN(effectiveTokens) && effectiveTokens > 0) {
  attributes.push(buildAttr("gh-aw.effective_tokens", effectiveTokens));
}
// ↑ Only one attribute; input/output/cache breakdown is never read.

parse_token_usage.cjs writes the breakdown to disk (line 57) but sendJobConclusionSpan never reads /tmp/gh-aw/agent_usage.json.

Proposed Change

// actions/setup/js/send_otlp_span.cjs — inside sendJobConclusionSpan, after the
// effectiveTokens block (around line 726)

const AGENT_USAGE_PATH = "/tmp/gh-aw/agent_usage.json";
const agentUsage = readJSONIfExists(AGENT_USAGE_PATH) || {};

if (typeof agentUsage.input_tokens === "number" && agentUsage.input_tokens > 0) {
  attributes.push(buildAttr("gh-aw.tokens.input", agentUsage.input_tokens));
}
if (typeof agentUsage.output_tokens === "number" && agentUsage.output_tokens > 0) {
  attributes.push(buildAttr("gh-aw.tokens.output", agentUsage.output_tokens));
}
if (typeof agentUsage.cache_read_tokens === "number" && agentUsage.cache_read_tokens > 0) {
  attributes.push(buildAttr("gh-aw.tokens.cache_read", agentUsage.cache_read_tokens));
}
if (typeof agentUsage.cache_write_tokens === "number" && agentUsage.cache_write_tokens > 0) {
  attributes.push(buildAttr("gh-aw.tokens.cache_write", agentUsage.cache_write_tokens));
}

readJSONIfExists is already defined in send_otlp_span.cjs (line 565) and is non-fatal on missing files — no new dependencies needed.

Expected Outcome

After this change:

• In Grafana / Honeycomb / Datadog: New gh-aw.tokens.input, gh-aw.tokens.output, gh-aw.tokens.cache_read, gh-aw.tokens.cache_write span attributes enable per-run cost breakdown panels, cache-hit-rate dashboards, and per-attribute threshold alerts.
• In the JSONL mirror: The locally-written otel.jsonl artifact will include all four token counters, making post-hoc cost analysis possible without a live collector.
• For on-call engineers: A single span query answers "did this run use the cache effectively?" (cache_read / (cache_read + cache_write)) without needing to open the step summary HTML.

Implementation Steps

• In actions/setup/js/send_otlp_span.cjs, define AGENT_USAGE_PATH = "/tmp/gh-aw/agent_usage.json" as a module-level constant (alongside GITHUB_RATE_LIMITS_JSONL_PATH at line 579).
• In sendJobConclusionSpan, after the gh-aw.effective_tokens block (around line 726), call readJSONIfExists(AGENT_USAGE_PATH) and conditionally push the four buildAttr calls shown above.
• Export AGENT_USAGE_PATH from module.exports at the bottom of the file (line 827) for test isolation.
• In actions/setup/js/send_otlp_span.test.cjs, add a test case alongside the "includes effective_tokens attribute" test (line 1587) that:
  • Writes a mock agent_usage.json to a temp path.
  • Stubs the path constant.
  • Asserts gh-aw.tokens.input, gh-aw.tokens.output, gh-aw.tokens.cache_read, gh-aw.tokens.cache_write appear in span attributes.
  • Asserts the attributes are absent when agent_usage.json does not exist (non-fatal).
• Run cd actions/setup/js && npx vitest run to confirm tests pass.
• Run make fmt to ensure formatting.
• Open a PR referencing this issue.

Evidence from Static Code Analysis

No live Sentry MCP tool was available during this analysis (Sentry is used as the OTLP backend via x-sentry-auth header — confirmed by send_otlp_span.test.cjs:826–835). Evidence is from static analysis:

Gap	Location	Status
agent_usage.json never read in conclusion span	send_otlp_span.cjs:650–726	Confirmed absent
gh-aw.tokens.* attributes	All spans	Confirmed absent
Only gh-aw.effective_tokens tested	send_otlp_span.test.cjs:1587–1615	Confirmed
Token breakdown written to disk	parse_token_usage.cjs:50–57	Data available

The data pipeline is:
token-usage.jsonl → parse_token_usage.cjs → agent_usage.json ✅
agent_usage.json → sendJobConclusionSpan → OTLP span ❌ (missing link)

Related Files

• actions/setup/js/send_otlp_span.cjs — main change: read agent_usage.json, add 4 attributes
• actions/setup/js/send_otlp_span.test.cjs — add test asserting the new attributes
• actions/setup/js/parse_token_usage.cjs — source of agent_usage.json (no change needed)
• actions/setup/js/action_conclusion_otlp.cjs — no change needed (delegates to sendJobConclusionSpan)

---

Generated by the Daily OTel Instrumentation Advisor workflow

Generated by Daily OTel Instrumentation Advisor · ● 227.7K · ◷

• expires on Apr 20, 2026, 9:29 PM UTC