[grafana-otel-advisor] OTel improvement: add gh-aw.detection.conclusion and gh-aw.detection.reason to conclusion spans

[github-actions]

OTel Instrumentation Improvement: Capture Detection Scan Outcome in Conclusion Spans

Analysis Date: 2026-05-05
Priority: High
Effort: Small (< 2h)

Problem

GH_AW_DETECTION_CONCLUSION and GH_AW_DETECTION_REASON are available as job-level environment variables in compiled workflows (injected from needs.detection.outputs.*). These values are consumed in multiple places for user-facing messages, PR labels, and GitHub comments — but they are never emitted as OTLP span attributes. As a result, a DevOps engineer querying Grafana, Honeycomb, or Datadog cannot distinguish between:

• A workflow that failed because the agent couldn't solve the task (agentConclusion=failure, no detection signal)
• A workflow that succeeded but was flagged by threat detection (agentConclusion=success, detectionConclusion=warning, detectionReason=threat_detected)
• A workflow that failed due to a detection engine error (detectionConclusion=failure, detectionReason=agent_failure)

These are categorically different failure modes requiring different remediation — yet in OTel backends today they are indistinguishable.

Why This Matters (DevOps Perspective)

With gh-aw.detection.conclusion and gh-aw.detection.reason as span attributes, on-call engineers can:

• Build a "Threat Detection Flag Rate" panel: count(spans where gh-aw.detection.conclusion="warning") grouped by gh-aw.detection.reason
• Create an alert that fires when threat detection failure rate (detectionReason=agent_failure) spikes — indicating a detection engine regression, not an agent regression
• Quickly triage whether an alert represents a true agent failure or a security scan false positive by filtering on gh-aw.detection.conclusion
• Measure the operational impact of the PR closure reason labeling feature (PR fix: two-checkpoint pre-flight validation + PR closure reason labeling #30253), since those closure reasons are driven by detectionReason

Without these attributes, the entire detection layer is invisible in the OTel backend — every notification job span looks identical regardless of whether threat detection fired.

Current Behavior

In compiled workflows, GH_AW_DETECTION_CONCLUSION and GH_AW_DETECTION_REASON are set as job environment variables:

# From .github/workflows/smoke-claude.lock.yml (lines 2574-2575, 2927-2928)
environment:
  GH_AW_DETECTION_CONCLUSION: ${{ needs.detection.outputs.detection_conclusion }}
  GH_AW_DETECTION_REASON: ${{ needs.detection.outputs.detection_reason }}

These values are read by four modules for user-facing messages but are absent from any span:

• actions/setup/js/generate_footer.cjs — generates PR closure caution alerts
• actions/setup/js/notify_comment_error.cjs — determines comment tone and content
• actions/setup/js/messages_footer.cjs — generates detection warning footers
• actions/setup/js/handle_detection_runs.cjs — routes detection outcomes

But actions/setup/js/send_otlp_span.cjs reads GH_AW_AGENT_CONCLUSION (line 1097) and never reads GH_AW_DETECTION_CONCLUSION or GH_AW_DETECTION_REASON:

// Current: actions/setup/js/send_otlp_span.cjs (lines 1095-1151)
// Agent conclusion is passed to downstream jobs via GH_AW_AGENT_CONCLUSION.
const agentConclusion = process.env.GH_AW_AGENT_CONCLUSION || "";
// ...
if (agentConclusion) {
  attributes.push(buildAttr("gh-aw.agent.conclusion", agentConclusion));
}
// GH_AW_DETECTION_CONCLUSION and GH_AW_DETECTION_REASON are never read here

Proposed Change

In sendJobConclusionSpan, after the agentConclusion block (around line 1151 in send_otlp_span.cjs), add:

// Detection scan outcome (security scanning / threat detection).
// Set as job env vars in compiled workflows from needs.detection.outputs.*
const detectionConclusion = process.env.GH_AW_DETECTION_CONCLUSION || "";
const detectionReason = process.env.GH_AW_DETECTION_REASON || "";
if (detectionConclusion) {
  attributes.push(buildAttr("gh-aw.detection.conclusion", detectionConclusion));
}
if (detectionReason) {
  attributes.push(buildAttr("gh-aw.detection.reason", detectionReason));
}

Also update the JSDoc environment variables list for sendJobConclusionSpan (~line 1010) to document the two new inputs.

Expected Outcome

After this change:

• In Grafana / Honeycomb / Datadog: Filter conclusion spans by gh-aw.detection.conclusion (values: warning, failure, success) and gh-aw.detection.reason (values: threat_detected, agent_failure, parse_error). Build panels for detection flag rate by reason.
• In the JSONL mirror (/tmp/gh-aw/otel.jsonl): Detection outcome visible in the artifact without a live collector, enabling offline post-mortem debugging.
• For on-call engineers: A span with gh-aw.detection.conclusion=warning and gh-aw.detection.reason=threat_detected immediately signals "security review needed" rather than "agent regression" — reducing MTTR by eliminating the need to cross-reference GitHub Actions logs.

Implementation Steps

• Add GH_AW_DETECTION_CONCLUSION and GH_AW_DETECTION_REASON to the JSDoc environment-variables list in sendJobConclusionSpan (~line 1010 in send_otlp_span.cjs)
• Add the two buildAttr calls after the agentConclusion block (~line 1151)
• Add a test case in send_otlp_span.test.cjs (in the sendJobConclusionSpan describe block, line 2051+) asserting gh-aw.detection.conclusion and gh-aw.detection.reason appear in attributes when the env vars are set
• Run cd actions/setup/js && npx vitest run to confirm tests pass
• Run make fmt to ensure formatting
• Open a PR referencing this issue

Evidence from Live Grafana Data

Grafana MCP tools were unavailable in this session (tool list returned empty). Evidence is based on static code analysis:

Code gap confirmed: GH_AW_DETECTION_CONCLUSION and GH_AW_DETECTION_REASON appear in send_otlp_span.cjs zero times (verified via grep), while GH_AW_AGENT_CONCLUSION appears 4 times and produces the gh-aw.agent.conclusion span attribute.

Propagation confirmed: Both env vars are set as job-level environment variables in at least 5 compiled workflow lock files (.github/workflows/*.lock.yml), confirming they are reliably available when the conclusion post-action runs.

Expected live span gap: Any Grafana query for gh-aw.detection.conclusion returns zero results today, even for runs where threat detection fired, because the attribute is never emitted.

Related Files

• actions/setup/js/send_otlp_span.cjs — primary change location (sendJobConclusionSpan, ~line 1151)
• actions/setup/js/send_otlp_span.test.cjs — test file (add new test case in sendJobConclusionSpan describe block, line 2051+)
• actions/setup/js/generate_footer.cjs — already reads GH_AW_DETECTION_REASON (reference for possible values: threat_detected, agent_failure, parse_error)
• actions/setup/js/notify_comment_error.cjs — already reads GH_AW_DETECTION_CONCLUSION (reference for possible values: warning, failure, success)

---

Generated by the Daily Grafana OTel Instrumentation Advisor workflow

Generated by Daily Grafana OTel Instrumentation Advisor · ● 391.9K · ◷

• expires on May 12, 2026, 4:40 AM UTC