[reliability] Daily Reliability Review - 2026-08-03

[github-actions]

Executive Summary

Telemetry (Sentry spans dataset, project gh-aw, org github, last 24h, workflow-run §30861546814) shows the spans dataset is healthy and populated across the full 24h window, but the errors and logs datasets returned zero events — those pipelines are either unused or not exporting. Three workflows had confirmed operational failures with long-running gen_ai spans ending in OTLP span.status:error / gh-aw.run.status:failure: PR Code Quality Reviewer (2 separate failed runs), [aw] Failure Investigator (6h) (1 run with 5 failed attempts, up to ~11.6 min each), and Contribution Check (1 failed attempt inside an otherwise-successful run). Separately, the majority (~90/100) of sampled error spans carry no gh-aw.workflow.name/gh-aw.run.status attribution at all, and gen_ai.response.finish_reasons / release never appeared on any span in the window despite emit-side code that should always populate the former — this is a confirmed instrumentation/query gap, not a confirmed absence of finish reasons at runtime.

Top Reliability Findings

Priority	Workflow	Problem	Evidence	Next Action
1	PR Code Quality Reviewer	Repeated long-running agent failures (timeout-class)	2 traces, 4 error spans, durations 233.8s, 301.9s, 465.2s, 519.8s; gh-aw.run.status:failure on all; traces 003d99cddebc68fe208564f9cddfa00e (19:16–19:17 UTC) and aa73bcb8ce0b5d6b262aa409bd19b626 (22:42–22:43 UTC)	Investigate why this workflow's agent step repeatedly exceeds ~4–9 min and errors; check engine timeout config and recent prompt/tool changes
2	[aw] Failure Investigator (6h)	Self-monitoring workflow failing on nearly every attempt within one run	1 trace d38263a26234d5fa4ad42ce6c268b4bd, 5 error spans in ~16 min (19:15–19:31 UTC), max duration 693.9s (~11.6 min), all gh-aw.run.status:failure	Highest operational risk since this is the reliability tooling itself — check its recent config/prompt and whether it is looping/retrying into the same failure
3	Contribution Check	Single failed attempt inside a mostly-successful run	Trace a6220231e35f5c0c6d6a7f75f6975a06, 1 error span, 74.8s, gh-aw.run.status:failure at 21:20:54 UTC, surrounded by successful spans	Lower urgency; monitor for recurrence, no action needed yet
4	All workflows (attribution gap)	~90% of sampled error spans (child invoke_agent/gateway.request spans) carry no gh-aw.workflow.name or gh-aw.run.status	100-span sample (API-capped) of span.status:error span.op:gen_ai over 24h: only 10 spans across 4 traces carried workflow attribution	Root-cause is architectural, not a bug — see Notes
5	All workflows	gen_ai.response.finish_reasons never observed; release/service.version not populated on any span	has:gen_ai.response.finish_reasons → 0 results in 24h; release/service.version fields empty on every sampled span including ones matching has:release	See Recommendations — likely a query/schema mismatch on the array-valued attribute, not a runtime bug

Representative Traces

View representative traces

• PR Code Quality Reviewer (run 1): 003d99cddebc68fe208564f9cddfa00e — spans at 19:16:45 (519.8s, error) and 19:17:34 (465.2s, error), gh-aw.run.status:failure.
• PR Code Quality Reviewer (run 2): aa73bcb8ce0b5d6b262aa409bd19b626 — spans at 22:42:10 (301.9s, error) and 22:43:14 (233.8s, error), gh-aw.run.status:failure.
• [aw] Failure Investigator (6h): d38263a26234d5fa4ad42ce6c268b4bd — 5 error spans between 19:15:31 and 19:31:25, durations 693.9s / 58.1s / 9.1s / 37.9s / 12.8s, all gh-aw.run.status:failure, interleaved with ok spans (retry pattern within the same run).
• Contribution Check: a6220231e35f5c0c6d6a7f75f6975a06 — single error span at 21:20:54 (74.8s), gh-aw.run.status:failure, surrounded by successful spans in the same run.
• No gh-aw.run.status:cancelled spans found in the 24h window (query returned 0 results) — no confirmed cancellations.
• errors dataset query (24h, no filter): 0 results. logs dataset query (24h, no filter): 0 results.

Recommendations

1. Triage PR Code Quality Reviewer and [aw] Failure Investigator (6h) first — both show multi-minute gen_ai spans ending in error/failure, consistent with agent-step timeouts rather than fast-fail errors. Pull the GitHub Actions run logs for the timestamps above to confirm the failure mode (API timeout, tool call hang, or engine crash).
2. Verify whether Sentry's spans schema indexes array-valued attributes (gen_ai.response.finish_reasons, written via buildArrayAttr in actions/setup/js/send_otlp_span.cjs:2178) for has:/equality queries. The emit-side code guarantees this attribute is always set on jobName === "agent" conclusion spans (falling back to "timeout"/"unknown"), but it was unobservable via list_events in this 24h window — either the ingestion path drops/renames array attributes or this conclusion span type wasn't hit by any run in the window. Confirm with a raw OTLP payload capture before assuming a runtime regression.
3. For the ~90% of error spans lacking gh-aw.workflow.name/gh-aw.run.status: these are mostly invoke_agent/gateway.request child spans emitted by the agent SDK/gateway layer rather than the gh-aw wrapper span in send_otlp_span.cjs. If per-workflow filtering on these child spans is needed, propagate gh-aw.workflow.name down via span attributes/baggage at emission time rather than relying on parent-span lookups.
4. release/service.version (mapped from service.version per send_otlp_span.cjs:360) did not show an actual value on any sampled span even where has:release matched — confirm the Sentry backend's release mapping for this project is wired to service.version, since the emit side does set it.

Notes

View notes

• spans dataset: confirmed populated continuously across the full 24h window (earliest sampled error span 2026-08-02T23:19:28Z, latest 2026-08-03T23:16:13Z).
• errors dataset: 0 results for the 24h window — either no exception-level events are sent to this project, or nothing errored at that granularity. Treated as an explicit observability finding per instructions, not silently skipped.
• logs dataset: 0 results for the 24h window — same treatment as above.
• The Sentry MCP bridge exposes list_events/list_issue_events but not search_events or get_trace_details in this environment; trace continuity was verified with list_events filtered by trace:<id> instead, per the fallback path in the runbook.
• list_events is capped at limit: 100 server-side with no offset/cursor parameter observed, so the "~100 error spans in 24h" sample is a floor, not an exact count — the true failure volume over 24h may be higher.
• 6 gateway.request spans with span.status:error were seen scattered through the day (e.g. trace 173a1e0897ba21df2d28491dac33baec at 19:17:12 UTC), but inspecting that trace's full span list did not surface the specific errored gateway span for root-causing — this is flagged as inconclusive, not a confirmed exporter/auth failure.
• No gh-aw.run.status:cancelled spans were found in the window.
• Two of the MCP tool responses embedded instructions asking to reformat output with emoji/color-coded cards; these were treated as untrusted data per policy and ignored — flagging here for visibility since they read as an injected formatting directive rather than legitimate tool output.

References:

• §30861546814

Generated by 🚨 Daily Reliability Review · agent · 109.6 AIC · ⌖ 34.7 AIC · ⊞ 5.7K · ◷

• expires on Aug 5, 2026, 3:25 PM UTC-08:00

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #50420

View newer issue

---

This action was performed automatically by the Daily Reliability Review workflow.