💡 Agent Session Telemetry Dashboard for Fleet-Wide Runtime Analytics

[github-actions[bot]]

Summary

Build a weekly agent session analytics report that aggregates runtime telemetry across the fleet: per-tier stage durations (p50/p95), timeout prevalence over total runs, model-chain fallback activation rates, engine fallback paths taken, and success/failure rates by tier and workflow. This bridges the gap between token cost reporting ("what does it cost?") and fleet health monitoring ("is anything broken?") with a third question: "how well is the agent fleet performing?"

Market Signal

• Gartner warns over 40% of agentic AI projects will be canceled by 2027 due to monitoring gaps (Microsoft Azure blog, June 23, 2026)
• Arize 2026 observability guide emphasizes agentic systems "fail in ways that look like success" — incorrect but well-formed outputs, unnecessary tool calls — and that fleet-level dashboards aggregating latency, error frequency, and token usage per task are essential
• MLflow 2026 monitoring guide highlights trace-based debugging as the primary mechanism for agentic systems — per-session traces rather than per-call logs
• FutureAGI canary research shows stratified health signals (latency, timeout rate, fallback activation, AND quality score) are essential for fleet promotion decisions
• Azure Copilot Observability Agent (GA June 23, 2026) correlates signals across agents, applications, and infrastructure — validating the multi-signal fleet analytics pattern

User Signal

• Issue [dev-lead timeout follow-up 3/3] Report timeout wall-rate as prevalence (over total dev-lead runs) #1030 explicitly requests "report timeout wall-rate as prevalence (over total dev-lead runs)"
• Issue dev-lead emits empty, model-less token records that pollute the "no price" count in the cost report #1009 reports empty, model-less token records polluting the cost report — symptom of missing session-level telemetry
• The timeout debugging effort ([dev-lead timeout A/3] Maximize runtime per stage (raise job + per-tier budgets + guardrail) #1017–feat: implement issue #1028 — [dev-lead timeout follow-up 1/3] Don't retry exit-124 at the engine level (in-run same-budget retry) #1033) required manual log analysis across multiple sessions — a session analytics dashboard would have surfaced the timeout pattern earlier
• TOKEN_LOG_FILE in engine.sh already captures per-call JSONL records but they are consumed only by the cost report, not analyzed for runtime performance patterns
• The TRIAGE_TIMEOUT_SEC / DEEP_TIMEOUT_SEC / AUDIT_TIMEOUT_SEC / ACTION_TIMEOUT_SEC / DUCK_TIMEOUT_SEC constants provide reference points for actual vs. budgeted stage duration analysis

Technical Opportunity

The raw data and infrastructure exist:

• engine.sh writes per-call JSONL records via TOKEN_LOG_FILE (when enabled) — extend with stage timing fields
• fleet_report.sh collects workflow-level pass/fail data — source for success/failure rates
• token_report.sh aggregates cost data — source for model usage patterns
• The session analytics report would aggregate along a new axis: per-session stage performance rather than per-model cost or per-workflow health
• The timeout constants provide the "budget" reference for each stage — compare actual duration vs. budget to measure headroom and identify stages running hot

Recommended telemetry schema extension:

{
  "session_id": "run-12345",
  "stage": "DEEP",
  "model": "claude-sonnet-4-6",
  "model_chain_position": 1,
  "fallback_from": null,
  "duration_sec": 847,
  "budget_sec": 2400,
  "exit_code": 0,
  "timestamp": "2026-07-03T10:15:00Z"
}

Assessment

Dimension	Score	Rationale
Feasibility	med	Raw data exists but needs schema extension + new aggregation script. Follows token_report.sh pattern.
Impact	high	Surfaces timeout prevalence, fallback frequency, and stage-duration regression — the exact questions the timeout debugging effort needed answered.
Urgency	med	The timeout handling landed (#1017–#1033), but the reporting ask (#1030) is still open and the aggregation gap remains.

Adversarial Review

Strongest objection: The project already has token_report.sh and fleet_report.sh. Adding a third reporting workflow creates more maintenance burden and CI cost without clear decision-support value.

Rebuttal: The existing reports answer different questions: fleet_report.sh = "is anything broken?"; token_report.sh = "what does it cost?" Neither answers "how well is the fleet performing?" — the per-session runtime analytics that would surface timeout prevalence trends, model-chain fallback frequency, and stage-duration regression. The timeout debugging effort (#1017–#1033) proved this gap is real: it took a multi-PR effort to even measure the problem because the runtime data wasn't aggregated into the right view.

Suggested Next Step

Define the session telemetry schema (stage, model, duration, exit_code, fallback_path), extend the TOKEN_LOG_FILE JSONL format to include stage timing data, and prototype a weekly aggregation script following the token_report.sh pattern. Start with the dev-lead workflow as the pilot consumer.