[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-06-25

[github-actions[bot]]

🤖 Copilot Agent Session Analysis — 2026-06-25

Executive Summary

• Sessions Analyzed: 50 (most recent agent/CI runs on github/gh-aw)
• Analysis Period: 2026-06-25 06:47–07:22Z (36-min burst)
• Completion Rate: 8% (4 success / 46 action_required / 0 failure) — +6pts vs 06-24 (2%)
• Average Duration: 1.50 min (46/50 zero-duration gate runs; the 4 substantive runs averaged 18.8 min)
• Experimental Strategy: None (probability roll = 68/100, standard analysis)
• Data Quality: ⚠️ Metadata-only — conversation transcripts unavailable (OAuth gap, 32nd+ consecutive day)

Key Metrics

Metric	Value	Trend
Total Sessions	50	→
Successful Completions	4 (8%)	↑ (vs 2% on 06-24)
Failed/Abandoned	46 action_required (92%), 0 failures	↓ improving
Average Duration	1.50 min	↑
Substantive (non-zero) Sessions	4 (8%)	↑
Loop Detection Rate	0 detectable (no transcripts)	→
Context Issues	N/A (metadata-only)	→
Orphaned Branches	0 / 6 active (0%)	→ NORMAL

📈 Session Trends Analysis

Completion Patterns

The 30-day completion curve continues its saw-tooth oscillation — sharp single-day spikes (peaks of 38–40% in early June) collapsing back to a 0–8% gate-sweep floor. Today's 8% is a modest recovery off yesterday's 2% trough, but the recent 7-day average (~9%) remains well below the early-June highs, confirming a softening trend. Zero failures today is the one unambiguously positive signal.

Duration & Efficiency

Duration stays strictly bimodal: a median of 0 min (dominated by instantaneous action_required gate sweeps) against a small cluster of 14–30 min substantive agent runs. The non-zero-session bars track completion almost exactly — every minute of real work today came from the same 4 runs that succeeded, reinforcing that "duration > 0" is a near-perfect proxy for "did real agent work happen."

Success Factors ✅

1. Agentic PR-comment runs (Addressing comment on PR): 4/4 = 100% success rate.
   • All four successes were Copilot cloud-agent runs responding to PR review comments (fix: stop codex harness retry loop draining tokens on exhausted rate-limit reconnects #41385, fix: use union of caller + worker permissions for call-workflow jobs #41387, refactor: consolidate triplicate merge helpers and add sliceutil.SortedKeys #41388, Remove non-strict override from AOAI API key smoke workflow and restore Copilot BYOK env passthrough #41401), one per distinct copilot/* branch.
   • Durations 14.6 / 14.6 / 16.2 / 29.8 min — substantive, non-trivial work.
2. Provenance inversion (recurring pattern): success is determined by who triggered the run, not the task text.
   • Cloud-agent / PR-comment runs convert; CI gate sweeps (Smoke CI, Agentic Commands, Q, Doc Build) never do — they return action_required by design.
3. Copilot assignment ⇒ no orphaning: all 6 open PRs are Copilot-assigned, sustaining a ~35-day streak of 0% orphan rate.

Failure Signals ⚠️

1. Gate-sweep dominance: 46/50 (92%) runs are zero-duration action_required — CI gates queued behind required approvals, not genuine agent failures. Inflates the apparent "failure" count.
2. Branch concentration: top-2 branches (retry-loop-drained-tokens-2 42%, remove-strict-false-and-fix-env-support 30%) account for 72% of all runs — a single branch's CI re-fires can dominate the daily picture.
3. Persistent observability gap: conversation transcripts have been unavailable for 32+ consecutive days (OAuth re-auth needed). True behavioral analysis (loops, reasoning, recovery) remains impossible.

Prompt Quality Analysis 📝

Conversation transcripts are unavailable (OAuth gap, 32nd+ day), so prompt-text quality cannot be scored this run. Inference from run metadata only:

• High-reliability task class: Addressing comment on PR #NNNN — scoped, contextual (a specific PR thread), explicit acceptance signal (the reviewer's comment). 100% success across the 4 observed.
• Zero-conversion task class: review/CI gate workflows (Q, Agentic Commands, Smoke CI, Doc Build) — action_required by design; these are gates, not prompts, and should not be read as prompt-quality failures.

No prompt-text examples can be shown or sanitized without transcripts.

Orphaned Branch Escalation Alerts 🚨

Branches with ≥5 simultaneous gate firings and no Copilot agent assigned for >2 hours.

Summary

• Orphaned Branches Today: 0 out of 6 active branches (0%)
• Historical Baseline: ~40% orphaned rate
• Status: ✅ NORMAL (well below the 50% elevated-waste threshold; ~35th consecutive healthy day)

Escalation Candidate Details

✅ No orphaned branches exceed the escalation threshold today.

Branch	PR	Gate Count	Wait Time	Severity	Recommended Action
(none)	—	—	—	—	—

All 6 open PRs (#41401, #41388, #41387, #41385, #41358, #41295) carry a Copilot assignee, and the only in-progress workflow runs at scan time were on main (3 runs) — no copilot/* branch had an active gate sweep waiting on an unassigned agent.

CI Waste Estimate

• Orphaned gate-hours today: 0 (no unassigned branches with active gates)
• Recoverable capacity: 0% — nothing to recover; assignment discipline is healthy.

Notable Observations — Diagnostics

Loop Detection

• Sessions with detectable loops: 0 (cannot be measured without transcripts; metadata shows no obvious retry storms)
• Substantive runs: 4, all single-pass completions

Tool Usage

• Tool-level usage is not observable without transcripts. Run-level workflow mix: Smoke CI (7), Agentic Commands (9), Q (9), Doc Build - Deploy (4), CGO (4), CWI (4), CJS (3), plus PR-comment cloud-agent runs.

Context Issues

• Not measurable this run (metadata-only). No clarification-request signals available.

Workflow Mix

• 6 branches, all copilot/*; top-2 = 72% of runs; 36-minute clustered burst (06:47–07:22Z).

Experimental Analysis

Standard analysis only — no experimental strategy this run (probability roll = 68 ≥ 30 threshold).

Actionable Recommendations

For Users Writing Task Descriptions

1. Anchor work to a specific PR thread when possible — the Addressing comment on PR #NNNN pattern converted 4/4 today and is the most reliable observed task class. Tie requests to a concrete PR + reviewer comment rather than free-floating instructions.
2. Expect gate runs to read as action_required — these are required-approval CI gates, not failures; don't interpret the 92% action_required rate as agent quality.

For System Improvements

1. Restore conversation-log access (HIGH impact): the OAuth/transcript gap is now 32+ days — the single largest blind spot. Until fixed, loop detection, reasoning quality, and context-confusion metrics are all unmeasurable.
2. De-noise the success metric (MEDIUM impact): separate "agent task outcomes" from "CI gate states" in the dashboard so the 8% completion figure reflects real agent work (4/4 = 100% of substantive runs succeeded) rather than gate-sweep dilution.

For Tool Development

1. Per-branch gate refire tracking (MEDIUM): with top-2 branches at 72%, a "refire ratio" (runs ÷ distinct workflows) per branch would distinguish broad CI activity from one branch re-firing the same gates — useful for spotting waste hotspots. Need: ~6 branches/day.

Historical Trends & Statistical Summary

Trends Over Time

• Completion rate: saw-tooth; recent 7d (06-19→25): 6, 0, 24, 4, 20, 2, 8 → avg ~9.1%, below early-June 38–40% peaks.
• Duration: persistently bimodal; substantive cluster 14–30 min, everything else ~0.
• Orphan rate: 0% sustained ~35 days vs 40% historical baseline — durable improvement in assignment discipline.

Statistical Summary

Total Sessions Analyzed:     50
Successful Completions:       4 (8%)
Action-required (gates):     46 (92%)
Failures:                     0 (0%)

Average Session Duration:    1.50 min
Median Session Duration:     0.00 min
Longest Session:            29.80 min
Shortest Session:            0.00 min
Substantive (non-zero):       4 sessions (8%)

Loop Detection:               0 (unmeasurable — no transcripts)
Context Issues:               N/A (metadata-only)
Branches:                     6 (all copilot/*), top-2 = 72%
Orphaned Branches:            0 / 6 (0%, NORMAL)
Conversation-log gap:        32+ consecutive days

Next Steps

• Restore Copilot conversation-log / OAuth access — unblocks true behavioral analysis (32+ day gap)
• Split agent-task outcomes from CI-gate states in reporting to de-dilute the completion metric
• Continue monitoring saw-tooth recovery; watch whether 7-day average climbs back toward double digits

References:

• §28153132249 — PR fix: use union of caller + worker permissions for call-workflow jobs #41387 comment run (success, 14.6m)
• §28153086222 — PR Remove non-strict override from AOAI API key smoke workflow and restore Copilot BYOK env passthrough #41401 comment run (success, 16.2m)
• §28152233592 — PR fix: stop codex harness retry loop draining tokens on exhausted rate-limit reconnects #41385 comment run (success, 29.8m)

Generated by 📊 Copilot Session Insights · 258 AIC · ⌖ 37.2 AIC · ⊞ 18.6K · ◷

• expires on Jun 26, 2026, 12:26 AM UTC-08:00