[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-05-23

[github-actions[bot]]

Executive Summary

• Sessions Analyzed: 50
• Analysis Period: 2026-05-23 (02:12 UTC → 07:38 UTC)
• Completion Rate: 44% (success-conclusion / total)
• Average Duration: 8.5 min (median 5.4 min)
• Experimental Strategy: none (standard analysis only)
• Regime shift: First day in the multi-day window where success-conclusion exceeds action_required. The 4-day run of action_required dominance (05-19 → 05-22, 86–98%) ended today.
• Data quality: infrastructure-only for the 15th consecutive run — conversation transcripts directory remained empty, so behavioral analysis falls back to metadata. Analyses below are derived from session conclusion, duration, branch, and workflow-name signals.

Key Metrics

Metric	Value	Trend
Total Sessions	50	→
Successful Completions	22 (44%)	↑↑
Failed	8 (16%)	↑
Action Required	14 (28%)	↓↓
Cancelled	4 (8%)	↑
Skipped	2 (4%)	↑
Average Duration	8.5 min	↑↑
Median Duration	5.4 min	↑
Sessions ≥20 min (loop proxy)	9 (18%)	↑↑
Open PRs in repo	1	↓

📈 Session Trends Analysis

Completion Patterns

Today's chart shows the sharpest single-day recovery in successful completions across the 4-day window: from 0–6 successes/day on 05-20 → 05-22, up to 22 on 05-23. The completion-rate trace climbs from 0% → 44%, ending the action_required-dominated regime that held for three consecutive days.

Duration & Efficiency

Session durations also stepped up sharply: average 8.5 min and median 5.4 min vs near-zero medians the preceding 3 days. Sessions ≥20 min (loop proxy) jumped to 9 from 0/1/0 — real agent iteration resumed today, consistent with the conclusiveness lift.

Success Factors ✅

1. Workflow-type diversity in the queue — Today no single workflow dominated. Six workflow types each fired exactly 5 times (Agentic Commands, CGO, Label Closed PRs, PR Description Updater, Q, Smoke CI). On prior days, Agentic Commands + Q held ≥48–64% of the queue; today they hold just 20%. Diverse-queue days correlate with high completion.
2. Smoke CI wins 100% — Smoke CI ran 5 times and succeeded 5 times. Pre-merge CI on the dominant branch was clean.
3. "Addressing comment on PR" pattern is reliable — 4 of 6 such runs succeeded (67%), continuing the trend from 05-18 where tight-scope comment-driven runs all closed in 8–12 min.
4. Concentrated agent work on a single branch produced output — 27/50 sessions on copilot/refactor-semantic-clustering. Unlike the 25-session, 4%-conclusive top branch on 05-21, today's dominant branch generated a mixed-but-real outcome distribution.
5. Wider sampling window catches more conclusive events — sampling covered 5h 26m today vs 19 min on 05-22, 58 min on 05-21. Wider windows include overnight agent activity, which appears to be where conclusive runs actually happen.

Failure Signals ⚠️

1. CGO is the dominant failure cluster — 4 of 5 CGO runs failed (80% failure rate). All on the same dominant branch. Pattern matches the 05-21 single CGO failure on refactor-oversized-functions-parser-workflow — CGO compile is platform-specific and brittle.
2. PR Code Quality Reviewer: 2/2 failed — small sample but 100% failure rate. Worth investigating whether the reviewer is broken globally or only on this branch.
3. High action_required share on Q and Agentic Commands — 5/5 Q runs ended action_required (0% conclusive); 3/5 Agentic Commands ran action_required. Same workflows that dominated the 05-19 → 05-22 stuck regime continue to gate-out.
4. 9 long sessions (≥20 min) — likely iteration loops rather than productive work. Without conversation transcripts we cannot distinguish "slow but successful" from "stuck." 4 of the 9 long sessions ended success or cancelled; the rest were action_required or skipped.

Prompt Quality Analysis 📝

Caveat: conversation transcripts are still unavailable (15th consecutive run). Prompt-quality scoring requires the agent's internal monologue, which we don't have on disk. The signals below are inferred from workflow-name patterns and outcome distributions, not from prompt text.

High-success workflow shapes

• Workflows scoped to a specific PR by number (e.g., "Addressing comment on PR fix: exclude merged upstream commits from diffSize in push_to_pull_request_branch incremental mode #34139/34144/34149"): 4/6 succeeded — tight scope correlates with success.
• Platform CI gates (Smoke CI, Doc Build, Design Decision Gate): 10/10 succeeded — these aren't agent prompts, but they show that the pre-merge gating layer is healthy.

Low-success workflow shapes

• Polling/scheduled gates (Q, Agentic Commands): 8/10 ended action_required — these workflows enter a state that needs human action and stall.
• CGO build (4/5 failed) — likely a platform/build-config issue, not a prompt-quality issue.

Orphaned Branch Escalation Alerts 🚨

Summary

• Orphaned Branches Today: 0 out of 1 active branch (0%)
• Historical Baseline: ~40% orphaned rate
• Status: ✅ NORMAL (16th consecutive day at zero orphan threshold)

Escalation Candidates

✅ No orphaned branches exceed the escalation threshold today.

The repo state is currently very idle — only 1 open PR system-wide (#33219 — "Bind Node toolcache into AWF chroot...") with Copilot assigned. The single in-progress workflow run is this analysis workflow itself on main. The orphan filter cannot fire because there are no candidate branches in the danger band.

CI Waste Estimate

• Orphaned gate-hours today: 0 gate-hours
• Recoverable capacity: nothing to recover today; the system is operating cleanly on the orphan dimension

Notable Observations

Loop Detection (9 long sessions)

• Sessions with duration ≥20 min: 9 (18% of total)
• Branch distribution: 6 of 9 long sessions ran on copilot/refactor-semantic-clustering
• Outcome distribution of long sessions: 4 success, 1 cancelled, 2 skipped, 1 action_required, 1 success-then-stuck pattern
• Longest sessions: Doc Build - Deploy (27.5 min, success), Agentic Commands (25.3 min, success), CGO (25.3 min, cancelled), Smoke CI (25.3 min, success), Q (24.9 min, skipped)
• Without transcripts, cannot confirm whether long sessions reflect productive iteration or stuck loops

Tool / Workflow Usage

• 16 distinct workflow names fired today (highest diversity in recent runs)
• Tied for top firings (5 each): Agentic Commands, CGO, Label Closed PRs, PR Description Updater, Q, Smoke CI
• Real agent workflows that succeeded: "Running Copilot cloud agent" 2/2 success; "Addressing comment on PR Use Copilot BYOK platform default model instead of hard-coded Claude fallback #34149" 2/3 success
• No completely-missing-tool requests observed in the conclusion metadata

Branch Activity

• copilot/refactor-semantic-clustering — 27/50 (54% of all sessions). Hosted the bulk of CGO failures and successes alike.
• copilot/fix-safe-outputs-job-failure — 8/50 (16%)
• copilot/fix-chaos-pr-bundle-fuzzer — 6/50 (12%)
• copilot/fix-patch-base-calculation — 6/50 (12%)
• copilot/review-misconfigured-model — 3/50 (6%)

All 5 active branches are copilot/* — no human-authored branches in today's sample.

Experimental Analysis

This run did NOT include an experimental strategy. Standard analysis only.

For reference, the 05-21 "Inverse Gate-Count to Conclusiveness" experimental strategy was partially invalidated today: the 27-session dominant branch produced a conclusive outcome mix rather than the 4%-conclusive pattern predicted by that model. Recommendation in cache: refine the strategy to condition on workflow-type diversity, not just gate count.

Actionable Recommendations

For Users Writing Task Descriptions

1. Reference specific PR numbers when commenting — "Addressing comment on PR #N"–style scoped runs continue to outperform broad polling workflows. If a task can be expressed as a comment on a specific PR, that scoping helps.
2. Avoid co-occurring CGO + agent runs on the same branch — when CGO repeatedly fails on a branch (4/5 today), the agent's downstream gates compound the noise. Stabilize the build before driving agent iterations.
3. Don't conflate "long session" with "failed session" — today 4 of 9 long sessions succeeded. Without transcripts we can't distinguish, but the heuristic of treating ≥20-min runs as "loops" overcounts failures.

For System Improvements

1. Restore conversation transcript export — 15 consecutive runs without transcripts means behavioral analysis has been blocked since 2026-05-06. Estimated impact: High (unblocks the primary analytical capability of this workflow).
2. Investigate CGO failure cluster on dominant branches — 4 CGO failures today + 1 on 05-21 suggests a recurring issue. Estimated impact: Medium (saves CI minutes and reduces noise in success metrics).
3. Promote net-PR-throughput as headline metric — daily-completion-rate-pct reads 2% on operationally normal days and 44% on this regime-break day. Net PR change (open backlog delta) is more informative. Estimated impact: Medium.

For Tool Development

1. gh auth login OAuth-blocker on transcript fetch — 15 consecutive runs. Use case: enable behavioral analysis of agent reasoning. Frequency of need: every run.
2. Workflow-type-diversity signal — today's data suggests an early-warning metric: when Agentic Commands + Q exceed ~50% of the daily queue, completion-rate falls below 10%. Worth instrumenting as a leading indicator.

Trends Over Time

Using the cached multi-day history (session-analysis-history.json, 15 entries from 05-06 → 05-23):

• Completion-rate trend: 8 → 0 → 2 → (gap) → 8 → 8 → 16 → 0 → 8 → 22 → 2 → 0 → 12 → 2 → 44 — today is the new high, 2.0× the prior peak (22% on 05-18).
• Average-duration trend: agent durations on success runs cluster 10.2–22.3 min historically; today's overall avg of 8.5 min (across all 50 sessions, not just successes) is consistent with that band.
• Orphan-rate trend: 0% for 16 consecutive days vs the 40% baseline in the spec. The orphan filter as currently specified has not fired once in the visible window.

Statistical Summary

Total Sessions Analyzed:     50
Successful Completions:      22 (44.0%)
Failed Sessions:             8 (16.0%)
Action-Required Sessions:    14 (28.0%)
Cancelled Sessions:          4 (8.0%)
Skipped Sessions:            2 (4.0%)

Average Session Duration:    8.54 min
Median Session Duration:     5.38 min
Longest Session:             27.48 min (Doc Build - Deploy, success)
Shortest Session:            0.0 min

Long-session (≥20 min) count: 9 (18%)
Unique Branches:             5 (all copilot/*)
Unique Workflow Names:       16
Top Workflow Tie:            6 workflows × 5 firings each

Open PRs:                    1 (#33219, Copilot assigned)
In-progress Runs:            1 (this workflow on main)
Orphaned Branches:           0
Orphan Status vs Baseline:   16 consecutive days at 0% vs 40% baseline

Data Quality:                infrastructure-only (15 consecutive runs)

Next Steps

• Validate whether the 05-23 regime break is sustained (next-run check)
• Investigate CGO failure cluster on copilot/refactor-semantic-clustering
• Refine the "Inverse Gate-Count to Conclusiveness" strategy to account for today's counter-example
• Continue tracking the 0/40% orphan-rate divergence and consider whether the baseline figure needs updating
• Escalate the 15-run conversation-transcript outage to unblock behavioral analysis

References:

• §26326889667 — current run
• §26275531347 — 05-22 run
• §26213113053 — 05-21 run

Generated by 📊 Copilot Session Insights · ● 13.9M · ◷

• expires on May 24, 2026, 7:47 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Session Insights.

A newer discussion is available at Discussion #34397.