[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-11-05

[github-actions[bot]]

🤖 Copilot Agent Session Analysis — 2026-11-05

Executive Summary

• Sessions Analyzed: 50 pre-fetched runs (06:26–06:42Z, 15.5 min) plus 94 coding-agent sessions from the last 24h
• Analysis Period: 2026-11-04 07:00Z → 2026-11-05 07:00Z
• Completion Rate: 64.0% raw (32/50); 100% once the 18 skipped Squad routes are excluded; 0 failures
• Average Duration: 1.8 min for the 50-run sample (gate-heavy). Actual coding-agent sessions average ~27 min
• Experimental Strategy: none (roll = 66, standard run)

Headline: The pre-fetched sample again holds almost no coding-agent work. 36 of the 50 runs (72%) are routing runs fired by the agent's own review-thread replies. Transcript recovery regressed to 0/50: the job-log endpoint now returns a DIFC policy 403 from inside the sandbox, so this report is based on metadata. To compensate, it adds the 24h coding-agent population (46 cloud-agent and 48 addressing-comment sessions).

Key Metrics

Metric	Value	Trend (vs 11-04)
Total Sessions (sample)	50	→
Successful Completions	32 (64.0%)	↓ from 39 (78.0%)
Failed/Abandoned	0 failed, 18 skipped (36.0%)	→ 0 failures again
Average Duration (sample)	1.8 min (median 0.7)	↓ from 5.0 min
Loop Detection Rate	not measurable (0 transcripts)	—
Context Issues	not measurable (0 transcripts)	—
Coding-agent sessions, 24h	94 (86 success, 91.5%)	↑ from 9
Cloud-agent PRs, 24h	26 merged · 15 open · 3 closed unmerged	↑

📈 Session Trends Analysis

Completion Patterns

Completion has stayed above the 30-day mean (49.9%) for 7 straight recorded days (64–90%), after a volatile October with a low of 4% on Oct 24. Today's 64% matches Nov 2–3. All 18 non-successes are skipped Squad routes, not failures. The real failure count has been 0 for two days running.

Duration & Efficiency

The sample's average fell to 1.8 min because 36 of its runs are 30–60 s routing jobs. That reflects the sample's composition, not faster agents. The 24h coding-agent population tells a different story: a ~27 min mean, and setup still accounts for 73–87% of short code-scanning agent runs. Loops have only been measurable on days with transcripts and were not measurable today.

Success Factors ✅

1. Scoped, single-concern fixes merge: 26 of 45 cloud-agent PRs from the last 24h are already merged (58%). Sampled merged titles each describe a single behavior change.
   • Success rate: 26/45 merged; 44/46 cloud-agent sessions finished successfully (95.7%)
   • Example: "Restore max-ai-credits: -1 budget disabling", "Prevent import read failures from masquerading as stale locks"
2. Review replies that cite the fixing commit: all 18 of the agent's review-thread replies today named the commit (b8686a6, 36d7525) and the regression coverage added. Reviewers can verify each reply in one hop.
   • Success rate: 18/18 replies on Warn when dynamic checkout refs expose host setup steps to untrusted code #74951 and Support deployment controls on workflow environments #74938 traced to a pushed commit
   • Example: "The warning now checks merged PreAgentSteps as well as CustomSteps... Fixed in b8686a6."
3. Warm dependency setup keeps agent time productive: the addressing-comment run on Prevent import read failures from masquerading as stale locks #74935 spent 82% of its 21.4 min in Processing Request. Dev-deps install took 27 s and recompile 93 s.
   • Setup share: 18.1%, compared with 73–87% for code-scanning agent runs
4. Merge-from-main before fixing: Warn when dynamic checkout refs expose host setup steps to untrusted code #74951, Support deployment controls on workflow environments #74938 and Add label-scoped title deduplication for create-issue #74941 each merged origin/main minutes before the fix commit, and all downstream CI gates passed.

Failure Signals ⚠️

1. 60-minute session cap cuts off broad tasks (NEW cloud_agent_60min_cap_wip): both cloud-agent cancellations hit 60.3 and 60.4 min and left [WIP] PRs. One ([WIP] Refactor functions exceeding length limits in various packages #74913) is a multi-file lint refactor, now closed. The other (feat(experiments): add outcome-backed metrics with adaptive maturity windows #74992), outcome-backed metrics, is still WIP.
   • Failure rate: 2/46 (4.3%); the longest successful session was 55.0 min (p90 42.9 min)
   • Example: "[WIP] Refactor functions exceeding length limits in various..."
2. Batch assignment outruns review (NEW batch_assignment_review_backlog): 21 cloud-agent sessions launched in the 19–20Z burst on 11-04 (PRs Restore max-ai-credits: -1 budget disabling #74933–Warn when dynamic checkout refs expose host setup steps to untrusted code #74951). Only 9 are merged; 12 are still open about 11h later, and those branches now generate merge-from-main commits and review-reply fan-out.
   • Open rate: 57% of the batch, compared with 3 of 25 sessions outside the batch
3. Review-reply fan-out dominates CI (agent_reply_comment_fanout, now CONFIRMED on its 2nd day): each agent reply fires Agentic Commands (a 30–55 s route job) and Squad (every job skipped).
   • Impact: 36 of 50 sampled runs (72%) came from 18 replies, compared with 12 runs from 6 replies on 11-04
4. Addressing-comment cancellations: 4 of 48 were cancelled (8.3%), three of them after 15–18 min on automation branches (actions-update-*, update-docs-astro-*, code-scanning-fix-*). The cause isn't visible without logs.
5. Investigation-only drafts close unmerged: 3 cloud-agent PRs closed without merging. All were drafts whose outcome was "no change needed" (Investigate clustered smoke-engine failures #74790, Verify reported command-injection findings are already mitigated #74788), plus the timed-out [WIP] Refactor functions exceeding length limits in various packages #74913.

Prompt Quality Analysis 📝

Per-Prompt Breakdown

Prompts and issue bodies aren't in the run metadata, and transcripts were unavailable, so this section uses PR titles and outcomes as a proxy.

High-Quality Prompt Characteristics

• Names a specific behavior or regression to restore or prevent: typical of the 26 merged PRs
• Bounded to one subsystem (checkout, environment, safe-outputs, budget): all 4 PRs on today's sample branches
• Review comments that point at a concrete code path: 18/18 resolved with a cited commit on Warn when dynamic checkout refs expose host setup steps to untrusted code #74951 and Support deployment controls on workflow environments #74938

Example High-Quality Prompt (inferred from the resulting PR):

Restore `max-ai-credits: -1` so it disables the budget again; add a regression test covering -1 and 0.

Low-Quality Prompt Characteristics

• Open-ended, multi-file scope ("functions exceeding length limits in various files"): 1 of the 2 sessions that hit the 60-min cap
• Feature-sized asks with no explicit acceptance slice ("support outcome-backed metrics in experiments"): the other capped session
• "Investigate..." or "verify..." asks with no stated deliverable: 2 of the 3 closed-unmerged drafts

Example Low-Quality Prompt:

Refactor functions exceeding length limits in various files.

Orphaned Branch Escalation Alerts 🚨

Branches with ≥5 simultaneous gate firings and no Copilot agent assigned for >2 hours.

Summary

• Orphaned Branches Today: 0 out of 26 open PRs (0.0%)
• Historical Baseline: 0.0% orphaned rate (30-day mean over 7 recorded days)
• Status: NORMAL

Escalation Candidate Details

Escalation Candidates

✅ No orphaned branches exceed the escalation threshold today.

7 runs were in progress in the last 6h: 4 on main and 1 Addressing comment each on #75112, #74077 and #75116. No PR branch had more than 1 concurrent gate, and 9 open PRs have no Copilot assignee, but none has active gates.

CI Waste Estimate

• Orphaned gate-hours today: 0 gates × 0h ≈ 0 CI-minutes wasted
• Recoverable capacity: none needed. The bigger waste is the 18 Squad and 18 Agentic Commands runs from reply fan-out, about 18 × 35 s ≈ 10.5 runner-min of routing in 15 min.

Notable Observations

Loop Detection and Session Diagnostics

Loop Detection

• Sessions with loops: not measurable (0 transcripts)
• Average loop count: n/a
• Common loop patterns: none detectable from metadata alone.

Tool Usage

• Most used tools: not observable today. From step metadata, every agent job ran the same setup chain: Node, Go, npm deps, dev deps, build, Recompile workflows, uv, gopls, TS language server, then MCP servers.
• Setup overhead per agent job:

Run	Job	Agent time	Setup share	Dev-deps + recompile
§41981713296 Code scanning #74951	7.4 min	57 s	87.2%	4.3 min
§41980651523 Code scanning #74938	9.25 min	150 s	73.0%	4.5 min
§41980664150 Addressing comment #74935	21.4 min	17.6 min	18.1%	2.0 min

• Missing tools: none requested (not observable)

Data Pipeline

• Transcripts: 0/50; session-logs-latest/ is empty (11-04 had 13/13 events.jsonl)
• NEW transcript_fetch_difc_block: gh api --allow-escape-sequences repos/{repo}/actions/jobs/{id}/logs now returns DIFC policy violation: response requires resource access (HTTP 403) for all 3 agent jobs. This replaces the earlier escape-sequence refusal. It is a policy boundary, so it was not worked around. Transcript capture has to happen in the pre-agent fetch step.

Context Issues

• Sessions with confusion: not measurable
• Clarification requests: 0 visible. All 18 review replies were resolutions, not questions.

Experimental Analysis

Standard analysis only. No experimental strategy this run (roll = 66).

Actionable Recommendations

For Users Writing Task Descriptions

1. Split anything that can't finish in about 45 minutes: both capped sessions were broad. Successful sessions peak at 55 min.
   • Before: "Refactor functions exceeding length limits in various files"
   • After: "Split compileWorkflow in pkg/workflow/compiler.go below 80 lines; keep behavior and tests unchanged" (one issue per package)
2. Give investigations a deliverable: "investigate" and "verify" asks produced drafts that closed unmerged.
   • Before: "Investigate clustered smoke-engine failures"
   • After: "Identify the root cause of the smoke-engine failures in runs X/Y; open a fix PR or comment with the failing step and evidence"
3. Stagger batch assignments to match review capacity: 21 assignments in 6 minutes produced 12 PRs still waiting about 11h later.
   • Example: assign 5–6 at a time and queue the next batch as earlier PRs merge

For System Improvements

1. Collapse reply fan-out: skip the Squad and Agentic Commands triggers for review-thread replies authored by the coding agent, or filter at the on: level so they don't create runs.
   • Potential impact: High (72% of today's sample, 1:2 reply-to-run amplification)
2. Restore transcript capture outside the sandbox: the 403 means the pre-agent fetch step needs to save agent job logs and events.jsonl artifacts into session-logs-latest/.
   • Potential impact: High (behavioral analysis has been blind for most of the last 50 days)
3. Skip Recompile workflows and the full dev-deps install for read-only agent runs (code-scanning findings): about 4.4 min of setup per run comes before 1–2.5 min of agent work.
   • Potential impact: Medium (about 155 code-scanning runs in 24h)

For Tool Development

1. Session time budget warnings: an in-session signal at about 45 min, so the agent commits partial progress and summarizes before the 60-min cap.
   • Frequency of need: 2 sessions in 24h
   • Use case: avoid [WIP] PRs with no handoff summary

Historical Trends and Statistical Summary

Trends Over Time

• Completion rate trend: 7 consecutive recorded days at or above 64% (10-30: 70, 10-31: 90, 11-01: 70, 11-02: 64, 11-03: 64, 11-04: 78, 11-05: 64), well above the 30-day mean of 49.9%. Failures are 0 on 11-04 and 11-05, after CGO and CJS reds on 11-01 to 11-03.
• Average duration trend: the sample average has stayed in a 1.7–5.7 min band since 10-29. It is driven by sample composition, not agent speed.
• Quality improvement: coding-agent outcomes are healthy (91.5% session success, 58% of PRs merged within 24h). The constraint has moved from agent reliability to review throughput and CI amplification.

Statistical Summary

Total Sessions Analyzed:     50 (+94 coding-agent sessions from 24h metadata)
Successful Completions:      32 (64.0%)
Failed Sessions:             0 (0.0%)
Abandoned Sessions:          18 (36.0%)  [skipped Squad routes]
In-Progress Sessions:        0 (0.0%)    [3 were running at fetch time; all succeeded]

Average Session Duration:    1.8 min    (24h coding agents: 27.1 min)
Median Session Duration:     0.7 min    (24h coding agents: ~26 min)
Longest Session:             21.5 min   (24h: 60.4 min, cancelled at cap)
Shortest Session:            0.0 min    (18 instant Squad skips)

Loop Detection:              n/a (0 transcripts)
Context Issues:              n/a (0 transcripts)
Tool Failures:               n/a; 0 failed jobs/steps in the 3 agent runs

High-Quality Prompts:        26/45 cloud-agent PRs merged (proxy)
Medium-Quality Prompts:      15/45 still open (insufficient signal)
Low-Quality Prompts:         4/45 (2 capped at 60 min + 2 investigation-only drafts)

Next Steps

• Gate Squad and Agentic Commands triggers on agent-authored review replies
• Move transcript capture into the pre-agent fetch step (sandbox job-log reads are DIFC-blocked)
• Review the 12 open PRs from the 11-04 19Z batch (Restore max-ai-credits: -1 budget disabling #74933–Warn when dynamic checkout refs expose host setup steps to untrusted code #74951)
• Re-scope feat(experiments): add outcome-backed metrics with adaptive maturity windows #74992 into smaller issues before re-assigning
• Schedule follow-up analysis in 24h

References:

• §41980664150: Addressing comment on Prevent import read failures from masquerading as stale locks #74935 (17.6 min agent time)
• §41981713296: Code scanning on Warn when dynamic checkout refs expose host setup steps to untrusted code #74951 (87% setup)
• §41982127279: this analysis run

---

Analysis generated automatically on 2026-11-05 at 07:15 UTC
Run ID: 41982127279
Workflow: Copilot Session Insights

Generated by 📊 Copilot Session Insights · claude · routed: opus55 high · ~215 AIC · ⌖ ~2 AIC · ⊞ 14.6K · ◷

• expires on Nov 5, 2026, 11:12 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Session Insights.

A newer discussion is available at Discussion #75355.