[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-11-06

[github-actions[bot]]

🤖 Copilot Agent Session Analysis — 2026-11-06

Executive Summary

• Sessions Analyzed: 50 (2 coding-agent sessions + 48 CI gates, reviewers and code-scanning runs on 5 copilot/* branches)
• Analysis Period: 2026-11-06 04:23–05:19 UTC (sample). 24h context: 2026-11-05 07:11 → 2026-11-06 07:11 UTC
• Completion Rate: 90.0% (45/50), tying the 10-31 high. 97.8% if you exclude skipped runs. Push-level: 20/20 clusters had at least one success
• Average Duration: 3.31 min (median 1.04 min)
• Experimental Strategy: none (roll 72, standard run)

Note

The main finding today is a Sous Chef no-op loop. PR Sous Chef posted the same Copilot assignment 33 times on #74985. The requested-work hash didn't change in 32 of them. The result was 15 Copilot sessions in 24h that did nothing (~75 runner-min). Every one of those replies said it was blocked on review threads that only a maintainer can resolve.

📈 Session Trends Analysis

Completion Patterns

Completion reached 90.0%, tying the 10-31 high. That is +26 pts vs 11-05 and +18.6 pts above the 7-day mean (71.4%). The 30-day mean is 56.2% (24 recorded days). Since 10-29 hard failures have stayed at 0–6 per day; most non-success runs were skipped or action_required gates, not real failures. Today's only red run (CGO on #74940) went green 41 minutes later after an agent fix.

Duration & Efficiency

The sample average (3.3 min) is below the 7-day mean (4.5 min) because short CI gates make up most of the sample. Real coding-agent sessions in the last 24h averaged 17.8–18.7 min. No session in the sample looped. The loop this time was outside the sample: the dispatcher kept re-requesting work on #74985, which the in-sample loop metric doesn't catch.

Key Metrics

Metric	Value	Trend
Total Sessions	50	→
Successful Completions	45 (90.0%)	↑ (64.0% on 11-05)
Failed/Abandoned	1 failure (2.0%) + 4 skipped (8.0%)	↓ (18 skipped on 11-05)
Average Duration	3.31 min (median 1.04)	↓ vs 7-day mean 4.48
Loop Detection Rate	0 (0%) in sample · 1 PR / 15 no-op sessions in 24h	⚠️ new loop type
Context Issues	15 dispatcher-side no-op sessions (24h) · transcripts unavailable	↑
Coding-agent sessions (24h)	71 · 95.7% success (67/70 completed)	↓ volume (94) · ↑ success (91.5%)

Success Factors ✅

1. Specific, single-intent human prompts: when a maintainer's @copilot comment asked for one concrete change, Copilot almost always produced a commit or a verified answer.

   • Success rate: 0/19 replies were no-ops; 14/19 (73.7%) landed a commit or change and 5 were verifications or a blocked push (24h)
   • Example: "remove 'Powered by' to save space, just a list of model names" → Done in d6f34ac (Show models powering auto in attribution footers #74985). "resolve the merge conflicts on this branch." → Resolved in 2bf81a3.

2. Small, focused cloud-agent tasks merge quickly: all 11 cloud-agent sessions succeeded, and every PR that merged was small.

   • Success rate: 7/7 merged PRs touched 1–12 files (median 3 files, +31 lines) and merged within ~2–3h. The only closed-unmerged PR was the largest: Extract shared Go and Node workflow setup #75299, a refactor with 37 files and +504/−442 lines.
   • Example: Test ZIP extraction limits against high-compression archives #75264, Clarify ai-moderator add-labels format #75263, Align lint and logs CLI options #75265 and Add Koka as an option in formal-methods instructions #75267 were created 00:55–00:57Z and all merged by 03:53Z.

3. CI failure dispatch with links to the failing jobs: when a Sous Chef dispatch was about failing CI and linked the failing jobs, one session fixed it.

   • Success rate: 1/1 in sample
   • Example: CGO failed on Preserve retry diagnostics in external threat-detection artifacts #74940 at 04:34Z (build-wasm, impacted-go-tests). Sous Chef dispatched at 04:54Z with job links, the addressing session ran 19.6 min, and CGO was green at 05:16Z.

4. CI gates on Copilot pushes are healthy: 20/20 push clusters had at least one success, and 19/20 were all green. CWI, CJS, Work Queue and code-scanning were 100% green across the sample.

Failure Signals ⚠️

1. Sous Chef re-dispatches with identical state (NEW): PR Sous Chef re-posts the same assignment while review threads stay unresolved. Copilot can't resolve threads, so each re-post ends in a no-op.

   • Failure rate: 15/32 (46.9%) of Copilot replies to Sous Chef were no-ops in 24h, all 15 on Show models powering auto in attribution footers #74985
   • Example: Show models powering auto in attribution footers #74985 got 33 posts over 19h. The requested-work hash was the same in 32/33, and the same (head, state) pair was dispatched up to 9 times in a row. All 15 no-op replies named threads r4475080603, r4475080618, r4475165420 and r4475165477, e.g. "This request will keep repeating until a maintainer resolves threads ... I can't resolve them from here."

2. Lock-file merge treadmill: generated .lock.yml conflicts drive both commits and prompts.

   • Failure rate: 5 of 19 human prompts (26%) were "resolve the merge conflicts", and all were lock-file conflicts. 6 of the 11 commits on Show models powering auto in attribution footers #74985 were merges from main.
   • Example: Preserve retry diagnostics in external threat-detection artifacts #74940 now carries 258 regenerated lock files (323 changed files) for a fix whose source change is small.

3. Setup overhead: setup steps take a large share of each agent run's runner time.

   • Failure rate: setup was 39.1% of the cloud-agent run and 24.1% of the addressing run. Recompile workflows took 154s and 142s, and Install development dependencies took 127s in the cloud-agent run.
   • Example: in the cloud-agent run 42163327734, 438s of 1119s went to setup before Processing Request started.

4. Sessions close to the 60-min cap: 3 addressing sessions ran 46.9, 52.2 and 55.7 min (Support GitHub App authentication for repo-memory #74937, Allow status-only reporters to read proven pre-agent custom outputs #74939, feat(experiments): add outcome-backed metrics with adaptive maturity windows #74992). None was cancelled at the cap today, but feat(experiments): add outcome-backed metrics with adaptive maturity windows #74992 is still the [WIP] PR left by a cap cancel on 11-05.

5. Infrastructure failures: 1 addressing session (Clarify strict-mode XPIA policy for nonsecret test/build state #74936) failed in Install development dependencies after 28.7 min. On Add label-scoped title deduplication for create-issue #74941 one session failed in Processing Request at 5.3 min, and one session was cancelled (fix: pin activation lock checks to immutable workflow revision #75228, 14.2 min). On 11-05 a ruleset (commit_refs_rules) rejected pushes on Preserve retry diagnostics in external threat-detection artifacts #74940; the fix Fix PR branch rejection for Copilot-review-only rulesets #75230 merged today at 05:48Z.

Prompt Quality Analysis 📝

Per-Prompt Breakdown

Classified from the 51 Copilot replies on PRs that had addressing-comment sessions in the last 24h:

Quality	Source	Replies	Productive
High	Direct human @copilot prompts	19 (37%)	19 (100%) not no-op · 14 (73.7%) landed a change
Medium	Sous Chef multi-item lists with new state	17 (33%)	17 (100%, incl. 4 "starting" acks)
Low	Sous Chef re-posts with unchanged state	15 (29%)	0 (0%)

High-Quality Prompt Characteristics

• One imperative action on one target: typical of the prompts that landed a change (e.g. resolve the merge conflicts on this branch., 5 of 5 landed)
• A concrete outcome the agent can check, such as "just a list of model names" or "inside the existing execution.json": present in the follow-ups that produced commits d6f34ac and 47e0fe75
• Constraint stated up front (e.g. "reuse the existing files already stored in artifacts"): led to a smaller design (Preserve retry diagnostics in external threat-detection artifacts #74940 folded retry telemetry into an existing artifact)

Example High-Quality Prompt:

@copilot try to reuse the existing json files that are already stored in artifacts instead of adding a new file

Low-Quality Prompt Characteristics

• Repeats items the agent already answered (review threads it replied to but can't resolve): 15/15 no-op sessions
• Hashes for head, work and state unchanged since the previous dispatch: 15/15 no-op sessions

Example Low-Quality Prompt (abridged):

@copilot Copilot assignment requested for the following outstanding work:
1. Update this branch with the latest main using make merge-main ...
2-7. Review (...): <threads already addressed in 80fd66f>
8. Fix failing check `JS Tests (shard 1/8)`
Sous-chef requested-head: 80fd66f...  requested-work: 071b79eb...  (identical to previous 8 dispatches)

Orphaned Branch Escalation Alerts 🚨

Branches with ≥5 simultaneous gate firings and no Copilot agent assigned for >2 hours.

Summary

• Orphaned Branches Today: 0 out of 17 open PRs (0.0%)
• Historical Baseline: 0.0% orphaned rate (30-day mean over 8 recorded days)
• Status: NORMAL (56th consecutive 0.0% day)

Escalation Candidate Details

Escalation Candidates

✅ No orphaned branches exceed the escalation threshold today.

11 runs were in progress across 6 branches. The highest was 3 gates on pelikhan/evals-decision-fallback (#75339, opened 06:41Z, under 1h old). Every other PR branch had at most 1 active run.

CI Waste Estimate

• Orphaned gate-hours today: 0 gates × 0h ≈ 0 CI-minutes wasted
• Recoverable capacity: none needed. The real waste today was the ~75 runner-min of no-op Sous Chef sessions on Show models powering auto in attribution footers #74985 (assigned, not orphaned).

Notable Observations

Loop Detection and Session Diagnostics

Loop Detection

• Sessions with loops: 0 in sample (0%). In the 24h window: 1 PR (Show models powering auto in attribution footers #74985) with 15 no-op re-dispatches
• Average loop count: 15 iterations on the one looping PR, at a ~20–45 min cadence (07:22Z → 16:32Z on 11-05)
• Common loop patterns: the dispatcher sees unresolved review threads, Copilot replies "nothing to change", and the threads remain unresolved, so the dispatcher posts again. The last no-op was at 16:32Z. The 16:54Z post had no @copilot mention, and the next dispatch (20:20Z) came after a human-requested conflict fix and carried a new work hash.

Tool Usage

• Transcripts: 0/50. gh api .../actions/jobs/{id}/logs returned DIFC policy violation (HTTP 403) for the second day in a row. I didn't try to work around it, and the pre-fetch logs directory was empty.

• Step-level time (from job step timings):

  Step	Cloud agent (42163327734)	Addressing (42165251685)
  Processing Request	681s	883s
  Recompile workflows	154s	142s
  Install development dependencies	127s	26s
  Build code	38s	18s
  Setup share of step time	39.1%	24.1%

• Missing tools: review-thread resolution. 14 of 15 no-op replies explicitly said Copilot couldn't resolve the threads.

Context Issues

• Sessions with confusion: 0 visible on the agent side. Copilot diagnosed the loop correctly every time. The confusion was on the dispatcher side: Sous Chef doesn't recognise that Copilot already answered.
• Run volume from PR edits: Agentic Commands was 15/50 (30%) of the sample. Every one was a pull_request (edited/opened) event triggered by Copilot's own PR updates; Squad was absent (it was 18 runs on 11-05). Over 24h Agentic Commands ran 397 times.
• Clarification requests: 0

24h Coding-Agent Outcomes

Type	Sessions	Success	Mean / median / max (min)	Outcome
Running Copilot cloud agent	11	11 (100%)	18.7 / 18.9 / 37.4	7 merged, 1 closed, 2 open, 1 no PR
Addressing comment	60	56 (2 fail, 1 cancel, 1 running)	17.8 / 16.4 / 55.7	21 sessions on #74985 alone
Code scanning AI findings (bot)	70	69 (1 running)	8.3 / 8.3 / 13.6	n/a
Copilot Code Review	36	34 (2 running)	11.5 / 9.0 / 26.8	n/a

Experimental Analysis

Standard analysis only, no experimental strategy this run (roll 72 ≥ 30). The promoted push_event_reattribution metric is still reported: push-level completion was 20/20 (100%), compared with the raw 90.0%.

Actionable Recommendations

For Users Writing Task Descriptions

1. Resolve review threads once Copilot has addressed them: unresolved threads are what make Sous Chef re-dispatch.

   • Example: on Show models powering auto in attribution footers #74985, resolving 4 threads after 80fd66f would have prevented 15 no-op sessions.

2. Use one action with a checkable outcome (direct prompts had 0 no-ops vs 46.9% for re-posted dispatcher lists):

   • Before: @copilot fix the attribution
   • After: @copilot remove "Powered by" from the auto attribution label; keep just the comma-separated model list

3. Keep cloud-agent tasks small: the 7 merged PRs had a median of 3 files; the 37-file refactor was closed.

   • Before: "Extract shared Go and Node workflow setup" across all workflows
   • After: "Extract the Go setup steps into a shared import and migrate the first 5 workflows"

For System Improvements

1. De-duplicate Sous Chef dispatches: skip re-dispatch when (requested-head, requested-work, requested-state) matches the last dispatch Copilot answered with a no-op. Alternatively, drop threads with a newer Copilot reply from requested-work, or escalate to the PR author instead.

   • Potential impact: High. 15 sessions and ~75 runner-min on one PR in one day, plus the reply-comment noise.

2. Skip or cache Recompile workflows in Copilot setup steps when no .github/workflows/*.md changed. It took ~2.5 min in both measured sessions; if that's typical, it costs ≈3 runner-hours/day across 71 sessions.

   • Potential impact: Medium-High

3. Capture transcripts in the pre-agent fetch step, since in-sandbox job-log reads are DIFC-blocked (2nd day).

   • Potential impact: Medium. It is a prerequisite for loop, tool and token analysis.

4. Filter Copilot PR-body progress edits from Agentic Commands (pull_request: edited by Copilot): 30% of today's sample.

   • Potential impact: Low-Medium

For Tool Development

1. Review-thread resolution for the coding agent (e.g. resolve after replying with a fix SHA)

   • Frequency of need: 15 sessions on 1 PR (24h), explicitly requested in 14 replies
   • Use case: close the loop after addressing review feedback so dispatchers and humans see an accurate thread state

2. Pre-flight push-rule check

   • Frequency of need: 1 PR (Preserve retry diagnostics in external threat-detection artifacts #74940, 11-05), fixed upstream by Fix PR branch rejection for Copilot-review-only rulesets #75230 today
   • Use case: catch commit_refs_rules rejections before the agent spends a session on commits it can't push

Historical Trends and Statistical Summary

Trends Over Time

• Completion rate trend: 90.0% today, compared with a 7-day mean of 71.4% (10-30..11-05) and a 30-day mean of 56.2%. Raw completion has been ≥64% for 8 straight recorded days (10-30..11-06).
• Average duration trend: 3.31 min, compared with a 7-day mean of 4.48 min. The sample is dominated by short gates; real agent sessions take ~18 min.
• Quality improvement: 24h coding-agent success rose from 91.5% (11-05) to 95.7%. There were no 60-min cap cancels, against 2 on 11-05.
• Orphans: 0.0% for the 56th consecutive recorded day.

Statistical Summary

Total Sessions Analyzed:     50
Successful Completions:      45 (90.0%)
Failed Sessions:              1 (2.0%)
Abandoned Sessions:           4 (8.0%)   [skipped PR Specification Reviewer]
In-Progress Sessions:         0 (0.0%)

Average Session Duration:   3.31 min
Median Session Duration:    1.04 min
Longest Session:           19.6 min   [Addressing comment on PR #74940]
Shortest Session:           0.03 min  [skipped gate, 2 s]

Loop Detection:            0 sessions in sample (0%); 15 no-op re-dispatches on #74985 (24h)
Context Issues:            15 dispatcher-side no-op sessions (24h)
Tool Failures:             n/a (transcripts DIFC-blocked); 1 job-step failure (CGO)

High-Quality Prompts:      19 (37%)   [24h, n=51 Copilot replies]
Medium-Quality Prompts:    17 (33%)
Low-Quality Prompts:       15 (29%)

Next Steps

• Add identical-state de-duplication to PR Sous Chef (owner: Sous Chef workflow maintainers)
• Decide on a process or tool for resolving review threads after Copilot addresses them
• Evaluate skipping Recompile workflows in Copilot setup when no workflow .md changed
• Move transcript capture into the pre-agent fetch step
• Follow-up analysis in 24h: confirm whether souschef_identical_state_redispatch recurs

---

Analysis generated automatically on 2026-11-06 at 07:20 UTC
Run ID: 42176060093
Workflow: Copilot Session Insights

Generated by 📊 Copilot Session Insights · claude · routed: opus55 xhigh · ~379 AIC · ⌖ ~2 AIC · ⊞ 14.6K · ◷

• expires on Nov 6, 2026, 11:25 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Session Insights.

A newer discussion is available at Discussion #75634.