[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-11-01

[github-actions[bot]]

🤖 Copilot Agent Session Analysis — 2026-11-01

Executive Summary

• Sessions Analyzed: 50 workflow runs on copilot/* branches. Of these, 3 are coding-agent sessions, 7 are review or code-scanning agent runs, and 40 are gates or Squad runs.
• Analysis Period: 2026-10-31 23:28Z → 2026-11-01 05:32Z (~6 h, 2 branches: PR audit: report failed continue-on-error evals/detection engine steps and add an evals summary #73556 and PR Record a not-evaluated evals marker instead of UNKNOWN answers when the judge never ran or failed #73704)
• Completion Rate: 70.0%. All 3 coding-agent sessions succeeded.
• Average Duration: 4.7 min (median 0.75 min). Coding-agent sessions averaged 39.9 min.
• Experimental Strategy: none (standard run, roll = 34)

Warning

No transcripts were available today (0 of 50). The last three days had them, but all of those came from review-bot workflows, and none of those ran in this window. None of the 10 dynamic agent runs had artifacts. In this analysis sandbox, gh also refused to download the cloud-agent job log because the response contains terminal escape sequences. It's not yet confirmed whether the pre-fetch step hits the same refusal. Because of this, the behavioral findings below come from job-step timings and PR and issue history, not from turn-by-turn data.

Three things need attention:

1. CGO has been failing on main since 2026-10-29 10:23Z (about 68 h). PR audit: report failed continue-on-error evals/detection engine steps and add an evals summary #73556 was merged on a head commit where impacted-go-tests had failed. With main red, CI no longer tells us whether an agent's fix works.
2. Review and code-scanning agents spend about two-thirds of their job time on setup. Across 7 runs, 41.1 of 62.5 job-minutes went to copilot-setup-steps before the agent started.
3. The test-quality-sentinel prefetch bug is still in the source (test-quality-sentinel.md:104-105). This is the third report in a row.

Key Metrics

Metric	Value	Trend (vs 2026-10-31)
Total Sessions	50	→
Successful Completions	35 (70.0%)	↓ from 90.0%
Failed/Abandoned	2 (4.0%), both CGO	↑ from 0
Average Duration	4.7 min (median 0.75)	↓ from 5.7 min
Loop Detection Rate	n/a (0 transcripts)	— (1 session on 10-31)
Context Issues	n/a (0 transcripts)	—
Coding-agent success	3/3 (100%)	→

The remaining runs were 6 action_required (12%) and 7 skipped (14%), all Squad or Agentic Commands follow-ups. None of them are failures.

📈 Session Trends Analysis

Completion Patterns

Completion has been at or above 70% for three days in a row (70.0 → 90.0 → 70.0%). That is the first such streak in the recorded window, which averages 51.3% over 20 recorded days. Today's drop from 90% is mostly a change in mix: 17 of the 50 runs were Squad and Agentic Commands follow-ups that stop at skipped or action_required. The only real failures were 2 CGO runs, and those match the red main baseline described below.

Duration & Efficiency

The average was 4.7 min, close to the last two days (5.4 and 5.7). The median fell to 0.75 min because 10 Squad runs took about 1 s each. Loops can only be counted when transcripts exist, so the loop panel has no data point for today.

Both charts use stacked panels that share the date axis instead of a secondary y-axis, so each panel has a single scale.

Success Factors ✅

1. Issues written as specs: Both coding tasks came from issues of about 19K characters. Each had 32–38 file paths, real failing run IDs, and numbered acceptance cases.
   • Success rate: 2/2 tasks produced a PR (audit: report failed continue-on-error evals/detection engine steps and add an evals summary #73556 merged; Record a not-evaluated evals marker instead of UNKNOWN answers when the judge never ran or failed #73704 opened after one 54.6-min session).
   • Example: on audit: report failed continue-on-error evals/detection engine steps and add an evals summary #73556 the agent kept a log-derived error message on purpose because the issue's acceptance case 6 required it. The numbered acceptance cases let it disagree with a reviewer and give a reason.
2. Review rounds that converged: On audit: report failed continue-on-error evals/detection engine steps and add an evals summary #73556 there were 3 Copilot Code Review rounds of about 11 min each and 2 "Addressing comment" sessions of 30.2 and 34.1 min. Each round targeted a smaller edge case: first the step-naming contract, then the fallback for a blank step name, then YAML quoting of that fallback.
   • Success rate: 5/5 review and address runs succeeded. The PR merged after about 10 h, 10 commits and 39 review comments.
   • Example: 2 of the 7 agent replies were reasoned disagreements ("Already handled...", "Keeping the message, deliberately..."), not fixes to every comment.
3. The maintainer ran dependent issues one after another: BinEval persists UNKNOWN answers when the evals judge never ran or failed #73551 was assigned to Copilot 28 s after audit: report failed continue-on-error evals/detection engine steps and add an evals summary #73556 merged, so the follow-up started from a main that already contained the first change.
   • Success rate: 1/1 (the cloud-agent session succeeded).
   • Example: the issues were filed 4.5 min apart but worked on one after the other.

Failure Signals ⚠️

1. CI red on main (main_ci_red_baseline): CGO on main has had no successful run since 2026-10-29 10:23Z. Both agent head commits on audit: report failed continue-on-error evals/detection engine steps and add an evals summary #73556 (7b883a68dd, b61cf648a2) failed impacted-go-tests with exit code 2. The PR was merged on b61cf648a2 without a re-run.
   • Failure rate: 2/3 CGO runs in this window (66.7%). The one passing CGO run was on Record a not-evaluated evals marker instead of UNKNOWN answers when the judge never ran or failed #73704.
   • Example: each "Fixed in <sha>" reply named a new regression test, but CI on that same commit could not confirm the fix.
2. Setup takes longer than the review itself (setup_overhead_dominates_review_agents): Before the agent starts, every dynamic agent job runs npm ci, make deps-dev, make build, make recompile || true, gopls and the TypeScript language server.
   • Setup share: 74.9% of code-scanning job time and 57.5% of Copilot Code Review job time, compared with 10.8% for coding-agent sessions.
   • Example: §41257501903 spent 6.2 min on setup and 43 s on the agent step.
3. Lock-file churn hides the real change: PR Record a not-evaluated evals marker instead of UNKNOWN answers when the judge never ran or failed #73704 changes 392 files across 2 commits. At least 254 of them are regenerated .lock.yml files, and they are in the same commit as the code change (faf247f27b).
   • Failure rate: n/a. This is a risk to review quality, not a failed run.
4. Approval pairs created by review bots (review_completion_triggers_approval_pair): 3 of 6 reviewer or scanner completions were followed 8–11 s later by a Squad + Agentic Commands pair that stopped at action_required with 0 duration.
   • This accounts for all 6 action_required runs today.

Prompt Quality Analysis 📝

Per-Prompt Breakdown

High-Quality Prompt Characteristics

Today's sample has 2 task prompts: issues #73552 → PR #73556 and #73551 → PR #73704. Both are high quality.

• Concrete failing run IDs: in 2/2 (100%) of successful sessions (1 and 4 run references)
• Explicit file paths: 2/2 (100%), with 32 and 38 distinct paths
• Numbered acceptance cases or steps: 2/2 (100%), with 11 and 6 numbered items
• Area labels (compiler, cli, threat-detection, model-routing): 2/2 (100%)

Example High-Quality Prompt (structure only, content shortened):

Title: audit: failed continue-on-error evals and threat-detection engine steps
       are invisible (pre-flight failures report as workflow_succeeded)
Evidence: run <id> — engine step failed pre-flight; audit says workflow_succeeded
Where:    pkg/cli/audit_*.go, pkg/workflow/evals_steps.go, pkg/constants/...
Acceptance cases:
  1. ...
  6. The finding names the failing step and its pre-flight message

Low-Quality Prompt Characteristics

• There were no low-quality prompts in today's sample (0 of 2).

Example Low-Quality Prompt (illustrative only, not from today's data):

Fix the audit so it reports evals failures.

It has no run IDs, no file paths and no acceptance cases. The agent would have to guess what "done" means, and a reviewer would have nothing to cite when disagreeing.

Orphaned Branch Escalation Alerts 🚨

Branches with ≥5 simultaneous gate firings and no Copilot agent assigned for >2 hours.

Summary

• Orphaned Branches Today: 0 out of 17 active branches (0.0%)
• Historical Baseline: 0.0% orphaned rate (30-day mean over 3 recorded days)
• Status: NORMAL

Escalation Candidate Details

Escalation Candidates

✅ No orphaned branches exceed the escalation threshold today. There were 3 in-progress runs, all on main (Daily Workflow Updater, PR Sous Chef and this workflow), so no PR branch had more than 0 active gates.

CI Waste Estimate

• Orphaned gate-hours today: 0 gates × 0 h ≈ 0 CI-minutes wasted
• Recoverable capacity: n/a. There were no critical or high branches.
• Related waste, not orphan-driven: make deps-dev and make recompile took 27.5 runner-minutes across 7 reviewer and scanner jobs in this 6 h window.

Notable Observations

Loop Detection and Session Diagnostics

Loop Detection

• Sessions with loops: n/a (0 transcripts)
• Average loop count: n/a
• Common loop patterns: Run metadata shows no retry loops. Every agent run was attempt 1, and 0 of 807 steps across the 10 agent jobs failed. The one multi-round pattern, on audit: report failed continue-on-error evals/detection engine steps and add an evals summary #73556, converged rather than repeated.

Tool Usage (step-level, from the jobs API)

Agent	Runs	Avg job	Avg setup	Avg agent step	Setup share
Coding agent (copilot-swe-agent)	3	39.6 min	4.3 min	35.2 min	10.8%
Copilot Code Review	3	11.0 min	6.3 min	4.6 min	57.5%
Code scanning AI (GHAS)	4	7.4 min	5.5 min	1.8 min	74.9%

• Most used tools: n/a (no transcripts)
• Tool success rates: 100% at step level (0 failed steps)
• Missing tools: transcript access for dynamic/copilot-swe-agent runs

Context Issues

• Sessions with confusion: n/a (0 transcripts)
• Common confusion points: none visible in PR threads. The 7 Squad replies each cite a commit SHA or a test name.
• Clarification requests: 0 visible in PR threads

Carried-over items

• test_sentinel_prefetch_always_fails: still at test-quality-sentinel.md:104-105. It wasn't triggered today because no Test Quality Sentinel runs fell in the window.
• inline_safeoutputs_quote_breakage: couldn't be observed today. Draft PR [safeoutputs] Clarify safeoutputs apostrophe-escaping scope to prevent shell syntax errors #73571 updates the apostrophe-escaping guidance.
• After the merge of PR audit: report failed continue-on-error evals/detection engine steps and add an evals summary #73556 at 04:37:05Z, a "Squad implement" run fired 9 s later and was skipped. No failure cascade followed, because nothing else was running on that branch.

Experimental Analysis

Standard analysis only — no experimental strategy this run (roll = 34).

Actionable Recommendations

For Users Writing Task Descriptions

1. Keep writing issues as specs: include failing run IDs, file paths and numbered acceptance cases.
   • Before: "Audit should report evals failures." After: "Run 409554... reports workflow_succeeded although the evals step failed pre-flight. Acceptance: (1) ... (6) the finding names the step and its message."
2. State dependencies between issues in writing: today the maintainer started BinEval persists UNKNOWN answers when the evals judge never ran or failed #73551 by hand once audit: report failed continue-on-error evals/detection engine steps and add an evals summary #73556 merged. Writing the order into the issue lets automation or another maintainer follow it.
   • Before: two related issues with no ordering. After: "Blocked by audit: failed continue-on-error evals and threat-detection engine steps are invisible (pre-flight failures report as workflow_succeeded) #73552 — start after its PR merges."
3. Ask for generated files in their own commit: when a compiler change regenerates lock files, keep the code and tests in one commit and the make recompile output in another, so reviewers can find the real change.
   • Before: 1 commit, 392 files. After: commit 1 with code and tests (~40 files), commit 2 with "recompile lock files".

For System Improvements

1. Get CGO passing on main again: main hasn't had a green CGO run in about 68 h. Until it does, a failing CI run on an agent PR says nothing about the agent's change, and PRs get merged on red.
   • Potential impact: High
2. Reduce setup time for review and scanning agents: copilot-setup-steps.yml runs make deps-dev (about 2 min) and make recompile || true (about 2 min) before every Copilot Code Review and code-scanning job. AGENTS.md already tells coding agents to run make recompile after changing workflows, so it may not need to run at setup. Caching dependencies would also help.
   • Potential impact: Medium (roughly 4 of every 7–11 job-minutes for reviewers)
3. Fix the Test Quality Sentinel prefetch: gh pr diff accepts only one positional argument, so PRs that change tests get a noop instead of a review.
   • Potential impact: Medium

For Tool Development

1. Transcripts for coding-agent runs: every transcript recovered so far came from gh-aw workflows (review bots and one smoke test). No copilot-swe-agent session has produced one.
   • Frequency of need: 3 sessions today (100% of coding-agent sessions)
   • Use case: loop, tool and context analysis for the sessions that actually write code. Check whether the pre-fetch step's job-log download fails on terminal escape sequences. copilot-session-data-fetch.md:228 would count that as an API error and point to token permissions instead.
2. Select sessions by agent type: the newest-50-runs window covered only 6 h and 2 branches, and 40 of the 50 runs were gates or Squad runs. Filtering on path (dynamic/copilot-swe-agent/*, dynamic/agents/*) would get about 50 real agent sessions per day.
   • Frequency of need: every run
   • Use case: comparable per-session metrics from day to day

Historical Trends and Statistical Summary

Trends Over Time

• Completion rate trend: 70.0% today, against a mean of 51.3% over 20 recorded days (Oct 3–Nov 1; Oct 4–13 is a gap in the cache). Today is the third day in a row at or above 70%. The range over the window is 4.0–90.0%.
• Average duration trend: 4.7 min today, compared with 5.4 and 5.7 min on the previous two days. The window's peak was Oct 26 (35.6 min).
• Quality improvement: Today's 2 prompts can't be compared with earlier days, because prompts weren't scored before. Start scoring now to build the baseline.
• Orphan rate: 0.0% on every recorded day (56 of 56).

Statistical Summary

Total Sessions Analyzed:     50
Successful Completions:      35 (70.0%)
Failed Sessions:              2 (4.0%)
Abandoned Sessions:           0 (0.0%)
In-Progress Sessions:         0 (0.0%)
(Action required:             6 (12.0%); Skipped: 7 (14.0%))

Average Session Duration:   4.7 min
Median Session Duration:    0.75 min
Longest Session:           54.8 min (Running Copilot cloud agent)
Shortest Session:           0.0 min (Squad, action_required)

Loop Detection:            n/a (0 transcripts)
Context Issues:            n/a (0 transcripts)
Tool Failures:             0 failed steps across 10 agent jobs

High-Quality Prompts:      2 (100%)
Medium-Quality Prompts:    0 (0%)
Low-Quality Prompts:       0 (0%)

Next Steps

• Find and fix the CGO failure on main, red since 2026-10-29 10:23Z
• Decide whether make recompile and make deps-dev belong in copilot-setup-steps.yml for every agent
• Check the pre-fetch step logs for the terminal-escape-sequence refusal
• Fix test-quality-sentinel.md:104-105
• Schedule follow-up analysis in 24 h

References:

• §41273833572 — cloud-agent session on Record a not-evaluated evals marker instead of UNKNOWN answers when the judge never ran or failed #73704 (54.6 min, 50.1 min of it on the agent step)
• §41261863781 — CGO failure on audit: report failed continue-on-error evals/detection engine steps and add an evals summary #73556's merge head b61cf648a2
• §41257501903 — code-scanning job with 89% setup and 43 s on the agent step

Analysis generated on 2026-11-01 at 07:10 UTC · Run §41281651199

Generated by 📊 Copilot Session Insights · claude · routed: opus55 xhigh · ~381 AIC · ⌖ ~2 AIC · ⊞ 14.6K · ◷

• expires on Nov 1, 2026, 11:14 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Session Insights.

A newer discussion is available at Discussion #74070.