[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-11-03

[github-actions[bot]]

🤖 Copilot Agent Session Analysis — 2026-11-03

Executive Summary

• Sessions Analyzed: 50 run records on copilot/* branches: 12 Copilot coding-agent sessions (9 Agent Merge, 3 Addressing comment), 19 CI gates, 4 code-scanning agents, 6 Agentic Commands, 9 Squad
• Analysis Period: 2026-11-03 05:55–06:37 UTC (~42 min window, 9 branches)
• Completion Rate: 64.0% (32/50). Excluding skipped and still-running runs: 80.0% (32/40)
• Average Duration: 4.9 min (median 3.3 min)
• Experimental Strategy: none (roll = 31, standard run)

Top findings

1. Agent Merge runs finish without doing anything (new pattern). 8 of 9 Agent Merge on PR #N sessions concluded success with no commit, comment, review, or merge. 0 of 9 PRs were merged.
2. main is red, so every PR's CGO is red. CGO on main has failed 7 pushes in a row since 04:47Z. PR CGO failed 4/4 at the same build-wasm step. The only Agent Merge that pushed a fix was on the one PR whose failure is not also red on main.
3. Cause of the empty transcripts found. Downloading the job log fails with the response contains terminal escape sequences; pass --allow-escape-sequences. That is why 0/12 coding-agent transcripts were available today (and on 11-01 and 11-02).
4. No orphaned branches: 0 of 13 open PRs (0.0%), against a historical baseline of 0.0%.

📈 Session Trends Analysis

Completion Patterns

Completion held at 64.0% for a second day, 11.3 points above the 20-day prior mean of 52.7% (range 4.0–90.0%). The last five days (70 → 90 → 70 → 64 → 64%) are the steadiest stretch in the window. This rate measures the mix of run types in the sample more than agent quality: 10 of today's 18 non-successes are Squad skips or approval-gated runs, not failures.

Duration & Efficiency

Average duration (4.9 min) has stayed between 3.9 and 5.7 min for five days, well below the 9.0-min prior mean. Loops rose to 2: both are Agent Merge re-runs on a PR head that hadn't changed. Within coding-agent sessions, about 4 minutes of setup comes before any agent work. That setup is 41.5% of all coding-agent runner time.

Key Metrics

Metric	Value	Trend (vs 11-02)
Total Sessions	50	→
Successful Completions	32 (64.0%)	→ (32)
Failed/Abandoned	18 (36.0%): 6 failed, 8 skipped, 2 approval-gated, 2 still running	→ (18)
Average Duration	4.9 min (median 3.3)	↑ (3.9 / 1.4)
Loop Detection Rate	2 (4.0% of runs; 16.7% of coding-agent sessions)	↑ (0)
Context Issues	not measurable (0 transcripts)	—
Coding-agent sessions	12 (10 completed, 2 running >45 min)	↑ (3)

Success Factors ✅

1. The PR's failure is its own, not inherited from main: the one Agent Merge that pushed a fix (§41588147254, 15.4 min of agent work) ran on Add safe-output identifiers and dependency failure propagation #74376. That PR has a failure that isn't red on main (JS Tests (shard 3/4); CJS on main passed at 06:21Z).
   • Success rate: 1/2 Agent Merge runs on PRs with their own failures pushed a commit, vs 0/7 on PRs whose failures match main
   • Example: commit ec4fb0321 "Isolate safe-output dependency tests from detection env"
2. Specific, checkable comment requests: "resolve the merge conflicts on this branch." on Restrict implicit network defaults to documented infrastructure #74364 produced a merge commit, regenerated locks, and a reply in 20.3 min (§41590834905).
   • Success rate: 1/1 completed Addressing comment sessions. Comment-triggered sessions have resolved 100% of the time for 3 straight recorded days.
   • Example: an earlier "make the safe output result identifier shorter" on Add safe-output identifiers and dependency failure propagation #74376 also produced a commit and a reply
3. Deterministic infrastructure gates stay green: CWI, Work Queue Compilation Integration, Doc Build - Deploy, and Code scanning AI findings passed 16/16 (100%).
   • Example: all four fired together on each push and averaged 1.2–8.7 min per workflow
4. Runners aren't the bottleneck: queue time was 0.08–0.25 min for all 12 coding-agent sessions.

Failure Signals ⚠️

1. Agent Merge silent no-op (new): 9 Agent Merge runs, triggered from the PR UI with no prompt. 8 ended success with no visible output, and none merged.
   • No-output rate: 88.9% (8/9); 57.5 runner-min, of which 32.4 min was setup
   • Example: Report failures when activation stops before agent startup #74365 got Agent Merge at 06:02, 06:25 and 06:33Z. Between the first two, a maintainer merged main into the branch by hand (06:18Z), which the agent hadn't done.
2. Inherited red main (3rd occurrence of this pattern): CGO on main has failed every push since aa028fa (04:47Z, unit tests A–C / S–Z). build-wasm golden tests have also failed since 29fcf91 (04:59Z, Remove telemetry hosts from runtime-selected ecosystems #74363). The lock regeneration in fix(ci): regenerate locks after telemetry domain removal #74418 didn't fix either.
   • Failure rate: PR CGO 4/4 (100%), all failing at Run wasm golden tests (Go string API)
   • Example: Prevent repeat-update conflicts in source-managed workflows #74359, Fix false dependency cycles for custom jobs after repo-memory push #74366 and Complete fix codemods for nested workflows and App-token permissions #74368 are unstable only because of build-wasm / impacted-go-tests
3. Re-running Agent Merge on an unchanged head: Report failures when activation stops before agent startup #74365 (on 72064e436) and Prevent repeat-update conflicts in source-managed workflows #74359 (on a4d00d5e5) each got a second Agent Merge on the same SHA.
   • Failure rate: 2/2 re-runs left no output (~13 runner-min)
   • Example: §41590621444 did 0.95 min of agent work after 4.25 min of setup
4. Overlapping sessions on one PR (new): Add safe-output identifiers and dependency failure propagation #74376 had 3 overlapping Copilot sessions (Agent Merge 06:09–06:29, Addressing comment from 06:25, Agent Merge 06:36–06:46). Complete fix codemods for nested workflows and App-token permissions #74368 had 2.
   • Both Addressing comment runs that shared a PR with another session were still in Processing Request after ~45 min (2/2). The one without overlap finished in 20 min. The sample is small.
   • Example: PR timeline copilot_work_started / copilot_work_finished events interleave and can't be matched to runs
5. Repeated lock-file conflicts (new): Restrict implicit network defaults to documented infrastructure #74364 regenerates workflow locks. It needed main merged in at 05:37Z and again at 06:48Z, while main landed 11 commits in 1 h 47 min.

Prompt Quality Analysis 📝

Per-Prompt Breakdown

High-Quality Prompt Characteristics

• States an end state you can check (conflicts resolved, identifier shortened): 2/2 such prompts seen today produced a commit and a reply
• One problem on one PR: 1/1 completed comment-triggered session today had a single-objective prompt
• The failure is the PR's own: 1/1 Agent Merge that committed ran on a PR whose failure isn't on main

Example High-Quality Prompt:

@copilot resolve the merge conflicts on this branch.

Low-Quality Prompt Characteristics

• No task text at all (Agent Merge from the UI): 9/12 coding-agent sessions; 8/9 of them left no output
• Delegation with no expected result ("run pr-finisher skill"): the session was still running at ~50 min

Example Low-Quality Prompt:

[Agent Merge triggered from the PR UI: no task text; the only red check is build-wasm, which is also red on main]

Distribution (12 coding-agent sessions): High 1 (8.3%) · Medium 2 (16.7%: "fix ci failures", "run pr-finisher skill") · Low / no prompt 9 (75.0%)

Orphaned Branch Escalation Alerts 🚨

Branches with ≥5 simultaneous gate firings and no Copilot agent assigned for >2 hours.

Summary

• Orphaned Branches Today: 0 out of 13 open PRs (0.0%)
• Historical Baseline: 0.0% orphaned rate (mean over 5 recorded days, 10-29 → 11-02)
• Status: NORMAL

Escalation Candidate Details

Escalation Candidates

✅ No orphaned branches exceed the escalation threshold today. 8 runs were in progress, and no branch had more than 3 (copilot/extend-safe-outputs-identifier). Every Copilot-authored open PR has Copilot assigned.

CI Waste Estimate

• Orphaned gate-hours today: 0 gates × 0 h = 0 CI-minutes
• Recoverable capacity: none from orphans. The waste is elsewhere:
  • 4 PR CGO runs against a main that was already red: ~36.6 runner-min
  • 8 Agent Merge runs with no output: ~57.5 runner-min

Notable Observations

Loop Detection and Session Diagnostics

Loop Detection

• Sessions with loops: 2 (16.7% of coding-agent sessions)
• Average loop count: 1 extra run per looping PR
• Common loop pattern: Agent Merge started again on the same head SHA after an earlier run left no output

Tool Usage

Transcripts were unavailable, so these figures come from Actions step timings (10 completed coding-agent sessions):

Step	Median	Range	Total
Checkout code	0.28 min	0.20–0.33	2.7 min
Set up Go	0.41 min	0.30–0.47	4.0 min
Install development dependencies	0.56 min	0.45–0.62	5.4 min
Build code	0.33 min	0.02–0.40	2.3 min
Recompile workflows	1.95 min	1.52–2.50	20.2 min
Setup total (before Processing Request)	3.97 min	3.60–4.73	40.6 min
Agent work (Processing Request)	3.71 min	0.95–16.53	56.2 min

• Missing tools: job-log or transcript access for dynamic/copilot-swe-agent runs (0/12 retrievable). These runs have no artifacts, and log download is refused because of escape sequences.

Context Issues

• Sessions with confusion: not measurable (0 transcripts)
• Indirect signal: 8 Agent Merge runs left no explanation on the PR, so reviewers can't tell "nothing to fix" from "couldn't fix"
• Clarification requests: 0 seen in PR timelines

Experimental Analysis

Standard analysis only, no experimental strategy this run (roll = 31; threshold < 30).

Actionable Recommendations

For Users Writing Task Descriptions

1. Fix main before using Agent Merge: when the only red checks are also red on main, the agent has nothing PR-specific to fix.
   • Example: before, 3 Agent Merge runs on Report failures when activation stops before agent startup #74365 with no output. After, fix the build-wasm golden regression on main (first red at 29fcf91), then run Agent Merge once.
2. Don't re-run on an unchanged head: if an Agent Merge left no commit and the head SHA hasn't moved, a second run will do the same.
   • Example: before, Agent Merge on Prevent repeat-update conflicts in source-managed workflows #74359 at 06:25 and 06:34Z, both on a4d00d5e5. After, push a change or post a specific comment first.
3. Name the check and the end state: say which failure is the PR's own and what "done" looks like.
   • Example: before, "@copilot fix ci failures". After, "@copilot fix the failing JS Tests (shard 3/4) check; build-wasm is red on main too, so ignore it."
4. Run one session per PR at a time: wait for the active session to finish before starting another on the same branch.

For System Improvements

1. Fix transcript fetching in .github/workflows/shared/copilot-session-data-fetch.md:228: the gh api .../actions/jobs/{id}/logs call exits 1 with the escape-sequence error, so the [cca-engine] fallback yields nothing.
   • Suggested fix: write the log to a file with escape sequences explicitly allowed, then strip ANSI/control characters before analysis. Log the gh stderr per run instead of counting it as an API error.
   • Potential impact: High (no turn, token, or tool data for 3 straight days)
2. Fix the build-wasm golden tests and unit tests on main: CGO has been red on 7 consecutive pushes since 04:47Z.
   • Potential impact: High (every PR's CGO is red; Agent Merge can't finish)
3. Make Recompile workflows optional in coding-agent setup: .github/workflows/copilot-setup-steps.yml:36 runs make recompile || true on every session. That is ~50% of setup (20.2 of 40.6 min today), and || true hides failures.
   • Options: skip it for merge or conflict-only sessions, or let the agent run it on demand
   • Potential impact: Medium (~2 min per session)
4. Have Agent Merge explain no-op runs (Copilot product feedback): post a short status such as "remaining failures match main; nothing to fix" so maintainers don't start it again.
   • Potential impact: Medium

For Tool Development

1. Transcript access for coding-agent runs: logs are blocked by the escape-sequence refusal, and these runs produce no artifacts.
   • Frequency of need: 12 sessions today (all of them)
   • Use case: turn, token, and tool analysis plus loop detection
2. Matching sessions to runs: copilot_work_started / copilot_work_finished timeline events carry no run ID.
   • Frequency of need: 3 PRs with overlapping sessions today
   • Use case: tracing which session pushed which commit

Historical Trends and Statistical Summary

Trends Over Time

• Completion rate trend: 64.0% today against a 20-day prior mean of 52.7%. The last 5 days averaged 71.6%, the most stable stretch in the 30-day window.
• Average duration trend: 4.9 min against a prior mean of 9.0 min. Five consecutive days fell between 3.9 and 5.7 min, and the 10-26 outlier (35.6 min) has not recurred.
• Population shift: coding-agent sessions rose to 12/50 (24%). 11-01 and 11-02 each recorded 3 comment-triggered sessions, and 10-31 had none. "Agent Merge" had appeared only once each on 09-04 and 09-22; today's 9 runs are the most on record.
• Recurring patterns: no transcripts for the 3rd day in a row (cause now confirmed); main CI red for the 3rd time on record; orphan rate 0.0% on every recorded day.
• Cache note: cache-memory history had no entries for 11-01 and 11-02. They were filled in from repo-memory so the orphan baseline covers 5 days.

Statistical Summary

Total Sessions Analyzed:     50
Successful Completions:      32 (64.0%)
Failed Sessions:              6 (12.0%)
Abandoned Sessions:          10 (20.0%)  [8 skipped, 2 approval-gated]
In-Progress Sessions:         2 (4.0%)

Average Session Duration:   4.9 min
Median Session Duration:    3.3 min
Longest Session:           20.4 min  (Addressing comment, PR #74364)
Shortest Session:           0.0 min  (Squad skips)

Loop Detection:             2 sessions (4.0%)
Context Issues:             n/a (0 transcripts)
Tool Failures:              n/a (0 transcripts)

High-Quality Prompts:       1 (8.3% of 12 coding-agent sessions)
Medium-Quality Prompts:     2 (16.7%)
Low-Quality Prompts:        9 (75.0%, no prompt / UI-triggered)

Next Steps

• Restore main CGO (build-wasm golden + unit tests A–C / S–Z) before more Agent Merge runs
• Patch the transcript fetch in copilot-session-data-fetch.md to handle the escape-sequence refusal
• Evaluate making make recompile optional in copilot-setup-steps.yml
• Re-check the two long-running Addressing comment sessions (§41589764768, §41589563872)
• Schedule follow-up analysis in 24 hours (confirm or drop agent_merge_silent_noop)

References:

• §41588147254: the one Agent Merge that pushed a fix (Add safe-output identifiers and dependency failure propagation #74376)
• §41590621444: Agent Merge re-run on an unchanged SHA, no output (Report failures when activation stops before agent startup #74365)
• §41589219047: CGO on main, red at build-wasm + unit tests

---

Analysis generated automatically on 2026-11-03 at 07:15 UTC
Run ID: 41591423008
Workflow: Copilot Session Insights

Generated by 📊 Copilot Session Insights · claude · routed: opus55 xhigh · ~336 AIC · ⌖ ~2 AIC · ⊞ 14.6K · ◷

• expires on Nov 3, 2026, 11:19 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Session Insights.

A newer discussion is available at Discussion #74786.