[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-11-07

[github-actions[bot]]

Warning

Threat detection engine error

Threat detection could not complete its analysis due to a tooling error. This is not a security finding.

What happened

The threat detection results could not be parsed.

Review the workflow run logs to troubleshoot the error.

🤖 Copilot Agent Session Analysis — 2026-11-07

Executive Summary

• Sessions Analyzed: 50 (window 2026-11-07 05:44Z → 06:20Z, 5 Copilot branches)
• Analysis Period: 50-run sample from today, plus a 24h coding-agent rollup (93 coding-agent sessions)
• Completion Rate: 80.0% (40/50); 97.6% when skipped runs are excluded (40/41)
• Average Duration: 2.1 min (median 0.6 min)
• Experimental Strategy: none (roll = 83, standard run)

The one failure today came from the PR's own code, not from the base branch. CGO failed on #75615, whose base commit 466663f was green on main. The cloud-agent session that pushed the change finished about 6 minutes before CGO reported back, so the agent never saw the red result. Separately, the 60-minute cloud-agent cap was hit again, and the resulting PR (#75444) was merged with [WIP] still in its title.

Key Metrics

Metric	Value	Trend (vs 11-06)
Total Sessions	50	→
Successful Completions	40 (80.0%)	↓ from 45 (90.0%)
Failed/Abandoned	1 failure + 9 skipped (20.0%)	↑ (skips 4 → 9)
Average Duration	2.1 min (median 0.6)	↓ from 3.3 min
Loop Detection Rate	0 (0%)	→
Context Issues	not measurable (0/50 transcripts)	→

📈 Session Trends Analysis

Completion Patterns

Completion has stayed at or above 46% for 10 straight recorded days (Oct 29 → Nov 07). The 25-day mean is 57%. Hard failures plus cancellations have been 0–1 per sample for 4 straight days, down from 5–6 on Nov 02–03. Today's 9 non-successes are almost all skips (6 Squad, 3 PR Specification Reviewer). These are expected skips, not regressions.

Duration & Efficiency

Sample durations keep shrinking: the average fell from 5.0 min (11-04) to 2.1 min today. The cause is that the sample is mostly ~35-second Agentic Commands routing runs (19/50), not that agents got faster. The real coding-agent cost shows up in the 24h rollup: cloud-agent runs averaged 26.1 min and addressing-comment runs 24.6 min. No loops were detected for the 3rd sample in a row.

Success Factors ✅

1. Tightly scoped, enumerated tasks merge fast: all 4 model-routing branches (model-routing-*, route-claude-code-workflows) named the exact workflows to change. All 4 PRs merged within about 3–8h.
   • Success rate: 4/4 merged (cloud-agent mean for this group 33.4 min)
   • Example: Route three Codex-on-Copilot workflows with economy model routing instead of pinned GPT Sol models #75460 "Route three Codex-on-Copilot workflows with economy model routing"
2. Single-defect fix-* tasks: 9 fix-* cloud-agent branches. 6 merged, 2 closed, 1 open draft.
   • Success rate: 6/8 resolved PRs merged (75%)
   • Example: Accept submission metadata in work queue delivery verification #75384 "Accept submission metadata in work queue delivery verification" (13.4 min session, merged in 5h)
3. Addressing-comment follow-ups land code: 31 of 32 PRs that got an addressing-comment session also got Copilot commits (171 commits across 65 sessions). The no-op re-dispatch loop seen yesterday (souschef_identical_state_redispatch) did not come back.
   • Success rate: 63/65 sessions succeeded (2 cancelled)
4. Cloud-agent reliability: 27/28 succeeded in 24h. Of 21 resolved PRs, 19 merged (90%).

Failure Signals ⚠️

1. Agent hands off before PR CI finishes (NEW, agent_session_ends_before_ci): the Document generated permissions and preserve author comments #75615 cloud agent (run 42343254472, 32.5 min) ended at about 06:06Z. CGO then failed at 06:12Z in build-wasm (wasm golden tests) and impacted-go-tests.
   • Not inherited from the base: base 466663f was green on main (05:27Z), and sibling PRs [file-diet] Split run artifact processor into focused files #75620 and Reject run-step expressions in shell re-evaluation contexts #75366 passed CGO.
   • The pre-PR gate scripts/agent-report-progress.sh doesn't run make test-wasm-golden. A plausible cause is a change to compiler output (the PR documents generated permissions) without regenerating the golden files. This is unconfirmed because the job log is blocked.
   • Failure rate: 1/3 CGO runs in the sample
2. 60-minute cap cancel, then a [WIP] title merged (cloud_agent_60min_cap_wip, now confirmed, 2nd occurrence): copilot/self-hosted-isolate-job-runtime was cancelled at 59.4 min. The re-run succeeded at 57.8 min, also near the cap. PR [WIP] Fix cleanup and isolation for self-hosted job runtimes #75444 was then merged with the title "[WIP] Fix cleanup and isolation for self-hosted job runtimes".
   • 3/28 cloud-agent runs (11%) exceeded 45 min
3. Reply fan-out inflates CI noise (agent_reply_comment_fanout, 4th day): Copilot's review-thread replies on Reject run-step expressions in shell re-evaluation contexts #75366 triggered 7 Agentic Commands and 6 Squad runs within 2 seconds at 05:56Z. That is 13 of the 50 sampled runs (26%). Each skipped Squad run uses the agent's reply text as its run title.
4. Setup overhead stays fixed: on average, 24.3% of each cloud-agent job is setup. Recompile workflows takes 122s and Install development dependencies takes 111s, about 3.9 min per run × 28 runs ≈ 1.8 runner-hours/day.
5. Tasks the agent concluded were blocked: Document upstream blocker for imported-engine threat detection #75386 ended as a draft "Document upstream blocker for imported-engine threat detection" and was closed unmerged. Fail explicitly on unsupported Copilot BYOK tool images #75379 was closed unmerged.

Prompt Quality Analysis 📝

Per-Prompt Breakdown

Prompt text isn't available in the session data, and transcripts are blocked. Prompt quality is therefore inferred from task titles, branch names and PR outcomes for the 27 cloud-agent branches in the last 24h.

High-Quality Prompt Characteristics

• Lists the exact targets (specific workflows, files or functions): 4/4 model-routing and 2/2 spec-librarian/static-analysis tasks merged
• One defect with an observable symptom ("fix-verify-output-id", "fix-details-open-escaping"): 6/8 resolved fix-* PRs merged
• Recurring automated templates (spdd-daily-spec-work-plan, spec-librarian-*): 12.3 min and 9.5 min sessions, both merged

Example High-Quality Prompt (reconstructed from outcome):

Route Designer Drift Audit, Daily Documentation Diagram and Chaos PR Bot
through Copilot model routing (economy tier). Update frontmatter only,
recompile, and verify lock files.

Low-Quality Prompt Characteristics

• Broad, multi-part infrastructure scope ("Fix cleanup and isolation for self-hosted job runtimes"): needed 2 sessions totalling 117 min, one of them cap-cancelled
• Depends on upstream behavior the agent can't change: Document upstream blocker for imported-engine threat detection #75386 ended with the agent documenting the blocker instead of fixing it

Example Low-Quality Prompt:

Fix cleanup and isolation for self-hosted job runtimes.

Orphaned Branch Escalation Alerts 🚨

Branches with ≥5 simultaneous gate firings and no Copilot agent assigned for >2 hours.

Summary

• Orphaned Branches Today: 0 out of 9 open PRs (0.0%)
• Historical Baseline: 0.0% orphaned rate (30-day mean over 9 recorded days)
• Status: NORMAL

Escalation Candidate Details

Escalation Candidates

✅ No orphaned branches exceed the escalation threshold today. All 4 in-progress runs from the last 6 hours are on main, and every one of the 9 open PRs has Copilot assigned.

CI Waste Estimate

• Orphaned gate-hours today: 0
• Recoverable capacity: none needed

Notable Observations

Loop Detection and Session Diagnostics

Loop Detection

• Sessions with loops: 0 (0%)
• Average loop count: 0
• Common loop patterns: none. Repeated sessions on the same PR all produced commits (e.g., Route Designer Drift Audit, Daily Documentation Diagram and Chaos PR Bundle Fuzzer with economy model routing on Copilot #75461: 7 addressing sessions → 13 Copilot commits; Route four pinned Codex workflows with cost/economy model routing #75470: 5 → 17).

Tool Usage

• Most used tools: not observable. Job-log transcripts were blocked for the 3rd consecutive day (gh api .../jobs/{id}/logs returns DIFC policy violation (HTTP 403)), and the pre-fetched logs/ directory was empty.
• Step timing (proxy): cloud-agent Processing Request averaged 19.6 min out of a 25.9 min job
• Missing tools: none reported

Context Issues

• Sessions with confusion: not measurable without transcripts
• Clarification requests: 0 observable

Workflow mix in sample

Workflow	Runs	Success	Skipped	Failed
Agentic Commands	19	19	0	0
Squad	6	0	6	0
Code scanning AI findings	5	5	0	0
PR Specification Reviewer	4	1	3	0
CGO	3	2	0	1
CWI	3	3	0	0
Other (7 workflows)	10	10	0	0

Experimental Analysis

Standard analysis only - no experimental strategy this run (roll = 83).

Actionable Recommendations

For Users Writing Task Descriptions

1. List the targets explicitly: name the exact workflows or files to change.
   • Before: "Fix cleanup and isolation for self-hosted job runtimes" → After: "In <file>, (a) remove the job-runtime directory in post-steps, (b) run as the effective runner user; add regression tests in <test file>"
2. Split broad infrastructure work into sub-tasks of about 30 minutes each. Runs over 45 min (11% today) are the ones at risk of the 60-min cap.
3. Say up front what is out of scope when a fix depends on upstream behavior, so the agent doesn't spend the session documenting a blocker (Document upstream blocker for imported-engine threat detection #75386).

For System Improvements

1. Add make test-wasm-golden to the change-scoped agent-report-progress gate whenever pkg/workflow compiler output changes. Alternatively, have the agent wait for or poll the PR's CGO result before ending the session.
   • Potential impact: High (prevents red pushes like Document generated permissions and preserve author comments #75615 that need another agent round)
2. Strip or block [WIP] titles at merge after a cap-cancelled session. A merge-time title check or a Sous Chef rule would have caught [WIP] Fix cleanup and isolation for self-hosted job runtimes #75444.
   • Potential impact: Medium
3. Filter Copilot's own review replies out of the Squad and Agentic Commands triggers. Today they produced 13/50 runs of noise, and this is the 4th day the pattern has appeared.
   • Potential impact: Medium
4. Cache the Recompile workflows and dev-dependency setup steps for the coding agent (≈ 3.9 min per run, ≈ 1.8 runner-h/day).
   • Potential impact: Medium

For Tool Development

1. Capture transcripts in the pre-agent fetch step: the in-sandbox log fetch is blocked by policy (DIFC 403, 3 days running), so behavioural metrics are unavailable.
   • Frequency of need: 50/50 sessions
   • Use case: tool success rates, loop and context-confusion detection

Historical Trends and Statistical Summary

Trends Over Time

• Completion rate trend: 10 consecutive recorded days ≥ 46%. Last 5 days: 64 → 64 → 78 → 64 → 90 → 80%. The 25-day mean is 57%.
• Average duration trend: 5.0 → 1.8 → 3.3 → 2.1 min. Samples are dominated by short routing runs.
• Quality improvement: hard failures fell from 5–6 per sample (11-02/03) to 0–1 (11-04 → 11-07). The red-main inheritance seen on 11-01 to 11-03 did not recur, because main CGO recovered by 05:27Z.
• Coding agents (24h): cloud agent 28 runs vs 11 yesterday. Mean duration rose from 18.7 to 26.1 min. Merged PRs rose from 7 to 19.

Statistical Summary

Total Sessions Analyzed:     50
Successful Completions:      40 (80.0%)
Failed Sessions:             1 (2.0%)
Abandoned Sessions:          0 (0.0%)
Skipped Sessions:            9 (18.0%)
In-Progress Sessions:        0 (0.0%)

Average Session Duration:   2.1 min
Median Session Duration:    0.6 min
Longest Session:           10.9 min
Shortest Session:          0.03 min

Loop Detection:            0 sessions (0%)
Context Issues:            n/a (0 transcripts)
Tool Failures:             n/a (0 transcripts)

24h Cloud Agent:           28 runs, 27 ok, 1 cap-cancel; mean 26.1 min, median 25.4 min
24h Addressing Comment:    65 runs, 63 ok, 2 cancelled; mean 24.6 min
24h Cloud-Agent PRs:       19 merged, 2 closed unmerged, 6 open

Next Steps

• Decide whether test-wasm-golden belongs in the change-scoped pre-PR gate
• Rename [WIP] Fix cleanup and isolation for self-hosted job runtimes #75444 (merged with [WIP] title) and add a merge-time [WIP] guard
• Filter Copilot-authored review replies from the Squad/Agentic Commands triggers
• Move transcript capture into the pre-agent fetch step (DIFC block, 3rd day)
• Follow-up analysis in 24h

---

Analysis generated automatically on 2026-11-07 at 07:10 UTC
Run ID: 42347916061
Workflow: Copilot Session Insights

Generated by 📊 Copilot Session Insights · claude · routed: opus55 high · ~184 AIC · ⌖ ~2 AIC · ⊞ 14.6K · ◷

• expires on Nov 7, 2026, 11:10 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Session Insights.

A newer discussion is available at Discussion #75858.