[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-11-08

[github-actions[bot]]

🤖 Copilot Agent Session Analysis — 2026-11-08

Executive Summary

• Sessions Analyzed: 103 coding-agent sessions (last 24h), plus step-level timing for the 24-session fetch sample
• Analysis Period: 2026-11-07 06:45 UTC → 2026-11-08 06:45 UTC (30-day trend: 1,951 sessions since 2026-10-09)
• Completion Rate: 97.0% (97 of 100 completed; 0 failures, 3 cancelled, 3 still running)
• Average Duration: 23.7 min (median 22.2 min)
• Experimental Strategy: none this run (standard analysis)

Important

Methodology change: today's completion rate can't be compared with earlier reports. The fetch module now selects only dynamic/copilot-swe-agent runs, so CI gates, Squad, and review bots are excluded. Earlier reports used a 50-run mix that included CI gates and ran 64–90%. The new series is 95–97% because it measures the agent itself.

The sample also shrank from 50 to 24 sessions. The repo-wide actions/runs?created=>=... query stops at 1,000 results: page 10 is full, page 11 is empty, and the oldest returned run is 00:46:09Z. In this repository that covers only about 6 hours. I filled the gap by querying the workflow directly (actions/workflows/180923337/runs).

Key Metrics

Trend arrows compare the last 24h with the previous 24h, using the same coding-agent-only method.

Metric	Value	Trend
Total Sessions	103 (45 cloud agent, 58 addressing comment)	↑ (93)
Successful Completions	97 (97.0% of completed)	→ (96.8%)
Failed/Abandoned	3 cancelled, 0 failed (3.0%)	→ (3.2%)
Average Duration	23.7 min	↓ (25.0 min)
Loop Detection Rate (repeat sessions)	16 (15.5%)	↑ (11.8%)
Context Issues	2 (3.4% of addressing sessions)	n/a (new metric)

📈 Session Trends Analysis

Completion Patterns

Coding-agent completion has held at 89.5–100% on every day in the last 30 except one: Oct 15 fell to 71.7%, with 13 of 46 runs failed or cancelled. The 30-day rate is 95.8% and the 7-day rate is 96.0%. Volume rises and falls with batch-assignment days (147 on Oct 9, 106 on Nov 3), but completion doesn't drop on high-volume days.

Duration & Efficiency

Mean daily duration has stayed between 18 and 29 minutes with no drift (30-day mean 23.1 min, median 20.9). Repeat dispatches spiked on Oct 9 (47) and Nov 5 (34; that spike was the #74985 Sous Chef redispatch loop). The last 24h had 16, mostly from maintainers iterating on #74077, #75659 and #75491.

The charts use two stacked panels instead of a secondary y-axis, and blue/orange instead of green/red. Green/red fails the deuteranopia (red-green colorblind) separation check.

Success Factors ✅

1. Specific prompts that state the desired outcome: the prompt names the artifact and the end state.
   • Commit rate: 90% (9 of 10 high-specificity human prompts)
   • Example: otel.md should not leak implementation details ... use a typespec model to generate js and go constants → commit 49ea314a in 20.4 min
2. Structured Sous Chef work lists: numbered requests (make merge-main, "Fix failing check X").
   • Commit rate: 100% of the 24 Sous Chef sessions that weren't interrupted (24 of 26 completed; the other 2 were cancelled when the PR merged)
   • Example: fix(linters): see through parentheses in tolowerequalfold and bufioscannererunchecked #75816, merge main plus a failing check → 2 commits in 20.4 min
3. Repository routines named in the prompt ("merge main, fix conflicts, recompile", "run pr-finisher skill", "fix ci failures").
   • Commit rate: 76.9% (10 of 13). All 3 misses were the stale-conflict prompt; excluding those it's 100%.
   • Example: run pr-finisher skill. Ensure aliases are supported → commit in 34.6 min
4. Cloud-agent tasks lead to merges: 34 of 44 cloud-agent branches from the last 24h are already merged (77.3%), 7 are still open, and 2 were closed without merging.
   • Success rate: 44 of 45 cloud sessions succeeded
5. The Sous Chef redispatch loop is gone: no PR received more than one identical Sous Chef post (27 PRs checked). On 11-06, 15 of 32 Sous Chef replies were no-ops; today it's 0 of 26.

Failure Signals ⚠️

1. Merge-main dominates Sous Chef dispatches: all 29 Sous Chef dispatches requested make merge-main, and 22 requested nothing else.
   • Cost: 19 completed merge-main-only sessions used 472.7 runner-minutes (about 20% of all coding-agent runner time in the last 24h)
   • Example: [linter-miner] Add splitlines-on-newline custom analyzer #75820, Improve safe-output helper test coverage #75821 and Suppress static-analysis false positives in shared workflow setup #75822 were opened at 04:20Z and all got merge-main dispatches at 06:41Z, about 2.3h later
2. Stale merge-conflict prompt: the canned prompt resolve the merge conflicts on this branch. was sent when the branch already had the latest merge.
   • No-op rate: 60% (3 of 5 uses)
   • Example: Remediate Codex budget and workflow failure boundaries #75491 got the prompt 4 times; Copilot answered "no unresolved merge conflicts, MERGE_HEAD absent" 3 times (16.2 runner-min)
3. Merge cancels an in-flight session: the maintainer merged the PR while a Sous Chef session was still merging main.
   • Affected: 2 of 29 Sous Chef sessions (6.9%), 28.4 runner-min
   • Example: [duplicate-code] refactor: share repeated harness output pattern matchers #75757 merged at 23:07:01Z; session cancelled at 23:07:08Z
4. 60-minute session cap: §42390680999 was cancelled at 59.4 min. It left [WIP] draft [WIP] Fix ignored errors and discarded results in lint findings #75705, which was later closed without merging.
   • Risk: 5 sessions in the last 24h ran 45 minutes or longer (4.9%); there were 7 cap cancellations in the last 7 days
5. Commit pushed but no PR opened: §42443139952 (copilot/update-stylus-workflow) succeeded and pushed 6eff5d445f, but the branch has no PR, so the work is stranded.
   • Frequency: 1 of 44 cloud-agent branches (2.3%)

Prompt Quality Analysis 📝

Per-Prompt Breakdown

I matched each of the 58 addressing-comment sessions to the latest non-Copilot @copilot comment in the 15 minutes before the run started (57 matched). The outcome is "commit" if Copilot pushed a commit during the run (plus 3 minutes), "reply only" if it only commented, and "silent" otherwise.

Prompt source	Sessions	Commit landed	Reply only	Cancelled	Running	Mean min
Human maintainer	27	22 (81.5%)	5	0	0	21.8
PR Sous Chef (gh-aw-bot)	29	24	0	2	3	25.8
Review bot (github-actions[bot])	1	1	0	0	0	36.4
Unattributed	1	1	0	0	0	12.9

How human prompts landed by specificity:

Specificity	Prompts	Commit landed
High: names an artifact and an outcome	10 (37.0%)	9 (90.0%)
Medium: a routine or command	14 (51.9%)	11 (78.6%)
Low: vague, or relies on missing context	3 (11.1%)	2 (66.7%)

Prompt length didn't predict the outcome: all 7 prompts under 40 characters landed commits.

High-Quality Prompt Characteristics

• States a concrete outcome or constraint: 9 of 22 (40.9%) human commit sessions
• Names a repository routine (merge main, recompile, pr-finisher, fix CI): 10 of 22 (45.5%) human commit sessions
• Numbered, machine-generated work list (Sous Chef): 24 of 24 Sous Chef sessions that weren't interrupted

Example High-Quality Prompt:

use typespec compiler instead of .py script

Result: commit a079c5dd in 14.4 min. It replaced the Python parser with the pinned TypeSpec compiler and emitter.

Low-Quality Prompt Characteristics

• Asks about state that no longer exists (stale conflict): 3 of 5 (60%) no-commit sessions
• Ambiguous term: 1 of 5 (20%)
• Constraint conflicts with the existing code: 1 of 5 (20%). On Exclude vendored YAML from ESLint custom rules #75806, Copilot explained why the vendored parser was needed; the follow-up prompt landed a commit 30 minutes later.

Example Low-Quality Prompt:

fix typescript configuration for otel

Result: Copilot confirmed the TypeScript config already passes typecheck and asked whether the prompt meant TypeSpec (8.5 min, no change).

Orphaned Branch Escalation Alerts 🚨

Branches with ≥5 simultaneous gate firings and no Copilot agent assigned for >2 hours.

Summary

• Orphaned Branches Today: 0 of 13 open PR branches (0.0%)
• Historical Baseline: 0.0% orphaned rate (30-day mean over 10 recorded days)
• Status: NORMAL

Escalation Candidate Details

Escalation Candidates

✅ No orphaned branches exceed the escalation threshold today. Of the 9 in-progress runs from the last 6 hours, 6 are on main and 3 are addressing-comment sessions (one each on #75820, #75821 and #75822). No PR branch has more than 1 active run.

CI Waste Estimate

• Orphaned gate-hours today: 0 gates × 0h ≈ 0 CI-minutes wasted
• Recoverable capacity: none needed. Every open Copilot PR has Copilot assigned. The only open PRs without an assignee are 3 bot PRs ([Safe Outputs Conformance] WTD-002: Accept current threat-caution delegation in conformance checker #75841, [ca] chore: update Claude Code CLI to 2.1.318 #75838, [model-inventory] Model alias inventory update - 2026-11-07 #75775) with no active gates.

Notable Observations

Loop Detection and Session Diagnostics

Loop Detection

• Sessions with loops: 16 of 103 (15.5%). A repeat session is the 3rd or later addressing-comment run on the same branch in one UTC day.
• Average loop count: 4.8 addressing sessions per repeating branch-day (6 branch-days; on 11-07, Remediate Codex budget and workflow failure boundaries #75491 had 8, Add declarative connectors and migrate native provider integrations #74077 had 6 and [otel-reference] Expand OpenTelemetry reference guide #75659 had 5)
• Common loop patterns: Most repeats were productive, with a maintainer iterating on a feature ([otel-reference] Expand OpenTelemetry reference guide #75659: TypeSpec migration over 5 sessions, 4 with commits). There was 1 real stuck loop: Remediate Codex budget and workflow failure boundaries #75491's repeated stale-conflict prompt.

Tool Usage

• Most used tools: unavailable. There were 0 transcripts for the 24 sampled sessions (no events.jsonl artifacts, fourth consecutive day), so I used step timing instead.
• Setup overhead: 23.8% of session time, about 5.4 min of the 22.8-min sample mean. That's 27.8% for cloud-agent sessions and 20.3% for addressing sessions. The biggest steps are Recompile workflows (mean 145s, max 226s) and Install development dependencies (77s). The worst case was the Playwright CLI bump: 6.6 min of setup for 1.6 min of agent work (78.1%).
• Missing tools: none observable without transcripts

Context Issues

• Sessions with confusion: 2 of 58 addressing sessions (3.4%)
• Common confusion points: TypeScript vs TypeSpec ([otel-reference] Expand OpenTelemetry reference guide #75659); removing a vendored module that runtime code depends on (Exclude vendored YAML from ESLint custom rules #75806)
• Clarification requests: 1. The other was reasoned pushback with an alternative.

Experimental Analysis

Standard analysis only; no experimental strategy this run.

Actionable Recommendations

For Users Writing Task Descriptions

1. Check branch state before sending the canned conflict prompt: if the PR shows a conflict, name the file, or ask for a fresh merge.
   • Before: resolve the merge conflicts on this branch. (no-op 3 of 4 times on Remediate Codex budget and workflow failure boundaries #75491)
   • After: merge latest main; conflict is in .github/workflows/daily-storify.lock.yml, regenerate it from source
2. Spell out terms that are easy to confuse: say which config file or system you mean.
   • Before: fix typescript configuration for otel
   • After: add a TypeSpec config (tspconfig.yaml) for the otel model and wire it into make build
3. When removing a dependency, say what replaces it.
   • Before: remove the vendored copy ... no third-party dependencies at runtime (Copilot pushed back)
   • After: remove the vendored YAML parser; don't parse workflows, validate immutable file hashes instead (landed on the next prompt)

For System Improvements

1. Fix the session fetch cap: in .github/workflows/shared/copilot-session-data-fetch.md, change line 129 to query actions/workflows/<copilot-swe-agent id>/runs, split into date windows under 1,000 results each.
   • Potential impact: High. The sample returns to 50 sessions instead of about 6 hours' worth.
2. Stop dispatching Copilot for merge-main-only work: let Sous Chef run a deterministic merge-and-recompile job (or call the update-branch API) and call Copilot only when real conflicts or failing checks remain.
   • Potential impact: High. About 470 runner-minutes a day, roughly 20% of coding-agent time.
3. Skip or cancel Sous Chef dispatch on PRs that are approved or about to be merged.
   • Potential impact: Medium. 2 cancelled sessions (28.4 runner-min) and less churn on PRs that are about to merge.
4. Reduce setup time: skip or cache Recompile workflows when the request doesn't touch .github/workflows/*.md.
   • Potential impact: Medium. About 145s × 103 sessions ≈ 4.1 runner-hours a day.
5. Guard the 60-minute cap: split tasks that run past 45 minutes, or open a PR from checkpointed progress before the cap.
   • Potential impact: Medium. 7 cap cancellations in 7 days, each leaving a [WIP] draft.

For Tool Development

1. Session event artifacts or transcript access: there are no events.jsonl artifacts and the job-log fallback returns nothing.
   • Frequency of need: 24 of 24 sessions (fourth consecutive day)
   • Use case: tool success rates, turn counts, and detecting real loops
2. "Branch already up to date" pre-check for conflict prompts.
   • Frequency of need: 3 sessions
   • Use case: answer the stale conflict request without starting a 5-minute agent session
3. Detection of commits without a PR.
   • Frequency of need: 1 session
   • Use case: catch work like copilot/update-stylus-workflow that's stranded on a branch

Historical Trends and Statistical Summary

Trends Over Time

• Completion rate trend: coding-agent-only completion is flat and high: 30-day 95.8%, 7-day 96.0%, 24h 97.0%. The old mixed metric (64–90% from 10-30 to 11-07) isn't continued.
• Average duration trend: stable. 30-day mean 23.1 min, 7-day 22.6, 24h 23.7; the previous 24h was 25.0.
• Quality improvement: the Sous Chef no-op rate fell from 46.9% on 11-06 to 0% today. The cloud-agent merge rate is 77.3% within 24h.

Statistical Summary

Total Sessions Analyzed:     103
Successful Completions:      97 (94.2%; 97.0% of completed)
Failed Sessions:             0 (0.0%)
Abandoned Sessions:          3 (2.9%)  [1 at the 60-min cap, 2 cancelled by PR merge]
In-Progress Sessions:        3 (2.9%)

Average Session Duration:    23.7 min
Median Session Duration:     22.2 min
Longest Session:             59.4 min (cancelled at cap; longest success 58.8 min)
Shortest Session:            4.6 min

Loop Detection:              16 sessions (15.5%)  [1 real stuck loop]
Context Issues:              2 sessions (1.9%)
Tool Failures:               n/a (no transcripts)

High-Quality Prompts:        10 (37.0% of 27 human prompts)
Medium-Quality Prompts:      14 (51.9%)
Low-Quality Prompts:         3 (11.1%)

Next Steps

• Fix the 1,000-result cap in copilot-session-data-fetch.md (use the workflow-scoped runs endpoint)
• Decide whether Sous Chef should handle merge-main-only work without Copilot
• Look into missing events.jsonl artifacts for coding-agent sessions
• Schedule a follow-up analysis in 24h to confirm the coding-agent-only baseline

---

Analysis generated on 2026-11-08 at 06:55 UTC

References:

• §42455621833: this analysis run
• §42390680999: cloud-agent session cancelled at the 60-minute cap
• §42443139952: cloud-agent session that pushed a commit with no PR

Generated by 📊 Copilot Session Insights · claude · routed: opus55 xhigh · ~348 AIC · ⌖ ~6 AIC · ⊞ 14.6K · ◷

• expires on Nov 8, 2026, 11:00 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Session Insights.

A newer discussion is available at Discussion #76171.