[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-10-31

[github-actions[bot]]

🤖 Copilot Agent Session Analysis — 2026-10-31

Executive Summary

• Sessions Analyzed: 50 (15 with full events.jsonl transcripts, the best coverage recorded so far)
• Analysis Period: 2026-10-31 02:27Z – 03:21Z (55 min, 5 branches, all copilot/model-routing-*)
• Completion Rate: 90.0% (45/50). This is the highest on record (previous max 80.0% on 10-21; prior 19-day mean 47.7%)
• Average Duration: 5.7 min (median 7.5 min)
• Experimental Strategy: none (standard run, roll = 82)

Important

The 90% figure counts zero coding-agent task sessions (provenance_inversion). All 45 successes are PR gates, reviewers, and dispatched validation runs that fanned out from five Copilot-authored model-routing PRs (#73384–#73388). All five PRs merged. The real signal today comes from the 15 reviewer transcripts. They show 75% first-try safe-output success, one false-positive REQUEST_CHANGES (later dismissed), and 15.5% of AI credits spent on 5 no-op test reviews.

Key Metrics

Metric	Value	Trend (vs 10-30)
Total Sessions	50	→
Successful Completions	45 (90.0%)	↑ from 35 (70.0%)
Failed/Abandoned	0 failed, 5 skipped (10.0%)	↓ from 15
Average Duration	5.7 min (median 7.5)	↑ from 5.4 (median 2.5)
Loop Detection Rate	1 session (2.0%; 1/15 transcripts)	→
Context Issues	1 (1/15 transcripts, 6.7%)	new metric
Transcripts / AI credits	15 / 359.7 AIU (24.0 per transcript)	↑ from 6 / 200.5 (33.4 per transcript)
Safe-output first-try success	18/24 attempts (75.0%), eventual 100%	new metric

📈 Session Trends Analysis

Completion Patterns

Completion has risen three snapshots in a row (46% → 70% → 90%) and is now 42 points above the prior mean. The rise reflects the workflow mix more than better agent behaviour. Today's window held only gate and reviewer workflows, which conclude success deterministically. There were none of the action_required Squad runs or smoke-matrix bundles that pulled 10-24 (4%) and 10-28 (8%) down.

Duration & Efficiency

The median (7.5 min) is above the mean (5.7 min) for the first time since 10-21. Most runs are ~8-minute agentic PR gates, and the 1-second Squad skips pull the mean down. Agent time is a small share of job time: Test Quality Sentinel averaged 0.26 min of agent activity inside an 8.5-min job, so a no-op gate is dominated by setup, detection, and conclusion overhead.

Charts use stacked panels with one y-scale each (no dual axis). Successful is blue and non-successful is orange, because the red/green pair fails the deuteranopia check (ΔE 4.1). Oct 4–13 has no analysis snapshots. On 10-29 the later of two snapshots is plotted.

Success Factors ✅

1. Prefetched PR context with a cheap model: PR Code Quality Reviewer (gpt-5.6-luna plus a haiku-5.5 grumpy-coder subagent) reads the prefetched pr-diff.patch instead of reconstructing git history.
   • Success rate: 5/5 reviews posted at 1.9 AIU and 0.77 agent-min on average, 30× cheaper than Matt Pocock Skills Reviewer (58.9 AIU)
   • Example: §41160124704 reviewed PR Route Design Decision Gate with model routing instead of pinning claude-opus-5.5 #73384 in 8 turns for 0.93 AIU
2. Deterministic gates stay deterministic: Stale Lock Files and PR Data Prefetch.
   • Success rate: 10/10, average 0.26 min
   • Example: Stale Lock Files verified the terminal-stylist lock in 21 s
3. Rich, quantified PR descriptions: all 5 Copilot-authored PR bodies (1,970–2,621 chars) lead with per-job AI-credit cost, name the detection and evals jobs, and mention the lock file.
   • Success rate: 5/5 merged; Matt Pocock gave 4 APPROVE and 1 COMMENT
   • Example: Route Design Decision Gate with model routing instead of pinning claude-opus-5.5 #73384 opens with "The pin also applies to threat detection and evals, which make up about three quarters of its roughly 79 AIC per run."
4. Heredoc / file fallback for safe outputs: every failed submission was recovered.
   • Success rate: 6/6 failed attempts recovered (eventual write success 100%)
   • Example: §41160124693 succeeded with cat <<'PAYLOAD' | safeoutputs submit_pull_request_review . after inline JSON failed
5. Cheap triage subagent: pr-triage on claude-haiku-4.5 classified each PR for 0.7–2.1 AIU, a small share of each Matt Pocock run's cost.

Failure Signals ⚠️

1. Inline single-quoted JSON breaks on apostrophes (inline_safeoutputs_quote_breakage, 3rd consecutive day)
   • Failure rate: 3/5 Matt Pocock runs failed the first submission (syntax error near unexpected token '(', exit 2); 4/10 Matt Pocock submission attempts failed overall
   • Example: the body text this isn't a novel/untested pattern ended the shell quote. One retry also hit -32602 unknown parameter 'comments'
2. Smart quote corrupted into a control character (NEW smart_quote_control_char)
   • Failure rate: 2/5 PR Code Quality Reviewer runs. gpt-5.6-luna emitted raw U+0019 where ’ (U+2019) belonged, so safeoutputs rejected the payload with Bad control character in string literal
   • Example: the router\x19s av... in §41160155864 and repository\x19s existing in §41160070746
3. Read-only reviewer runs a full rebuild (NEW reviewer_heavy_build_in_review)
   • Failure rate: 1/5 Matt Pocock runs, adding 10.7 min. This was the longest session today (19.2 min job)
   • Example: §41160201422 ran git stash && make recompile, polled read_bash 5 times (30–180 s each), then killed it. The interrupted recompile truncated agentics-maintenance.yml in the sandbox (−968 lines), and the agent had to revert it. This duplicated the Stale Lock Files gate, which had already passed
4. Lock-file job-scope misread produces a false REQUEST_CHANGES (NEW false_positive_request_changes_job_scope)
   • Failure rate: 1/5 PR Code Quality Reviewer verdicts (20%)
   • Example: on Route Daily Go Function Namer via model routing instead of pinning claude-sonnet-5 #73385 the reviewer read the evals job's GH_AW_PI_MODEL as the agent job's model and claimed routing was not applied. It saw the same head (3db7fe7c2) as the maintainer's rebuttal, so this was not a stale-head race. The review was dismissed and the PR merged
5. Bash allowlist friction (NEW bash_allowlist_cut_denial)
   • Failure rate: 5/5 PR Code Quality Reviewer runs had exactly one denied call: cut ×3, python3 ×1, [ ... ] ×1. One Test Quality Sentinel run was also denied a test -f ... || echo probe

Prompt Quality Analysis 📝

Per-Prompt Breakdown

The "prompts" today are the five Copilot-authored PR descriptions that every reviewer read. All five rate high quality (5/5).

High-Quality Prompt Characteristics

• States the per-job cost breakdown (agent vs detection vs evals) with numbers: 5/5 PRs, all merged
• Names the lock file and the jobs affected by the change: 5/5
• Includes the literal frontmatter change in a fenced yaml block: 4/5. The only PR without one (Route Daily Go Function Namer via model routing instead of pinning claude-sonnet-5 #73385) is the only PR that drew a false REQUEST_CHANGES (n = 1, suggestive only)

Example High-Quality Prompt (PR #73384, excerpt):

Design Decision Gate pins `copilot/claude-opus-5.5`. The pin also applies to
threat detection and evals, which make up about three quarters of its roughly
79 AIC per run. This PR switches the workflow to model routing (goal: cost,
mode: balanced). Detection and evals then fall back to their default models...

Low-Quality Prompt Characteristics

• No low-quality PR descriptions in today's sample
• Gap seen in reviewer behaviour, not in the prompt: no PR body cited the post-change dispatched run as validation. The rebuttal on Route Daily Go Function Namer via model routing instead of pinning claude-sonnet-5 #73385 cited §41160411938 (routed to gpt-5.6-luna). Putting that evidence in the PR body would likely have prevented both the false REQUEST_CHANGES and the 10.7-min self-verification rebuild

Orphaned Branch Escalation Alerts 🚨

Branches with ≥5 simultaneous gate firings and no Copilot agent assigned for >2 hours.

Summary

• Orphaned Branches Today: 0 out of 9 open PRs (0.0%)
• Historical Baseline: insufficient history. Only 2 recorded days carry orphan_rate (10-29, 10-30, both 0.0%). The 10-30 report's "30-day baseline" was a miscount
• Status: NORMAL (0.0% is below the 50% fallback threshold)

Escalation Candidate Details

Escalation Candidates

✅ No orphaned branches exceed the escalation threshold today. 4 in-progress runs; the most on any PR branch was 1 (PR #73452). 4/9 open PRs have Copilot assigned. The 5 model-routing branches analyzed above were all merged before this check ran.

CI Waste Estimate

• Orphaned gate-hours today: 0 gates × 0 h ≈ 0 CI-minutes wasted
• Recoverable capacity: n/a

Notable Observations

Loop Detection and Session Diagnostics

Loop Detection

• Sessions with loops: 1 (2.0% of sessions; 6.7% of transcripts)
• Average loop count: 5 polling iterations (read_bash on one shell) in §41160201422
• Common loop patterns: wait-polling on a long build. No identical tool call repeated ≥3× in any transcript (max 2). Safe-output retry chains (2–3 attempts in 5 runs) are counted as error recovery, not loops

Tool Usage

• Most used tools: bash 222 (70%), view 42, grep 16, task 10, read_bash 9, glob / read_agent / task_complete 5 each, create 2, stop_bash 1. Total 317 calls in 274 turns
• Tool success rates: 98.1% reported (6 denials) vs 95.6% effective. 18 bash calls exited non-zero yet reported success=true (hidden_shell_failure_masking): 8 genuine (6 safe-output submits, 2 git diff HEAD~1 exit 128 in a shallow clone) and 10 benign (grep no-match, absent-marker probes)
• Lock-file spelunking: 8/10 reviewer sessions spent 19 calls grepping .lock.yml for per-job model or routing settings
• Missing tools: none requested. Denied commands: cut, python3, [, test

Context Issues

• Sessions with confusion: 1 (the Route Daily Go Function Namer via model routing instead of pinning claude-sonnet-5 #73385 job-scope misread)
• Common confusion points: multi-job lock files (agent vs detection vs evals env); git base-commit discovery in partial clones (promisor remote and AWF CA errors in §41160070697)
• Clarification requests: 0 (autopilot)

Cost Concentration

Workflow	Runs	AIU (share)	AIU/run	Turns/run	Agent min / job min
Matt Pocock Skills Reviewer	5	294.3 (81.8%)	58.9	34.6	5.87 / 12.14
Test Quality Sentinel	5	55.7 (15.5%)	11.1	6.6	0.26 / 8.49
PR Code Quality Reviewer	5	9.7 (2.7%)	1.9	13.6	0.77 / 8.48

Matt Pocock AIU/run fell from 90.4 (10-30) to 58.9. Each run still re-reads ~42k tokens of context per turn (95% cache hits).

Experimental Analysis

Standard analysis only. No experimental strategy this run (roll = 82 ≥ 30).

A workflow-level experiment was visible in the data. Test Quality Sentinel's model_size A/B (issue #43530) served claude-sonnet-5 3× (13.4 AIU, 4 turns each) and claude-haiku-4.5 2× (7.8 AIU, 7–14 turns). Haiku took 2–3.5× more turns, including a self-verification checklist, but cost 42% less per no-op. Both arms reached the same correct decision.

Actionable Recommendations

For Users Writing Task Descriptions

1. Put the literal frontmatter change in a fenced yaml block: reviewers anchor on it instead of grepping 1,000+-line lock files.
   • Example: Before: "switches to model routing (goal: cost)". After: a fenced block with engine: { id: pi, model-routing: { goal: cost, mode: economy } }
2. Cite the post-change validation run in the PR body: this removes the reviewer's urge to rebuild.
   • Example: Before: "Recompiled the lock file." After: "Dispatched run §41160411938 on this branch routed the agent to gpt-5.6-luna; evals/detection keep their default aliases."
3. Say which job keeps which model: lock files contain several jobs, so state it per job.
   • Example: "GH_AW_PI_MODEL: evals remains only in the evals job; the agent job uses modelRouting."

For System Improvements

1. Skip Test Quality Sentinel deterministically when test-files.txt is empty: the prefetch already computes it. 5/5 runs today were no-ops costing 55.7 AIU and ~42 runner-min. Also fix the still-present gh pr diff "$PR_NUMBER" -- <files> call at test-quality-sentinel.md:104-105 (verified: accepts at most 1 arg(s)). It will silently no-op the next PR that changes tests.
   • Potential impact: High
2. Harden safe-output submission: tell Matt Pocock Skills Reviewer to use the field-stdin form (cat body.md | safeoutputs submit_pull_request_review --body .). Have the safeoutputs CLI reject or normalize C0 control characters with a pointed hint ("U+0019 found, did you mean ’?"). 6/24 first attempts failed across 5/10 reviewer runs.
   • Potential impact: High
3. Forbid build commands in read-only reviewers: add guidance in mattpocock-skills-reviewer.md that make and git stash are out of scope and lock-file freshness belongs to the Stale Lock Files gate.
   • Potential impact: Medium (10.7 min and a sandbox mutation in 1/5 runs)
4. Add cut to the PR Code Quality Reviewer bash allowlist: this removes 3 of its 5 denials.
   • Potential impact: Low (1 turn per run)
5. Surface shell exit codes in tool.execution_complete.success: 18 masked non-zero exits make reported tool health 2.5 points rosier than reality.
   • Potential impact: Medium (observability)

For Tool Development

1. Per-job model/engine summary in the PR prefetch: a jobs → model / routing / env table derived from the lock file.
   • Frequency of need: 8 sessions (19 tool calls)
   • Use case: verifying model-routing changes without grepping multi-job YAML; would have prevented the Route Daily Go Function Namer via model routing instead of pinning claude-sonnet-5 #73385 false positive
2. Base-commit diff without git history: expose base SHA plus the prefetched patch path prominently, or deepen the fetch.
   • Frequency of need: 1 session today (2 exit-128 failures, promisor and CA errors); ~11 turns in 1 session on 10-30
   • Use case: git diff HEAD~1 in shallow or partial clones

Historical Trends and Statistical Summary

Trends Over Time

• Completion rate trend: 46.0% (10-29 eve) → 70.0% (10-30) → 90.0% (10-31). The new high is driven by the workflow mix (gate-only window), not by coding-agent outcomes
• Average duration trend: 1.7 → 5.4 → 5.7 min; median 0.1 → 2.5 → 7.5 min as skip-heavy days gave way to gate-heavy ones
• Quality improvement: transcript coverage 3 → 6 → 15. Matt Pocock AIU/run 90.4 → 58.9. inline_safeoutputs_quote_breakage persists (2/2 → 3/5). test_sentinel_prefetch_always_fails is still unfixed in source

Statistical Summary

Total Sessions Analyzed:     50
Successful Completions:      45 (90.0%)
Failed Sessions:              0 (0.0%)
Abandoned Sessions:           5 (10.0%)  [skipped Squad / Squad implement]
In-Progress Sessions:         0 (0.0%)

Average Session Duration:   5.7 min
Median Session Duration:    7.5 min
Longest Session:           19.2 min  (Matt Pocock Skills Reviewer, #73387)
Shortest Session:           0.02 min (Squad implement, skipped)

Loop Detection:             1 session (2.0%)
Context Issues:             1 session (6.7% of transcripts)
Tool Failures:              6 reported + 8 masked genuine (of 317 calls)

High-Quality Prompts:       5 (100%)  [PR descriptions, n=5]
Medium-Quality Prompts:     0 (0%)
Low-Quality Prompts:        0 (0%)

Next Steps

• Fix test-quality-sentinel.md:104-105 and add a deterministic "no test files → skip agent" gate
• Move Matt Pocock Skills Reviewer to --body . field-stdin submission; add a C0-control-char guard to safeoutputs
• Add cut to the PR Code Quality Reviewer allowlist; add a "no builds" rule to Matt Pocock Skills Reviewer
• Prototype a per-job model summary in the PR prefetch
• Schedule follow-up analysis in 1 day (track smart_quote_control_char recurrence and the first test-touching PR)

References:

• §41160201422: Matt Pocock reviewer make recompile polling loop (19.2 min)
• §41160140423: false-positive REQUEST_CHANGES on Route Daily Go Function Namer via model routing instead of pinning claude-sonnet-5 #73385
• §41160155864: U+0019 control-character safe-output failure

Analysis generated automatically on 2026-10-31 · Run ID: §41177460552

Generated by 📊 Copilot Session Insights · claude · routed: opus55 xhigh · ~311 AIC · ⌖ ~2 AIC · ⊞ 14.6K · ◷

• expires on Oct 31, 2026, 11:14 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Session Insights.

A newer discussion is available at Discussion #73734.