[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-10-30

[github-actions[bot]]

🤖 Copilot Agent Session Analysis — 2026-10-30

Executive Summary

• Sessions Analyzed: 50 runs (plus 6 full event transcripts)
• Analysis Period: 2026-10-30 00:10 UTC – 05:20 UTC (two active PR branches)
• Completion Rate: 70.0% (35/50), +24 pts vs the 10-29 evening snapshot (46.0%) and well above the 30-day mean (41.1%)
• Average Duration: 5.4 min (median 2.5 min)
• Experimental Strategy: none (roll = 93, standard run)

Important

Root-caused bug: Test Quality Sentinel never reviews PRs that change tests. Its pre-fetch step calls gh pr diff "$PR_NUMBER" -- <test files> (.github/workflows/test-quality-sentinel.md:104-109). gh pr diff accepts at most one positional argument, so this call always fails with accepts at most 1 arg(s), received 2. The step then writes test-prefetch-unavailable.txt and the agent calls noop. Both sentinel runs today hit this path, on PR #73052 (9 changed test files) and PR #73078 (1 changed test file). The same applies to every PR that changes a test file, and it fails silently: the workflow concludes success.

Key Metrics

Metric	Value	Trend (vs 10-29 evening)
Total Sessions	50	→
Successful Completions	35 (70.0%)	↑ (from 23)
Failed/Abandoned	5 (10.0%): 4 failure, 1 cancelled	↑ (from 2)
Gate-waiting / skipped	10 (20.0%): 6 action_required, 4 skipped	↓ (from 25)
Average Duration	5.4 min	↑ (from 1.7 min)
Loop Detection Rate	1 of 6 transcripts (16.7%)	→ (1 of 3)
Context Issues	1 of 6 transcripts (16.7%)	→
Tool success (reported / effective)	98.1% / 95.0% (159 calls)	↑ (95.9% / 93.2%)

📈 Session Trends Analysis

Completion Patterns

At 70%, completion is the 4th-highest of the 30 recorded days. That reflects the population, not a behavior change: today's 50 runs are mostly review bots and Agentic Commands runs on two PRs (#73052 and #73078), which usually succeed, and there was no smoke-matrix burst like the one that drove 10-28 down to 8%. Day-to-day swings of ±40 pts are still driven by which workflows land in the 50-run window. The Oct 4–13 gap is missing history, not zero.

Duration & Efficiency

Median duration (2.5 min) is back in its usual 0–5 min band. The average is pulled up by one outlier: Addressing comment on PR #73052 was cancelled at 60.3 min, a timeout-shaped end for the only long-running coding-agent session in the window. Loop tracking has existed only since 10-29, when transcripts returned; one looping session was found on each day.

Success Factors ✅

1. Small-model orchestration with a critic subagent: PR Code Quality Reviewer (gpt-5.6-luna orchestrator + claude-haiku-5.5 grumpy-coder subagent) produced substantive reviews on both PRs: REQUEST_CHANGES on Include canonical issue IDs in duplicate close intents #73078 and a COMMENT with a correctness concern on Support engine.model-routing for Claude on native Anthropic and Codex on native OpenAI #73052.
   • Success rate: 2/2 (100%), at 2.5–6.4 AIU and 16–22 turns
   • Example: flagged that the new canonical-ID path in close_issue.cjs:370 still sends a duplicate request
2. Pre-fetched PR context consumed in turn 0: all 4 reviewer runs with valid pre-fetch (pr-meta.json, pr-diff.patch) started from it, and none had to rediscover the diff via the API.
   • Success rate: 4/4 (100%) published review output
3. Independent reviewers converging: on Include canonical issue IDs in duplicate close intents #73078, the Matt Pocock Skills Reviewer and the PR Code Quality Reviewer both independently anchored comments at close_issue.cjs line 370, a strong signal that the issue is real.
4. Recovery after a failed safe-output call: when the inline safeoutputs call failed, both runs recovered by writing the JSON to a heredoc file and retrying.
   • Recovery rate: 2/2 (100%)

Failure Signals ⚠️

1. Test Quality Sentinel pre-fetch always fails when tests change (see callout above)
   • Failure rate: 2/2 sentinel runs (100% false-negative "skip")
   • Example: Support engine.model-routing for Claude on native Anthropic and Codex on native OpenAI #73052 changed model_routing_native_test.go, awf_model_routing.test.cjs and 7 other test files, and the sentinel posted noop: unable to fetch test file diff
   • Note: the 10-29 insight "sentinel spends tokens to noop because no test files changed" was likely the same bug
2. Inline single-quoted JSON breaks on apostrophes: both Matt Pocock Skills Reviewer runs' first safeoutputs submit_pull_request_review exited 2, because words like haven't and team's in the review body closed the single quote.
   • Failure rate: 2/2 (100%) first attempts; each retry costs one turn plus a large re-sent payload
3. Shell failures hidden behind success=true: 5 bash calls exited non-zero (codes 1, 2, 2, 128, 1) while tool.execution_complete reported success=true.
   • Reported 98.1% vs effective 95.0% tool success
4. Exploration overhead in skills review: one Matt Pocock run used 56 turns and 3.6M input tokens (117 AIU, 58% of today's total).
   • About 11 turns hunting a git base commit for a schema diff, ending in fatal: could not fetch ... from promisor remote (partial clone)
   • 4 turns finding that vitest/node_modules are not installed
   • It re-read the diff after a pr-triage subagent had already triaged it
5. CI failures concentrated on one branch: all 4 failure conclusions are CJS + CGO on copilot/support-engine-model-routing, on both pushes (00:10 and 01:11 UTC). The PR itself is red, not the harness.

Prompt Quality Analysis 📝

Per-Prompt Breakdown

High-Quality Prompt Characteristics

All 6 transcripts are workflow-authored prompts with explicit output contracts, so prompt text did not separate outcomes. Data and tooling did.

• Pre-fetched artefacts named by path: 6/6 (100%) of transcripts
• Explicit safe-output contract (review / comment / noop): 6/6
• Bounded subagent role (pr-triage, grumpy-coder): 3/3 of the runs that used one produced output

Example High-Quality Prompt pattern:

Read /tmp/gh-aw/agent/pr-diff.patch and pr-meta.json first. Delegate a bounded
risk-mining pass to the grumpy-coder agent; only comment on changed lines;
finish with exactly one submit_pull_request_review.

Low-Quality Prompt Characteristics

• Guard-clause instructions that trust a pre-fetch marker without surfacing it: in 2/2 failed (noop) sessions, the prompt said "if test-prefetch-unavailable.txt exists, noop", so a deterministic pre-fetch bug became a silent skip
• Open-ended "verify the schema change" with no base SHA provided: 1/1 looping session

Example Low-Quality Prompt pattern:

If the pre-fetch marker exists, read the reason and immediately call noop.

Orphaned Branch Escalation Alerts 🚨

Branches with ≥5 simultaneous gate firings and no Copilot agent assigned for >2 hours.

Summary

• Orphaned Branches Today: 0 out of 22 open-PR branches (0.0%)
• Historical Baseline: 0.0% orphaned rate (30-day mean over 30 recorded days)
• Status: NORMAL

Escalation Candidate Details

Escalation Candidates

✅ No orphaned branches exceed the escalation threshold today. Only 2 workflow runs were in progress: one on main (this workflow) and one Addressing comment on PR #73052 on copilot/support-engine-model-routing, which has Copilot assigned. The maximum gate count on any branch is 1.

CI Waste Estimate

• Orphaned gate-hours today: 0 gates × 0h ≈ 0 CI-minutes wasted
• Recoverable capacity: n/a

Notable Observations

Loop Detection and Session Diagnostics

Loop Detection

• Sessions with loops: 1 of 6 transcripts (16.7%)
• Average loop length: ~11 turns (one loop)
• Common loop pattern: repeated git diff <sha> <sha> -- schemas/ and git cat-file -e <sha> probes across 6 candidate base commits in a partial clone; no exact-duplicate commands, but no progress either

Tool Usage

• Most used tools: bash 98 (61.6%), view 39 (24.5%), grep 7, task 3, read_agent 3, glob 2, create 2, read_bash 2, task_complete 2, write_agent 1
• Tool success rates: 156/159 reported (98.1%); 151/159 effective (95.0%) after counting non-zero shell exits
• Reported failures: 1 permission denial (grep -rn ... pkg outside the bash allowlist) and 2 Path does not exist on PR-referenced smoke workflow files
• Subagents: pr-triage (haiku-4.5, 16 tool calls, 191k tokens), grumpy-coder ×2 (haiku-5.5, 4–9 tool calls). One grumpy-coder returned an empty first response and needed a write_agent nudge
• Missing tools: none requested; vitest binary absent in the reviewer sandbox

Token & Cost Profile (6 transcripts)

Workflow	Runs	Turns	Input tokens	AIU	Models
Matt Pocock Skills Reviewer	2	56 / 23	3.60M / 0.93M	117.3 / 63.5	claude-sonnet-5 (+haiku-4.5 subagent)
PR Code Quality Reviewer	2	22 / 16	0.98M / 0.39M	6.4 / 2.5	gpt-5.6-luna + claude-haiku-5.5
Test Quality Sentinel	2	4 / 4	0.12M / 0.12M	5.4 / 5.4	claude-haiku-4.5

Total: 125 turns, 6.13M input tokens (92.2% cache reads), 110k output tokens, about 200.5 AIU. Matt Pocock Skills Reviewer accounts for 90% of AIU while producing the same number of reviews as PR Code Quality Reviewer, which accounts for 4.4%.

Context Issues

• Sessions with confusion: 1 of 6 (16.7%): PR Code Quality Reviewer tried to view two smoke workflow files that do not exist in the workspace
• Clarification requests: 0

Experimental Analysis

Standard analysis only - no experimental strategy this run (roll = 93).

Actionable Recommendations

For Users Writing Task Descriptions

1. Give the base SHA whenever you ask an agent to verify a diff: the 11-turn loop came from inferring a base commit in a partial clone.
   • Before: "Verify the schema changes in this PR"
   • After: "Compare schemas/audit.schema.json between $BASE_SHA and HEAD (both are pre-fetched)"
2. Ask for a file-based payload for long safe-output bodies: avoids apostrophe breakage.
   • Before: safeoutputs submit_pull_request_review '<long JSON>'
   • After: write the review body to a file with a quoted heredoc and pass it via stdin
3. Make pre-fetch guard clauses loud: a pre-fetch fallback should report the failure (for example via report_incomplete or a visible comment), not a silent noop.

For System Improvements

1. Fix the Test Quality Sentinel pre-fetch (.github/workflows/test-quality-sentinel.md:104): replace gh pr diff "$PR_NUMBER" -- <files> with a full gh pr diff "$PR_NUMBER" filtered to test files (or git diff "$BASE_SHA...HEAD" -- <files>), then make recompile. Add a contract test in pkg/cli/pr_prefetch_cache_contract_test.go that rejects extra positional args to gh pr diff.
   • Potential impact: High (restores test review on every test-changing PR)
2. Surface non-zero shell exits in tool.execution_complete: analytics and agents both treat success=true as truth while the exit code is buried in result text.
   • Potential impact: Medium
3. Rebalance the Matt Pocock Skills Reviewer model: claude-sonnet-5 costs about 20x the luna+haiku review stack for comparable output. Consider a luna/haiku orchestrator, or caps on turns and git exploration.
   • Potential impact: Medium-High (~90% of today's reviewer AIU)
4. Investigate the 60-min cancellation of Addressing comment on PR #73052.
   • Potential impact: Medium

For Tool Development

1. Unshallow / blob-filter-aware base-commit helper: needed when a reviewer must diff against a historical base.
   • Frequency of need: 1 session
   • Use case: schema diff verification in skills review
2. Node test toolchain in the reviewer sandbox (or an explicit "tests unavailable" marker).
   • Frequency of need: 1 session (4 turns lost)
   • Use case: running the vitest suite touched by the PR

Historical Trends and Statistical Summary

Trends Over Time

• Completion rate trend: 70.0% today vs 30-day mean 41.1% (range 4–80%). The last four snapshots climb from 8.0% to 30.0% to 46.0% to 70.0%, mostly from population mix, not agent behavior.
• Average duration trend: 5.4 min, down from the 10-29 morning outlier (90.3 min) and close to the 30-day typical range (1–15 min).
• Quality improvement: transcript coverage recovered for the 2nd straight day (3 then 6 of 50), but it still covers only review-bot workflows (review_bot_log_coverage_gap persists). hidden_shell_failure_masking is confirmed on a 2nd day, and the Test Quality Sentinel has moved from sonnet-5 to haiku-4.5 (13.3 to 5.4 AIU per noop).

Statistical Summary

Total Sessions Analyzed:     50
Successful Completions:      35 (70.0%)
Failed Sessions:             4 (8.0%)
Abandoned Sessions:          1 (2.0%)   [cancelled at 60.3 min]
Gate-waiting/Skipped:        10 (20.0%)
In-Progress Sessions:        0 (0.0%)

Average Session Duration:    5.4 min
Median Session Duration:     2.5 min
Longest Session:             60.3 min
Shortest Session:            0.0 min

Loop Detection:              1 of 6 transcripts (16.7%)
Context Issues:              1 of 6 transcripts (16.7%)
Tool Failures:               8 occurrences (3 reported + 5 hidden)

High-Quality Prompts:        6 of 6 transcripts (100%)
Medium-Quality Prompts:      0 (0%)
Low-Quality Prompts:         0 (0%)

Next Steps

• Fix gh pr diff usage in test-quality-sentinel.md and recompile (highest priority)
• Decide whether pre-fetch fallbacks should fail loudly instead of noop
• Evaluate a cheaper model stack for Matt Pocock Skills Reviewer
• Schedule follow-up analysis in 1 day to confirm the sentinel reviews a test-changing PR after the fix

---

Analysis generated automatically on 2026-10-30 at 07:20 UTC
Run ID: 41016078391
Workflow: Copilot Session Insights

Generated by 📊 Copilot Session Insights · claude · routed: opus55 high · ~214 AIC · ⌖ ~2 AIC · ⊞ 14.6K · ◷

• expires on Oct 30, 2026, 11:18 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Session Insights.

A newer discussion is available at Discussion #73453.