[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-11-04

[github-actions[bot]]

🤖 Copilot Agent Session Analysis — 2026-11-04

Executive Summary

• Sessions Analyzed: 50 workflow runs on copilot/* branches (05:28Z–06:38Z), including 13 agent transcripts (first full transcript coverage after 0/50 yesterday). As context, I also looked at the 9 Copilot cloud-agent sessions from the past 24h, which all fell before the 50-run window.
• Analysis Period: 2026-11-04 05:28Z → 06:38Z (run window); 2026-11-03 06:00Z → 2026-11-04 07:10Z (cloud-agent outcome context)
• Completion Rate: 78.0% (39/50). 0 failures, 10 action_required, 1 in progress
• Average Duration: 5.0 min (median 7.0 min)
• Experimental Strategy: none (roll = 36, standard run)

📈 Session Trends Analysis

Completion Patterns

Completion rose from 64% to 78%, the third-best day in the 22 recorded days (after 90% on Oct 31 and 80% on Oct 21). There were zero failed runs (as on Oct 31); every non-completed run is an approval gate (action_required) or still in progress. Since Oct 29 the rate has stayed in a 46–90% band, while Oct 14–28 swung between 4% and 80%.

Duration & Efficiency

Mean duration has stayed between 3.9 and 5.7 min since Oct 30. Today's median (7.0 min) sits above the mean because this window holds almost no near-zero runs: only the 10 approval gates are instant, and no runs were skipped. No loops were detected in any of the 13 transcripts (0 repeated identical tool calls), down from 2 yesterday.

Key Metrics

Metric	Value	Trend
Total Sessions	50 (13 with transcripts)	→
Successful Completions	39 (78.0%)	↑ (64.0%)
Failed/Abandoned	0 failed / 10 action_required (20.0%)	↓ failures (6 → 0)
Average Duration	5.0 min (median 7.0)	↑ (4.9 / 3.3)
Loop Detection Rate	0 (0%)	↓ (2)
Context Issues	1 of 9 cloud-agent sessions (11%)	→

Success Factors ✅

1. Scanner-derived issues with exact file:line, rule ID, root cause and suggested fix: these cloud-agent tasks merged quickly.
   • Success rate: 6/8 cloud-agent PRs merged in 24h (75%), median time to merge 2h15m (range 1h33m–2h44m)
   • Example: [static-analysis] fix(shellcheck): suppress SC2086 on intentional hippo CLI arg splitting (about 7.9k chars, with path and expected result) → Suppress intentional SC2086 in hippo MCP script #74742 merged
2. Prefetched PR diff plus a delegated specialist subagent: PR Code Quality Reviewer reads pr-diff.patch, hands review to a grumpy-coder subagent (Haiku 5.5), then posts.
   • Success rate: 4/4 runs posted a review; agent time 0.5–1.4 min; average 13 turns, 18 tool calls
   • Example: run 41781678103 finished in 9 turns with a clean COMMENT verdict
3. Independent reviewers converging on the same defect: on Normalize labels before security filtering #74752, PR Code Quality Reviewer and Matt Pocock Skills Reviewer each found, unprompted by the other, that sanitizing before matching lets blocked mention-style labels bypass the blocked-pattern check.
   • Success rate: 1/1. An "Addressing comment" session pushed fix d4918e8a 16 min after dispatch
   • Example: both reviews were REQUEST_CHANGES with inline comments on safe_output_validator.cjs
4. Prompt-cache reuse: 90.6% of the 4.59M input tokens across the 13 transcripts were cache reads.
   • Example: Test Quality Sentinel averaged 174k input tokens but only 958 output tokens and 0.93 AIU per run

Failure Signals ⚠️

1. A duplicate issue led to an empty PR: Suppress runner-guard false positive in Agy installation #74751 (issue [static-analysis] Suppress RGS-018 false positive in credentials-check.yml #73435) targeted the same Agy install step that Suppress RGS-018 for checksum-verified Agy install #74712 (issue [static-analysis] Suppress RGS-018 false positive on checksum-verified Agy install #73121) had fixed 81 min earlier. The PR had 0 changed files and was closed.
   • Failure rate: 1/8 cloud-agent sessions (12.5%)
   • Example: [static-analysis] Suppress RGS-018 false positive in credentials-check.yml #73435 is a "protected files" fallback issue for credentials-check.yml:126, the same RGS-018 finding as [static-analysis] Suppress RGS-018 false positive on checksum-verified Agy install #73121
2. U+0019 control character in inline safeoutputs JSON (smart_quote_control_char, recurring since Oct 31). The model writes U+0019 where a typographic apostrophe belongs, and the CLI rejects the payload with Bad control character in string literal.
   • Failure rate: 2/4 PR Code Quality Reviewer runs (gpt-5.6-luna); each recovered after 1 retry
3. TLS failures on git history operations: git calls that trigger a fetch from github.com (git diff HEAD~1, git show, merge-base origin/main) fail with SSL certificate verification failed ... /run/awf-lib/trust/ca-certificates.crt.
   • Failure rate: 4/13 transcripts (31%); 3 of the 4 are Haiku-routed Matt Pocock runs
   • Example: reviewers then fall back to the prefetched pr-diff.patch, which already contained the diff
4. Opposite reviewer verdicts on one PR: on Suppress intentional SC2086 in hippo MCP script #74742, PR Code Quality Reviewer requested changes (unquoted $INPUT_ARGS breaks quoted multi-word arguments), while Matt Pocock (routed to luna) approved after a shellcheck probe. The PR merged at 07:06Z with the REQUEST_CHANGES review still on record.
   • Failure rate: 1/4 reviewed PRs had conflicting verdicts
5. Setup dominates reviewer runtime: agent time is 17% of runner wall time across the 13 transcribed runs (19.0 of about 112 min).
   • Test Quality Sentinel: 0.37 min agent / 7.8 min wall (5%); PR Code Quality Reviewer: 0.91 / 7.7 (12%); Matt Pocock: 3.4 / 10.5 (32%)

Prompt Quality Analysis 📝

Per-Prompt Breakdown

High-Quality Prompt Characteristics

Based on the linked issues of 5 of the 8 cloud-agent PRs (small sample):

• File path present: 5/5 issues (100%), all resulting PRs merged or under active review
• Rule ID plus root cause plus proposed fix: 4/5 (80%). All 4 static-analysis issues are about 7–8k characters and include expected behavior
• Explicit "expected/should" outcome: 4/5 (80%). 3 of those 4 PRs merged; the 4th (Suppress runner-guard false positive in Agy installation #74751) was a duplicate target

Example High-Quality Prompt:

[static-analysis] Suppress RGS-018 false positive in daily-workflow-security-model
- Finding: <file>:<line> RGS-018 (high)
- Root cause: checksum hex coincidentally matches a payload signature
- Fix: add `# runner-guard:ignore RGS-018` with rationale, following existing convention
- Expected: scanner treats the step as a false positive; no recompile needed

Low-Quality Prompt Characteristics

• No explicit acceptance criteria: 1/5 ([uk-ai-resilience] Tier C: blocked-label check bypass via unnormalised labels (code scanning 1061) #74548, about 1.9k chars). It was the only PR to draw REQUEST_CHANGES from two reviewers and needed a follow-up session
• Issue overlapping an already-merged fix (no dedup check): 1/5 ([static-analysis] Suppress RGS-018 false positive in credentials-check.yml #73435) → empty PR

Example Low-Quality Prompt:

[uk-ai-resilience] Tier C: blocked-label check bypass via unnormalised labels
(describes the bypass, but does not state which representation — raw or sanitized —
blocked/allowed patterns must match against, nor the expected test matrix)

Orphaned Branch Escalation Alerts 🚨

Branches with ≥5 simultaneous gate firings and no Copilot agent assigned for >2 hours.

Summary

• Orphaned Branches Today: 0 out of 22 open PRs (0.0%). In-progress runs span 4 branches; the busiest is main with 8.
• Historical Baseline: 0.0% orphaned rate (mean over 6 recorded days; fewer than 30 available)
• Status: NORMAL

Escalation Candidate Details

Escalation Candidates

✅ No orphaned branches exceed the escalation threshold today.

Informational: fix/cs-1063-b8126f7add549835 has 7 in-progress runs but no open PR, so it is outside the escalation scope. The only open Copilot PR, #74752, has 1 in-progress gate and Copilot assigned.

CI Waste Estimate

• Orphaned gate-hours today: 0 gates × 0 h ≈ 0 CI-minutes wasted
• Recoverable capacity: none needed

Notable Observations

Loop Detection and Session Diagnostics

Loop Detection

• Sessions with loops: 0 (0%)
• Average loop count: 0
• Common loop patterns: none. The highest turn counts are the Haiku-routed Matt Pocock runs (15–24 turns). Those turns go to deliberate verification, not repetition: one run started a targeted go test and waited on it with sleep 60, another wrote Node probe scripts and tried vitest.

Tool Usage

• Most used tools: bash 108 (49%), view 71 (32%), skill 13, glob 5, task 5, task_complete 5, grep 4, read_agent 4, create 2
• Tool success rates: 215/219 (98.2%). 3 of the 4 failures are a benign view probe for test-prefetch-unavailable.txt in Test Quality Sentinel (the file is absent when the prefetch succeeds). The 4th is 1 permission denial: the grumpy-coder subagent ran a compound git diff ...; cmp ... outside the workflow's bash allowlist.
• Missing tools: none requested. vitest is not installed under actions/setup/js/node_modules, so one reviewer fell back to hand-written assert scripts.
• Model routing: Matt Pocock Skills Reviewer (model-routing: goal=cost, mode=economy) went to claude-haiku-5.5 in 3/4 runs (15–24 turns, 3.5–4.8 min, 3.3–3.5 AIU, about 40k output tokens) and to gpt-5.6-luna in 1/4 (11 turns, 0.8 min, 3.2 AIU). In that luna run the pr-triage subagent on claude-haiku-4.5 cost 1.98 AIU for 2 requests, 62% of the run. Economy routing did not lower cost per review today.

Context Issues

• Sessions with confusion: 1/9 cloud-agent sessions (Suppress runner-guard false positive in Agy installation #74751: correctly found nothing to change, but still opened an empty PR)
• Common confusion points: overlapping issues for the same scanner finding
• Clarification requests: 0 across 13 transcripts

Event fan-out

• 6 agent reply comments on Normalize labels before security filtering #74752 review threads triggered 12 workflow runs within 2 s (6 Agentic Commands succeeded, 6 Squad skipped).
• All 10 action_required runs are approval gates on copilot/* branches: 6 Squad/Agentic Commands runs triggered by review comments, and 4 CI gates (CWI, CJS, CGO, Work Queue) on Clarify required message argument for push_to_pull_request_branch #74740's head at 06:37Z. Clarify required message argument for push_to_pull_request_branch #74740 merged at 06:49Z.

Experimental Analysis

Standard analysis only - no experimental strategy this run (roll = 36).

Actionable Recommendations

For Users Writing Task Descriptions

1. State the representation that checks must match against: security-normalization tasks need an explicit acceptance matrix.
   • Example: Before: "blocked-label check bypass via unnormalised labels". After: "Blocked and allowed patterns must match both the trimmed raw label and the sanitized/truncated value; add tests for zero-width, fullwidth, and >64-char inputs."
2. Check for an existing or merged fix before assigning a scanner issue: link related issues ([static-analysis] Suppress RGS-018 false positive on checksum-verified Agy install #73121 ↔ [static-analysis] Suppress RGS-018 false positive in credentials-check.yml #73435) and close duplicates first.
   • Example: Before: assign the protected-files fallback issue as-is. After: "Superseded by Suppress RGS-018 for checksum-verified Agy install #74712 if the RGS-018 ignore is already present at credentials-check.yml:126."
3. Keep the static-analysis issue template: file:line, rule ID, root cause, proposed fix, and expected outcome. Issues in this format merged in 4/4 cases.

For System Improvements

1. Sanitize C0 control characters in the safeoutputs CLI JSON parser (map U+0019 to ' or escape U+0000–U+001F inside strings), or tell gpt-5.6-luna reviewers to use ASCII apostrophes.
   • Potential impact: Medium (removes a retry turn in about 50% of PR Code Quality Reviewer runs)
2. Stop opening empty PRs: when the cloud agent finds no diff, report no-op on the issue instead.
   • Potential impact: Medium
3. Reconcile conflicting reviewer verdicts: when one agentic reviewer approves and another requests changes on the same head SHA, post a summary or block auto-merge until a human resolves it.
   • Potential impact: High (Suppress intentional SC2086 in hippo MCP script #74742 merged with an unresolved correctness concern about quoted arguments)
4. Revisit model-routing: economy for Matt Pocock Skills Reviewer: Haiku-routed runs cost as much as the luna run and take 4–6× longer, and the haiku-4.5 pr-triage subagent costs about 1 AIU per request.
   • Potential impact: Medium
5. Reduce reviewer setup overhead: agent time is only 5–32% of runner wall time.
   • Potential impact: Medium

For Tool Development

1. Git history inside the AWF sandbox: either fetch the base commit during setup or trust the AWF CA for git, so git diff HEAD~1 and merge-base work.
   • Frequency of need: 4 sessions
   • Use case: reviewers comparing base and head files
2. Preinstall vitest for actions/setup/js in reviewer workflows that may run targeted JS tests.
   • Frequency of need: 1 session
   • Use case: verifying a suspected regression with the repository's own tests

Historical Trends and Statistical Summary

Trends Over Time

• Completion rate trend: 78.0% today vs 64.0% yesterday. The 7-day mean (Oct 29–Nov 04) is 68.9%, compared with 47.6% for Oct 14–28.
• Average duration trend: stable at 3.9–5.7 min since Oct 30 (Oct 14–28 ranged 1.1–35.6 min)
• Quality improvement: transcript coverage recovered (conversation_logs_empty → resolved; the files are now {run_id}-events.jsonl). test_sentinel_prefetch_always_fails did not recur (5/5 sentinels prefetched cleanly). agent_merge_silent_noop was not observed (no Agent Merge runs in the last 24h).

Statistical Summary

Total Sessions Analyzed:     50
Successful Completions:      39 (78.0%)
Failed Sessions:             0 (0.0%)
Abandoned Sessions:          10 (20.0%)  [action_required approval gates]
In-Progress Sessions:        1 (2.0%)

Average Session Duration:   5.0 min
Median Session Duration:    7.0 min
Longest Session:           14.5 min  (Running Copilot Code Review, #74752)
Shortest Session:           0.0 min  (action_required gates)

Loop Detection:            0 sessions (0%)
Context Issues:            1 session (#74751, of 9 cloud-agent sessions in 24h)
Tool Failures:             4 occurrences (3 benign probes, 1 permission denial)

High-Quality Prompts:      4 (80%)  [of 5 sampled cloud-agent source issues]
Medium-Quality Prompts:    1 (20%)
Low-Quality Prompts:       0 (0%)

Cloud-agent sessions (24h): 8 runs + 1 addressing-comment; mean 13.1 min, median 13.2 min
Cloud-agent PR outcomes:    6 merged, 1 closed (empty duplicate), 1 open
Transcript tokens:          4.59M input (90.6% cache read), 154.5k output, 24.3 AIU total

Next Steps

• Review recommendations with team
• Implement high-priority prompt improvements (dedup scanner issues; acceptance matrix for security fixes)
• Consider system enhancements for recurring issues (safeoutputs control-char sanitization, reviewer verdict reconciliation)
• Schedule follow-up analysis in 24 hours

---

Analysis generated automatically on 2026-11-04 at 07:15 UTC
Run ID: 41787195873
Workflow: Copilot Session Insights

Generated by 📊 Copilot Session Insights · claude · routed: opus55 high · ~220 AIC · ⌖ ~2 AIC · ⊞ 14.6K · ◷

• expires on Nov 4, 2026, 11:15 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Session Insights.

A newer discussion is available at Discussion #75127.