[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-11-02

[github-actions[bot]]

🤖 Copilot Agent Session Analysis — 2026-11-02

Executive Summary

• Sessions Analyzed: 50 workflow runs on 4 copilot/* branches. 3 were Copilot coding-agent sessions, and 1 had a full agent transcript.
• Analysis Period: 2026-11-01 22:27 UTC to 2026-11-02 06:04 UTC. 49 of the 50 runs started between 22:27 and 23:49.
• Completion Rate: 64.0% (32/50) raw, 86.5% (32/37) when the 13 skipped Squad runs are left out.
• Average Duration: 3.9 min (median 1.4 min)
• Experimental Strategy: none. This was a standard run (roll 53).

All 3 coding-agent sessions succeeded, and all 3 PRs behind today's branches merged (#73886, #73819, #73839). The most useful finding comes from the one transcript. A run on the copilot/route-linter-miner branch moved its main agent to the economy model, but its 3 subagents stayed pinned to claude-sonnet-5.5 and used 74.9% of the run's AI units (AIU) with 26% of the requests. Separately, #73886 merged while every CGO run on its branch was red (4 of 4).

📈 Session Trends Analysis

Completion Patterns

Completion fell for the second day in a row (90% → 70% → 64%) but is still 12 points above the 30-day mean of 51.9%. The drop comes from Squad skips (13, up from 5 on 10-31) and 4 CI-gate failures. Of the 37 runs that actually executed, 86.5% succeeded.

Duration & Efficiency

Average and median durations stay in the low range seen since 10-29 (3.9 / 1.4 min). The mix of short runs pulls the average down: 13 Squad skips took about 1 second each and 11 Agentic Commands averaged 0.8 min. The 3 coding-agent sessions averaged 21.9 min. No loops were found in today's only transcript.

Key Metrics

Metric	Value	Trend (vs 11-01)
Total Sessions	50	→
Successful Completions	32 (64.0%)	↓ from 35 (70.0%)
Failed/Abandoned	18 (36.0%): 4 failure, 1 cancelled, 13 skipped	↑ from 15
Average Duration	3.9 min	↓ from 4.7 min
Loop Detection Rate	0/1 transcripts (0%)	→ (sample of 1)
Context Issues	1/1 transcripts (minor)	n/a
Copilot coding-agent sessions	3/3 success, avg 21.9 min	→ (3/3 on 11-01)
Transcript coverage	1/50	↓ from 15/50 on 10-31

Success Factors ✅

1. Coding-agent sessions answering PR review comments: all 3 Addressing comment on PR runs succeeded (2 on Upgrade AWF to v0.29.5 and generate Docker enclave integration #73819, 1 on audit: count only the audited attempt's evals invocations, once each, in the evals summary #73886), and both PRs merged.

   • Success rate: 100% (3/3)
   • Example: the 9 Squad reply runs on audit: count only the audited attempt's evals invocations, once each, in the evals summary #73886 show 5 fixes, each naming the new regression test (for example EvalsSummaryUnknownAttemptMixedEvidence), plus 4 reasoned declines ("Not changing this in this PR... listed as a possible follow-up"). The PR merged 37 minutes after the first reply.

2. Briefed subagents in a fixed order: the transcript run handed work to discussion-miner, then code-pattern-scanner, then linter-writer, with briefs of 724, 656, and 1,798 chars. All 3 subagents finished in 3 min 20 s, and the work landed in a single PR.

   • Success rate: 3/3 subagents completed, and 1/1 PR opened on the first create_pull_request call.

3. Explicit file paths in git add: before committing, the agent restored a clobbered tracked binary and then staged only named paths. This kept a 12.3 MB file out of the PR (see Failure Signals).

4. High prompt-cache reuse: cache reads were 95.8% of input-side tokens (3.04M cache-read vs 1,650 fresh input and 130,933 cache-write).

5. No permission friction: 85 of 85 permission requests were approved, with 0 denials. On 10-31, every PR Code Quality Reviewer run (5/5) had at least one denial.

Failure Signals ⚠️

1. Subagent model pin dominates cost (new): the transcript ran on the copilot/route-linter-miner branch, which became Linter Miner: use model routing (cost/economy) instead of pinning claude-sonnet-5.5 #73839 "use model routing (cost/economy) instead of pinning claude". The main agent was routed to gpt-5.6-luna, but all 3 subagents loaded claude-sonnet-5.5 through agent_definition_default.

   • Impact: Sonnet made 20 requests for 27.75 AIU, against Luna's 58 requests for 9.32 AIU. That is 74.9% of the 37.06 AIU total, at about 8.6× the per-request cost.
   • Example: subagent.completed ... "modelSelectionSource":"agent_definition_default" appears 3 times.

2. Final validation gate cannot pass in the agent sandbox: the agent ran make agent-report-progress twice, and it exited non-zero both times.

   • The 1st run failed with unable to determine merge-base from BASE_REF=origin/main, even after unshallowing. This is the 3rd straight day of the partial_clone_git_history_dead_end pattern.
   • The 2nd run (BASE_REF=HEAD) failed because golangci-lint could not be installed (make install-golangci-lint, Error 1).
   • Also seen: make fmt failed in fmt-cjs, and make golint-custom exited 3 on findings that already exist across the repo. The agent polled that last step for about 4.5 min.
   • Failure rate: 4 of 4 gate-style make targets the agent ran exited non-zero. The PR was opened anyway, with a note in the summary.

3. Tracked build artifact gets clobbered (new): main tracks a 12.3 MB ELF executable at the repo root named linters (last changed by Add regexpdynamicpattern linter: flag non-constant regexp compile patterns #50674, and before that by feat(linters): add uncheckedflushreturn analyzer + fix existing violations #49862 and feat(linters): add stringsjoinone analyzer #47288). Running go build ./cmd/linters without -o overwrites it.

   • Frequency: the agent had to run git checkout linters twice in 1 run.

4. Merged with a red gate (2nd day): audit: count only the audited attempt's evals invocations, once each, in the evals summary #73886 merged at 23:06 with CGO impacted-go-tests failing on its head f10d0ce, on TestFallbackAWFilesMatchesLocalAWDirectory. All 4 CGO runs on the branch failed. On 11-01 the same thing happened with audit: report failed continue-on-error evals/detection engine steps and add an evals summary #73556.

   • Upgrade AWF to v0.29.5 and generate Docker enclave integration #73819 handled it correctly: CGO failed twice (TestActionsLockJSONFieldsAreSorted), passed at 23:42, and the PR merged at 23:49.

5. Hidden shell failures: tool.execution_complete reported success for 7 bash calls that actually exited non-zero.

   • Effective tool success rate: 89.8% (88/98), against 96.9% (95/98) as reported.

Prompt Quality Analysis 📝

Per-Prompt Breakdown

Limited sample: none of today's 3 coding-agent sessions has a transcript, so their prompts couldn't be scored. The points below come from the one transcript and from the review-reply text in the Squad run titles.

High-Quality Prompt Characteristics

• Named target paths and conventions: the linter-writer brief named pkg/linters/largefunc as the convention to follow, the analyzer name, the URL, and the Doc wording. That subagent finished in 1 min 54 s, and its own summary matched the brief item for item.
• Review comments that describe a concrete scenario: 5 of the 9 Squad replies on audit: count only the audited attempt's evals invocations, once each, in the evals summary #73886 came back with a code fix and a named test (for example EvalsSummarySharedJobLogLocation for "two evals jobs resolve to the same log file").
• Scope stated up front: "I did not modify any existing linter implementation" shows the brief set clear limits, and the agent respected them.

Example High-Quality Prompt (paraphrased from the transcript):

Implement a new Go analysis linter package following the pkg/linters/largefunc
conventions. Analyzer name: osremoveuncheckedcleanup. Doc: one sentence beginning
"reports". Register it in registry.go and spec_test.go. Do not modify existing linters.

Low-Quality Prompt Characteristics

• Validation steps that can't succeed in the sandbox: the prompt required make agent-report-progress without saying how to resolve BASE_REF in a shallow checkout, so the agent spent 2 turns finding the workaround.
• Unstated tool schema: the agent guessed the todos SQL columns, and 2 of its 5 sql calls failed before it read PRAGMA table_info.

Example Low-Quality Prompt:

Run make agent-report-progress before opening the PR.

(This gives no BASE_REF, ignores the shallow checkout, and assumes golangci-lint can be installed.)

Orphaned Branch Escalation Alerts 🚨

Branches with ≥5 simultaneous gate firings and no Copilot agent assigned for >2 hours.

Summary

• Orphaned Branches Today: 0 out of 10 open PRs (0.0%)
• Historical Baseline: 0.0% orphaned rate (mean over 4 prior recorded days)
• Status: NORMAL

Escalation Candidate Details

Escalation Candidates

✅ No orphaned branches exceed the escalation threshold today. At 07:14 UTC there was 1 in-progress run in the last 6 hours, and it was on main. No open PR branch had any active gate.

CI Waste Estimate

• Orphaned gate-hours today: 0 gates × 0 h ≈ 0 CI-minutes wasted
• Recoverable capacity: none needed

Notable Observations

Loop Detection and Session Diagnostics

Loop Detection

• Sessions with loops: 0 of 1 transcript (0%)
• Average loop count: 0
• Common loop patterns: no repeated cycles. The nearest thing was polling make golint-custom with read_bash (delay 120 s), which was waiting rather than looping.

Tool Usage

• Most used tools: bash 71, view 11, sql 5, task 3, read_bash 3, edit 2, skill/glob/create 1 each (98 total, 78 turns)
• Tool success rates: 96.9% as reported (3 failures: 2 sql, 1 edit "No match found"); 89.8% effective once the 7 hidden non-zero bash exits are counted
• Session timing: about 12.7 min in the agent (API time 5.6 min = 44%), 23.1 min for the whole workflow run. Subagents took 3.3 min of that; make fmt took 78 s, and golint-custom took about 4.5 min of polling.
• Missing tools: golangci-lint could not be installed. The run filed a report_missing_data for a cache-memory miss on its first run of proposed-linters.json.

Context Issues

• Sessions with confusion: 1 of 1 (minor), from guessing the todos table schema
• Common confusion points: the SQL tool schema, and which BASE_REF to use in a shallow clone
• Clarification requests: 0

CI Gate Health (per workflow, a standing metric)

Workflow	Runs	Success	Failure	Avg min
Agentic Commands	11	11	0	0.8
Squad (+ implement)	13	0	0 (13 skipped)	0.0
CGO	4	1	3	6.9
CJS	3	2	1	4.9
CWI	4	4	0	2.5
Code scanning AI findings	4	3	0 (1 cancelled, superseded)	7.2
Doc Build - Deploy	3	3	0	2.9
Work Queue Compilation Integration	3	3	0	1.4
Addressing comment on PR	3	3	0	21.9
Linter Miner	1	1	0	23.1
Harbor Conformance	1	1	0	2.5

• Correction to the 11-01 report: CGO on main passed at 01:47, 02:46, and 04:20 UTC today, so the "red on main since 10-29" streak has ended. main went red again from 05:48 UTC, at the current HEAD f37d3b9, on a different test: TestPortfolioAnalystAvoidsInvalidSentryAggregations. Because main runs only impacted tests, its green runs don't prove audit: count only the audited attempt's evals invocations, once each, in the evals summary #73886's failing test was harmless. I couldn't re-run that test here because the sandbox firewall blocks the Go toolchain download.
• A Squad implement run fired at 06:04 on copilot/support-claude-on-vertex-ai, the branch of Support Claude on Vertex AI with GCP WIF #72273, which was closed without merging. It was skipped and cost about 1 s.

Experimental Analysis

Standard analysis only, with no experimental strategy this run (roll 53 ≥ 30).

Actionable Recommendations

For Users Writing Task Descriptions

1. Ask for a named regression test in each review comment: every audit: count only the audited attempt's evals invocations, once each, in the evals summary #73886 fix that named a test merged cleanly.

   • Example: write "Add a test where two evals jobs share one log file and both are counted" rather than "handle the shared-log case".

2. Give validation commands that work in a shallow checkout: say which BASE_REF to use, or name the narrower check you actually want.

   • Example: write "Run go test ./pkg/linters/... and go build -o /tmp/linters ./cmd/linters" rather than "Run make agent-report-progress".

3. Say where build output should go: always pass -o /tmp/... when building cmd/linters.

   • Example: write go build -o /tmp/gh-aw/agent/linters ./cmd/linters rather than go build ./cmd/linters.

For System Improvements

1. Route subagent models too, not just the main agent: Linter Miner's agent definitions still set claude-sonnet-5.5 as the default. Pricing those 20 subagent requests at Luna's per-request rate would bring the run from about 37 to about 12.5 AIU (−66%). That is a projection only; subagent output quality on the cheaper model hasn't been tested.

   • Potential impact: High

2. Remove the tracked linters binary from the repo root: it is a 12.3 MB ELF on main, rewritten by 3 linter PRs since July. Find out why it is tracked, then git rm --cached it and ignore it.

   • Potential impact: Medium (repo size, agent turns, and the risk of committing it by accident)

3. Make the final validation gate pass in agent sandboxes: have agent-report-progress fall back to a usable BASE_REF when merge-base can't be resolved. Also make sure golangci-lint and the fmt-cjs dependencies are available in the sandbox, or skip those steps explicitly.

   • Potential impact: High (on 11-02, 4 of 4 gate targets failed for reasons that had nothing to do with the change)

4. Don't merge over a red CGO: this has happened 2 days in a row (audit: report failed continue-on-error evals/detection engine steps and add an evals summary #73556, audit: count only the audited attempt's evals invocations, once each, in the evals summary #73886). Consider making CGO impacted-go-tests a required check, or adding a Squad/merge guard.

   • Potential impact: Medium

5. Don't run Squad implement on closed PRs: it fired on the branch of closed-unmerged Support Claude on Vertex AI with GCP WIF #72273.

   • Potential impact: Low

For Tool Development

1. Transcripts for coding-agent sessions: none of today's 3 Addressing comment sessions produced a transcript. Coverage fell from 15/50 to 1/50.

   • Frequency of need: 3 sessions today (3 on 11-01)
   • Use case: scoring prompt quality and spotting loops on the sessions that matter most

2. Show the SQL todos schema in the tool description: this would avoid failed INSERTs from guessing columns.

   • Frequency of need: 1 session (2 failed calls)
   • Use case: tracking task progress inside the agent

3. Make tool.execution_complete report a non-zero exit as success=false (hidden_shell_failure_masking, first seen 10-29)

   • Frequency of need: 7 calls in 1 session
   • Use case: accurate tool success metrics, and earlier agent self-correction

Historical Trends and Statistical Summary

Trends Over Time

• Completion rate trend: 90.0% (10-31) → 70.0% (11-01) → 64.0% (11-02). The 30-day mean is 51.9% with a range of 4–90%, so today is above average and the swings are driven by how many runs were skipped or gates fired, not by agent quality.
• Average duration trend: 1.7 to 5.7 min every day since 10-29. The 10-26 outlier of 35.6 min has not recurred.
• Quality improvement: coding-agent sessions went 3/3 for the 2nd day running. Transcript coverage is unstable (3 → 6 → 15 → 0 → 1). Permission denials went from 5/5 PCQR runs on 10-31 to 0 today, though today's agent was a different one.

Statistical Summary

Total Sessions Analyzed:     50
Successful Completions:      32 (64.0%)
Failed Sessions:             4 (8.0%)   [CGO x3, CJS x1]
Abandoned Sessions:          14 (28.0%) [13 skipped Squad, 1 cancelled]
In-Progress Sessions:        0 (0.0%)

Average Session Duration:    3.9 min
Median Session Duration:     1.4 min
Longest Session:             28.2 min (Addressing comment on PR #73886)
Shortest Session:            0.02 min (Squad implement, skipped)

Loop Detection:              0 sessions (0% of 1 transcript)
Context Issues:              1 session (minor)
Tool Failures:               3 reported + 7 hidden = 10 occurrences

High-Quality Prompts:        n/a (no coding-agent transcripts)
Medium-Quality Prompts:      n/a
Low-Quality Prompts:         n/a

AIU (1 transcript):          37.06 (Sonnet subagents 27.75 / Luna main 9.32)
Tokens:                      1,650 in / 3,043,671 cache-read / 130,933 cache-write / 26,928 out

Next Steps

• Route Linter Miner subagents through model routing and compare AIU and output quality
• Find out why the root linters binary is tracked, then untrack it
• Fix agent-report-progress for shallow checkouts and sandbox lint installs
• Decide whether CGO impacted-go-tests should block merges
• Schedule follow-up analysis in 24 hours

---

Analysis generated automatically on 2026-11-02 at 07:30 UTC
Run ID: 41402196393
Workflow: Copilot Session Insights

Generated by 📊 Copilot Session Insights · claude · routed: opus55 high · ~235 AIC · ⌖ ~2 AIC · ⊞ 14.6K · ◷

• expires on Nov 2, 2026, 11:24 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Session Insights.

A newer discussion is available at Discussion #74442.