fix(copilot-session-insights): restore transcript fetch after 42-day gap

[Copilot]

Copilot session transcript logs have been empty for 42 consecutive days, causing behavioral analytics (loop detection, context confusion) to fall back to run-metadata-only inference.

Two bugs identified:

Bug 1 — branch name extraction broken since ~Jul 8 (primary)
Session numbers were extracted by stripping non-numerics from branch names. Copilot switched from copilot/issue-123 to descriptive slugs (copilot/fix-mcp-gateway-docker-daemon-access), producing empty strings that caused [ -n "$session_number" ] to skip all 50 sessions — zero files downloaded.

Bug 2 — auth failure (older, pre-Jul 8)
gh agent-task view --log doesn't have a --log flag and requires an OAuth token. Cached files from May–July contained "this command requires an OAuth token" error messages rather than transcript data.

Changes

• shared/copilot-session-data-fetch.md — replaces the broken loop with a gh api job-log approach: filters sessions-list.json to actual agent runs (conclusion != "action_required", skipping ~47 CI gate runs per day), fetches job logs via gh api repos/$REPO/actions/jobs/$job_id/logs (works with standard GITHUB_TOKEN), then greps for [cca-engine] turn= lines as the transcript

  # Before (broken): extracted session number from branch slug → always empty
  session_number=$(echo "$branch" | sed 's/copilot\///' | sed 's/[^0-9]//g')
  if [ -n "$session_number" ]; then  # always false for descriptive slugs
  
  # After: use run_id directly, skip CI gate runs, fetch via API
  jq -r '.[] | select(.conclusion != "action_required") | "\(.id) \(.head_branch)"' sessions-list.json \
    | while read -r run_id branch; do
        job_id=$(gh api "repos/$GITHUB_REPOSITORY/actions/runs/${run_id}/jobs" --jq '.jobs[0].id')
        gh api "repos/$GITHUB_REPOSITORY/actions/jobs/${job_id}/logs" \
          | grep "\[cca-engine\] turn=" > "${run_id}-conversation.txt"

• copilot-session-insights.md — adds grep * and wc * to the bash tool allow-list; updates the prompt's transcript format description to match [cca-engine] turn= line structure

• scripts/check-stale-lock-files.sh — fixes a false failure: shared components under shared/ and skills/ compile into parent workflow lock files rather than producing standalone .lock.yml files; these paths are now excluded from the stale-lock check

[github-actions]

PR Triage

Category: bug | Risk: medium | Priority: medium | Score: 48/100 (impact:25, urgency:15, quality:8) | Action: batch_review | Batch: pr-batch:draft-review

Restores transcript fetch after 42-day data gap by fixing branch-name extraction regression (descriptive slug to session-number). Medium impact/urgency. Still a draft; grouped with other draft PRs.

Generated by 🔧 PR Triage Agent · 31.2 AIC · ⌖ 8.39 AIC · ⊞ 5.4K · ◷

[github-actions]

Great detective work identifying and fixing the 42-day transcript fetch gap! 🔍 The root cause analysis covering both the branch name extraction bug and the auth failure is thorough and well-documented.

One thing that would help:

• Add tests — the new gh api job-log approach in shared/copilot-session-data-fetch.md and the stale-lock-file fix in scripts/check-stale-lock-files.sh would benefit from test coverage to prevent regressions.

If you'd like a hand, you can assign this prompt to your coding agent:

Add test coverage for the transcript fetch changes in shared/copilot-session-data-fetch.md:
1. Test that runs with conclusion != "action_required" are included
2. Test that the [cca-engine] turn= grep pattern extracts correct lines
3. Test the check-stale-lock-files.sh exclusion of shared/ and skills/ paths

Generated by ✅ Contribution Check · 141.9 AIC · ⌖ 14.9 AIC · ⊞ 6.2K · ◷

[github-actions]

🤖 PR Triage

Field	Value
Category	bug
Risk	medium
Priority Score	40 / 100
Recommended Action	defer

Score breakdown: Impact 20/50 · Urgency 12/30 · Quality 8/20

Rationale: Draft PR fixing 42-day transcript fetch gap (+105/-99 lines). No CI. Still in draft — defer until author marks ready and CI runs pass.

Generated by 🔧 PR Triage Agent · 151.2 AIC · ⌖ 7.52 AIC · ⊞ 5.4K · ◷

[github-actions]

✅ PR Code Quality Reviewer completed the code quality review.

[github-actions]

✅ Test Quality Sentinel completed test quality analysis.

No test files were added or modified in this PR. Test Quality Sentinel skipped. Changes are workflow definitions and shell script updates only.

[github-actions]

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

[github-actions]

✅ Design Decision Gate 🏗️ completed the design decision gate check.

No ADR enforcement needed: PR #44822 does not have the 'implementation' label and has 0 new lines of code in business logic directories (threshold: 100).

[Copilot]

Pull request overview

Restores Copilot session telemetry retrieval through GitHub Actions job logs and updates consumers accordingly.

Changes:

• Fetches [cca-engine] telemetry using run and job IDs.
• Updates session-insights tooling and prompts.
• Adjusts stale-lock filtering and regenerated workflows.

Show a summary per file

File	Description
scripts/check-stale-lock-files.sh	Excludes shared components from direct lock checks.
.github/workflows/shared/copilot-session-data-fetch.md	Fetches telemetry from Actions job logs.
.github/workflows/copilot-session-insights.md	Updates tools and telemetry instructions.
.github/workflows/copilot-session-insights.lock.yml	Regenerates the insights workflow.
.github/workflows/copilot-opt.lock.yml	Regenerates the optimizer workflow.
.github/workflows/agentic-auto-upgrade.yml	Changes the weekly upgrade schedule.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 6/6 changed files
• Comments generated: 5
• Review effort level: Medium

[github-actions]

Review: fix(copilot-session-insights): restore transcript fetch after 42-day gap

The root-cause analysis is correct and the fix is well-structured. The switch from gh agent-task view --log (broken OAuth + numeric-branch assumption) to gh api .../jobs/{id}/logs (standard GITHUB_TOKEN, run-ID-based) directly addresses both identified bugs.

One correctness issue flagged inline: the conclusion != "action_required" filter includes null-conclusion runs (still queued/in-progress), which can produce empty or partial transcripts. Narrowing to conclusion == "success" or .conclusion == "failure" would close this gap.

Everything else looks solid:

• CI gate exclusion logic is correct and well-commented
• Raw log cleanup (rm -f *-raw.log) prevents disk bloat
• check-stale-lock-files.sh fix correctly excludes shared/ and skills/ subdirectories
• Allow-list additions (grep *, wc *) are minimal and appropriate
• Documentation updates accurately reflect the new transcript format

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · 24.5 AIC · ⌖ 4.43 AIC · ⊞ 4.8K

Comments that could not be inline-anchored

.github/workflows/shared/copilot-session-data-fetch.md:167

The filter select(.conclusion != "action_required") also includes runs with conclusion: null (still queued/in-progress). Fetching job logs for an active run may return an incomplete or empty log, causing a silent Warning: No [cca-engine] conversation lines found.

Consider filtering to only terminal states:

jq -r '.[] | select(.conclusion == "success" or .conclusion == "failure") | "\(.id) \(.head_branch)"'

This avoids wasted API calls for pending runs and prevents partial tra…

[github-actions]

REQUEST_CHANGES — The core fix (replacing broken gh agent-task view --log + numeric-extraction with direct job-log API calls) is correct and addresses the root cause well. However, three issues will cause the same silent-empty-transcript failure mode to recur, and one will silently fetch wrong transcripts.

Blocking issues (3 medium + 1 high)

High — jobs[0].id position-based selection

Assumes one job per run and stable ordering. A second job or changed job order will silently fetch the wrong log. Fix: select by job name.

Medium — blacklist vs whitelist for conclusion

!= action_required lets through cancelled, timed_out, startup_failure — runs with no transcript. Whitelist success or failure instead.

Medium — no rate-limit throttling

50 runs × 2 API calls each with no sleep. GitHub secondary rate limits will hit on backfill runs and produce silent 403 → empty files, reproducing the original bug.

Medium — no log-truncation detection

GitHub silently truncates large job logs. Late-session turns will be lost with no warning.

🔎 Code quality review by PR Code Quality Reviewer · 52.1 AIC · ⌖ 4.79 AIC · ⊞ 5.4K
_{Comment /review to run again}

[github-actions]

jobs[0].id silently assumes a single job and stable ordering: if this workflow gains a second job or the API returns jobs in a non-deterministic order, the wrong job's logs are fetched with no diagnostic.

💡 Suggested fix

Filter by job name instead of relying on position:

job_id=$(gh api "repos/$GITHUB_REPOSITORY/actions/runs/${run_id}/jobs" \
  --jq '.jobs[] | select(.name == "Copilot Coding Agent") | .id' 2>/dev/null || true)

This is resilient to job reordering and future addition of setup/cleanup jobs.

[github-actions]

conclusion != action_required is a blacklist — it passes cancelled, timed_out, startup_failure, and skipped runs that will have no transcript, wasting one or two API calls per ghost run and silently producing empty files.

💡 Suggested fix

Whitelist the known-good conclusions instead:

jq -r '.[] | select(.conclusion == "success" or .conclusion == "failure") | "\(.id) \(.head_branch)"' \
  /tmp/gh-aw/agent/session-data/sessions-list.json | while read -r run_id branch; do

cancelled and timed_out runs never complete a full agent turn loop, so their job logs will contain no [cca-engine] lines worth analyzing anyway.

[github-actions]

No rate-limit throttling across up to 50 runs × 2 API calls each: rapid-fire requests will hit GitHub's secondary rate limits after a 42-day backfill catch-up, causing silent 403 failures and empty transcripts — the same symptom the PR set out to fix.

💡 Suggested fix

Add a small sleep between iterations to stay within secondary rate limits:

  fi
  sleep 1  # avoid GitHub secondary rate limits (burst protection)
done

Alternatively, add explicit 403 detection:

if ! gh api "repos/$GITHUB_REPOSITORY/actions/jobs/${job_id}/logs" \
  > "/tmp/gh-aw/agent/session-data/logs/${run_id}-raw.log" 2>&1; then
  echo "  Warning: gh api failed for job $job_id (rate limit or expired?)"
  rm -f "/tmp/gh-aw/agent/session-data/logs/${run_id}-raw.log"
  continue
fi

[github-actions]

GitHub truncates large job logs — for long agent sessions the tail of the log (where later turns appear) may be silently cut, producing a partial transcript with no indication of truncation.

💡 Suggested fix

After fetching the raw log, check for a known end-of-job marker to detect truncation:

if grep -q 'Post job cleanup' "/tmp/gh-aw/agent/session-data/logs/${run_id}-raw.log" 2>/dev/null; then
  echo "  Log appears complete (cleanup marker found)"
else
  echo "  Warning: log may be truncated for run $run_id — transcript may be partial"
fi

Alternatively, surface the final turn number in the summary count so reviewers can spot unexpectedly low turn counts.

[github-actions]

Skills-Based Review 🧠

Applied /diagnosing-bugs, /tdd, and /codebase-design — commenting rather than blocking; the core fix is correct but three issues are worth addressing.

📋 Key Themes & Highlights

Key Themes

• Fragile job selection (.jobs[0]): position-based job ID lookup will silently pick the wrong job if the run structure changes, reproducing the empty-transcript problem without any visible error
• Overly broad run filter (conclusion != "action_required"): admits in_progress/queued runs; status == "completed" is the correct guard for completed job logs
• No regression test: the 42-day silent outage stemmed from an undetected assumption change; both bugs would benefit from a minimal CI check anchoring the new filter logic

Positive Highlights

• ✅ Root cause correctly identified and addressed: replaced broken sed-based session-number extraction with run_id-based approach
• ✅ Authentication issue diagnosed and fixed: gh api with standard GITHUB_TOKEN instead of OAuth-gated gh agent-task view --log
• ✅ CI gate filtering is well-commented and reduces wasted API calls (~47 per day)
• ✅ Transcript format documentation updated to match actual [cca-engine] turn= log structure — precise and actionable for downstream analysis
• ✅ Raw log files cleaned up after transcript extraction

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · 52.2 AIC · ⌖ 4.84 AIC · ⊞ 6.6K
_{Comment /matt to run again}

Comments that could not be inline-anchored

.github/workflows/shared/copilot-session-data-fetch.md:106

.github/workflows/shared/copilot-session-data-fetch.md:106

[/diagnosing-bugs] .jobs[0] hard-codes position: if a run ever gains a setup/wrapper job before the Copilot agent job, the wrong job ID is silently selected and all transcripts for that run are empty.

<details>
<summary>💡 Filter by job name instead</summary>

job_id=$(gh api "repos/$GITHUB_REPOSITORY/actions/runs/${run_id}/jobs" \
  --jq '.jobs[] | select(.name | ascii_downcase | contains("copilot")) | .id' \
  2>/dev/null | head -1 || true)

This makes the intent explicit a…

.github/workflows/shared/copilot-session-data-fetch.md:98

[/diagnosing-bugs] The filter conclusion != "action_required" may also admit in_progress and queued runs whose job logs aren't yet complete. Fetching logs for an in-progress run risks an empty or truncated transcript and wastes an API call.

<details>
<summary>💡 Narrow to terminal conclusions</summary>

jq -r '.[] | select(.conclusion == "success" or .conclusion == "failure" or .conclusion == "timed_out") | "\(.id) \(.head_branch)"' ...

Or equivalently, require `status =…

.github/workflows/shared/copilot-session-data-fetch.md:219

[/tdd] There is no regression test covering Bug 1 (branch slug extraction) or Bug 2 (OAuth requirement). Without a test, the next Copilot branch-naming change could silently re-introduce the same 42-day outage.

<details>
<summary>💡 Minimal regression signal</summary>

Even a lightweight CI check would help — for example, a unit test for the jq filter:

# Verify action_required runs are excluded
echo '[{"conclusion":"action_required"},{"conclusion":"success"}]'   | jq '[.[] | selec…

</details>

<details><summary>scripts/check-stale-lock-files.sh:79</summary>

**[/codebase-design]** The exclusion list (`shared/`, `skills/`) is hardcoded as path-prefix patterns. If a new compiled-only subdirectory is added (e.g. `templates/`), this check will silently false-positive again until someone remembers to add another `grep -v` line.

&lt;details&gt;
&lt;summary&gt;💡 Consider a declaration-based approach&lt;/summary&gt;

A small marker file (e.g. `.compiled-only`) in each such directory would make the property self-documenting and allow the script to discover exclusions dynam…

</details>

[github-actions]

🎉 This pull request is included in a new release.

Release: v0.82.8