Copilot session analysis workflow already exists and meets requirements

[Copilot]

The requested workflow for analyzing Copilot agent sessions and extracting insights already exists at .github/workflows/copilot-session-insights.md and fully implements the problem statement requirements.

Existing Implementation

The copilot-session-insights.md workflow provides:

• Scheduled execution: Daily at 8:00 AM Pacific (16:00 UTC) + manual dispatch
• Session acquisition: Downloads ~50 agent sessions via gh agent-task CLI
• Persistent cache: Stores historical analysis, strategies, and patterns in cache-memory
• Experimental analysis: 30% probability of trying novel analytical approaches each run
• Automated reporting: Publishes findings as GitHub Discussion in "audits" category

Configuration

engine: claude
tools:
  cache-memory: true
  github:
    allowed: [search_pull_requests, pull_request_read, list_pull_requests]
  bash:
    - "gh agent-task list *"
    - "gh agent-task view *"
safe-outputs:
  create-discussion:
    category: "audits"

Analysis Phases

The workflow prompt defines systematic analysis covering:

• Completion rates and loop detection
• Prompt structure and quality indicators
• Context confusion and error recovery patterns
• Tool usage effectiveness
• Experimental strategies (semantic clustering, temporal analysis, code quality metrics)

No changes required—workflow is compiled and production-ready.

Original prompt

create a github agentic workflow that analyzes Copilot agent sessions and extracts insights

Objective

Create a recurring GitHub Agentic Workflow that downloads and analyzes approximately 50 Copilot agent sessions per run. The workflow should automatically identify behavioral patterns, inefficiencies, or improvement opportunities across sessions and publish a summarized report as a GitHub discussion. The long-term goal is to enhance the quality and efficiency of Copilot agentic task prompts based on observed trends.

Context

Copilot agent sessions are recorded interactions between users and AI-driven agents. By periodically sampling and analyzing these sessions, the workflow can surface insights into prompt structure, agent reasoning loops, error recovery, and successful task completions. Storing findings and experimentation strategies in cache memory allows the agent to evolve its analytical approach over time.

Requirements

• Schedule workflow to run daily at 8:00 AM Pacific Time (or on manual dispatch).
• Use the GitHub CLI (gh) to list and download up to 50 Copilot agent sessions.
• Store downloaded session files in a temporary working directory.
• Maintain a cache-memory to store:
  • Past analysis results.
  • Discovered strategies and heuristics.
  • Novel analysis methods attempted in previous runs.
• Ensure ~30% of runs experiment with a new analysis strategy.
• Summarize insights and recommendations in a GitHub Discussion under a configurable category.
• Use the repository’s GH_TOKEN for authenticated API calls.
• Ensure compliance with GitHub Actions runtime and API rate limits.

Phases

Phase 0: Setup and Prerequisites

• Confirm that GH_TOKEN is available in the environment.
• Install and authenticate the GitHub CLI (gh auth status).
• Initialize or restore cache-memory files (e.g., /tmp/agent-analysis-cache.json).
• Create a temporary working directory for session downloads.

Phase 1: Data Acquisition

• Use gh copilot sessions list to retrieve recent session metadata.
• Randomly sample or limit to the most recent 50 sessions.
• Download session logs to the temp directory.
• Record metadata such as timestamps, agent model versions, and task categories.

Phase 2: Session Analysis

• Iterate through each downloaded session file.
• Apply multiple analysis strategies, for example:
  • Detect repetitive or looping agent responses.
  • Identify sessions that resulted in aborted or incomplete tasks.
  • Cluster prompts by structure or keywords.
  • Detect signs of confusion or missing context.
• Cache the outcomes of each strategy.
• Introduce exploration: with 30% probability, attempt a novel analytical heuristic (e.g., clustering prompt embeddings, new scoring metric).

Phase 3: Insight Synthesis

• Aggregate all observations into concise categories:
  • Success factors — patterns associated with successful completions.
  • Failure signals — common indicators of confusion or inefficiency.
  • Prompt quality indicators — phrasing, keywords, or structure linked to higher success.
• Summarize recommendations for agent improvement.
• Include pseudo-code or logic sketches for new analytical methods discovered during exploration.

Phase 4: Reporting

• Generate a human-readable Markdown report summarizing:
  • Key insights.
  • Statistical summaries (e.g., success ratios, failure clusters).
  • Any actionable recommendations (e.g., prompt wording fixes, new instruction additions).
• Post this report to a GitHub Discussion titled:

  Daily Copilot Agent Session Analysis — [YYYY-MM-DD]
  

• Tag the discussion with labels such as agent-insights, daily-report, and prompt-analysis.

Acceptance Criteria

• Workflow runs automatically once per day or on manual dispatch.
• Successfully downloads and analyzes ~50 sessions per run.
• Caches prior results and uses them to evolve strategies.
• Posts a well-structured GitHub Discussion summarizing insights.
• Discussion includes at least one actionable recommendation per run.

Constraints

• Do not include or expose sensitive session data.
• The analysis should not execute or replay session commands.
• Each run must complete within GitHub Actions’ maximum runtime limit.
• No actual implementation code is written by the assistant; only specifications and pseudo-code logic.

Note

Custom agent used: create-agentic-workflow
Design agentic workflows using GitHub Agentic Workflows (gh-aw) extension with interactive guidance on triggers, tools, and security best practices.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.