diff --git a/.github/aw/report.md b/.github/aw/report.md index 237b24841b9..435d4858382 100644 --- a/.github/aw/report.md +++ b/.github/aw/report.md @@ -4,7 +4,7 @@ description: Guidelines for creating agentic workflows that generate reports — # Report Generation -Consult this file when creating an agentic workflow that generates reports — recurring status updates, audits, analysis summaries, or any structured output posted as a GitHub issue, discussion, or comment. +For agentic workflows that generate reports — status updates, audits, summaries, or any structured output posted as a GitHub issue, discussion, or comment. ## Choosing the Right Output Type @@ -14,18 +14,18 @@ Consult this file when creating an agentic workflow that generates reports — r | Inline update on an existing issue or PR | `add-comment` with `hide-older-comments` | | Discussion-based report (only when explicitly requested) | `create-discussion` with `close-older-discussions` | -Use `create-issue` by default for reports — issues are familiar, searchable, and support the full close/expire cleanup mechanism. Only use `create-discussion` when the user explicitly asks for it. +Default to `create-issue` — issues are searchable and support full close/expire cleanup. Only use `create-discussion` when explicitly requested. ## Automatic Cleanup -Reports accumulate over time. Always configure automatic cleanup when the workflow runs on a schedule or recurs. +Always configure cleanup for scheduled or recurring reports. -- **`expires`**: Auto-closes the issue or discussion after a time period (e.g. `7` days, `2w`, `1m`). Use when reports become stale after a fixed window. -- **`close-older-issues: true`**: Closes previous issues from the same workflow before creating a new one. Requires `title-prefix` or `labels` to identify matching issues. -- **`close-older-discussions: true`**: Closes older discussions with the same title prefix or labels as "OUTDATED". Requires `title-prefix` or `labels`. -- **`hide-older-comments: true`**: Minimizes previous comments from the same workflow before posting a new one. Useful for rolling status updates on the same issue or PR. +- **`expires`**: Auto-closes after a time period (e.g. `7`, `2w`, `1m`). Use when reports go stale after a fixed window. +- **`close-older-issues: true`**: Closes previous issues from the same workflow. Requires `title-prefix` or `labels`. +- **`close-older-discussions: true`**: Closes older matching discussions as "OUTDATED". Requires `title-prefix` or `labels`. +- **`hide-older-comments: true`**: Minimizes previous comments from the same workflow. Useful for rolling status updates. -**Default recommendation for recurring reports:** use `create-issue` with `close-older-issues: true` and a stable `title-prefix` so only the latest report is active. +**Recommended for recurring reports**: `create-issue` with `close-older-issues: true` and a stable `title-prefix`. ```yaml safe-outputs: @@ -97,13 +97,11 @@ Keep critical information visible (summary, critical issues, key metrics). ## Avoiding Mentions and Backlinks -Reports often reference issues, PRs, or users. Without filtering, `@username` sends a notification and `#123` creates a cross-reference backlink on that issue or PR — adding noise every time the report runs. +Without filtering, `@username` notifies users and `#123` creates cross-reference backlinks on that issue/PR — noise every run. -Use the built-in safe-outputs filtering options to suppress this automatically: - -- **`mentions: false`** — escapes all `@mentions` in AI-generated output so no notifications are sent. -- **`allowed-github-references: []`** — escapes all `#123` / `owner/repo#123` references so no backlinks are created on referenced items. -- **`max-bot-mentions: 0`** — neutralizes bot-trigger phrases such as `fixes #123` or `closes #456` that would otherwise close referenced issues. +- **`mentions: false`** — escapes all `@mentions`, no notifications sent. +- **`allowed-github-references: []`** — escapes all `#123` / `owner/repo#123` references, no backlinks. +- **`max-bot-mentions: 0`** — neutralizes bot-trigger phrases like `fixes #123` / `closes #456`. ```yaml safe-outputs: @@ -117,4 +115,4 @@ safe-outputs: expires: 30 ``` -These options apply globally to all safe-output types (issues, comments, discussions) and are the recommended way to keep reports from polluting unrelated items. +Applies globally to all safe-output types (issues, comments, discussions). diff --git a/.github/aw/runbooks/workflow-health.md b/.github/aw/runbooks/workflow-health.md index f2499ab7cc4..3314c8e65cd 100644 --- a/.github/aw/runbooks/workflow-health.md +++ b/.github/aw/runbooks/workflow-health.md @@ -4,47 +4,23 @@ ### Missing Tool Configurations -**Symptoms**: -- Error messages containing "missing-tool" or "tool not found" -- Workflow fails when attempting to access GitHub APIs -- Agent cannot perform GitHub operations (read issues, create PRs, etc.) - -**Common Causes**: -- GitHub MCP server not configured in workflow frontmatter -- Missing toolsets specification -- Incorrect toolset names +Errors like "missing-tool" or "tool not found"; agent cannot perform GitHub operations. Cause: GitHub MCP server not configured in frontmatter, missing/incorrect `toolsets:`. ### Authentication and Permission Errors -**Symptoms**: -- HTTP 403 (Forbidden) errors -- "Resource not accessible" errors -- Token scope errors - -**Common Causes**: -- Missing `permissions:` block in workflow frontmatter -- Insufficient token permissions for requested operations -- GITHUB_TOKEN not passed to custom actions +HTTP 403, "Resource not accessible", or token scope errors. Cause: missing or insufficient `permissions:` block; GITHUB_TOKEN not passed to custom actions. ### Input/Secret Validation Failures -**Symptoms**: -- MCP Scripts action fails -- Environment variable not available -- Template expression evaluation errors - -**Common Causes**: -- MCP Scripts action not configured -- Missing required secrets -- Incorrect secret references +MCP Scripts action fails; env var unavailable; template expression errors. Cause: MCP Scripts not configured, missing required secrets, or incorrect secret references. ## Investigation Steps ### Step 1: Analyze Workflow Logs -Use the `gh aw logs` command to download and analyze workflow logs: +Download and analyze with `gh aw logs`: -> **Note**: The commands below are meant to be run from a local machine or a Copilot coding agent session. If you include `gh aw logs` or `gh aw audit` as steps inside a generated workflow, you must add `actions: read` to `permissions:` and install the extension with the `setup-cli` action before calling these commands — see [Logs and Metrics](../github-agentic-workflows.md#logs-and-metrics) for details. +> **Note**: Run these commands locally or from a Copilot coding agent session. If you include `gh aw logs` / `gh aw audit` as workflow steps, add `actions: read` to `permissions:` and install the extension with the `setup-cli` action first — see [Logs and Metrics](../github-agentic-workflows.md#logs-and-metrics). ```bash # Download logs from last 24 hours @@ -57,31 +33,21 @@ gh aw logs --run-id -o /tmp/workflow-logs gh aw logs --workflow --start-date -7d ``` -**What to look for**: -- Error messages in the "Run AI Agent" step -- Missing-tool errors -- HTTP error codes (401, 403, 404, 500) -- Stack traces or exception details +Look in the "Run AI Agent" step for missing-tool errors, HTTP codes (401/403/404/500), and stack traces. ### Step 2: Identify Missing-Tool Errors -Missing-tool errors typically appear in this format: +Missing-tool errors look like: ``` Error: Tool 'github:read_issue' not found Error: missing tool configuration for mcpscripts-gh ``` -To identify which tools are missing: - -1. Check the workflow `.md` file for the `tools:` section -2. Compare with similar working workflows -3. Verify the tool is properly configured in frontmatter +Check the workflow's `tools:` section, compare with similar working workflows, and verify the tool is configured in frontmatter. ### Step 3: Verify MCP Server Configurations -Check if the workflow has proper MCP server configuration: - ```aw --- tools: @@ -90,74 +56,34 @@ tools: --- ``` -Use `gh aw mcp inspect ` to verify MCP server configuration: +Verify with: ```bash -# Inspect MCP servers for a workflow -gh aw mcp inspect - -# List all workflows with MCP servers -gh aw mcp list +gh aw mcp inspect # Inspect MCP servers for a workflow +gh aw mcp list # List all workflows with MCP servers ``` ### Step 4: Check Permissions Configuration -Verify the workflow has required permissions: - ```aw --- permissions: - contents: read # For reading repository files - issues: write # For creating/updating issues - pull-requests: write # For creating/updating PRs - actions: read # For accessing workflow runs + contents: read # repository files + issues: write # create/update issues + pull-requests: write # create/update PRs + actions: read # access workflow runs --- ``` -Common permission requirements: -- **Reading issues**: `issues: read` -- **Creating issues**: `issues: write` -- **Reading PRs**: `pull-requests: read` -- **Creating PRs**: `pull-requests: write` -- **Reading workflow runs**: `actions: read` +Match the permission scope to the operation: read for queries, write for create/update. ## Resolution Procedures ### Adding GitHub MCP Server to Workflows -**Problem**: Workflow fails with missing GitHub tool errors. - -**Solution**: Add GitHub MCP server configuration to the workflow frontmatter. +Add `tools.github.toolsets` to the workflow frontmatter, then `gh aw compile .md` and `gh aw mcp inspect ` to verify. -1. Open the workflow `.md` file -2. Add or update the `tools:` section: - -```aw ---- -tools: - github: - toolsets: [default] ---- -``` - -3. Compile the workflow: - -```bash -gh aw compile .md -``` - -4. Verify the configuration: - -```bash -gh aw mcp inspect -``` - -**Available toolsets**: -- `default`: repositories, issues, pull requests, and common operations -- `repos`: repository management tools -- `issues`: issue operations -- `pull_requests`: PR operations -- `actions`: GitHub Actions workflow tools +**Available toolsets**: `default` (repos + issues + pull_requests + common), `repos`, `issues`, `pull_requests`, `actions`. **Example**: Dev workflow with GitHub MCP server @@ -183,13 +109,7 @@ Analyze repository issues and provide insights. ### Configuring MCP Scripts and Safe-Outputs -**Problem**: Workflow fails with missing mcpscripts-gh or safe-output errors. - -**Solution**: Configure mcp-scripts and safe-outputs in the workflow. - -#### Adding MCP Scripts - -MCP Scripts securely pass GitHub context to AI agents: +MCP Scripts pass GitHub context to the agent as env vars: ```aw --- @@ -201,11 +121,7 @@ mcp-scripts: --- ``` -The mcp-scripts are automatically made available to the agent as environment variables. - -#### Adding Safe-Outputs - -Safe-outputs enable AI agents to create GitHub resources: +Safe-outputs let the agent create GitHub resources: ```aw --- @@ -219,118 +135,27 @@ safe-outputs: --- ``` -**Example**: Complete workflow with mcp-scripts and safe-outputs - -```aw ---- -description: Issue triage workflow -on: - issues: - types: [opened] -permissions: - contents: read - issues: write -engine: copilot -tools: - github: - toolsets: [default] -mcp-scripts: - issue: - title: ${{ github.event.issue.title }} - body: ${{ github.event.issue.body }} - number: ${{ github.event.issue.number }} -safe-outputs: - create-issue: - labels: ["ai-generated", "triage"] ---- - -# Issue Triage Agent - -Analyze the issue and determine appropriate labels and priority. -``` - ### Testing Workflow Fixes -After making changes, test the workflow: - -1. **Compile the workflow**: - -```bash -gh aw compile .md -``` - -2. **Trigger manually** (if `workflow_dispatch` is enabled): - ```bash -gh workflow run .lock.yml -``` - -3. **Monitor the run**: - -```bash -# Get the run ID -gh run list --workflow=.lock.yml --limit 1 - -# Watch the run -gh run watch - -# Download logs if it fails -gh aw logs --run-id +gh aw compile .md # compile +gh workflow run .lock.yml # trigger (if workflow_dispatch) +gh run list --workflow=.lock.yml --limit 1 # get run ID +gh run watch # monitor +gh aw logs --run-id # logs on failure ``` -4. **Verify success**: - - Check that no missing-tool errors occur - - Verify the agent completes successfully - - Confirm any created resources (issues, PRs, discussions) +Verify: no missing-tool errors, agent completes, expected resources created. ## Case Study: DeepReport Incident Response -Three failing workflows were fixed: +Three failures fixed: -**Weekly Issue Summary** — missing `actions: read` permission. Added and recompiled. - -**Dev Workflow** — "Tool 'github:read_issue' not found" (GitHub MCP server not configured): - -```aw -tools: - github: - toolsets: [default] -``` - -**Daily Copilot PR Merged** — "missing tool configuration for mcpscripts-gh": - -```aw -mcp-scripts: - pull_request: - number: ${{ github.event.pull_request.number }} - title: ${{ github.event.pull_request.title }} -``` - -## Quick Reference - -### Essential Commands - -```bash -# Download recent workflow logs -gh aw logs --start-date -1d -o /tmp/logs - -# Inspect MCP configuration -gh aw mcp inspect - -# List all workflows with MCP servers -gh aw mcp list - -# Compile workflow after changes -gh aw compile .md - -# Trigger workflow manually -gh workflow run .lock.yml - -# Watch workflow execution -gh run watch -``` +- **Weekly Issue Summary** — missing `actions: read` permission. +- **Dev Workflow** — "Tool 'github:read_issue' not found": added `tools.github.toolsets: [default]`. +- **Daily Copilot PR Merged** — "missing tool configuration for mcpscripts-gh": added `mcp-scripts.pull_request` with `number`/`title` from `github.event.pull_request`. -### Common Configuration Patterns +## Common Configuration Patterns **Basic GitHub integration**: ```aw