Skip to content

[caveman] Optimize instruction verbosity — workflow-health, report (2026-05-19) #33393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 1 commit into from
May 19, 2026
Merged

[caveman] Optimize instruction verbosity — workflow-health, report (2026-05-19) #33393

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 13 additions & 15 deletions .github/aw/report.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Guidelines for creating agentic workflows that generate reports —

# Report Generation

Consult this file when creating an agentic workflow that generates reports — recurring status updates, audits, analysis summaries, or any structured output posted as a GitHub issue, discussion, or comment.
For agentic workflows that generate reports — status updates, audits, summaries, or any structured output posted as a GitHub issue, discussion, or comment.

## Choosing the Right Output Type

Expand All @@ -14,18 +14,18 @@ Consult this file when creating an agentic workflow that generates reports — r
| Inline update on an existing issue or PR | `add-comment` with `hide-older-comments` |
| Discussion-based report (only when explicitly requested) | `create-discussion` with `close-older-discussions` |

Use `create-issue` by default for reports — issues are familiar, searchable, and support the full close/expire cleanup mechanism. Only use `create-discussion` when the user explicitly asks for it.
Default to `create-issue` — issues are searchable and support full close/expire cleanup. Only use `create-discussion` when explicitly requested.

## Automatic Cleanup

Reports accumulate over time. Always configure automatic cleanup when the workflow runs on a schedule or recurs.
Always configure cleanup for scheduled or recurring reports.

- **`expires`**: Auto-closes the issue or discussion after a time period (e.g. `7` days, `2w`, `1m`). Use when reports become stale after a fixed window.
- **`close-older-issues: true`**: Closes previous issues from the same workflow before creating a new one. Requires `title-prefix` or `labels` to identify matching issues.
- **`close-older-discussions: true`**: Closes older discussions with the same title prefix or labels as "OUTDATED". Requires `title-prefix` or `labels`.
- **`hide-older-comments: true`**: Minimizes previous comments from the same workflow before posting a new one. Useful for rolling status updates on the same issue or PR.
- **`expires`**: Auto-closes after a time period (e.g. `7`, `2w`, `1m`). Use when reports go stale after a fixed window.
- **`close-older-issues: true`**: Closes previous issues from the same workflow. Requires `title-prefix` or `labels`.
- **`close-older-discussions: true`**: Closes older matching discussions as "OUTDATED". Requires `title-prefix` or `labels`.
- **`hide-older-comments: true`**: Minimizes previous comments from the same workflow. Useful for rolling status updates.

**Default recommendation for recurring reports:** use `create-issue` with `close-older-issues: true` and a stable `title-prefix` so only the latest report is active.
**Recommended for recurring reports**: `create-issue` with `close-older-issues: true` and a stable `title-prefix`.

```yaml
safe-outputs:
Expand Down Expand Up @@ -97,13 +97,11 @@ Keep critical information visible (summary, critical issues, key metrics).

## Avoiding Mentions and Backlinks

Reports often reference issues, PRs, or users. Without filtering, `@username` sends a notification and `#123` creates a cross-reference backlink on that issue or PR — adding noise every time the report runs.
Without filtering, `@username` notifies users and `#123` creates cross-reference backlinks on that issue/PR — noise every run.

Use the built-in safe-outputs filtering options to suppress this automatically:

- **`mentions: false`** — escapes all `@mentions` in AI-generated output so no notifications are sent.
- **`allowed-github-references: []`** — escapes all `#123` / `owner/repo#123` references so no backlinks are created on referenced items.
- **`max-bot-mentions: 0`** — neutralizes bot-trigger phrases such as `fixes #123` or `closes #456` that would otherwise close referenced issues.
- **`mentions: false`** — escapes all `@mentions`, no notifications sent.
- **`allowed-github-references: []`** — escapes all `#123` / `owner/repo#123` references, no backlinks.
- **`max-bot-mentions: 0`** — neutralizes bot-trigger phrases like `fixes #123` / `closes #456`.

```yaml
safe-outputs:
Expand All @@ -117,4 +115,4 @@ safe-outputs:
expires: 30
```

These options apply globally to all safe-output types (issues, comments, discussions) and are the recommended way to keep reports from polluting unrelated items.
Applies globally to all safe-output types (issues, comments, discussions).
237 changes: 31 additions & 206 deletions .github/aw/runbooks/workflow-health.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,47 +4,23 @@

### Missing Tool Configurations

**Symptoms**:
- Error messages containing "missing-tool" or "tool not found"
- Workflow fails when attempting to access GitHub APIs
- Agent cannot perform GitHub operations (read issues, create PRs, etc.)

**Common Causes**:
- GitHub MCP server not configured in workflow frontmatter
- Missing toolsets specification
- Incorrect toolset names
Errors like "missing-tool" or "tool not found"; agent cannot perform GitHub operations. Cause: GitHub MCP server not configured in frontmatter, missing/incorrect `toolsets:`.

### Authentication and Permission Errors

**Symptoms**:
- HTTP 403 (Forbidden) errors
- "Resource not accessible" errors
- Token scope errors

**Common Causes**:
- Missing `permissions:` block in workflow frontmatter
- Insufficient token permissions for requested operations
- GITHUB_TOKEN not passed to custom actions
HTTP 403, "Resource not accessible", or token scope errors. Cause: missing or insufficient `permissions:` block; GITHUB_TOKEN not passed to custom actions.

### Input/Secret Validation Failures

**Symptoms**:
- MCP Scripts action fails
- Environment variable not available
- Template expression evaluation errors

**Common Causes**:
- MCP Scripts action not configured
- Missing required secrets
- Incorrect secret references
MCP Scripts action fails; env var unavailable; template expression errors. Cause: MCP Scripts not configured, missing required secrets, or incorrect secret references.

## Investigation Steps

### Step 1: Analyze Workflow Logs

Use the `gh aw logs` command to download and analyze workflow logs:
Download and analyze with `gh aw logs`:

> **Note**: The commands below are meant to be run from a local machine or a Copilot coding agent session. If you include `gh aw logs` or `gh aw audit` as steps inside a generated workflow, you must add `actions: read` to `permissions:` and install the extension with the `setup-cli` action before calling these commands — see [Logs and Metrics](../github-agentic-workflows.md#logs-and-metrics) for details.
> **Note**: Run these commands locally or from a Copilot coding agent session. If you include `gh aw logs` / `gh aw audit` as workflow steps, add `actions: read` to `permissions:` and install the extension with the `setup-cli` action first — see [Logs and Metrics](../github-agentic-workflows.md#logs-and-metrics).

```bash
# Download logs from last 24 hours
Expand All @@ -57,31 +33,21 @@ gh aw logs --run-id <run-id> -o /tmp/workflow-logs
gh aw logs --workflow <workflow-name> --start-date -7d
```

**What to look for**:
- Error messages in the "Run AI Agent" step
- Missing-tool errors
- HTTP error codes (401, 403, 404, 500)
- Stack traces or exception details
Look in the "Run AI Agent" step for missing-tool errors, HTTP codes (401/403/404/500), and stack traces.

### Step 2: Identify Missing-Tool Errors

Missing-tool errors typically appear in this format:
Missing-tool errors look like:

```
Error: Tool 'github:read_issue' not found
Error: missing tool configuration for mcpscripts-gh
```

To identify which tools are missing:

1. Check the workflow `.md` file for the `tools:` section
2. Compare with similar working workflows
3. Verify the tool is properly configured in frontmatter
Check the workflow's `tools:` section, compare with similar working workflows, and verify the tool is configured in frontmatter.

### Step 3: Verify MCP Server Configurations

Check if the workflow has proper MCP server configuration:

```aw
---
tools:
Expand All @@ -90,74 +56,34 @@ tools:
---
```

Use `gh aw mcp inspect <workflow-name>` to verify MCP server configuration:
Verify with:

```bash
# Inspect MCP servers for a workflow
gh aw mcp inspect <workflow-name>

# List all workflows with MCP servers
gh aw mcp list
gh aw mcp inspect <workflow-name> # Inspect MCP servers for a workflow
gh aw mcp list # List all workflows with MCP servers
```

### Step 4: Check Permissions Configuration

Verify the workflow has required permissions:

```aw
---
permissions:
contents: read # For reading repository files
issues: write # For creating/updating issues
pull-requests: write # For creating/updating PRs
actions: read # For accessing workflow runs
contents: read # repository files
issues: write # create/update issues
pull-requests: write # create/update PRs
actions: read # access workflow runs
---
```

Common permission requirements:
- **Reading issues**: `issues: read`
- **Creating issues**: `issues: write`
- **Reading PRs**: `pull-requests: read`
- **Creating PRs**: `pull-requests: write`
- **Reading workflow runs**: `actions: read`
Match the permission scope to the operation: read for queries, write for create/update.

## Resolution Procedures

### Adding GitHub MCP Server to Workflows

**Problem**: Workflow fails with missing GitHub tool errors.

**Solution**: Add GitHub MCP server configuration to the workflow frontmatter.
Add `tools.github.toolsets` to the workflow frontmatter, then `gh aw compile <workflow-name>.md` and `gh aw mcp inspect <workflow-name>` to verify.

1. Open the workflow `.md` file
2. Add or update the `tools:` section:

```aw
---
tools:
github:
toolsets: [default]
---
```

3. Compile the workflow:

```bash
gh aw compile <workflow-name>.md
```

4. Verify the configuration:

```bash
gh aw mcp inspect <workflow-name>
```

**Available toolsets**:
- `default`: repositories, issues, pull requests, and common operations
- `repos`: repository management tools
- `issues`: issue operations
- `pull_requests`: PR operations
- `actions`: GitHub Actions workflow tools
**Available toolsets**: `default` (repos + issues + pull_requests + common), `repos`, `issues`, `pull_requests`, `actions`.

**Example**: Dev workflow with GitHub MCP server

Expand All @@ -183,13 +109,7 @@ Analyze repository issues and provide insights.

### Configuring MCP Scripts and Safe-Outputs

**Problem**: Workflow fails with missing mcpscripts-gh or safe-output errors.

**Solution**: Configure mcp-scripts and safe-outputs in the workflow.

#### Adding MCP Scripts

MCP Scripts securely pass GitHub context to AI agents:
MCP Scripts pass GitHub context to the agent as env vars:

```aw
---
Expand All @@ -201,11 +121,7 @@ mcp-scripts:
---
```

The mcp-scripts are automatically made available to the agent as environment variables.

#### Adding Safe-Outputs

Safe-outputs enable AI agents to create GitHub resources:
Safe-outputs let the agent create GitHub resources:

```aw
---
Expand All @@ -219,118 +135,27 @@ safe-outputs:
---
```

**Example**: Complete workflow with mcp-scripts and safe-outputs

```aw
---
description: Issue triage workflow
on:
issues:
types: [opened]
permissions:
contents: read
issues: write
engine: copilot
tools:
github:
toolsets: [default]
mcp-scripts:
issue:
title: ${{ github.event.issue.title }}
body: ${{ github.event.issue.body }}
number: ${{ github.event.issue.number }}
safe-outputs:
create-issue:
labels: ["ai-generated", "triage"]
---

# Issue Triage Agent

Analyze the issue and determine appropriate labels and priority.
```

### Testing Workflow Fixes

After making changes, test the workflow:

1. **Compile the workflow**:

```bash
gh aw compile <workflow-name>.md
```

2. **Trigger manually** (if `workflow_dispatch` is enabled):

```bash
gh workflow run <workflow-name>.lock.yml
```

3. **Monitor the run**:

```bash
# Get the run ID
gh run list --workflow=<workflow-name>.lock.yml --limit 1

# Watch the run
gh run watch <run-id>

# Download logs if it fails
gh aw logs --run-id <run-id>
gh aw compile <workflow-name>.md # compile
gh workflow run <workflow-name>.lock.yml # trigger (if workflow_dispatch)
gh run list --workflow=<workflow-name>.lock.yml --limit 1 # get run ID
gh run watch <run-id> # monitor
gh aw logs --run-id <run-id> # logs on failure
```

4. **Verify success**:
- Check that no missing-tool errors occur
- Verify the agent completes successfully
- Confirm any created resources (issues, PRs, discussions)
Verify: no missing-tool errors, agent completes, expected resources created.

## Case Study: DeepReport Incident Response

Three failing workflows were fixed:
Three failures fixed:

**Weekly Issue Summary** — missing `actions: read` permission. Added and recompiled.

**Dev Workflow** — "Tool 'github:read_issue' not found" (GitHub MCP server not configured):

```aw
tools:
github:
toolsets: [default]
```

**Daily Copilot PR Merged** — "missing tool configuration for mcpscripts-gh":

```aw
mcp-scripts:
pull_request:
number: ${{ github.event.pull_request.number }}
title: ${{ github.event.pull_request.title }}
```

## Quick Reference

### Essential Commands

```bash
# Download recent workflow logs
gh aw logs --start-date -1d -o /tmp/logs

# Inspect MCP configuration
gh aw mcp inspect <workflow-name>

# List all workflows with MCP servers
gh aw mcp list

# Compile workflow after changes
gh aw compile <workflow-name>.md

# Trigger workflow manually
gh workflow run <workflow-name>.lock.yml

# Watch workflow execution
gh run watch <run-id>
```
- **Weekly Issue Summary** — missing `actions: read` permission.
- **Dev Workflow** — "Tool 'github:read_issue' not found": added `tools.github.toolsets: [default]`.
- **Daily Copilot PR Merged** — "missing tool configuration for mcpscripts-gh": added `mcp-scripts.pull_request` with `number`/`title` from `github.event.pull_request`.

### Common Configuration Patterns
## Common Configuration Patterns

**Basic GitHub integration**:
```aw
Expand Down
Loading