Agent Persona Exploration - 2026-04-30

[github-actions[bot]]

Persona Overview

• Agent: agentic-workflows (create-agentic-workflow custom agent)
• Scenarios Tested: 4
• Average Quality Score: 4.55/5.0
• Run: §25146547353

Key Findings

• 🟢 Trigger selection was excellent across all scenarios — agents correctly used pull_request with paths: filters, workflow_run, schedule, and other appropriate triggers (5/5 average)
• 🟢 Tool selection was correct for all scenarios — GitHub MCP toolsets were always preferred over direct API access (5/5 average)
• 🟢 Safe-outputs were always configured correctly — add-pr-comment, create-issue, create-discussion as appropriate
• 🔴 Critical security gap: 3 of 4 scenarios incorrectly added write permissions to the agent job (e.g., pull-requests: write, discussions: write) instead of keeping the agent read-only and relying solely on safe-outputs
• 🟢 Prompts were consistently well-structured, actionable, and included helpful output format templates

Top Patterns

1. Most common trigger: pull_request: types: [opened, synchronize] for PR-based workflows
2. Tool default: github: toolsets: [default] (DevOps scenario correctly used specific toolsets [actions, issues, repos])
3. Security pattern inconsistency: Agents split on whether to add write permissions directly vs. keep read-only

View High Quality Responses (Top 2)

DevOps — Deployment Incident Reporter (5.0/5.0)

• Used workflow_run: types: [completed] trigger — correct and precise
• Kept agent job permissions read-only (actions: read, contents: read) ✅
• Used specific GitHub toolsets: [actions, issues, repos] instead of generic [default]
• Added max: 1 + expires: 1h rate limiting on create-issue safe-output — preventing issue flooding
• Prompt included conditional early-exit logic ("if conclusion is not failure, stop")

Backend Engineer — DB Migration Review (4.4/5.0)

• Excellent use of paths: filter on pull_request trigger to scope to SQL files only
• Comprehensive migration safety checklist in prompt (critical vs. warning vs. informational)
• Well-structured output template for PR comment

View Areas for Improvement

Critical: Write permissions on agent job (3/4 scenarios affected)

The agent repeatedly added write permissions directly to the agent job:

# ❌ Generated (incorrect)
permissions:
  pull-requests: write   # Should NOT be here

The correct pattern per .github/aw/create-agentic-workflow.md:

# ✅ Correct — agent stays read-only
permissions:
  pull-requests: read
  contents: read

safe-outputs:
  add-pr-comment:    # Writes go through safe-outputs only

The DevOps scenario was the only one that got this right — likely because its write operation (create-issue) was clearly separated from reading GitHub data.

Minor: Generic toolset [default] vs. specific toolsets

Most scenarios defaulted to toolsets: [default] which works but is broader than necessary. Scoped toolsets (e.g., [issues, pull_requests]) would follow least-privilege principles better.

Recommendations

1. Strengthen the permission model guidance in .github/aw/create-agentic-workflow.md — add a prominently-placed "permission quick-reference table" showing which safe-output requires which read permission (e.g., add-pr-comment → pull-requests: read, never write). The current guidance exists but agents still generate write permissions 75% of the time.

2. Add a "safe-outputs permission mapping" section to .github/aw/github-agentic-workflows.md — explicitly document that all safe-outputs operate independently of the agent job's permission scope, and that adding write permissions to the agent job is always wrong when safe-outputs are in use.

3. Encourage specific GitHub toolsets over [default] — the DevOps scenario demonstrated better least-privilege by using [actions, issues, repos]. Consider adding guidance to scope toolsets to only what the workflow needs.

References:

• §25146547353

Generated by Agent Persona Explorer · ● 921.9K · ◷

[github-actions[bot]]

This discussion has been marked as outdated by Agent Persona Explorer.

A newer discussion is available at Discussion #29387.