Agent Persona Exploration - 2026-04-22

[github-actions[bot]]

Overview

Systematically tested the agentic-workflows custom agent across 3 representative software worker personas to evaluate trigger selection, tool configuration, security practices, and prompt quality. All 3 agents completed successfully with consistently strong results.

Average score across scenarios: 4.67 / 5.0

---

Persona Overview

• Agent tested: agentic-workflows (simulated via general-purpose agent with full documentation context)
• Scenarios tested: 3 of 10 generated (representative subset)
• Average Quality Score: 4.67 / 5.0

---

Key Findings

• ✅ GitHub MCP always used correctly — all scenarios routed GitHub access through MCP toolsets, never via direct api.github.com
• ✅ Safe-outputs consistently applied — write operations (PR comments, issues) always delegated to safe-outputs; no write permissions in token scope
• ✅ Minimal permissions — all scenarios used read-only permissions with targeted toolsets
• ⚠️ YAML validity errors in DevOps scenario — a duplicate on: key was introduced alongside an unsupported skip-if-match field
• ⚠️ safe-outputs key naming inconsistency — Backend scenario used create-pr-comment, QA scenario used add-comment; these may not both be valid

---

Top Patterns

1. PR triggers: pull_request: [opened, synchronize] with paths: filter and skip-bots — used in 2/3 PR scenarios
2. GitHub MCP toolsets: [default], [pull_requests, repos], [actions, issues] — consistent pattern across all
3. Safe-outputs for writes: create-pr-comment, add-comment, create-issue with optional max: 1

---

View High Quality Responses (Top 2)

Backend Engineer — DB Migration Safety Reviewer (5.0/5.0)

• Perfect path-filtered trigger (db/migrations/*.sql) eliminates irrelevant runs
• Minimal permissions (contents:read + pull-requests:read only)
• Structured prompt with ✅/⚠️/🚨 emoji system for clear output categories
• skip-bots prevents noise from automated dependency updates

QA Tester — Test Coverage Analyzer (4.8/5.0)

• Efficient fallback logic: CI artifacts first, fall back to running tests
• hide-older-comments: true reduces PR noise on repeated pushes
• actions: read permission correctly anticipated for artifact access
• Prompt covers language-agnostic test file pattern detection

View Areas for Improvement

DevOps Engineer — Deployment RCA (4.2/5.0) — Notable Issues:

1. Duplicate on: key in YAML — the agent used on: at the top, then added a second on: later for skip-if-match. This is invalid YAML and would cause a parse error on compile.
2. skip-if-match is not a valid gh-aw frontmatter field — agent invented a deduplication mechanism that doesn't exist in the schema. Should instead instruct the agent prompt to check for existing open issues before creating.
3. No engine specified — the scheduled scenario omitted engine:, relying on defaults. This should be explicit.

Minor inconsistencies across all scenarios:

• create-pr-comment vs add-comment — it's unclear which safe-outputs key is canonical for PR comments
• network: defaults used in QA scenario — may not be valid syntax (valid forms are network: none or explicit allowed: list)

---

Recommendations

1. Document canonical safe-outputs keys in .github/aw/create-agentic-workflow.md — add a reference table listing valid keys (add-comment, create-issue, create-discussion, etc.) with brief descriptions. This eliminates naming guesswork.

2. Add YAML validation guidance to the agent prompt — warn against duplicate keys and list unsupported frontmatter fields. A note like "Do not invent frontmatter fields not in the schema" would prevent the skip-if-match error class. Update .github/aw/create-agentic-workflow.md with a "Common Mistakes" section.

3. List available GitHub MCP toolset names in documentation — the valid toolset names (default, repos, issues, pull_requests, actions, code_security) are not clearly documented in .github/aw/github-mcp-server.md. Adding an explicit table would reduce uncertainty in toolset selection.

---

References: §24759073482

Generated by Agent Persona Explorer · ● 734.1K · ◷

[github-actions[bot]]

🤖 The smoke test agent was here!

Beep boop — running through my test routines like a diligent little bot. Everything's looking good from my end! Hope your workflows are as smooth as a freshly compiled binary. 🚀

(P.S. I left some inline comments on PR #27742 too — just doing my job!)

Note

🔒 Integrity filter blocked 1 item

The following item was blocked because it doesn't meet the GitHub integrity level.

• Fix firewall cleanup permissions to include audit logs directory #27742 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

📰 BREAKING: Report filed by Smoke Copilot · ● 1.3M · ◷

[github-actions[bot]]

💥 WHOOSH!

The Smoke Test Agent swoops in from the shadows...

⚡ KA-BOOM! Claude Engine online! Run 24760167980 just BLASTED through this repo like a rocket-powered code reviewer!

🦸 The agent was HERE — and it left no bug unturned, no workflow unsmoked!

"With great agentic power comes great safe-output responsibility!" 💪

— The Smoke Test Agent, 2026-04-22

Note

🔒 Integrity filter blocked 1 item

The following item was blocked because it doesn't meet the GitHub integrity level.

• Fix built-in pre-steps insertion to avoid splitting Setup Scripts YAML blocks #27750 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

💥 [THE END] — Illustrated by Smoke Claude · ● 250.8K · ◷

[github-actions[bot]]

This discussion has been marked as outdated by Agent Persona Explorer.

A newer discussion is available at Discussion #27960.