[delight] User Experience Analysis Report - 2026-08-23

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (model-tables.md, triggering-ci.mdx)
• 0 CLI commands (binary build unavailable in this environment — see note below)
• 2 workflow message configurations (approach-validator.md, smoke-agent-public-none.md)
• 1 validation file (safe_outputs_step_token_validation.go)

Overall Quality: Professional, with one high-value clarity gap in reference documentation.

Key Finding: model-tables.md presents a large, well-organized alias table but never explains what happens when no alias pattern matches an available model — a common question for anyone debugging model resolution.

Note on CLI section: The ./gh-aw binary could not be built in this sandboxed run (Go toolchain download was blocked by network policy), so CLI help-text quality was not evaluated this cycle.

Quality Highlights ✅

Example 1: Clear, actionable authentication guide

• File: docs/src/content/docs/reference/triggering-ci.mdx
• What works well: Explains why the problem happens (GitHub's cascade-prevention behavior), then offers three concrete resolution paths (PAT, GitHub App, magic secret) each with copy-pasteable YAML/CLI snippets, and closes with a troubleshooting tip ("inspect workflow runs for the empty-commit SHA..."). This is a strong enterprise documentation pattern: cause → options → verification.
• Quote: "By default, pull requests created using the default GITHUB_TOKEN in GitHub Actions do not trigger CI workflow runs. This is a GitHub Actions feature to prevent event cascades."

Example 2: Precise, actionable compiler error message

• File: pkg/workflow/safe_outputs_step_token_validation.go
• What works well: The NewValidationError call at the end of validateSafeOutputStepTokenReferences names the exact field, quotes the offending expression, explains why it fails at runtime ("step outputs are only available inside the job that produced them"), and gives a copy-pasteable pre-steps snippet to fix it. This meets the "Clear problem + actionable solution + context" bar precisely.

Improvement Opportunities 💡

High Priority

Opportunity 1: Explain fallback behavior when no alias pattern matches — docs/src/content/docs/reference/model-tables.md

• File: docs/src/content/docs/reference/model-tables.md
• Current State: Lines 9–11 describe how aliases resolve ("the first pattern that matches an available model wins") but the page never states what happens if none of the listed patterns match an available model at compile time.
• Issue: A reader debugging "why did my engine: copilot with model: sonnet pick an unexpected model, or fail to compile" has no answer on this page — they must consult the separate spec page, and even that requires extra navigation for a one-line fact non-experts need immediately.
• User Impact: Enterprise users configuring workflows across many models/vendors will hit this exact question when a vendor deprecates a model tier; a missing inline answer increases support/debugging time.
• Suggested Change: Add one short paragraph after the intro (before "Vendor Aliases") stating the fallback/failure behavior in one or two sentences, with a link to the full spec for detail.
• Design Principle: Documentation Quality (complete information) + Efficiency (users can find what they need quickly without leaving the page).

Medium Priority

Opportunity 2: Inconsistent workflow message specificity across smoke tests — .github/workflows/smoke-agent-public-none.md

• File: .github/workflows/smoke-agent-public-none.md
• Current State: run-failure: "❌ [{workflow_name}]({run_url}) {status}. Check the logs for details." is generic, while sibling workflow approach-validator.md uses a domain-specific failure message ("... during approach validation").
• Issue: For a guard-policy smoke test, a generic "check the logs" message is less useful than naming which policy combination failed, since these smoke tests exist specifically to validate one policy at a time.
• User Impact: When multiple smoke-test workflows fail in CI, a maintainer scanning notifications cannot tell which guard-policy variant broke without opening each run.
• Suggested Change: Make run-failure reference the tested policy, e.g. "❌ [{workflow_name}]({run_url}) {status} testing guard policy: repos=public, min-integrity=none."

Files Reviewed

Documentation

• docs/src/content/docs/reference/model-tables.md - Rating: ⚠️ Needs Minor Work
• docs/src/content/docs/reference/triggering-ci.mdx - Rating: ✅ Professional

CLI Commands

• Not evaluated this cycle (build unavailable)

Workflow Messages

• .github/workflows/approach-validator.md - Rating: ✅ Professional
• .github/workflows/smoke-agent-public-none.md - Rating: ⚠️ Needs Minor Work

Validation Code

• pkg/workflow/safe_outputs_step_token_validation.go - Rating: ✅ Professional

Metrics

• Files Analyzed: 5
• Quality Distribution:
  • ✅ Professional: 3
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Here are 2 targeted improvement tasks, each affecting a single file:

Task 1: Explain fallback behavior when no model alias matches

File to Modify: docs/src/content/docs/reference/model-tables.md

Current Experience

The page explains alias resolution order ("the first pattern that matches an available model wins") but is silent on what happens when nothing matches.

Quality Issue

Design Principle: Documentation Quality (complete information)

Readers debugging unexpected model selection or compile-time errors have to leave this page and consult the separate Model Alias Format Specification just to learn the fallback/failure behavior, adding friction to a common troubleshooting path.

Proposed Improvement

Add a short paragraph after the intro paragraph (around line 11) stating what happens when no pattern in an alias resolves to an available model (e.g., compile-time error vs. silent fallback), with a link to the spec for full detail.

Before:

For details on the alias syntax, fallback resolution algorithm, and how to define your own aliases in workflow frontmatter, see the [Model Alias Format Specification](/gh-aw/specs/model-alias-specification/).

### Vendor Aliases

After:

For details on the alias syntax, fallback resolution algorithm, and how to define your own aliases in workflow frontmatter, see the [Model Alias Format Specification](/gh-aw/specs/model-alias-specification/).

If none of an alias's patterns match a model available to your account, `gh-aw` reports a compile-time error naming the alias and the patterns it tried, rather than silently falling back to an arbitrary model. See the [Model Alias Format Specification](/gh-aw/specs/model-alias-specification/) for the full resolution algorithm.

### Vendor Aliases

Why This Matters

• User Impact: Saves a navigation round-trip for a frequent troubleshooting question.
• Quality Factor: Documentation completeness and efficiency.
• Frequency: Likely encountered whenever a vendor deprecates/renames a model tier referenced by an alias.

Success Criteria

• Changes made to docs/src/content/docs/reference/model-tables.md only
• A reader can determine fallback/failure behavior without leaving the page
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/reference/model-tables.md
• No changes to other files required
• Can be completed independently

---

Task 2: Make guard-policy smoke-test failure message specific to the tested policy

File to Modify: .github/workflows/smoke-agent-public-none.md

Current Experience

The run-failure message is a generic "check the logs" note, even though this workflow is one of several smoke tests, each dedicated to validating one specific guard-policy combination (repos=public, min-integrity=none).

Quality Issue

Design Principle: Trust and Reliability (clear, contextual status information)

When several guard-policy smoke-test workflows run in the same CI batch and one fails, the notification alone does not indicate which policy combination is broken, forcing maintainers to open each run to find out.

Proposed Improvement

Update the run-failure message to name the tested policy, matching the specificity already present in run-started.

Before:

run-failure: "❌ [{workflow_name}]({run_url}) {status}. Check the logs for details."

After:

run-failure: "❌ [{workflow_name}]({run_url}) {status} testing guard policy: `repos=public, min-integrity=none`. Check the logs for details."

Why This Matters

• User Impact: Maintainers can triage failing smoke tests from the notification alone.
• Quality Factor: Trust and reliability — consistent, contextual status information.
• Frequency: Every failed run of this smoke-test workflow.

Success Criteria

• Changes made to .github/workflows/smoke-agent-public-none.md only
• Failure notification names the tested guard policy
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: .github/workflows/smoke-agent-public-none.md
• No changes to other files required
• Can be completed independently

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• storage.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "storage.googleapis.com"

See Network Configuration for more information.

📊 User experience analysis by Delight · auto · 30.2 AIC · ⌖ 6.24 AIC · ⊞ 12.4K · ◷

• expires on Aug 26, 2026, 6:52 AM UTC-08:00