[delight] User Experience Analysis Report - 2026-08-07

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation file(s): docs/src/content/docs/reference/dependencies.md, docs/src/content/docs/reference/playwright.md
• 0 CLI command(s) — no gh-aw binary available in this environment to inspect --help output; skipped this run
• 2 workflow message configuration(s): .github/workflows/spec-librarian.md, .github/workflows/smoke-update-cross-repo-pr.md
• 1 validation file(s): pkg/workflow/tools_validation.go

Overall Quality: Generally professional. Documentation is clear and well-structured; workflow messages are consistent and appropriately scoped. One concrete gap found in a validation error message.

Key Finding: The bash tool anonymous-syntax error in pkg/workflow/tools_validation.go omits the offending workflow's name from the message users actually see, even though the name is available and used in the (invisible-by-default) debug log — this creates unnecessary friction for repos with multiple workflows.

Quality Highlights ✅

Example 1: Actionable, well-formed validation error

• File: pkg/workflow/tools_validation.go
• What works well: The error clearly states the problem ("anonymous syntax 'bash:' is not supported"), lists all three valid alternatives, and points to an automated fix command (gh aw fix).
• Quote/Reference: "invalid bash tool configuration: anonymous syntax 'bash:' is not supported. Use 'bash: true' ... Run 'gh aw fix' to automatically migrate" (line 22)

Example 2: Clear reference documentation structure

• File: docs/src/content/docs/reference/playwright.md
• What works well: Presents CLI mode as the clear recommendation with a rationale (token-efficient, no Docker overhead, direct localhost access), immediately followed by copy-pasteable YAML and bash examples. Deprecated MCP mode is clearly marked and paired with an explanation of why it's discouraged rather than just a label.
• Quote/Reference: "CLI mode is recommended for all new workflows — it is token-efficient (no large MCP schemas in context), avoids Docker overhead, and lets the agent reach local dev servers via localhost directly." (line 12)

Improvement Opportunities 💡

High Priority

Opportunity 1: Missing workflow name in bash validation error - Single File Improvement

• File: pkg/workflow/tools_validation.go
• Current State: validateBashToolConfig (lines 12–27) receives workflowName as a parameter and includes it in the internal debug log (line 21, only visible with DEBUG=*), but the errors.New(...) returned to the user on line 22 never mentions which workflow triggered the failure.
• Issue: In repositories with many workflow files, a compile error that doesn't name the offending file forces the user to search manually or re-run with debug logging enabled to find the source of the problem.
• User Impact: Slower troubleshooting, especially in CI where gh aw compile may process dozens of workflows — users see a generic error with no indication of which .md file needs the fix.
• Suggested Change: Interpolate workflowName into the returned error message, e.g. prefix it with workflow %q: .
• Design Principle: Trust and Reliability — "Clear error messages with actionable solutions."

Files Reviewed

Documentation

• docs/src/content/docs/reference/dependencies.md - Rating: ✅
• docs/src/content/docs/reference/playwright.md - Rating: ✅

CLI Commands

• (skipped — no compiled gh-aw binary available in this environment)

Workflow Messages

• spec-librarian - Rating: ✅
• smoke-update-cross-repo-pr - Rating: ✅

Validation Code

• pkg/workflow/tools_validation.go - Rating: ⚠️

Metrics

• Files Analyzed: 5
• Quality Distribution:
  • ✅ Professional: 4
  • ⚠️ Needs Minor Work: 1
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Here are 1 targeted improvement task, affecting a single file:

Task 1: Include the workflow name in the bash anonymous-syntax error

File to Modify: pkg/workflow/tools_validation.go

Current Experience

When a workflow uses the unsupported anonymous bash: syntax, validateBashToolConfig (lines 12–27) returns a generic error that never names the offending workflow file. The workflow name is already passed into the function and is written to the debug log (line 21), but that log line is invisible unless the user runs with DEBUG=* set.

Quality Issue

Design Principle: Trust and Reliability — clear, actionable error messages.

In a repository with many workflow files, gh aw compile (or CI compilation) fails with a message that gives no indication of which file to fix, forcing users to either guess or re-run with debug logging just to locate the source.

Proposed Improvement

Prefix the returned error with the workflow name so the message is self-contained without requiring debug mode.

Before (line 22):

return errors.New("invalid bash tool configuration: anonymous syntax 'bash:' is not supported. Use 'bash: true' (enable all commands), 'bash: false' (disable), or 'bash: [\"cmd1\", \"cmd2\"]' (specific commands). Run 'gh aw fix' to automatically migrate")

After:

return fmt.Errorf("workflow %q: invalid bash tool configuration: anonymous syntax 'bash:' is not supported. Use 'bash: true' (enable all commands), 'bash: false' (disable), or 'bash: [\"cmd1\", \"cmd2\"]' (specific commands). Run 'gh aw fix' to automatically migrate", workflowName)

(Requires adding "fmt" to the imports, in place of or alongside "errors".)

Why This Matters

• User Impact: Users immediately know which workflow file to fix without re-running with DEBUG=*.
• Quality Factor: Trust and Reliability / Clarity and Precision.
• Frequency: Triggered on every compile of any workflow using the legacy anonymous bash: syntax — a common migration scenario across multi-workflow repos.

Success Criteria

• Changes made to pkg/workflow/tools_validation.go only
• Returned error message includes the workflow name
• Existing tests in pkg/workflow/tools_validation_test.go still pass (they assert on substring "anonymous syntax 'bash:' is not supported", which remains present)
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: pkg/workflow/tools_validation.go
• No changes to other files required
• Can be completed independently

📊 User experience analysis by Delight · auto · 32.4 AIC · ⌖ 3.25 AIC · ⊞ 12.1K · ◷

• expires on Aug 10, 2026, 7:10 AM UTC-08:00