[delight] User Experience Analysis Report - 2026-09-16

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (reference/model-tables.md, reference/footers.md)
• 2 CLI command surfaces (gh aw env get/env update, gh aw completion)
• 2 workflow message configurations (smoke-copilot.md, smoke-cursor.md)
• 1 validation file (pkg/workflow/tools_validation_github.go)

Overall Quality: Mostly professional. Documentation and validation error messages are strong; one documentation page has a readability gap and one CLI surface is missing usage examples.

Key Finding: docs/src/content/docs/reference/footers.md contains a single 260-word paragraph (line 107) describing template variables with no visual breaks, making it hard to scan for enterprise users who need to quickly find a specific placeholder.

Quality Highlights ✅

Example 1: Excellent Error Messages in GitHub Tool Validation

• File: pkg/workflow/tools_validation_github.go
• What works well: Every validation error follows a consistent, actionable pattern — states what's wrong, why it's not supported, and shows a corrected YAML example inline.
• Quote/Reference: "'tools.github.read-only: false' is not supported because the GitHub MCP server always operates in read-only mode. Remove the 'read-only' field or set it to 'true'. Example:\ntools:\n github:\n read-only: true"
• This is a textbook example of the "Clear problem statement + actionable solution + example" checklist.

Example 2: Well-Structured CLI Help for completion and audit

• File: pkg/cli/completion_command.go, pkg/cli/audit_command.go
• What works well: Long descriptions explain exactly what the command produces and enumerate accepted input formats; Example blocks show 5+ realistic, commented invocations covering common and edge-case usage (e.g., diffing multiple runs, extracting a specific failing step).

Improvement Opportunities 💡

High Priority

Opportunity 1: Break Up Dense Variable Reference Paragraph

• File: docs/src/content/docs/reference/footers.md
• Current State: Line 107 is a single unbroken paragraph (~260 words) listing over a dozen template variables ({workflow_name}, {agentic_workflow_url}, {ai_credits_suffix}, {ai_model}, etc.) with inline explanations, mixed together without visual separation.
• Issue: Violates "Documentation Quality" and "Clarity and Precision" principles — a wall of text forces users to read the entire paragraph linearly to find one variable's meaning, rather than scanning a table.
• User Impact: Enterprise users customizing footer messages must re-read the whole paragraph each time they look up a placeholder, slowing down configuration and increasing the chance of using the wrong variable.
• Suggested Change: Convert the variable list into a Markdown table (Variable | Description) with the surrounding narrative sentences trimmed to short intro/outro text.
• Design Principle: Documentation Quality (well-organized with clear hierarchy) and Efficiency (users can find what they need quickly).

Medium Priority

Opportunity 2: Missing Usage Examples for env get / env update Subcommands

• File: pkg/cli/env_command.go
• Current State: NewEnvCommand, newDefaultsGetCommand, and newDefaultsUpdateCommand (lines 124–214) each define Short and Long help text but no Example field, unlike most other commands in the CLI (e.g., add, audit, completion, checks).
• Issue: Users must infer correct flag combinations (--scope org --org my-org, --visibility all, etc.) from prose alone; every other reviewed command provides copy-pasteable examples.
• User Impact: Slower onboarding for an already flag-heavy command (4 scope-related flags plus dry-run/yes), increasing the chance of malformed invocations for org/enterprise-scoped operations.
• Suggested Change: Add an Example field to each of the three cobra.Command definitions in this file, covering repo/org/ent scope get and update flows, mirroring the style already used elsewhere in the CLI.
• Design Principle: Efficiency and Productivity (direct paths to outcomes, minimize cognitive load).

Files Reviewed

Documentation

• docs/src/content/docs/reference/model-tables.md - Rating: ✅
• docs/src/content/docs/reference/footers.md - Rating: ⚠️

CLI Commands

• gh aw completion (pkg/cli/completion_command.go) - Rating: ✅
• gh aw env get / gh aw env update (pkg/cli/env_command.go) - Rating: ⚠️

Workflow Messages

• .github/workflows/smoke-copilot.md - Rating: ✅
• .github/workflows/smoke-cursor.md - Rating: ✅

Validation Code

• pkg/workflow/tools_validation_github.go - Rating: ✅

Metrics

• Files Analyzed: 7
• Quality Distribution:
  • ✅ Professional: 5
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Here are 2 targeted improvement tasks, each affecting a single file:

Task 1: Convert Dense Variable Paragraph into a Table - docs/src/content/docs/reference/footers.md

File to Modify: docs/src/content/docs/reference/footers.md

Current Experience

Line 107 packs ~15 template variables and their descriptions into one 260-word paragraph under "Customizing Footer Messages", mixing narrative sentences with placeholder definitions.

Quality Issue

Design Principle: Documentation Quality (well-organized with clear hierarchy)

Users scanning for a single variable (e.g., what {ai_credits_suffix} outputs) must read the entire paragraph rather than jump directly to the relevant row, increasing time-to-answer for a reference page whose entire purpose is quick lookup.

Proposed Improvement

Replace the single paragraph with a short intro sentence plus a Markdown table listing each variable and its description, keeping the existing content (no information is removed).

Before:

The `messages.footer` template supports variables like `{workflow_name}`, `{agentic_workflow_url}`, `{run_url}`, `{triggering_number}`, `{triggering_type}`, `{ai_credits_suffix}`, and more. `{triggering_type}` is `issue`, `PR`, or `discussion`, matching the type of item that triggered the run. `{agentic_workflow_url}` links directly to the agentic workflow file view for the run (equivalent to `{run_url}/agentic_workflow`), while `{run_url}` links to the plain Actions run page. `{ai_credits_suffix}` is a pre-formatted, always-safe suffix (e.g. `" · sonnet46 12.4 AIC"` or `""`) that you can place directly before `{history_link}`. ...

After:

The `messages.footer` template supports the following variables:

| Variable | Description |
|----------|--------------|
| `{workflow_name}` | Name of the workflow that produced the item |
| `{run_url}` | Link to the plain Actions run page |
| `{agentic_workflow_url}` | Link to the agentic workflow file view (equivalent to `{run_url}/agentic_workflow`) |
| `{triggering_number}` | Number of the issue/PR/discussion that triggered the run |
| `{triggering_type}` | One of `issue`, `PR`, or `discussion` |
| `{ai_credits_suffix}` | Pre-formatted, always-safe suffix (e.g. `" · sonnet46 12.4 AIC"` or `""`) to place before `{history_link}` |
| `{ai_model}` | Full model name (e.g. `claude-sonnet-4.6`) |
| `{ai_model_short}` | Compact model identifier (e.g. `sonnet46`) |
| `{ai_credits}` | Raw numeric AI-credit cost |
| `{ai_credits_formatted}` | Formatted AI-credit cost |
| `{ai_credits_unit}` | Always `AIC` |
| `{agent_ai_credits_formatted}` | Formatted AI-credit cost for the agent step |
| `{evals_ai_credits_formatted}` | Formatted AI-credit cost for evals |
| `{threat_detection_ai_credits_formatted}` | Formatted AI-credit cost for threat detection |
| `{detection_conclusion}` | Threat-detection conclusion |
| `{detection_reason}` | Threat-detection reason |

See [Custom Messages](/gh-aw/reference/safe-outputs/#custom-messages-messages) for complete documentation.

Why This Matters

• User Impact: Enterprise users configuring custom branding can scan the table in seconds instead of parsing a dense paragraph.
• Quality Factor: Documentation Quality / visual hierarchy.
• Frequency: This is the canonical reference page for footer customization; every team enabling custom branding will read it.

Success Criteria

• Changes made to docs/src/content/docs/reference/footers.md only
• All 15+ variables preserved with equivalent descriptions, now in tabular form
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/reference/footers.md
• No changes to other files required
• Can be completed independently

---

Task 2: Add Usage Examples to env get / env update Commands - pkg/cli/env_command.go

File to Modify: pkg/cli/env_command.go

Current Experience

newDefaultsGetCommand (lines ~146–173) and newDefaultsUpdateCommand (lines ~178–214) each define detailed Long prose about scope resolution and flags, but neither sets an Example field on the cobra.Command. Nearly every other gh aw command (e.g., add, audit, checks, domains, completion) includes an Example block with 2-5 realistic invocations.

Quality Issue

Design Principle: Efficiency and Productivity (minimize cognitive load, provide direct paths to outcomes)

Without examples, users must manually translate the scope-resolution prose into correct flag combinations for repo/org/ent scopes, which is more error-prone than copying a working example, especially for the org/ent flows that require extra flags (--org, --enterprise, --visibility).

Proposed Improvement

Add an Example field to both newDefaultsGetCommand and newDefaultsUpdateCommand, following the existing style ( + constants.CLIExtensionPrefix + env ... with trailing # comments).

Before:

cmd := &cobra.Command{
    Use:   "get [file]",
    Short: "Download default compiler variables into a YAML file",
    Long: `Download compiler defaults into a YAML file.
...
- ent scope requires --enterprise <slug>.`,
    Args: cobra.MaximumNArgs(1),
    RunE: func(cmd *cobra.Command, args []string) error {

After:

cmd := &cobra.Command{
    Use:   "get [file]",
    Short: "Download default compiler variables into a YAML file",
    Long: `Download compiler defaults into a YAML file.
...
- ent scope requires --enterprise <slug>.`,
    Example: `  ` + string(constants.CLIExtensionPrefix) + ` env get                                   # Download repo defaults to file.yml
  ` + string(constants.CLIExtensionPrefix) + ` env get defaults.yml --repo owner/repo    # Download a specific repo's defaults
  ` + string(constants.CLIExtensionPrefix) + ` env get org.yml --scope org --org my-org  # Download org-scoped defaults`,
    Args: cobra.MaximumNArgs(1),
    RunE: func(cmd *cobra.Command, args []string) error {

(Apply the analogous pattern to newDefaultsUpdateCommand, covering repo, org with --visibility, and --dry-run usage.)

Why This Matters

• User Impact: Reduces trial-and-error for a flag-heavy, multi-scope command, especially for less-common org/enterprise flows.
• Quality Factor: CLI help consistency and completeness.
• Frequency: env get/env update are the primary entry points for bulk-managing compiler defaults across repos/orgs; used by platform teams during onboarding and governance changes.

Success Criteria

• Changes made to pkg/cli/env_command.go only
• Both get and update subcommands have an Example field with at least 2 realistic invocations each
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: pkg/cli/env_command.go
• No changes to other files required
• Can be completed independently

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• storage.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "storage.googleapis.com"

See Network Configuration for more information.

📊 User experience analysis by Delight · copilot · auto · 95.9 AIC · ⌖ 10.1 AIC · ⊞ 12.5K · ◷

• expires on Sep 19, 2026, 6:56 AM UTC-08:00