[delight] User Experience Analysis Report - 2026-09-09

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (blog/2026-09-09-agent-of-the-day.md, reference/auth-projects.mdx)
• 2 CLI commands (gh aw compile, gh aw run)
• 2 workflow message configurations (craft.md, smoke-test-tools.md)
• 1 validation file (pkg/workflow/engine_inline_definition_validation.go)

Overall Quality: Professional overall, with one notable inconsistency in error-message completeness and one workflow whose failure message lacks actionable guidance.

Key Finding: engine_inline_definition_validation.go gives a full inline engine: example for the token-url error but omits equivalent examples for the sibling client-id, client-secret, and header-name errors in the same oauth-client-credentials branch — an inconsistency that increases friction for users debugging OAuth engine auth.

Quality Highlights ✅

Example 1: Excellent reference documentation

• File: docs/src/content/docs/reference/auth-projects.mdx
• What works well: Clear problem statement up front ("Why a separate token is needed"), concrete PAT scope tables for both user-owned and org-owned projects, a copy-pasteable gh aw secrets set + YAML example, and a "Learn More" cross-link section. Precise, enterprise-appropriate tone throughout.
• Quote/Reference: "The standard GITHUB_TOKEN provided to every GitHub Actions workflow has repository-level scope only... any workflow step that reads project fields or writes updates must supply a token with explicit Projects permissions." (lines 11-12)

Example 2: Well-documented CLI command

• File: cmd/gh-aw/main.go (run command, lines 292-317)
• What works well: The Long description explains interactive-mode behavior, restricts scope clearly ("only works with workflows that have workflow_dispatch triggers"), and the Example block lists 10 realistic, incrementally more advanced invocations (from bare run to --json output).

Improvement Opportunities 💡

High Priority

Opportunity 1: Inconsistent OAuth validation error examples - Single File Improvement

• File: pkg/workflow/engine_inline_definition_validation.go
• Current State: In validateEngineAuthDefinition (lines ~137-150), the token-url missing-field error (line 139) includes a full Example: block showing the complete engine.provider.auth YAML shape. The three sibling checks in the same oauth-client-credentials case — missing client-id (line 142), missing client-secret (line 145), and missing header-name (line 149) — return terse one-line errors with no example at all.
• Issue: A user who fixes the token-url error (guided by the example) but then hits the very next validation error for client-id gets no comparable guidance, breaking the trust and consistency principle — the same validation function teaches differently depending on which field happens to be missing.
• User Impact: Enterprise users debugging OAuth engine configuration have to reverse-engineer the correct YAML shape for client-id/client-secret/header-name themselves, or scroll back up to the first error's example and guess whether it still applies.
• Suggested Change: Add the same Example: block (or a shared example constant) to the client-id, client-secret, and header-name error messages, matching the pattern already used for token-url.
• Design Principle: Trust and Reliability — consistent experience across touchpoints.

Medium Priority

Opportunity 2: Failure message lacks actionable next step

• File: .github/workflows/craft.md
• Current State: Line 47 — run-failure: "🛠️ Forge cooling down! [{workflow_name}]({run_url}) {status}. The anvil awaits another attempt..."
• Issue: Unlike the smoke-test-tools.md failure message, which names what went wrong ("detected missing tools: {status}"), craft.md's failure message is purely decorative — it doesn't point the user toward the run logs or suggest a next action beyond a vague "another attempt."
• User Impact: When workflow generation fails, users must click through to the run URL themselves with no hint about what to check first, adding an avoidable extra step for a professional/enterprise audience that values efficient error recovery.
• Suggested Change: Add a concrete pointer, e.g., "🛠️ Forge cooling down! [{workflow_name}]({run_url}) {status}. Check the run logs above for the compilation error, then retry with /craft."
• Design Principle: Trust and Reliability — clear error messages with actionable solutions.

Files Reviewed

Documentation

• docs/src/content/docs/blog/2026-09-09-agent-of-the-day.md - Rating: ✅
• docs/src/content/docs/reference/auth-projects.mdx - Rating: ✅

CLI Commands

• gh aw compile - Rating: ✅
• gh aw run - Rating: ✅

Workflow Messages

• .github/workflows/craft.md - Rating: ⚠️
• .github/workflows/smoke-test-tools.md - Rating: ✅

Validation Code

• pkg/workflow/engine_inline_definition_validation.go - Rating: ⚠️

Metrics

• Files Analyzed: 7
• Quality Distribution:
  • ✅ Professional: 5
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Here are 2 targeted improvement tasks, each affecting a single file:

Task 1: Add missing examples to OAuth engine-auth validation errors

File to Modify: pkg/workflow/engine_inline_definition_validation.go

Current Experience

In validateEngineAuthDefinition, the oauth-client-credentials case returns four separate validation errors for four required fields (token-url, client-id, client-secret, header-name). Only the first one (token-url) includes a full YAML example of a correctly configured engine.provider.auth block. The other three are one-liners with just a See: docs link.

Quality Issue

Design Principle: Trust and Reliability

Users iteratively fixing validation errors expect consistent guidance at each step. Providing a rich example for the first error and none for the next three creates an inconsistent, confusing debugging experience — especially since a user fixing token-url will likely hit client-id next and lose the example context.

Proposed Improvement

Add the same Example: block to the client-id, client-secret, and header-name error messages (or extract it into a shared local variable to avoid duplication).

Before:

if auth.ClientIDRef == "" {
    return fmt.Errorf("engine auth: strategy 'oauth-client-credentials' requires 'auth.client-id' to be set.\n\nSee: %s", constants.DocsEnginesURL)
}
if auth.ClientSecretRef == "" {
    return fmt.Errorf("engine auth: strategy 'oauth-client-credentials' requires 'auth.client-secret' to be set.\n\nSee: %s", constants.DocsEnginesURL)
}
// For oauth, header-name is required (the token must go somewhere).
if auth.HeaderName == "" {
    return fmt.Errorf("engine auth: strategy 'oauth-client-credentials' requires 'auth.header-name' to be set (e.g. 'api-key' or 'Authorization').\n\nSee: %s", constants.DocsEnginesURL)
}

After:

const oauthAuthExample = "\n\nExample:\nengine:\n  runtime:\n    id: codex\n  provider:\n    auth:\n      strategy: oauth-client-credentials\n      token-url: (auth.example.com/redacted)      client-id: MY_CLIENT_ID_SECRET\n      client-secret: MY_CLIENT_SECRET_SECRET\n      header-name: Authorization"

if auth.ClientIDRef == "" {
    return fmt.Errorf("engine auth: strategy 'oauth-client-credentials' requires 'auth.client-id' to be set.%s\n\nSee: %s", oauthAuthExample, constants.DocsEnginesURL)
}
if auth.ClientSecretRef == "" {
    return fmt.Errorf("engine auth: strategy 'oauth-client-credentials' requires 'auth.client-secret' to be set.%s\n\nSee: %s", oauthAuthExample, constants.DocsEnginesURL)
}
// For oauth, header-name is required (the token must go somewhere).
if auth.HeaderName == "" {
    return fmt.Errorf("engine auth: strategy 'oauth-client-credentials' requires 'auth.header-name' to be set (e.g. 'api-key' or 'Authorization').%s\n\nSee: %s", oauthAuthExample, constants.DocsEnginesURL)
}

Why This Matters

• User Impact: Users configuring OAuth-based custom engines get consistent, copy-pasteable guidance no matter which required field they miss first.
• Quality Factor: Trust and Reliability, Documentation Quality (inline error docs)
• Frequency: Every custom-engine author using oauth-client-credentials auth hits this validation path at least once during setup.

Success Criteria

• Changes made to pkg/workflow/engine_inline_definition_validation.go only
• All four oauth-client-credentials field-missing errors include the same example block
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: pkg/workflow/engine_inline_definition_validation.go
• No changes to other files required
• Can be completed independently

---

Task 2: Add actionable guidance to craft.md failure message

File to Modify: .github/workflows/craft.md

Current Experience

Line 47 defines the run-failure message as: "🛠️ Forge cooling down! [{workflow_name}]({run_url}) {status}. The anvil awaits another attempt..." — purely stylistic, with no hint about what to check or do next.

Quality Issue

Design Principle: Trust and Reliability

Compare this to smoke-test-tools.md's failure message, which explicitly says "detected missing tools: {status}" — giving the reader concrete information. Craft.md's message gives no comparable signal, forcing users to open the run and search for the cause themselves.

Proposed Improvement

Extend the failure message to point users toward the run logs and the retry mechanism (the /craft slash command).

Before:

run-failure: "🛠️ Forge cooling down! [{workflow_name}]({run_url}) {status}. The anvil awaits another attempt..."

After:

run-failure: "🛠️ Forge cooling down! [{workflow_name}]({run_url}) {status}. Check the run logs above for the compilation error, then retry with /craft."

Why This Matters

• User Impact: Users hitting a failed craft run get a direct next step instead of a vague retry prompt.
• Quality Factor: Trust and Reliability, Professional Communication
• Frequency: Occurs on every failed /craft invocation (compilation errors, invalid requests, etc.).

Success Criteria

• Changes made to .github/workflows/craft.md only
• run-failure message names the concrete next action (check logs, retry with /craft)
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: .github/workflows/craft.md
• No changes to other files required (note: after editing, run make recompile to regenerate craft.lock.yml, but that regeneration is a build artifact, not a manual second-file edit)
• Can be completed independently

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• storage.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "storage.googleapis.com"

See Network Configuration for more information.

📊 User experience analysis by Delight · copilot · auto · 94.2 AIC · ⌖ 8.31 AIC · ⊞ 12.5K · ◷

• expires on Sep 12, 2026, 6:55 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #59999.