[delight] User Experience Analysis Report - 2026-09-15

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (docs/src/content/docs/experimental/awf-reflect.md, docs/src/content/docs/patterns/chat-ops.md)
• 0 CLI commands (the gh-aw binary could not be built in this run — no network access to fetch the pinned Go toolchain — so CLI help text was not sampled)
• 2 workflow message configurations (.github/workflows/smoke-cursor.md, .github/workflows/dataflow-pr-discussion-dataset.md)
• 1 validation file (pkg/workflow/strict_mode_env_validation.go)

Overall Quality: Mostly professional. Documentation and validation code are strong; one factual inconsistency was found in a pattern doc's example walkthrough.

Key Finding: docs/src/content/docs/patterns/chat-ops.md describes an example that references steps.sanitized.outputs.text, but the embedded example code block does not actually use that variable anywhere — readers following the example literally will not find what the prose promised.

Quality Highlights ✅

Example 1: Clear, well-structured experimental API doc

• File: docs/src/content/docs/experimental/awf-reflect.md
• What works well: Uses a :::caution[Experimental] admonition up front to set expectations, documents the full response shape field-by-field, and gives a numbered "Recommended selection flow" that is directly actionable. The > [!IMPORTANT] callout about not hardcoding upstream URLs is a good example of proactively preventing a common integration mistake.
• Quote/Reference: "6. If models is null, retry discovery with bounded backoff (for example, every 3 seconds up to 5 attempts) before failing." — concrete guidance rather than vague advice.

Example 2: Precise, actionable strict-mode error messages

• File: pkg/workflow/strict_mode_env_validation.go
• What works well: Error messages name the exact offending section (env vs engine.env), list the specific secret references found, explain why it matters (leak risk to the agent sandbox), and point to a documentation link and an alternative (engine-specific secret configuration). This meets the "actionable solution + context" bar well.
• Quote/Reference: "strict mode: secrets detected in '%s' section will be leaked to the agent container. Found: %s. Use engine-specific secret configuration instead. See: https://github.github.com/gh-aw/reference/engines/"

Improvement Opportunities 💡

High Priority

Opportunity 1: Fix example/prose mismatch in ChatOps pattern doc

• File: docs/src/content/docs/patterns/chat-ops.md
• Current State: Line 25 states: "The example also references the triggering content via steps.sanitized.outputs.text, which strips injection attempts, excessive content, and untrusted mentions." However, the fenced example (lines 27–51) contains no reference to steps.sanitized.outputs.text anywhere in its prompt body ("When someone types /review in a pull request comment, perform a thorough analysis of the changes...").
• Issue: The prose over-promises what the example demonstrates. A user copying the example verbatim and looking for the sanitized-text usage pattern will not find it, undermining trust in the doc's accuracy.
• User Impact: Enterprise users adopting ChatOps patterns rely on the example being a faithful reference implementation. A description/code mismatch forces the reader to hunt elsewhere or assume the docs are stale — costing time and reducing confidence in the whole page.
• Suggested Change: Either (a) add a line to the example prompt body that explicitly reads ${{ steps.sanitized.outputs.text }} (e.g., "Analyze the following triggering comment: ${{ steps.sanitized.outputs.text }}"), or (b) if intentionally omitted for brevity, revise line 25's prose to stop claiming the example uses it and instead present it as an optional technique with its own short snippet.
• Design Principle: Trust and Reliability — documentation must accurately describe what the accompanying example does.

Medium Priority

Opportunity 2: Reduce ellipsis overuse in Cursor smoke-test workflow messages

• File: .github/workflows/smoke-cursor.md
• Current State (lines 49–51): run-started: "🖱️ Cursor initializing... [{workflow_name}]({run_url}) begins on this {event_type}...", run-failure: "⚠️ [{workflow_name}]({run_url}) {status}. Cursor encountered unexpected challenges..."
• Issue: Two of the three status messages end with a trailing "..." with no follow-up, giving a vague, unfinished tone ("encountered unexpected challenges..." doesn't say what challenges or what to do next) that reads as casual/dramatic rather than a precise enterprise status update.
• User Impact: Status comments/messages are often the first thing a reviewer scans on a PR or issue; vague language like "unexpected challenges..." forces the user to open the run logs to learn anything actionable, adding friction for a routine smoke test.
• Suggested Change: Replace with direct, information-bearing phrasing, e.g. run-started: "🖱️ [{workflow_name}]({run_url}) starting Cursor smoke test on this {event_type}." and run-failure: "⚠️ [{workflow_name}]({run_url}) {status}. See run logs for details."
• Design Principle: Professional Communication — messages should be direct and information-bearing rather than trailing off.

Files Reviewed

Documentation

• docs/src/content/docs/experimental/awf-reflect.md - Rating: ✅
• docs/src/content/docs/patterns/chat-ops.md - Rating: ⚠️ (prose/example mismatch)

CLI Commands

• None sampled this run (build environment lacked network access to fetch the pinned Go toolchain for ./gh-aw)

Workflow Messages

• .github/workflows/smoke-cursor.md - Rating: ⚠️
• .github/workflows/dataflow-pr-discussion-dataset.md - Rating: ✅

Validation Code

• pkg/workflow/strict_mode_env_validation.go - Rating: ✅

Metrics

• Files Analyzed: 5
• Quality Distribution:
  • ✅ Professional: 3
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Here are 2 targeted improvement tasks, each affecting a single file:

Task 1: Fix Example/Prose Mismatch in ChatOps Pattern Doc

File to Modify: docs/src/content/docs/patterns/chat-ops.md

Current Experience

Line 25 of the doc reads:

"The example also references the triggering content via steps.sanitized.outputs.text, which strips injection attempts, excessive content, and untrusted mentions — see Context Text."

But the fenced example immediately below it (lines 27–51) contains no usage of steps.sanitized.outputs.text anywhere in its frontmatter or prompt body.

Quality Issue

Design Principle: Trust and Reliability

Documentation that claims an example demonstrates a technique, but the example doesn't actually show it, erodes user confidence and wastes time as users search the example in vain for the described pattern.

Proposed Improvement

Add an explicit reference to ${{ steps.sanitized.outputs.text }} inside the example prompt body so the example matches its description.

Before:

# Code Review Assistant

When someone types /review in a pull request comment, perform a thorough analysis of the changes.

Examine the diff for potential bugs, security vulnerabilities, performance implications, code style issues, and missing tests or documentation.

Create specific review comments on relevant lines of code and add a summary comment with overall observations and recommendations.

After:

# Code Review Assistant

When someone types /review in a pull request comment, perform a thorough analysis of the changes.

Triggering comment (sanitized): ${{ steps.sanitized.outputs.text }}

Examine the diff for potential bugs, security vulnerabilities, performance implications, code style issues, and missing tests or documentation.

Create specific review comments on relevant lines of code and add a summary comment with overall observations and recommendations.

Why This Matters

• User Impact: Users copying this example as a starting point for their own ChatOps workflows get an accurate, complete reference implementation instead of a doc that describes a feature the code doesn't use.
• Quality Factor: Documentation accuracy / trust and reliability.
• Frequency: This is a featured pattern doc linked from multiple other pages (IssueOps, DispatchOps, LabelOps cross-references), so it likely sees regular traffic from users building ChatOps workflows.

Success Criteria

• Changes made to docs/src/content/docs/patterns/chat-ops.md only
• The example prompt body references ${{ steps.sanitized.outputs.text }}, matching the prose in line 25
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/patterns/chat-ops.md
• No changes to other files required
• Can be completed independently

---

Task 2: Tighten Vague Status Messages in Cursor Smoke Test Workflow

File to Modify: .github/workflows/smoke-cursor.md

Current Experience

Lines 49 and 51 use trailing ellipses and vague phrasing that don't convey concrete information:

run-started: "🖱️ Cursor initializing... [{workflow_name}]({run_url}) begins on this {event_type}..."
run-failure: "⚠️ [{workflow_name}]({run_url}) {status}. Cursor encountered unexpected challenges..."

Quality Issue

Design Principle: Professional Communication

Trailing "..." with no resolution, and phrases like "encountered unexpected challenges..." read as dramatic filler rather than direct status reporting. Enterprise users scanning PR/issue status comments benefit from concise, complete statements that either convey the outcome or point to where to find it.

Proposed Improvement

Replace vague, unfinished phrasing with direct statements that either complete the thought or point to actionable next steps (run logs).

Before:

  messages:
    footer: "> 🖱️ *[{workflow_name}]({run_url}) — Powered by Cursor*{ai_credits_suffix}{history_link}"
    run-started: "🖱️ Cursor initializing... [{workflow_name}]({run_url}) begins on this {event_type}..."
    run-success: "🚀 [{workflow_name}]({run_url}) Cursor delivered."
    run-failure: "⚠️ [{workflow_name}]({run_url}) {status}. Cursor encountered unexpected challenges..."

After:

  messages:
    footer: "> 🖱️ *[{workflow_name}]({run_url}) — Powered by Cursor*{ai_credits_suffix}{history_link}"
    run-started: "🖱️ [{workflow_name}]({run_url}) starting Cursor smoke test on this {event_type}."
    run-success: "🚀 [{workflow_name}]({run_url}) Cursor smoke test passed."
    run-failure: "⚠️ [{workflow_name}]({run_url}) {status}. See run logs for details."

Why This Matters

• User Impact: Reviewers scanning issue/PR comments get a clear, complete status at a glance instead of a message that trails off without saying what happened or what to do next.
• Quality Factor: Professional Communication — clear status, actionable on failure.
• Frequency: This workflow runs on a schedule (every 2 days) plus on-demand via slash command and PR label events, so these messages are seen frequently by maintainers.

Success Criteria

• Changes made to .github/workflows/smoke-cursor.md only
• run-started and run-failure messages no longer end with an unresolved ellipsis
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: .github/workflows/smoke-cursor.md
• No changes to other files required
• Can be completed independently

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• storage.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "storage.googleapis.com"

See Network Configuration for more information.

📊 User experience analysis by Delight · copilot · auto · 37 AIC · ⌖ 8.14 AIC · ⊞ 12.5K · ◷

• expires on Sep 18, 2026, 6:52 AM UTC-08:00

[cbi58105-crypto]

Code Review Assistant

When someone types /review in a pull request comment, perform a thorough analysis of the changes.

Examine the diff for potential bugs, security vulnerabilities, performance implications, code style issues, and missing tests or documentation.

Create specific review comments on relevant lines of code and add a summary comment with overall observations and recommendations.

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #61368.