[delight] User Experience Analysis Report - 2026-09-08

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files
• 0 CLI commands (binary not available in this environment; skipped)
• 2 workflow message configurations
• 1 validation file

Overall Quality: Mostly professional with one notable tone inconsistency.

Key Finding: The smoke-copilot-aoai-entra.md workflow uses tabloid-style "BREAKING NEWS" status messages that clash with the enterprise-appropriate tone used elsewhere in the codebase (e.g., pr-code-quality-reviewer.md).

Quality Highlights ✅

Example 1: Clear, well-scoped documentation

• File: docs/src/content/docs/experimental/awf-reflect.md
• What works well: Concise purpose statement, explicit experimental warning, numbered "Recommended selection flow" with concrete retry guidance ("retry discovery with bounded backoff, for example every 3 seconds up to 5 attempts"), and a runnable curl example.
• Quote/Reference: "Do not hardcode direct upstream model API URLs in shared workflow logic. All inference requests should go through the AWF gateway so usage remains controllable and observable for cost control, tracking, and optimization."

Example 2: Actionable validation errors

• File: pkg/workflow/engine_driver_validation.go (validateEngineScriptFilename)
• What works well: Each validation error includes a specific problem statement, a concrete YAML example showing the fix, and a documentation link — following the "clear problem + actionable solution + example" pattern.
• Quote/Reference: "Remove the surrounding whitespace from the script name.\n\nExample:\nengine:\n %s: driver.cjs\n\nSee: %s"

Improvement Opportunities 💡

High Priority

Opportunity 1: Inconsistent tone in workflow status messages - Single File Improvement

• File: .github/workflows/smoke-copilot-aoai-entra.md
• Current State: Lines 150-155 define a "messages" block using tabloid/breaking-news framing: "📰 BREAKING: ... Sources say the story is developing...", "📰 VERDICT: ... This is a developing story. 🎤", "📰 DEVELOPING STORY: ... Our correspondents are investigating the incident...".
• Issue: These messages read as playful/parody copy rather than professional status updates. Compared to pr-code-quality-reviewer.md's consistent, businesslike messages ("🔎 ... is reviewing code quality...", "✅ ... completed the code quality review."), this workflow's voice is jarring and inconsistent with the rest of the fleet.
• User Impact: Enterprise users (including auditors/reviewers scanning PR/issue comment history) may find "BREAKING NEWS"-style language unprofessional or confusing when trying to quickly assess run status, especially in failure states where clarity matters most.
• Suggested Change: Replace tabloid framing with clear, status-first language while retaining a single light touch (the 📰 emoji is fine to keep as a namespace/identity marker), e.g.: run-started: "📰 [{workflow_name}]({run_url}) is investigating this {event_type}...", run-success: "✅ [{workflow_name}]({run_url}) completed successfully. All systems operational.", run-failure: "⚠️ [{workflow_name}]({run_url}) {status}. Investigating the incident.".
• Design Principle: Professional Communication (business-appropriate tone, consistent voice across touchpoints).

Medium Priority

Opportunity 2: Retry guidance could specify units more explicitly

• File: docs/src/content/docs/experimental/awf-reflect.md
• Current State: Step 6 says "retry discovery with bounded backoff (for example, every 3 seconds up to 5 attempts)".
• Issue: Minor — the parenthetical example is good, but the doc doesn't state a total timeout bound, so integrators must compute it (15s) themselves.
• User Impact: Low; mostly a clarity nicety for implementers.
• Suggested Change: Add explicit total bound, e.g., "...up to 5 attempts (15 seconds total) before failing."
• Design Principle: Clarity and Precision.

Files Reviewed

Documentation

• docs/src/content/docs/specs/checkout-behavior-specification.md - Rating: ✅
• docs/src/content/docs/experimental/awf-reflect.md - Rating: ✅

CLI Commands

• (skipped — gh-aw binary not built in this environment)

Workflow Messages

• .github/workflows/smoke-copilot-aoai-entra.md - Rating: ⚠️
• .github/workflows/pr-code-quality-reviewer.md - Rating: ✅

Validation Code

• pkg/workflow/engine_driver_validation.go - Rating: ✅

Metrics

• Files Analyzed: 5
• Quality Distribution:
  • ✅ Professional: 4
  • ⚠️ Needs Minor Work: 1
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Here are 1 targeted improvement task, affecting a single file:

Task 1: Align smoke test status messages with enterprise tone - Improve .github/workflows/smoke-copilot-aoai-entra.md

File to Modify: .github/workflows/smoke-copilot-aoai-entra.md

Current Experience

Lines 150-155 define workflow status messages using a "BREAKING NEWS" tabloid-journalist persona:

    messages:
      append-only-comments: true
      footer: "> 📰 *BREAKING: Report filed by [{workflow_name}]({run_url})*{ai_credits_suffix}{history_link}"
      run-started: "📰 BREAKING: [{workflow_name}]({run_url}) is now investigating this {event_type}. Sources say the story is developing..."
      run-success: "📰 VERDICT: [{workflow_name}]({run_url}) has concluded. All systems operational. This is a developing story. 🎤"
      run-failure: "📰 DEVELOPING STORY: [{workflow_name}]({run_url}) reports {status}. Our correspondents are investigating the incident..."

Quality Issue

Design Principle: Professional Communication

This messaging style is inconsistent with the businesslike tone used in comparable workflows (e.g. pr-code-quality-reviewer.md's run-started/run-success/run-failure messages). Phrases like "Sources say the story is developing..." and "Our correspondents are investigating the incident..." read as parody rather than a status update, which can confuse readers scanning PR/issue history for run state, especially in failure cases.

Proposed Improvement

Keep the 📰 emoji as a lightweight visual identity marker but replace the tabloid narration with clear, direct status language.

Before:

      footer: "> 📰 *BREAKING: Report filed by [{workflow_name}]({run_url})*{ai_credits_suffix}{history_link}"
      run-started: "📰 BREAKING: [{workflow_name}]({run_url}) is now investigating this {event_type}. Sources say the story is developing..."
      run-success: "📰 VERDICT: [{workflow_name}]({run_url}) has concluded. All systems operational. This is a developing story. 🎤"
      run-failure: "📰 DEVELOPING STORY: [{workflow_name}]({run_url}) reports {status}. Our correspondents are investigating the incident..."

After:

      footer: "> 📰 *Report filed by [{workflow_name}]({run_url})*{ai_credits_suffix}{history_link}"
      run-started: "📰 [{workflow_name}]({run_url}) is investigating this {event_type}..."
      run-success: "✅ [{workflow_name}]({run_url}) completed successfully. All systems operational."
      run-failure: "⚠️ [{workflow_name}]({run_url}) {status}. Investigating the incident."

Why This Matters

• User Impact: Readers scanning PR/issue comments for smoke-test status get a clear, unambiguous signal instead of parsing tabloid narration.
• Quality Factor: Professional Communication / Trust and Reliability (consistent voice across the workflow fleet).
• Frequency: This workflow runs on a schedule (every 2 days) plus on-demand via slash command and label command, so its messages are seen regularly.

Success Criteria

• Changes made to .github/workflows/smoke-copilot-aoai-entra.md only
• messages block uses consistent, professional status language across run-started, run-success, run-failure, and footer
• Quality rating improves from ⚠️ Needs Minor Work to ✅ Professional

Scope Constraint

• Single file only: .github/workflows/smoke-copilot-aoai-entra.md
• No changes to other files required
• Can be completed independently (Note: after editing, run make recompile to regenerate the corresponding .lock.yml)

📊 User experience analysis by Delight · copilot · auto · 25 AIC · ⌖ 10.1 AIC · ⊞ 12.3K · ◷

• expires on Sep 11, 2026, 6:51 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #59760.