[delight] UX Analysis Report – 2026-07-17: Smoke Test Messaging & Rate Limiting Docs

[github-actions[bot]]

User Experience Analysis Report - 2026-07-17

Executive Summary

Today's analysis focused on:

• 2 documentation files
• 2 workflow message configurations
• 1 validation test file

Overall Quality: Needs minor work

Key Finding: The smoke-codex.md workflow messages use theatrical/mystical language ("ancient spirits stir", "prophecy is fulfilled", "mystical journey") that is inappropriate for an enterprise smoke test tool — users need clear, professional status communication, not narrative theater.

Quality Highlights ✅

Example 1: Custom Steps and Jobs Documentation

• File: docs/src/content/docs/reference/steps-jobs.md
• What works well: Excellent structure — clear section hierarchy for steps:, pre-agent-steps:, post-steps:, and jobs:. The supported fields table with concise descriptions is highly scannable. The "Jobs and Steps" order-of-execution map provides transparency into compiler behavior, which builds user trust.
• Quote/Reference: "Use this map to see where compiler-inserted steps land for each job type." — sets precise expectations before the detail.

Example 2: Rate Limiting Controls Documentation

• File: docs/src/content/docs/reference/rate-limiting-controls.md
• What works well: Defense-in-depth framing in the opening sentence is effective. The safe output limits table is well-structured with a "Purpose" column. Bypass exceptions are enumerated clearly with bullet points.

Improvement Opportunities 💡

High Priority

Opportunity 1: Unprofessional Theatrical Tone in Smoke Test Messages - smoke-codex.md

• File: .github/workflows/smoke-codex.md
• Current State: Messages use mystical/oracle metaphors throughout:
  • run-started: "🔮 The ancient spirits stir... awakens to divine this {event_type}..."
  • run-success: "✨ The prophecy is fulfilled... its mystical journey. The stars align. 🌟"
  • run-failure: "🌑 The shadows whisper... The oracle requires further meditation..."
  • footer: "🔮 The oracle has spoken through..."
• Issue: The oracle/prophecy/mystical metaphor is charming in a personal project but undermines credibility in an enterprise smoke test. Users reviewing CI status need to immediately understand state without decoding metaphor. "The oracle requires further meditation" does not communicate failure clearly.
• User Impact: Enterprise teams triaging failed smoke tests must parse poetic language to determine severity. The mystical framing is inconsistent with the professional communication found in other gh-aw workflows and can reduce trust in the tool's status reporting.
• Suggested Change: Replace theatrical metaphors with professional status language that preserves the emoji-light aesthetic used in other internal workflows (e.g., craft.md's "Crafted with care").
• Design Principle: Professional Communication — enterprise users need clear, unambiguous status signals.

Medium Priority

Opportunity 2: Best Practices Section Remains a Dense Paragraph (pending since 2026-07-05)

• File: docs/src/content/docs/reference/rate-limiting-controls.md
• Current State (lines near end of file): "Start with conservative limits and increase as needed. Use environments for high-risk operations (workflow dispatches, cross-repo operations, production systems). Layer multiple controls: rate limiting with concurrency, timeouts with stop-after, safe output limits with environments. Monitor workflow runs, safe output logs, and rate limit cancellations to identify needed adjustments."
• Issue: Four distinct best-practice recommendations are crammed into an unbroken paragraph. Adjacent sections use bullet lists and code blocks. This inconsistency slows scanning and makes the recommendations easy to skip.
• User Impact: Users configuring rate limiting for the first time will benefit most from these recommendations, but the wall-of-text format discourages engagement with exactly the guidance that would prevent misconfiguration.
• Suggested Change: Convert to a numbered list with bold lead-ins:
  1. Start conservative — begin with low limits and increase based on observed usage.
  2. Use environments for high-risk operations — workflow dispatches, cross-repo operations, and production systems should require manual approval.
  3. Layer controls — combine rate limiting with concurrency, timeouts with stop-after, and safe output limits with environments.
  4. Monitor regularly — review workflow runs, safe output logs, and rate-limit cancellations to identify adjustments.
• Design Principle: Efficiency and Productivity — scannable lists reduce cognitive load.

Files Reviewed

Documentation

• docs/src/content/docs/reference/rate-limiting-controls.md — Rating: ⚠️ Needs Minor Work
• docs/src/content/docs/reference/steps-jobs.md — Rating: ✅ Professional

CLI Commands

• (Not evaluated this run — binary not yet built in workspace)

Workflow Messages

• .github/workflows/smoke-codex.md — Rating: ❌ Needs Significant Work
• .github/workflows/craft.md — Rating: ⚠️ Needs Minor Work (footer/messages slightly gamified but acceptable)

Validation Code

• pkg/workflow/safe_outputs_allowed_labels_validation_test.go — Rating: ✅ Professional (test names are descriptive, test structure clean, no user-facing messages to evaluate)

Metrics

• Files Analyzed: 5
• Quality Distribution:
  • ✅ Professional: 2
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 1

---

🎯 Actionable Tasks

Here are 2 targeted improvement tasks, each affecting a single file:

---

Task 1: Replace Theatrical Messages with Professional Status Language — smoke-codex.md

File to Modify: .github/workflows/smoke-codex.md

Current Experience

The messages: block uses an oracle/mystical theme that is inconsistent with enterprise expectations:

• run-started: "🔮 The ancient spirits stir... {workflow_name} awakens to divine this {event_type}..."
• run-success: "✨ The prophecy is fulfilled... {workflow_name} has completed its mystical journey. The stars align. 🌟"
• run-failure: "🌑 The shadows whisper... {workflow_name} {status}. The oracle requires further meditation..."
• footer: "> 🔮 The oracle has spoken through {workflow_name}{ai_credits_suffix}{history_link}"

Quality Issue

Design Principle: Professional Communication

"The oracle requires further meditation" tells engineers nothing actionable about the failure. The theatrical framing adds noise during incident triage and undermines the professional tone expected of a CI/CD testing tool used in enterprise repositories.

Proposed Improvement

Before:

messages:
  footer: "> 🔮 *The oracle has spoken through [{workflow_name}]({run_url})*{ai_credits_suffix}{history_link}"
  run-started: "🔮 The ancient spirits stir... [{workflow_name}]({run_url}) awakens to divine this {event_type}..."
  run-success: "✨ The prophecy is fulfilled... [{workflow_name}]({run_url}) has completed its mystical journey. The stars align. 🌟"
  run-failure: "🌑 The shadows whisper... [{workflow_name}]({run_url}) {status}. The oracle requires further meditation..."

After:

messages:
  footer: "> 🧪 *Smoke test run by [{workflow_name}]({run_url})*{ai_credits_suffix}{history_link}"
  run-started: "🧪 Smoke test in progress — [{workflow_name}]({run_url}) is validating Codex engine functionality for this {event_type}."
  run-success: "✅ Smoke test passed — [{workflow_name}]({run_url}) completed successfully. Codex engine is operational."
  run-failure: "❌ Smoke test failed — [{workflow_name}]({run_url}) {status}. Review the [run log]({run_url}) for details."

Why This Matters

• User Impact: Engineers triaging a failed run immediately understand the state from the message without decoding metaphor.
• Quality Factor: Professional Communication + Clarity and Precision
• Frequency: Every smoke test run (triggered twice daily + on PR label + on slash command)

Success Criteria

• Changes made to .github/workflows/smoke-codex.md only
• All four message fields updated (footer, run-started, run-success, run-failure)
• Messages use clear status verbs (in progress / passed / failed) rather than metaphorical states
• Quality rating improves from ❌ to ✅

Scope Constraint

• Single file only: .github/workflows/smoke-codex.md
• No changes to other files required
• Can be completed independently

---

Task 2: Convert Best Practices Paragraph to Numbered List — rate-limiting-controls.md

File to Modify: docs/src/content/docs/reference/rate-limiting-controls.md

Current Experience

The "Best Practices" section near the end of the file is a single dense paragraph:

"Start with conservative limits and increase as needed. Use environments for high-risk operations (workflow dispatches, cross-repo operations, production systems). Layer multiple controls: rate limiting with concurrency, timeouts with stop-after, safe output limits with environments. Monitor workflow runs, safe output logs, and rate limit cancellations to identify needed adjustments."

Quality Issue

Design Principle: Efficiency and Productivity

Four distinct, independent best practices are merged without visual separation. Every other section in this document uses code blocks, tables, and bullet lists. The inconsistency signals lower importance and makes these recommendations easy to miss — exactly the opposite of their intent.

Proposed Improvement

Before:

## Best Practices

Start with conservative limits and increase as needed. Use environments for high-risk operations (workflow dispatches, cross-repo operations, production systems). Layer multiple controls: rate limiting with concurrency, timeouts with stop-after, safe output limits with environments. Monitor workflow runs, safe output logs, and rate limit cancellations to identify needed adjustments.

After:

## Best Practices

1. **Start conservative** — begin with low limits and increase based on observed usage patterns.
2. **Use environments for high-risk operations** — workflow dispatches, cross-repo operations, and production systems should require manual approval via a GitHub Environment.
3. **Layer controls** — combine rate limiting with concurrency groups, pair `timeout-minutes` with `stop-after`, and back safe output limits with environment approvals.
4. **Monitor regularly** — review workflow run history, safe output logs, and rate-limit cancellations to identify when limits need adjustment.

Why This Matters

• User Impact: New users configuring rate limiting for the first time scan this list to avoid misconfigurations. Clear enumeration makes each recommendation independently actionable.
• Quality Factor: Visual hierarchy + Efficiency
• Frequency: High — this is a reference page users consult during initial setup and when diagnosing runaway workflows.

Success Criteria

• Changes made to docs/src/content/docs/reference/rate-limiting-controls.md only
• Single paragraph converted to 4-item numbered list with bold lead-ins
• All four recommendations preserved with equivalent content
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/reference/rate-limiting-controls.md
• No changes to other files required
• Can be completed independently

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• storage.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "storage.googleapis.com"

See Network Configuration for more information.

📊 User experience analysis by Delight · 48.1 AIC · ⌖ 5.62 AIC · ⊞ 8.4K · ◷

• expires on Jul 20, 2026, 7:12 AM UTC-08:00