[delight] User Experience Analysis Report - 2026-09-12

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (patterns/orchestrator-ops.md, reference/mcp-gateway.md)
• 0 CLI commands (binary not available in this environment for --help sampling)
• 2 workflow message configurations (smoke-crush.md, smoke-pi.md)
• 1 validation file (dangerous_permissions_validation_test.go / its source dangerous_permissions_validation.go)

Overall Quality: Professional overall, with one concrete broken-link defect found in the MCP Gateway specification.

Key Finding: The MCP Gateway spec (docs/src/content/docs/reference/mcp-gateway.md) references a non-existent documentation host (docs.github.com) instead of the project's actual docs site (github.github.com), producing two broken example URLs that enterprise users may copy directly into their configuration.

Quality Highlights ✅

Example 1: Clear Error Messaging in Permission Validation

• File: pkg/workflow/dangerous_permissions_validation.go
• What works well: formatDangerousPermissionsError produces a message that states the problem ("must not have write permissions"), explains the reasoning (writes must go through safe-outputs with a scoped GitHub App token), links to documentation, lists every offending scope explicitly, and provides a ready-to-paste corrected permissions: block.
• Quote: "To fix this issue, remove write permissions from the agent job and use safe-outputs instead. If read access is still needed, keep the read permission: ..." — this is a textbook actionable, non-blaming enterprise error message.

Example 2: Consistent, Professional Workflow Messaging

• File: .github/workflows/smoke-pi.md / smoke-crush.md
• What works well: Both workflows use a consistent messages block (run-started, run-success, run-failure, footer) with a distinct but professional persona per engine (⚡ Crush, 🥧 Pi), predictable status communication, and a footer that always links back to the run for traceability.

Improvement Opportunities 💡

High Priority

Opportunity 1: Broken Documentation Host in MCP Gateway Spec

• File: docs/src/content/docs/reference/mcp-gateway.md
• Current State: Lines 680 and 717 (inside the "Custom Server Types" §4.1.4 examples) reference "https://docs.github.com/gh-aw/schemas/mcp-scripts-config.schema.json".
• Issue: docs.github.com is not this project's documentation host. Per docs/astro.config.mjs (site: "https://github.github.com", base: "/gh-aw/") and every other reference across the codebase (e.g. pkg/workflow/dangerous_permissions_validation.go, runs_on_validation.go, agent_validation.go), the correct host is github.github.com. The equivalent schema is correctly referenced elsewhere in the docs as /gh-aw/schemas/mcp-scripts-config.schema.json on the github.github.com host (see docs/src/content/docs/specs/mcp-scripts-specification.md).
• User Impact: This is a formal specification document (RFC-2119 style, "MUST"/"SHALL" language) meant to be authoritative. An enterprise user copying this example into a real customSchemas config would get a 404 when the gateway attempts to fetch the schema for validation, causing a fail-fast configuration error with a confusing root cause.
• Suggested Change: Replace both occurrences of https://docs.github.com/gh-aw/schemas/mcp-scripts-config.schema.json with https://github.github.com/gh-aw/schemas/mcp-scripts-config.schema.json.
• Design Principle: Trust and Reliability — accurate, working examples in an authoritative specification document.

Files Reviewed

Documentation

• docs/src/content/docs/patterns/orchestrator-ops.md - Rating: ✅
• docs/src/content/docs/reference/mcp-gateway.md - Rating: ⚠️ (one broken-link defect, otherwise thorough and well-structured)

Workflow Messages

• .github/workflows/smoke-crush.md - Rating: ✅
• .github/workflows/smoke-pi.md - Rating: ✅

Validation Code

• pkg/workflow/dangerous_permissions_validation.go (+ its test) - Rating: ✅

Metrics

• Files Analyzed: 5
• Quality Distribution:
  • ✅ Professional: 4
  • ⚠️ Needs Minor Work: 1
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Task 1: Fix Broken Documentation Host URL in MCP Gateway Spec

File to Modify: docs/src/content/docs/reference/mcp-gateway.md

Current Experience

The "Custom Server Types" section (§4.1.4) provides two JSON configuration examples showing how to register a custom customSchemas entry. Both examples (around lines 680 and 717) use the URL:

https://docs.github.com/gh-aw/schemas/mcp-scripts-config.schema.json

Quality Issue

Design Principle: Trust and Reliability

docs.github.com is not this project's documentation host — it resolves to GitHub's general product documentation site, not the gh-aw docs. The project's actual docs site (per docs/astro.config.mjs: site: "https://github.github.com", base: "/gh-aw/") is github.github.com/gh-aw/. Every other reference in the codebase (Go source error messages, other spec pages like mcp-scripts-specification.md) correctly uses github.github.com. This is the only place using the wrong host, and it appears twice in the same file.

Proposed Improvement

Before (lines 680, 717):

  "customSchemas": {
    "safeinputs": "https://docs.github.com/gh-aw/schemas/mcp-scripts-config.schema.json"
  }

After:

  "customSchemas": {
    "safeinputs": "https://github.github.com/gh-aw/schemas/mcp-scripts-config.schema.json"
  }

Why This Matters

• User Impact: Prevents a confusing 404/fetch-failure when a user copies this example verbatim, since §4.1.4's "Validation Behavior" states schema fetch failures MUST result in configuration validation errors — the user would hit an opaque failure caused purely by a documentation typo.
• Quality Factor: Technical accuracy and trust in an authoritative, RFC-2119-style specification document.
• Frequency: This is the only documented example of the customSchemas / custom server type feature, so any user adopting this feature is likely to copy this exact URL pattern.

Success Criteria

• Both occurrences of docs.github.com/gh-aw/schemas/mcp-scripts-config.schema.json replaced with github.github.com/gh-aw/schemas/mcp-scripts-config.schema.json in docs/src/content/docs/reference/mcp-gateway.md
• No other references to the incorrect host remain in the file
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/reference/mcp-gateway.md
• No changes to other files required
• Can be completed independently

📊 User experience analysis by Delight · copilot · auto · 45.6 AIC · ⌖ 10 AIC · ⊞ 12.5K · ◷

• expires on Sep 15, 2026, 6:48 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #60634.

[github-actions[bot]]

Automated smoke test interaction: verifying discussion comment tool (Copilot, run 34765331160).

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · copilot · auto · 63.8 AIC · ⌖ 3.62 AIC · ⊞ 10.1K · ◷
_{Comment /smoke-copilot to run again
Add label smoke to run again}