[delight] User Experience Analysis Report - 2026-08-25

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files
• 2 workflow message configurations
• 1 validation test file (informational; source of user-facing messages not directly modifiable here)

Overall Quality: Professional, with room for targeted polish.

Key Finding: The MCP Scripts reference doc's Troubleshooting section is too thin relative to the rest of the page's depth — it lists four generic bullet points without any concrete remediation steps, undermining an otherwise excellent, example-rich document.

Quality Highlights ✅

Example 1: MCP Scripts reference documentation

• File: docs/src/content/docs/reference/mcp-scripts.md
• What works well: Clear security warning callout upfront (read-only guidance), consistent per-language examples (JavaScript/Shell/Python/Go), a runnable "Complete Example" section, and a "Related Documentation" cross-link cluster.
• Quote/Reference: "MCP Scripts run outside the agent sandbox and must only implement READ-ONLY operations." — precise, actionable, and appropriately urgent.

Example 2: Approach Validator workflow messages

• File: .github/workflows/approach-validator.md
• What works well: run-success message gives a clear next step ("Review the report and react with ✅ or ❌"), not just a status confirmation.
• Quote/Reference: "✅ [{workflow_name}]({run_url}) completed the approach validation. Review the report and react with ✅ or ❌."

Improvement Opportunities 💡

High Priority

Opportunity 1: Thin, non-actionable Troubleshooting section - Single File Improvement

• File: docs/src/content/docs/reference/mcp-scripts.md
• Current State: Lines 271-276 list four bullet points ("Tool Not Found", "Script Errors", "Secret Not Available", "Large Output") each with a single vague clause and no example command, log excerpt, or fix.
• Issue: The rest of the page is rich with copy-pasteable YAML examples, but Troubleshooting — the section users hit when something breaks — offers no concrete diagnostic steps, unlike the detailed guidance elsewhere on the page.
• User Impact: Enterprise users debugging a failed mcp-script tool have to search elsewhere (workflow logs, source code) instead of finding an answer in the doc meant to solve exactly this problem.
• Suggested Change: Expand each bullet with a short cause + fix, e.g., point to the exact log line to search for, or the config field to check.
• Design Principle: Documentation Quality (complete, practical, addresses real troubleshooting needs)

Medium Priority

Opportunity 2: Generic, repetitive message wording across gate workflows

• File: .github/workflows/design-decision-gate.md
• Current State: Lines 48-52 define run-started/run-success/run-failure messages that only restate the workflow name and generic verbs ("is checking...", "completed the...check", "during...check") without surfacing outcome-specific information (e.g., whether an ADR was found, generated, or is missing).
• Issue: Compared to approach-validator.md's run-success message (which tells the user what to do next), this workflow's messages are purely status-echoing and don't reduce the reader's need to open the run/PR to learn the actual result.
• User Impact: Reviewers scanning PR comments across many gate workflows get low-information notifications and must always click through, slowing triage.
• Suggested Change: Add a brief actionable hint to run-success, e.g., "...Review the generated ADR draft in this PR." (Deferred to a separate task below only if scoped to this file alone.)
• Design Principle: Professional Communication (contextual, actionable messaging)

Files Reviewed

Documentation

• docs/src/content/docs/reference/mcp-scripts.md - Rating: ⚠️ Needs Minor Work
• docs/src/content/docs/blog/2026-01-13-meet-the-workflows.md - Rating: ✅ Professional

Workflow Messages

• .github/workflows/approach-validator.md - Rating: ✅ Professional
• .github/workflows/design-decision-gate.md - Rating: ⚠️ Needs Minor Work

Validation Code

• pkg/workflow/git_tool_validation_integration_test.go - Rating: ✅ Professional (test-only file; the tested behavior — auto-injecting git commands rather than erroring — is itself a strong UX choice that avoids blocking users with a configuration error)

Metrics

• Files Analyzed: 5
• Quality Distribution:
  • ✅ Professional: 3
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Here are 2 targeted improvement tasks, each affecting a single file:

Task 1: Add actionable diagnostic steps to MCP Scripts Troubleshooting section

File to Modify: docs/src/content/docs/reference/mcp-scripts.md

Current Experience

Lines 271-276 contain a bare bulleted list:

## Troubleshooting

- **Tool Not Found**: Verify tool name matches exactly
- **Script Errors**: Check workflow logs for syntax errors
- **Secret Not Available**: Confirm secret name in repository/org settings
- **Large Output**: Agent reads file path from response

Each bullet names a symptom but stops short of telling the reader where to look or what to change.

Quality Issue

Design Principle: Documentation Quality — completeness and practical detail.

The rest of the page provides full YAML snippets for every concept (JavaScript, shell, Python, Go tools, env vars, timeouts). The Troubleshooting section is the one place readers turn to when something is broken, yet it is the least detailed part of the document — a stark quality drop-off right when users need the most help.

Proposed Improvement

Expand each bullet to name the concrete artifact to check (tool name in frontmatter vs. agent call, workflow run logs panel, repo/org secrets settings page, and the returned file-path field) so troubleshooting doesn't require leaving the page.

Before:

## Troubleshooting

- **Tool Not Found**: Verify tool name matches exactly
- **Script Errors**: Check workflow logs for syntax errors
- **Secret Not Available**: Confirm secret name in repository/org settings
- **Large Output**: Agent reads file path from response

After:

## Troubleshooting

- **Tool Not Found**: The tool name the agent calls must exactly match the key under `mcp-scripts:` in frontmatter (case-sensitive). Check the agent's tool list in the run logs against your `mcp-scripts:` block.
- **Script Errors**: Open the workflow run logs and search for the mcp-scripts server output (`MCP Scripts` step) for a stack trace or syntax error; the line number in the trace maps directly to your `script:`/`run:`/`py:`/`go:` block.
- **Secret Not Available**: Confirm the secret referenced in `env:` (e.g., `${{ secrets.API_KEY }}`) is defined at the repository or organization level under **Settings > Secrets and variables > Actions**, and that its name matches exactly.
- **Large Output**: When output exceeds 500 characters, the agent receives a file path instead of inline content (see [Large Output Handling](#large-output-handling)); read that file if the agent's response looks truncated.

Why This Matters

• User Impact: Lets enterprise users resolve mcp-scripts issues without digging through source code or opening a support request.
• Quality Factor: Documentation Quality / completeness.
• Frequency: Every mcp-scripts author eventually hits at least one of these four failure modes.

Success Criteria

• Changes made to docs/src/content/docs/reference/mcp-scripts.md only
• Each Troubleshooting bullet names a concrete location/action to check
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/reference/mcp-scripts.md
• No changes to other files required
• Can be completed independently

---

Task 2: Make design-decision-gate success message outcome-specific

File to Modify: .github/workflows/design-decision-gate.md

Current Experience

Lines 48-52:

  messages:
    footer: "> 🏗️ *ADR gate enforced by [{workflow_name}]({run_url})*{ai_credits_suffix}{history_link}"
    run-started: "🔍 [{workflow_name}]({run_url}) is checking for design decision records on this {event_type}..."
    run-success: "✅ [{workflow_name}]({run_url}) completed the design decision gate check."
    run-failure: "❌ [{workflow_name}]({run_url}) {status} during design decision gate check."

The run-success message only confirms the check ran; it never tells the reader whether an ADR was found, missing, or drafted — the actual outcome that matters to a PR author.

Quality Issue

Design Principle: Professional Communication — messages should be contextual and give next steps, not just echo status.

Compare to approach-validator.md, which tells readers exactly what to do next ("Review the report and react with ✅ or ❌"). This workflow's run-success message gives zero indication of the actual gate result, forcing every reader to open the PR comment/run to find out what happened.

Proposed Improvement

Add a pointer to where the outcome is documented (the PR comment/generated ADR), consistent with how approach-validator.md guides the reader.

Before:

    run-success: "✅ [{workflow_name}]({run_url}) completed the design decision gate check."

After:

    run-success: "✅ [{workflow_name}]({run_url}) completed the design decision gate check. See the comment below for the result and any generated ADR draft."

Why This Matters

• User Impact: Reviewers scanning multiple PR checks get a one-line cue on where to find the actual outcome instead of a content-free confirmation.
• Quality Factor: Professional Communication / actionable messaging.
• Frequency: Runs on every labeled/ready-for-review PR touching gated paths — a frequent touchpoint.

Success Criteria

• Changes made to .github/workflows/design-decision-gate.md only
• run-success message points to where the result/ADR draft can be found
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: .github/workflows/design-decision-gate.md
• No changes to other files required
• Can be completed independently

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

• github.github.io
• storage.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.github.io"
    - "storage.googleapis.com"

See Network Configuration for more information.

📊 User experience analysis by Delight · copilot · auto · 45.9 AIC · ⌖ 10.1 AIC · ⊞ 12.3K · ◷

• expires on Aug 28, 2026, 7:05 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #56070.