[delight] User Experience Analysis Report - 2026-08-04

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (third-party-agent.md, ai-issue-triage.md)
• 2 workflow message configurations (smoke-update-cross-repo-pr.md, discussion-task-miner.md)
• 1 validation file (dangerous_permissions_validation_test.go / dangerous_permissions_validation.go)

Overall Quality: Professional overall, with one confirmed broken documentation reference in a user-facing compiler error message.

Key Finding: The dangerous-permissions compile error points users to docs/safe-outputs.md, a path that does not exist in this repository — every other validator in the codebase correctly links to https://github.github.com/gh-aw/reference/safe-outputs/.

Quality Highlights ✅

Example 1: Third-party agent integration guide

• File: docs/src/content/docs/guides/third-party-agent.md
• What works well: Concrete, runnable example (OpenCode engine definition) shown in full before explaining configuration steps; clear "Related documentation" section with contextual links; practical secret-setup instructions with exact navigation path ("Settings → Secrets and variables → Actions").
• Quote/Reference: "Use a repository-relative path such as shared/opencode.md for a vendored definition, or owner/repo/.github/workflows/opencode-engine.md@v1.2.14 to import a published one — pin remote imports to a tag or SHA to control when you pick up new versions." — precise, actionable, and anticipates the security question of pinning.

Example 2: Discussion Task Miner workflow messages

• File: .github/workflows/discussion-task-miner.md
• What works well: messages block uses a consistent voice across run-started/run-success/run-failure, sparingly uses one emoji per message tied to the workflow's purpose (⛏️/✅/⚠️), and run-failure explicitly directs the user to "review the logs" — an actionable next step.

Improvement Opportunities 💡

High Priority

Opportunity 1: Broken documentation link in dangerous-permissions error message

• File: pkg/workflow/dangerous_permissions_validation.go
• Current State: Line 85 of formatDangerousPermissionsError reads: lines = append(lines, "which uses a scoped GitHub App token. See: docs/safe-outputs.md")
• Issue: docs/safe-outputs.md does not exist anywhere in the repository. Every comparable validator (e.g. strict_mode_permissions_validation.go:37, compiler_main_job_helpers.go:344, compiler_activation_permissions.go:246) links instead to the published reference URL https://github.github.com/gh-aw/reference/safe-outputs/. This is the only write-permission-related error in the codebase using a broken/inconsistent path instead of the canonical published URL.
• User Impact: A user who hits this fairly common compile-time error (adding contents: write etc. directly to a workflow) and clicks/copies the referenced path finds nothing — increasing friction exactly at a security-relevant decision point.
• Suggested Change: Replace the dead relative path with the canonical published reference URL, matching the pattern used elsewhere in the codebase.
• Design Principle: Trust and Reliability — error messages must point to real, working resources.

Files Reviewed

Documentation

• docs/src/content/docs/guides/third-party-agent.md - Rating: ✅
• docs/src/content/docs/guides/ai-issue-triage.md - Rating: ✅

Workflow Messages

• .github/workflows/smoke-update-cross-repo-pr.md - Rating: ✅ (internal smoke test, appropriately terse)
• .github/workflows/discussion-task-miner.md - Rating: ✅

Validation Code

• pkg/workflow/dangerous_permissions_validation.go - Rating: ⚠️ (broken doc reference)

Metrics

• Files Analyzed: 5
• Quality Distribution:
  • ✅ Professional: 4
  • ⚠️ Needs Minor Work: 1
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Here is 1 targeted improvement task affecting a single file:

Task 1: Fix broken documentation link in write-permissions compile error

File to Modify: pkg/workflow/dangerous_permissions_validation.go

Current Experience

When a workflow author adds a write permission (e.g. contents: write) directly to the agent job, gh aw compile fails with a formatted error (see formatDangerousPermissionsError, lines 81-100) that tells the user to consult docs/safe-outputs.md for the recommended safe-outputs pattern. That path does not exist in the repository or published docs site.

Quality Issue

Design Principle: Trust and Reliability

Pointing users to a non-existent reference at the exact moment they need guidance to resolve a security-relevant compile error undermines trust in the tool's error messages and wastes the user's time searching for the right doc.

Proposed Improvement

Update line 85 to reference the canonical, working reference URL already used consistently elsewhere in the codebase (e.g. strict_mode_permissions_validation.go:37).

Before:

lines = append(lines, "which uses a scoped GitHub App token. See: docs/safe-outputs.md")

After:

lines = append(lines, "which uses a scoped GitHub App token. See: https://github.github.com/gh-aw/reference/safe-outputs/")

Why This Matters

• User Impact: Users hitting this error get a working link to the actual safe-outputs documentation instead of a dead path.
• Quality Factor: Trust and Reliability — accurate, actionable error messages.
• Frequency: This is a common first-time error for new workflow authors who initially reach for direct permissions instead of safe-outputs.

Success Criteria

• docs/safe-outputs.md reference replaced with https://github.github.com/gh-aw/reference/safe-outputs/ in pkg/workflow/dangerous_permissions_validation.go
• Existing unit test in pkg/workflow/dangerous_permissions_validation_test.go continues to pass (it does not assert on this specific substring, so no test update needed, but consider adding one)
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: pkg/workflow/dangerous_permissions_validation.go
• No changes to other files required
• Can be completed independently

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• storage.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "storage.googleapis.com"

See Network Configuration for more information.

📊 User experience analysis by Delight · auto · 30.9 AIC · ⌖ 3.96 AIC · ⊞ 12.1K · ◷

• expires on Aug 7, 2026, 7:35 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #50588.