[delight] User Experience Analysis Report - 2026-08-18

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation file(s): model-alias-specification.md, 2026-06-08-migrating-from-effective-tokens-to-ai-credits.md
• 2 workflow message configuration(s): pr-triage-agent.md, skillet.md
• 1 validation file: pkg/workflow/repo_memory_validation.go

Overall Quality: Generally professional. The specification doc and validation errors are strong models of enterprise-grade clarity. The main gaps are in AI-generated status messages, where failure states lack actionable next steps.

Key Finding: The run-failure message in pr-triage-agent.md tells users a run failed but gives no concrete next step, forcing them to dig through logs unassisted — a missed opportunity to reduce triage friction on an automation-heavy workflow.

Quality Highlights ✅

Example 1: Actionable, example-rich validation errors

• File: pkg/workflow/repo_memory_validation.go
• What works well: Every validation error names the offending value, states the constraint, and supplies a corrected example inline (Design Principle: Trust and Reliability / Documentation Quality).
• Quote/Reference: Line 42 — "branch-prefix must be at least 4 characters long, got %d. Example: branch-prefix: my-bot"

Example 2: Well-structured formal specification

• File: docs/src/content/docs/specs/model-alias-specification.md
• What works well: Clear TOC, versioned change log, worked resolution example (Appendix A) with a full trace — gives implementors a concrete reference instead of prose alone.
• Quote/Reference: Lines 862-898, the full Resolve(...) trace for deep-think?temperature=0.1.

Improvement Opportunities 💡

High Priority

Opportunity 1: Actionless failure message - pr-triage-agent.md

• File: .github/workflows/pr-triage-agent.md
• Current State: Line 53 — run-failure: "❌ PR triage failed! [{workflow_name}]({run_url}) {status}. Some PRs may not be triaged."
• Issue: The message reports failure but gives no guidance on what to do next (check logs? re-run? known transient cause?). Compare to run-success, which tells the user exactly where to look ("Check the issue for detailed report.").
• User Impact: On failure, enterprise users (often triaging dozens of PRs on a schedule) are left to manually open the run log with no hint of where to start, increasing time-to-resolution.
• Suggested Change: Add a concrete next step pointing at the run log, consistent with the success message's pattern of directing users to a resource.
• Design Principle: Trust and Reliability — "Clear error messages with actionable solutions."

Medium Priority

Opportunity 2: Missing verification step after fix command - migration blog post

• File: docs/src/content/docs/blog/2026-06-08-migrating-from-effective-tokens-to-ai-credits.md
• Current State: The post instructs gh aw fix --write (lines 1035-1039) but does not show what a successful update looks like or how to confirm the migration took effect.
• Issue: Users following the migration guide have no way to validate the outcome before moving on, which is a documentation-completeness gap for an operational change to spend-metric reporting.
• User Impact: Increases support questions/uncertainty for teams performing this migration across many workflows.
• Suggested Change: Add a short "Verify the change" step showing sample gh aw audit output with the new AIC field.
• Design Principle: Documentation Quality — "Complete and accurate... practical examples that reflect real use cases."

Files Reviewed

Documentation

• docs/src/content/docs/specs/model-alias-specification.md - Rating: ✅
• docs/src/content/docs/blog/2026-06-08-migrating-from-effective-tokens-to-ai-credits.md - Rating: ⚠️

Workflow Messages

• .github/workflows/pr-triage-agent.md - Rating: ⚠️
• .github/workflows/skillet.md - Rating: ✅

Validation Code

• pkg/workflow/repo_memory_validation.go - Rating: ✅

Metrics

• Files Analyzed: 5
• Quality Distribution:
  • ✅ Professional: 3
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Task 1: Improve pr-triage-agent.md run-failure message

File to Modify: .github/workflows/pr-triage-agent.md

Current Experience

Line 53 shows a failure message with no actionable guidance:

run-failure: "❌ PR triage failed! [{workflow_name}]({run_url}) {status}. Some PRs may not be triaged."

Quality Issue

Design Principle: Trust and Reliability

The message states the failure but stops short of telling the user where to look or what to check, unlike the paired run-success message which explicitly says "Check the issue for detailed report."

Proposed Improvement

Add a concrete next step referencing the run log.

Before:

run-failure: "❌ PR triage failed! [{workflow_name}]({run_url}) {status}. Some PRs may not be triaged."

After:

run-failure: "❌ PR triage failed! [{workflow_name}]({run_url}) {status}. Some PRs may not be triaged — check the run log linked above for the failing step before re-running."

Why This Matters

• User Impact: Gives users an immediate, actionable first step instead of an open-ended failure notice.
• Quality Factor: Trust and Reliability / actionable error messaging.
• Frequency: This message fires on every failed scheduled run (every ~6h), so the improvement compounds over time.

Success Criteria

• Changes made to .github/workflows/pr-triage-agent.md only
• run-failure message includes a concrete next step
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: .github/workflows/pr-triage-agent.md
• No changes to other files required
• Can be completed independently

---

Task 2: Add verification step to AI Credits migration blog post

File to Modify: docs/src/content/docs/blog/2026-06-08-migrating-from-effective-tokens-to-ai-credits.md

Current Experience

The post tells readers to run gh aw fix --write (around line 1038) but never shows what success looks like, so there's no way to confirm the migration worked before moving on.

Quality Issue

Design Principle: Documentation Quality — complete information, practical examples.

Readers following an operational migration guide need a way to validate the outcome; without it, uncertainty and support requests increase.

Proposed Improvement

Add a short "Verify the change" subsection after the gh aw fix --write code block showing expected gh aw audit output referencing AIC.

Before:

For repositories that need automatic workflow updates, run:

```bash
gh aw fix --write

Metric reference

**After:**
```markdown
For repositories that need automatic workflow updates, run:

```bash
gh aw fix --write

Verify the change

Confirm the update took effect by re-running the audit:

gh aw audit

Output should now report cost using AI Credits, for example:
Cost: 12.4 AIC ($0.124) instead of an Effective Tokens-only figure.

Metric reference

**Why This Matters**
- **User Impact**: Gives readers a concrete way to confirm the migration succeeded, reducing follow-up support questions.
- **Quality Factor**: Documentation completeness / practical examples.
- **Frequency**: Every team migrating from ET to AIC will follow this post once, but uncertainty at this step is a common documentation-completeness gap.

**Success Criteria**
- [ ] Changes made to `docs/src/content/docs/blog/2026-06-08-migrating-from-effective-tokens-to-ai-credits.md` only
- [ ] A "Verify the change" subsection with example output is added after the `gh aw fix --write` block
- [ ] Quality rating improves from ⚠️ to ✅

**Scope Constraint**
- **Single file only**: `docs/src/content/docs/blog/2026-06-08-migrating-from-effective-tokens-to-ai-credits.md`
- No changes to other files required
- Can be completed independently


<!-- gh-aw-tracker-id: delight-daily -->

> 📊 *User experience analysis by [Delight](https://github.com/github/gh-aw/actions/runs/32151040092)* · auto · 35.6 AIC · ⌖ 9.47 AIC · ⊞ 12.3K · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+%22gh-aw-workflow-call-id%3A+github%2Fgh-aw%2Fdelight%22&type=discussions)
> - [x] expires <!-- gh-aw-expires: 2026-08-21T15:03:31.332Z --> on Aug 21, 2026, 7:03 AM UTC-08:00

<!-- gh-aw-agentic-workflow: Delight, gh-aw-tracker-id: delight-daily, engine: copilot, model: auto, id: 32151040092, workflow_id: delight, run: https://github.com/github/gh-aw/actions/runs/32151040092 -->

<!-- gh-aw-workflow-id: delight -->
<!-- gh-aw-workflow-call-id: github/gh-aw/delight -->

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #54031.