[delight] User Experience Analysis Report - 2026-09-14

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (docs/src/content/docs/blog/2026-09-14-weekly-update.md, docs/src/content/docs/gallery/metrics-analytics.md)
• 2 workflow message configurations (smoke-pydantic.md, pr-sous-chef.md)
• 1 validation file (pkg/workflow/safe_outputs_needs_validation.go)
• CLI command review was skipped this run: the gh-aw binary could not be built in this sandbox (Go toolchain download to go1.26.8 was blocked by network policy), so no CLI help-text sample could be captured.

Overall Quality: Generally professional. Workflow message blocks are consistent and appropriately branded; the gallery doc is clear and concise. The one gap worth fixing is in a validation error message that omits the actual list of valid job names it is asking the user to choose from.

Key Finding: validateSafeOutputsNeedsField in pkg/workflow/safe_outputs_needs_validation.go tells users "Expected one of the workflow's custom jobs" but never lists what those jobs actually are, forcing the user to search their own frontmatter to find valid needs targets.

Quality Highlights ✅

Example 1: Consistent, branded workflow messages

• File: .github/workflows/smoke-pydantic.md
• What works well: The messages block uses a consistent 🐍 emoji, informative run-state text, and a footer that credits the engine — professional and easy to scan without being noisy.
• Quote/Reference: run-success: "🐍 [{workflow_name}]({run_url}) Pydantic AI delivered."

Example 2: Concise, scoped documentation

• File: docs/src/content/docs/gallery/metrics-analytics.md
• What works well: Explains the security posture up front ("read-only GitHub permissions and only the noop safe output") and links to three relevant reference pages instead of duplicating their content.
• Quote/Reference: "The agent receives read-only GitHub permissions and has only the noop safe output, so collection cannot directly mutate issues or pull requests."

Improvement Opportunities 💡

High Priority

Opportunity 1: Error message omits the actual list of valid job names

• File: pkg/workflow/safe_outputs_needs_validation.go
• Current State: Lines 45-52 and 53-60 return errors like "safe-outputs.%s: unknown job %q. Expected one of the workflow's custom jobs. Example: safe-outputs.%s: [secrets_fetcher]" — the example job name secrets_fetcher is a hardcoded placeholder, not derived from the user's workflow.
• Issue: The function already builds a customJobs map of every valid target (line 33-40) but discards it after computing the count for a debug log — the actual valid job names are never surfaced to the user in the error.
• User Impact: When a user mistypes a needs job name, they get a generic message with an unrelated example job name instead of the list of jobs that actually exist in their own workflow, forcing a manual scan through the frontmatter to self-diagnose.
• Suggested Change: Sort and join the keys of customJobs into the error message so users see their actual available options.
• Design Principle: Trust and Reliability — actionable error messages with correct, contextual information.

Medium Priority

Opportunity 2: Metrics gallery example lacks an output sample

• File: docs/src/content/docs/gallery/metrics-analytics.md
• Current State: The doc describes writing "a compact JSON snapshot to metrics/YYYY-MM-DD.json" but never shows what that snapshot looks like.
• Issue: Readers evaluating this pattern for their own repo cannot tell what fields to expect (e.g., per-workflow run counts, durations) without running the workflow themselves.
• User Impact: Slower adoption decisions; enterprise readers scanning the gallery for applicable patterns typically want to see the shape of the output before committing to a workflow.
• Suggested Change: Add a short fenced JSON example of a metrics/2026-01-15.json snapshot showing 2-3 representative fields.
• Design Principle: Documentation Quality — practical examples that reflect real use cases.

Files Reviewed

Documentation

• docs/src/content/docs/blog/2026-09-14-weekly-update.md - Rating: ✅
• docs/src/content/docs/gallery/metrics-analytics.md - Rating: ⚠️

CLI Commands

• Skipped this run (Go toolchain unavailable in sandbox to build gh-aw)

Workflow Messages

• .github/workflows/smoke-pydantic.md - Rating: ✅
• .github/workflows/pr-sous-chef.md - Rating: ✅

Validation Code

• pkg/workflow/safe_outputs_needs_validation.go - Rating: ⚠️

Metrics

• Files Analyzed: 5
• Quality Distribution:
  • ✅ Professional: 3
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Here are 2 targeted improvement tasks, each affecting a single file:

Task 1: List actual valid job names in the safe-outputs needs validation error

File to Modify: pkg/workflow/safe_outputs_needs_validation.go

Current Experience

When a workflow references an unknown job in safe-outputs.needs, the compiler returns an error naming a hardcoded example job (secrets_fetcher) that may not exist anywhere in the user's workflow, instead of the actual job names that would be valid. The valid job set is already computed in-function (customJobs, lines 33-40) but never surfaced.

Quality Issue

Design Principle: Trust and Reliability

Users must manually re-read their own frontmatter's jobs: section to figure out what a valid needs target looks like, adding an unnecessary diagnostic step to a compile error that could have been self-explanatory.

Proposed Improvement

Sort the customJobs map keys and include them in both error messages so the message is self-contained.

Before:

if !setutil.Contains(customJobs, need) {
    safeOutputsNeedsValidationLog.Printf("Validation failed: %q is not a known custom job", need)
    return fmt.Errorf(
        "safe-outputs.%s: unknown job %q. Expected one of the workflow's custom jobs. Example: safe-outputs.%s: [secrets_fetcher]",
        fieldName,
        need,
        fieldName,
    )
}

After:

if !setutil.Contains(customJobs, need) {
    safeOutputsNeedsValidationLog.Printf("Validation failed: %q is not a known custom job", need)
    available := make([]string, 0, len(customJobs))
    for jobName := range customJobs {
        available = append(available, jobName)
    }
    sort.Strings(available)
    if len(available) == 0 {
        return fmt.Errorf(
            "safe-outputs.%s: unknown job %q. This workflow has no custom jobs defined; add one under 'jobs:' before referencing it here",
            fieldName,
            need,
        )
    }
    return fmt.Errorf(
        "safe-outputs.%s: unknown job %q. Available custom jobs: %s",
        fieldName,
        need,
        strings.Join(available, ", "),
    )
}

(Requires adding "sort" and "strings" to the existing import block; the reserved-job-name branch above it can be left as-is or updated similarly for consistency.)

Why This Matters

• User Impact: Users see their actual valid options immediately, with no need to cross-reference frontmatter.
• Quality Factor: Trust and Reliability — accurate, actionable error messages.
• Frequency: Triggered every time a safe-outputs.needs typo or stale reference occurs during workflow compilation, which is common when jobs are renamed or removed.

Success Criteria

• Changes made to pkg/workflow/safe_outputs_needs_validation.go only
• Error message lists actual available job names (or a clear "no custom jobs defined" message when none exist)
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: pkg/workflow/safe_outputs_needs_validation.go
• No changes to other files required
• Can be completed independently

---

Task 2: Add a sample output snapshot to the metrics-analytics gallery doc

File to Modify: docs/src/content/docs/gallery/metrics-analytics.md

Current Experience

The doc explains that the workflow writes "a compact JSON snapshot to metrics/YYYY-MM-DD.json" (line ~35) but gives no indication of its structure, leaving readers to infer the schema or run the workflow to find out.

Quality Issue

Design Principle: Documentation Quality — practical examples that reflect real use cases

Gallery pages exist to help readers quickly decide whether a pattern fits their needs; without a sample of the output, readers can't evaluate whether the snapshot format meets their downstream analysis needs (e.g., a dashboard or another workflow consuming this data).

Proposed Improvement

Add a short fenced JSON example directly after the paragraph describing repo memory, showing a minimal but representative snapshot.

Before:

Repository memory lets later workflows compare snapshots without adding reports to the default branch. The agent receives read-only GitHub permissions and has only the `noop` safe output, so collection cannot directly mutate issues or pull requests.

## Learn More

After:

Repository memory lets later workflows compare snapshots without adding reports to the default branch. The agent receives read-only GitHub permissions and has only the `noop` safe output, so collection cannot directly mutate issues or pull requests.

A snapshot in `metrics/2026-01-15.json` might look like:

\`\`\`json
{
  "collection_window": "2026-01-14T00:00:00Z/2026-01-15T00:00:00Z",
  "repository": "octo/example",
  "workflows": [
    { "name": "ci.yml", "total_runs": 42, "success": 40, "failed": 2, "avg_duration_seconds": 187 }
  ]
}
\`\`\`

## Learn More

Why This Matters

• User Impact: Readers can immediately assess whether the schema fits their trend-analysis or dashboard needs without running the workflow first.
• Quality Factor: Documentation Quality — concrete, practical examples.
• Frequency: Every reader evaluating this gallery entry before adoption.

Success Criteria

• Changes made to docs/src/content/docs/gallery/metrics-analytics.md only
• A representative JSON snapshot example is added after the repo-memory explanation paragraph
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/gallery/metrics-analytics.md
• No changes to other files required
• Can be completed independently

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• storage.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "storage.googleapis.com"

See Network Configuration for more information.

📊 User experience analysis by Delight · copilot · auto · 58.7 AIC · ⌖ 10.2 AIC · ⊞ 12.5K · ◷

• expires on Sep 17, 2026, 7:05 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #61155.