📚 Documentation Noob Test Report - February 11, 2026

[github-actions]

Summary

Date of test: February 11, 2026
Test method: Automated documentation review + build analysis
Pages reviewed: Home page, Quick Start, Creating Workflows, CLI Commands, Overview

Overall impression as a new user:
The documentation shows strong foundational content with clear examples and good structure, but is significantly hampered by extensive broken internal links (30+ broken link errors during build) that would frustrate new users trying to navigate between related topics. The getting started path is well-designed, but link breakage creates a fragmented experience.

---

🔴 Critical Issues Found

1. Extensive Broken Internal Links (BLOCKING)

Issue: The documentation build process reports 30+ broken internal links across key pages, preventing navigation between related concepts.

Affected pages:

• /gh-aw/ (home) - Links to Quick Start and Creating Workflows broken
• /gh-aw/setup/quick-start/ - Links to workflow-structure, common-issues, creating-workflows broken
• /gh-aw/setup/creating-workflows/ - Links to workflow-structure and CLI run command broken
• /gh-aw/introduction/overview/ - Links to workflow-structure and safe-outputs broken
• /gh-aw/introduction/architecture/ - Links to network, threat-detection, safe-outputs broken
• /gh-aw/reference/tools/ - Multiple broken GitHub tools links
• Multiple examples pages with broken references

Build error excerpt:

05:34:25 ▶ gh-aw/setup/quick-start/
05:34:25   ├─ /gh-aw/reference/workflow-structure/ - invalid link
05:34:25   ├─ /gh-aw/troubleshooting/common-issues/ - invalid link
05:34:25   └─ /gh-aw/setup/creating-workflows/ - invalid link

Impact: Users following the getting started guide will encounter dead links when trying to learn about:

• Workflow structure and format
• Troubleshooting common issues
• Tools and permissions
• Network access and threat detection
• Safe outputs and inputs

Recommendation: PRIORITY 1 - Fix all broken internal links before any other documentation work. Users cannot effectively learn if core navigation is broken.

---

2. Missing Content Pages

Based on the link errors, these referenced pages appear to be missing or incorrectly named:

• /gh-aw/reference/workflow-structure/ (referenced from 5+ pages)
• /gh-aw/troubleshooting/common-issues/ (referenced from Quick Start)
• /gh-aw/reference/tokens/ (referenced from SideRepoOps pattern)
• /gh-aw/setup/cli/#run (anchor missing or incorrect)
• /gh-aw/reference/tools/#github-tools-github (anchor issues)
• /gh-aw/reference/safe-outputs/#security-and-sanitization (anchor missing)

Impact: Core conceptual pages that are frequently referenced simply don't exist or have incorrect paths.

Recommendation: Either create the missing pages or update all references to point to existing pages.

---

3. Link Validator Blocking Builds

Issue: The starlight-links-validator plugin is configured with strict settings that block successful builds when any links are broken:

starlightLinksValidator({
    errorOnRelativeLinks: true,
    errorOnLocalLinks: true,
}),

Impact:

• Cannot build documentation locally without disabling the validator
• Cannot deploy documentation with current link state
• Development workflow is blocked by validation errors

Recommendation:

1. Short-term: Temporarily set validation to warning mode instead of error mode during the link fixing phase
2. Long-term: Fix all broken links, then re-enable strict validation

---

🟡 Confusing Areas

1. "Frontmatter" Terminology Without Context

Location: Home page example, Quick Start, multiple reference pages

Issue: The documentation immediately uses the term "frontmatter" (line 40 in overview.mdx) without defining it for non-Jekyll/markdown users. A first-time user who hasn't used static site generators won't understand this concept.

Example from overview:

The YAML section at the top is called frontmatter

Recommendation: Add a tooltip or brief inline explanation on first use:

The YAML section at the top is called **frontmatter** (the configuration metadata at the beginning of a markdown file)

---

2. "Lock File" Concept Needs Earlier Explanation

Location: Home page (line 94), Overview page (line 42)

Issue: The .lock.yml concept is mentioned early but not explained until you click through to the FAQ. New users don't know why there are two files (.md and .lock.yml) or what the lock file does.

Quote:

The gh aw cli augments this with a lock file

Recommendation: Add a brief inline explanation:

The `gh aw compile` command generates a `.lock.yml` file (the compiled GitHub Actions workflow) from your markdown file, similar to how package-lock.json works in npm.

---

3. "Safe Outputs" Terminology Overload

Location: Multiple pages (home, overview, quick start)

Issue: The term "safe outputs" appears frequently but the concept is complex and scattered across multiple explanations. New users see it everywhere but don't immediately understand:

• What makes an output "safe"
• Why this matters
• What happens if they don't use safe outputs

Recommendation: Add a prominent callout on the Quick Start page:

> [!NOTE]  
> **Why "Safe Outputs"?**  
> Workflows run with read-only permissions by default. To allow the AI to create issues, PRs, or comments, you explicitly list which write operations are allowed using `safe-outputs`. This prevents the AI from accidentally modifying your repository in unintended ways.

---

4. Prerequisites Assume GitHub Actions Knowledge

Location: Quick Start (lines 28-33)

Issue: The prerequisites mention "GitHub Actions enabled" but don't explain what this means or how to verify it beyond a generic Settings link.

Current text:

- **GitHub Actions** enabled - Check in [Settings → Actions]

Recommendation: Add specific verification steps:

- **GitHub Actions** enabled - Go to Settings → Actions → General and verify "Allow all actions and reusable workflows" is selected. If you see "Actions permissions," your repository already has Actions enabled.

---

5. Video Links Without Fallback

Location: Home page, Quick Start, Creating Workflows

Issue: The documentation embeds videos but doesn't provide:

• Text descriptions of what the video shows
• Alternative text-based instructions for users who can't watch videos
• Transcript or step-by-step breakdown

Example:

(Video
  src="/gh-aw/videos/install-and-add-workflow-in-cli.mp4"
  aspectRatio="16:9"
/)

Recommendation: Add a text-based alternative:

<details>
<summary>Can't watch the video? Click for step-by-step instructions</summary>

1. Run `gh extension install github/gh-aw`
2. Navigate to your repository
3. Run `gh aw add-wizard githubnext/agentics/daily-repo-status`
4. Follow the interactive prompts
</details>

---

6. "Add-Wizard" Command Name

Location: Quick Start (line 54)

Issue: The command name gh aw add-wizard is not immediately intuitive. New users might expect gh aw add or gh aw setup.

Current:

gh aw add-wizard githubnext/agentics/daily-repo-status

Recommendation: Either:

1. Add an alias gh aw setup that points to add-wizard, or
2. Add a note: "The add-wizard command walks you through an interactive setup"

---

7. Repository Format Not Clear

Location: CLI Commands page, references throughout

Issue: When commands reference repositories, the format isn't consistently explained. Is it owner/repo, github.com/owner/repo, or a URL?

Example from CLI page:

gh aw logs workflow --repo github.enterprise.com/owner/repo

Recommendation: Add a consistent format note at the top of CLI Commands:

> [!TIP]  
> **Repository Format:** Commands accept repositories in the format `owner/repo` (e.g., `github/gh-aw`). For GitHub Enterprise Server, prefix with the hostname: `github.enterprise.com/owner/repo`.

---

🟢 What Worked Well

1. Clear Command Hierarchy

The CLI Commands page has an excellent "Most Common Commands" section at the top that immediately shows new users the 6 commands they'll use 90% of the time. This is outstanding UX design.

2. Progressive Disclosure

The Quick Start guide has a clear time estimate (10 minutes) and breaks steps into manageable chunks. This sets proper expectations.

3. Visual Examples

The home page example of the Daily Issues Report workflow is perfect - it's simple, understandable, and shows immediate value. The combination of markdown + expected output is very helpful.

4. Video Integration

Including videos in the getting started flow is excellent for visual learners, even if fallback text would help.

5. Multiple Authoring Paths

The Creating Workflows page shows 3 different ways to create workflows (GitHub web, coding agent, chatbot), acknowledging different user preferences. This is very thoughtful.

6. Security-First Messaging

The consistent emphasis on read-only defaults and safe outputs throughout the documentation is great for building trust with new users.

---

Recommendations

Quick Wins (Implement First)

1. Fix all broken internal links - This is blocking users from navigating the docs
2. Create missing reference pages or update links to point to existing content
3. Add inline tooltips for "frontmatter", "lock file", and "safe outputs" on first use
4. Add a Prerequisites verification section with actual commands to check GitHub Actions status
5. Set link validator to warning mode during the fix phase

Medium-Term Improvements

6. Add text alternatives for videos with step-by-step instructions
7. Create a Glossary callout box for the home page defining key terms upfront
8. Add repository format clarification at the top of CLI Commands
9. Create a "Concepts" page that explains frontmatter, lock files, safe outputs, and compilation early in the learning path
10. Add troubleshooting inline in Quick Start for common installation issues

Long-Term Enhancements

11. Interactive tutorial - Consider adding an interactive walkthrough using something like @instructure/canvas-rce or similar
12. Error message reference - Document common error messages users will encounter and how to fix them
13. Migration guide - For users coming from traditional GitHub Actions, explain how agentic workflows differ
14. Performance guide - Help users understand workflow execution time and costs

---

Testing Methodology Note

Due to technical limitations with the Playwright browser environment connecting to the local preview server, this report was generated through:

1. Static analysis of markdown source files
2. Build process analysis to identify broken links
3. Content review of key getting started pages
4. Structure analysis of navigation and information architecture

The broken link findings are from actual build errors, while UX observations are based on content review as a first-time user would experience it.

---

Action Items

• CRITICAL: Fix all 30+ broken internal links reported by link validator
• CRITICAL: Create missing reference pages or redirect links to existing content
• Add inline definitions for "frontmatter", "lock file", "safe outputs"
• Add GitHub Actions verification steps to Prerequisites
• Add text alternatives for video content
• Create a central Concepts/Glossary page
• Update CLI commands page with repository format guidance
• Add troubleshooting callouts in Quick Start

---

Labels: documentation, user-experience, automated-testing, needs-triage

---

Note: This was intended to be a discussion, but discussions could not be created due to permissions issues. This issue was created as a fallback.

AI generated by Documentation Noob Tester

• expires on Feb 18, 2026, 5:43 AM UTC