📚 Documentation Noob Test Report - January 18, 2026

[github-actions[bot]]

Documentation Noob Test Report - January 18, 2026

Executive Summary

As a complete beginner approaching GitHub Agentic Workflows for the first time, I conducted a thorough review of the documentation structure and content. The documentation has many strengths but several areas could significantly improve the new user experience.

Overall Impression: The documentation is comprehensive and technically accurate, but assumes significant prior knowledge. A true beginner without GitHub Actions experience would struggle to get started.

Pages Reviewed

1. Home page (index.mdx)
2. Quick Start (setup/quick-start.md)
3. CLI Commands (setup/cli.md)
4. Creating Workflows (setup/agentic-authoring.mdx)
5. Navigation structure and glossary

Total documentation pages: 111

---

🔴 Critical Issues (Block Getting Started)

1. Missing "What is GitHub Actions?" Prerequisite

Location: Quick Start page, Prerequisites section

Problem: The prerequisites assume users know what GitHub Actions is, but many beginners coming from AI/LLM backgrounds may not have GitHub Actions experience.

Impact: HIGH - Users can't understand the compilation model without understanding GitHub Actions basics.

Evidence:

Prerequisites:
- ✅ GitHub Actions enabled in your repository

This mentions GitHub Actions but doesn't explain:

• What GitHub Actions are
• Why they're needed
• How to verify they're enabled
• What happens if they're not enabled

Recommended Fix:
Add an expandable section:

:::tip[New to GitHub Actions?]
GitHub Actions is GitHub's built-in automation platform that runs code in response to events (like pushes, issues, or schedules). Agentic Workflows compile to GitHub Actions, so they run securely on GitHub's infrastructure.

**Check if Actions are enabled:**
1. Go to your repository on GitHub
2. Look for the "Actions" tab in the navigation bar
3. If you don't see it, go to Settings → Actions → General and enable it

[Learn more about GitHub Actions →](https://docs.github.com/en/actions)
:::

---

2. Unclear PAT Permissions Flow

Location: Quick Start, Step 4 - Add an AI secret

Problem: The PAT creation steps are buried and confusing. The "Can't find Copilot Requests permission?" callout suggests common problems but the main flow should prevent these issues.

Impact: CRITICAL - Users will fail at this step and abandon the setup.

Evidence from Quick Start:

3. Add permissions:
   - In "Account permissions" (not Repository permissions), find "Copilot Requests"
   - Set to "Access: Read"

Confusing elements:

1. Why "Account permissions" vs "Repository permissions"? (Not explained upfront)
2. What if I accidentally select "Repository access: Only select repositories"? (Copilot Requests won't appear)
3. The troubleshooting callout appears AFTER the main instructions, but users hit these issues DURING setup

Recommended Fix:

1. Add a visual diagram or screenshot showing the correct permission location
2. Reorder instructions to prevent common mistakes:

   **⚠️ Critical**: Follow this exact order or the permission won't appear:
   1. Token name: "Agentic Workflows Copilot"
   2. Resource owner: YOUR PERSONAL ACCOUNT (not an org)
   3. Repository access: "Public repositories" or "All repositories" (NOT "Only select repositories")
   4. NOW scroll to "Account permissions" section (below Repository permissions)
   5. Find "Copilot Requests" → Set to "Access: Read"

---

3. "Frontmatter" Term Undefined on First Use

Location: Quick Start, "Understanding Your First Workflow" section

Problem: The term "frontmatter" is used without definition for new users who may not be familiar with markdown conventions.

Impact: MEDIUM-HIGH - Users don't understand the workflow structure.

Evidence:

- **Frontmatter** (YAML configuration section between --- markers)

While this includes a brief explanation, it comes AFTER several earlier uses of the term and assumes knowledge of YAML.

Recommended Fix:
Add a clear definition at first use with visual example:

:::note[What is Frontmatter?]
Frontmatter is the configuration section at the top of your workflow file, written in YAML format and enclosed between `---` markers:

\`\`\`markdown
---
# This is frontmatter - workflow configuration
on: daily
permissions: read
---

# This is the markdown body - your instructions to the AI
Create a daily status report...
\`\`\`
:::

---

4. Compilation Concept Not Intuitive

Location: Quick Start, "How Agentic Workflows Work" section

Problem: The analogy "like writing code in Python that gets compiled to machine code" may not resonate with non-technical users or those from scripting backgrounds.

Impact: MEDIUM - Users don't understand why two files exist or when to compile.

Current explanation:

Think of it like writing code in a high-level language (Python, JavaScript) 
that gets compiled to machine code.

Recommended Fix:
Use a more accessible analogy:

### Why two files?

Think of it like Google Docs vs PDF:
- **`.md` file** = Your Google Doc (editable, human-friendly)
- **`.lock.yml` file** = The PDF (final version that runs)
- **Compilation** = The "Export to PDF" step

You edit the `.md` file (like a Google Doc), then run `gh aw compile` to create the `.lock.yml` file (like exporting to PDF) that GitHub Actions can execute.

🎯 **Rule**: Always edit `.md`, never edit `.lock.yml`

---

5. Missing Visual Workflow Diagram

Location: Quick Start page

Problem: No visual diagram showing the end-to-end flow from creating a workflow to seeing results.

Impact: HIGH - Visual learners struggle to understand the overall process.

Recommended Fix:
Add a simple diagram at the top of Quick Start:

┌─────────────┐      ┌──────────────┐      ┌─────────────┐      ┌──────────────┐
│   Create    │  →   │   Compile    │  →   │   Commit    │  →   │  AI Executes │
│  .md file   │      │ gh aw compile│      │  & push     │      │   & Posts    │
└─────────────┘      └──────────────┘      └─────────────┘      └──────────────┘
     You write           Validates             GitHub              Creates issue
   natural language      and hardens          Actions runs         with report

---

🟡 Confusing Areas (Slow Down Learning)

1. Jargon Overload in Quick Start

Location: Throughout Quick Start

Terms used without definition:

• "Agentic workflow" (used before defined)
• "Safe outputs" (linked but not explained inline)
• "Coding agent" (vs "AI agent" - why two terms?)
• "Lock file" (git lock file? something else?)
• "MCP server" (appears in init step without context)

Example from Step 2:

gh aw init

This command installs agents and tools for GitHub Copilot.

Questions a beginner would have:

• What agents? What tools?
• Do I need this if I don't use GitHub Copilot?
• What does MCP server mean? (mentioned in CLI docs but not explained here)

Recommended Fix:
Add inline definitions or a "Key Terms" sidebar:

:::note[Key Terms You'll See]
- **Agentic workflow**: A workflow file where AI makes decisions (vs traditional fixed scripts)
- **Safe outputs**: Pre-approved actions the AI can take (like creating issues)
- **Compilation**: Converting your .md file to GitHub Actions YAML
- **Lock file**: The compiled .lock.yml file (like a lockfile in package managers)
:::

---

2. CLI Commands Page Overwhelming for Beginners

Location: setup/cli.md

Problem: The CLI Commands page is 490+ lines with 20+ commands. While it has a "Most Common Commands" section at the top, a beginner looking for "how to run my workflow" might get lost.

Recommended Fix:

1. Keep the "Most Common Commands" section ✅ (this is good!)
2. Add a "First Time Here?" callout:

   :::tip[First Time Here?]
   **Just getting started?** You only need these 3 commands:
   1. `gh aw init` - One-time setup
   2. `gh aw add <workflow>` - Download a workflow
   3. `gh aw run <workflow>` - Test it
   
   Once you're comfortable, explore the full command reference below.
   :::

---

3. "Copilot Requests" Permission Name Confusion

Location: Quick Start, PAT creation

Problem: The permission is called "Copilot Requests" (not "Copilot Access" or "Copilot API"), which isn't intuitive.

Impact: Users search for "Copilot" and miss this permission.

Recommended Fix:
Add a visual marker:

4. Scroll to **"Account permissions"** and look for:

🔍 Search for: "Copilot Requests"

(Not "Copilot", not "GitHub Copilot" - the exact text is "Copilot Requests")

---

4. Unclear What Happens After gh aw run

Location: Quick Start, Step 5

Problem: After running gh aw run daily-team-status, users don't know:

• How long it takes
• Where to see progress
• What to do if it fails
• Where the issue will appear

Current text:

After a few moments, check the status:

"A few moments" is vague. Is it seconds? Minutes? Hours?

Recommended Fix:

### Step 5 — Run and Verify

Trigger the workflow:

\`\`\`bash
gh aw run daily-team-status
\`\`\`

**What happens next:**
1. ⏱️ **Execution time**: Usually 1-3 minutes depending on your repository size
2. 🔍 **Watch progress**: Click the GitHub Actions URL shown in the output
3. ✅ **When complete**: A new issue will appear in your Issues tab
4. ❌ **If it fails**: Run `gh aw logs daily-team-status` to see what went wrong

**Where to find your report:**
- Go to your repository on GitHub
- Click "Issues" tab
- Look for an issue with label `[team-status]`

---

5. "Agentic Authoring" vs "Creating Workflows"

Location: Navigation sidebar and page titles

Problem: The page is titled "Creating Workflows" but the URL is agentic-authoring and the sidebar says "Creating Workflows". This terminology inconsistency is confusing.

Recommended Fix:
Pick one term:

• Option 1: "Creating Workflows" everywhere (simpler for beginners)
• Option 2: "Agentic Authoring" everywhere (more specific but requires explanation)

I recommend Option 1 for beginners, with a note:

title: Creating Workflows with AI

> This guide shows you how to write workflows using GitHub Copilot's custom agent. 
> This approach is sometimes called "agentic authoring" because you're giving 
> instructions to an AI agent.

---

6. Examples Don't Show Expected Output

Location: Quick Start, Daily Issues Report example

Problem: The example shows the workflow code but not what the output looks like.

Recommended Fix:
Add a collapsible "Example Output" section:

<details>
<summary>📝 Example Output (click to expand)</summary>

When this workflow runs, it creates an issue like this:

**Title**: [team-status] Daily Team Status - January 18, 2026

**Body**:

🌟 Daily Team Status

Recent Activity

• 3 new issues opened
• 5 pull requests merged
• 2 discussions started

Team Highlights

...

</details>

---

🟢 What Works Well

1. "Most Common Commands" Section in CLI

The CLI page opens with the 5 most essential commands. This is perfect for beginners!

2. Clear Warning About AI Decisions

The warning at the top of Quick Start is excellent:

> Using agentic workflows means giving AI agents the ability to make decisions

This sets proper expectations.

3. Expandable Tips and Callouts

The use of :::tip and :::note callouts helps organize information hierarchically.

4. Prerequisites Checklist

The ✅ checklist format for prerequisites is clear and actionable:

- ✅ GitHub CLI (`gh`) installed
- ✅ GitHub account with admin or write access

5. Code Examples with Wrap

Using wrap attribute on code blocks (bash wrap) helps with long commands.

---

Recommendations

Quick Wins (High Impact, Low Effort)

1. ✅ Add "New to GitHub Actions?" expandable section to Quick Start prerequisites
2. ✅ Add visual workflow diagram at top of Quick Start
3. ✅ Define "frontmatter" inline on first use with a visual example
4. ✅ Add "Example Output" collapsible to show what workflows produce
5. ✅ Clarify "Copilot Requests" permission with exact search text

Medium Priority (Important for Comprehension)

6. Improve compilation analogy - use "Google Docs vs PDF" instead of "Python vs machine code"
7. Add "Key Terms" sidebar to Quick Start with inline definitions
8. Expand Step 5 guidance - explain timing, where to find results, how to troubleshoot
9. Add visual/screenshot for PAT permissions location
10. Consolidate "agentic authoring" terminology - pick one term and stick with it

Long-term Improvements

11. Create video walkthrough - 5-minute "Your First Workflow" screencast
12. Add troubleshooting decision tree - "Workflow failed? Start here" flowchart
13. Create a glossary quick reference card - one-page PDF with all key terms
14. Add more "Example Output" sections across guides to show what success looks like
15. Consider progressive disclosure - beginner/intermediate/advanced tabs for different skill levels

---

User Journey Analysis

Persona: Alex, a Developer New to AI Workflows

Background:

• 3 years of web development experience
• Familiar with Git and GitHub repos
• Never used GitHub Actions
• Excited about AI but cautious about automation

Journey Through Documentation:

1. Landing on home page ✅

   • Clear tagline: "Automate GitHub tasks safely using AI agents"
   • Sees "Getting Started" button - clicks it

2. Reading Quick Start ⚠️

   • Stumble rejig docs #1: "Prerequisites" mention GitHub Actions but don't explain what they are
   • Stumble Add workflow: githubnext/agentics/weekly-research #2: Step 2 says "init installs agents and tools" - what agents? what tools?
   • Recovery: Continues anyway, trusts the process

3. Creating PAT ❌

   • Blocked: Can't find "Copilot Requests" permission
   • Reason: Selected "Only select repositories" instead of "Public repositories"
   • Recovery: Reads troubleshooting callout, starts over

4. Running workflow ⚠️

   • Confusion: gh aw run executes but Alex doesn't know what's happening
   • Question: "How long should this take?"
   • Action: Opens GitHub Actions tab (not mentioned in docs) to watch progress

5. Seeing results ✅

   • Issue appears! Alex is excited
   • Wants to customize it but unclear where to start

Pain Points:

• Too much assumed knowledge about GitHub Actions
• PAT setup flow needs visual guidance
• Need more "what to expect" guidance at each step

---

Testing Notes

Test Environment:

• Date: January 18, 2026
• Documentation commit: Latest (111 pages)
• Build: Successful (Astro + Starlight)
• Browser: Chromium (Playwright)

Coverage:

• ✅ Home page (index.mdx)
• ✅ Quick Start (setup/quick-start.md)
• ✅ CLI Commands (setup/cli.md)
• ✅ Creating Workflows (setup/agentic-authoring.mdx)
• ✅ Navigation structure
• ⏭️ Examples (briefly reviewed structure)
• ⏭️ Reference pages (not deep-dive tested)

Methodology:
Since Playwright couldn't access the preview server due to network restrictions, I analyzed source markdown files and built HTML directly. This provided authentic content review from a beginner's perspective.

---

Labels: documentation, user-experience, automated-testing, beginner-friendly, getting-started

AI generated by Documentation Noob Tester