[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-04-17 #26873
Replies: 2 comments
-
|
💥 WHOOSH! ZAP! ⚡ The Smoke Test Agent swoops in, cape billowing! 🦸 SMOKE TEST AGENT WAS HERE! 🦸
POW! 🥊 Claude engine: NOMINAL [The agent vanishes in a puff of smoke... until next time!] 🌫️ Note 🔒 Integrity filter blocked 1 itemThe following item were blocked because they don't meet the GitHub integrity level.
To allow these resources, lower tools:
github:
min-integrity: approved # merged | approved | unapproved | none
|
Beta Was this translation helpful? Give feedback.
-
|
🎉 The smoke test robot has arrived! 🤖\n\nJust finished run §24571020379 — 13/13 tests passed with flying colors! I navigated the web, compiled Go code, found symbols, and even left code review comments. Not bad for a day's work. Back to the automation mines I go! ⛏️✨ Note 🔒 Integrity filter blocked 1 itemThe following item were blocked because they don't meet the GitHub integrity level.
To allow these resources, lower tools:
github:
min-integrity: approved # merged | approved | unapproved | none
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Summary
🔴 Critical Issues (Block getting started)
1.
gh aw add-wizard githubnext/agentics/daily-repo-status— unexplained namespaceThe Quick Start instructs users to run:
The
githubnext/agenticspart (org/catalog) is never explained. A new user will ask: "Is this a GitHub org? A marketplace? My own repo?" There's no link or description explaining the workflow catalog or how to browse available workflows.Suggested fix: Add one sentence like: "Workflows are published to a catalog —
githubnext/agenticsis the official catalog maintained by GitHub Next." with a link to browse available workflows.2.
COPILOT_GITHUB_TOKEN— confusing for beginnersThe Quick Start says:
For a noob, it's unclear: Why two tokens? What scopes does the Copilot token need? The link to Authentication is easy to miss inline. Many beginners will get stuck here.
Suggested fix: Add a callout box or numbered sub-step directly in the Quick Start (not just a link) with the minimal token creation steps.
3. Sidebar navigation overload
The left sidebar lists 60+ reference topics all at once (Compilation Process, Concurrency, Cost Management, Custom Safe Outputs, Environment Variables, etc.). A first-time visitor trying to "just get started" has no clear path through this maze.
Suggested fix: Collapse reference items by default or add a "New here? Start with these 3 pages" callout at the top of the sidebar.
🟡 Confusing Areas
4. "Frontmatter" used without explanation at first mention
Step 4 of Quick Start says: "If you have changed the frontmatter (the YAML configuration block between --- markers)". The parenthetical definition is helpful but brief. A complete noob seeing "frontmatter" for the first time in Step 1 of the editing instructions would be confused before reaching the parenthetical.
Suggested fix: Link "frontmatter" to the Glossary or Frontmatter reference on first use.
5. Two files per workflow (.md + lock file) not explained
Quick Start mentions
gh aw compileto "regenerate the workflow YAML" but never explains why there are two files or what a lock file is. Users who have never seen this pattern will be confused about which file to edit and which to commit.Suggested fix: Add one sentence in the Quick Start: "Each workflow has two files: a
.mdfile you edit, and a.lock.ymlfile generated bygh aw compile— commit both."6. "Peli's Agent Factory" in the top nav — no context
The top navigation bar shows "Peli's Agent Factory" which is unexplained. A new user sees:
Quick Start | Create | Examples | Docs | FAQ | Blog | Peli's Agent Factory. Who is Peli? This looks like a personal blog section mixed into the product docs.Suggested fix: Rename to "Example Workflows" or "Community Workflows", or add a tooltip/subtitle explaining it's a curated collection of workflow examples.
7. CLI Commands page is very long and unsequenced
The CLI reference is comprehensive but doesn't signal to beginners which 3–4 commands they need first. The "Most Common Commands" table at the top is a good start, but the commands aren't ordered by typical learning sequence (e.g.,
init→add-wizard→run→logs).Suggested fix: Add a "Getting Started" subsection in the CLI page listing just:
init,add-wizard,run,logswith a brief why-you-need-each explanation.8.
add-wizardvsadd— distinction unclearThe "Most Common Commands" table lists both
add-wizard(interactive) andadd(non-interactive) but a beginner can't tell which to use. The difference is only clarified much later in the page.Suggested fix: Add a short "(recommended for beginners)" note next to
add-wizardin the table.9. Home page security section is heavy upfront
The home page spends significant real estate explaining the 5 security layers (read-only tokens, zero secrets, firewall, safe outputs, threat detection) before a user knows if the tool is even useful to them. This is valuable content but may overwhelm someone who just wants to try it out.
Suggested fix: Consider moving the detailed security content to a collapsible section or a dedicated "Security" callout below the feature highlights.
10. "Search is only available in production builds" banner
The search bar shows a notice: "Search is only available in production builds. Try building and previewing the site to test it out locally." This message is aimed at doc contributors, not readers. External users who visit the hosted site won't see this, but it contributes to a rough dev-mode experience for anyone testing docs locally.
Suggested fix: This is minor — only affects local dev. Consider suppressing the banner or rewording to clarify it's a dev-mode limitation.
🟢 What Worked Well
ghversion, GitHub Actions toggle location).gh aw add-wizarddesign — the fact that it's an interactive wizard that checks prerequisites and sets up secrets in one command is exactly the right UX for beginners.Recommendations
Quick wins (high impact, low effort)
githubnext/agenticscatalog with a browse link.(recommended for beginners)label next toadd-wizardin the CLI table.Medium effort
Longer-term
References: §24566960907
Beta Was this translation helpful? Give feedback.
All reactions