fix(cli): ship complete agent diagnostic skill surface - #1034
Conversation
rickylabs
added
type:fix
area:cli
area:docs
status:plan
|
[PHASE: PLAN] Research and Design are locked for separate-session PLAN-EVAL. Findings
Next
Run artifact: |
|
@openhands-agent model=openrouter/qwen/qwen3.7-max provider=openrouter effort=xhigh output=pr-comment iterations=300 use harness SKILL
Run PLAN-EVAL only for PR #1034 / branch Read, in order:
Spot-check at least one load-bearing research finding against the tree. Walk every Plan-Gate box, OUTPUT CONTRACT (mandatory — verdict first)
|
OpenHands Agent — CompletedOPENHANDS_VERDICT: PASS Model: PLAN-EVAL Summary: PR #1034 — Agent Init Skill SurfaceSummaryCompleted PLAN-EVAL for PR #1034 (branch Verdict: PASS — all Plan-Gate boxes satisfied; implementation may begin. ChangesNo implementation changes made. This was a plan-only evaluation run. Read and evaluated:
ValidationSpot-checked load-bearing research findings against the repository tree:
Responses to Review CommentsPosted formal PLAN-EVAL verdict comment on PR #1034: #1034 (comment) Comment begins with Remaining RisksNone blocking. The plan correctly identifies and addresses:
The two commit slices are well-ordered: source/content/tests first, then generated artifact + full validation. This prevents false stale-barrel passes and keeps generated evidence traceable. Archetype selection (6 — CLI / Tooling) and scope overlay (docs) are correct for this bounded content/installer/guidance change. OPENHANDS_VERDICT: PASS Run: https://github.com/rickylabs/netscript/actions/runs/30714594170 |
|
[PHASE: PLAN-EVAL] [VERDICT: PASS] Plan-Gate ChecklistAll boxes satisfied:
Archetype and Scope
Correct selection. The plan changes embedded skill content, installer-generated assets, and shipped Markdown, not the CLI command surface or public API. Open-Decision Sweep (Evaluator)Independent sweep found no additional decisions that would force rework when deferred. The plan correctly identifies the two "must resolve now" items (regex parsing and version bump) and addresses them in the commit slices. VerdictPASS — implementation may begin. OPENHANDS_VERDICT: PASS |
rickylabs
added
status:impl
and removed
status:plan-eval
|
[PHASE: IMPL] S1 proves the source bundle is complete and symptom-discoverable.
Next: regenerate the embedded barrel, execute the exact scoped gates, and verify a fresh installed artifact. |
Orchestrator note — this PR overlaps #1034/#1030 and must be sequenced#1034 (issue #1023, agent-init skill surface) and #1030 (issue #1026, Aspire agent wiring) were Measured divergence between the two branches on shared files:
Each slice independently authored its own version of the shared skill surface, so this is a Sequencing: land one, rebase the other onto it, and re-verify that an agent can still find |
|
[PHASE: IMPL] S2 proves the generated embedded bundle and scoped gates.
Next: separate-session IMPL-EVAL. |
|
@openhands-agent model=openrouter/qwen/qwen3.7-max provider=openrouter effort=xhigh output=pr-comment iterations=500 use harness SKILL
Run IMPL-EVAL only for PR #1034 / branch Read Verify that PLAN-EVAL passed before implementation, the Design slices match the commit trail, the Independently run or inspect the exact scoped gates:
Do not run scaffold.runtime; it is explicitly N/A. Preserve lock hygiene. Verify the PR carries Post a PR comment beginning exactly OUTPUT CONTRACT (mandatory — verdict first)
|
OpenHands Agent — CompletedOPENHANDS_VERDICT: PASS Model: IMPL-EVAL Summary — PR #1034Run ID: fix-1023-agent-init-skill-surface--skills-discoverability SummaryPerformed IMPL-EVAL for PR #1034 (branch ChangesNo implementation changes made. This was a read-only evaluation per harness protocol. The evaluator verified existing commits without modifying source code. ValidationAll four scoped validation gates executed independently: Gate 1: Type checkingdeno run -A .llm/tools/run-deno-check.ts --root packages/cli --ext ts,tsxResult: PASS
Gate 2: Lintingdeno lint packages/cliResult: PASS
Gate 3: Feature testsdeno test -A packages/cli/src/public/features/agent/init/Result: PASS
Gate 4: Asset barrel freshnessdeno task check:assets-barrelResult: PASS
Additional verificationSymptom routes (all reachable from symptom-indexed playbook):
AGENTS guidance:
Dangling routes:
PR metadata:
scaffold.runtime: Explicitly N/A per plan, PR body, and issue scope. No scaffold/plugin/DB/Aspire-helper changes. Responses to review commentsNo review comments required response. The implementation satisfies the approved plan and all archetype gates. Remaining risksNone. The implementation is complete, tested, and ready for merge. Minor drift recorded in worklog:
ConclusionThe implementation satisfies the approved plan and archetype gates:
OPENHANDS_VERDICT: PASS Run: https://github.com/rickylabs/netscript/actions/runs/30715280587 |
Owner waived the open-model evaluator lane for the 0.0.3 fix train (2026-08-01); the Opus supervisor performs both passes. The slice's OpenHands/Qwen run 30714594170 is recorded as corroboration, not as the authority. IMPL-EVAL discharges the PLAN-EVAL condition: the route-integrity extractor is demonstrated red on origin/main content (three dangling aspire routes) and green on HEAD. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUCp2k2VFnH3vydCXHN3uV
|
@openhands-agent model=openrouter/minimax/minimax-m3 output=pr-comment iterations=400 use harness SKILL
Act as a cheap-and-quick documentation accuracy evaluator. Do not edit source, documentation,
Keep the iteration budget small. Prefer one to three decisive manual checks over broad exploration, |
🤖 Augment PR SummarySummary: Fixes Changes:
🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
|
|
||
| ```sh | ||
| aspire stop --all --non-interactive --nologo | ||
| docker ps -aq | xargs -r docker rm -f |
There was a problem hiding this comment.
skills/help.md:166: docker ps -aq | xargs -r docker rm -f will forcibly remove all Docker containers on the machine (including unrelated ones), which is risky on shared/dev hosts. Consider scoping or qualifying this cleanup guidance so it doesn’t encourage deleting non-NetScript containers.
Severity: high
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
There was a problem hiding this comment.
Fixed in 707e8d235 — and treated as the principle, not the line.
docker ps -aq | xargs -r docker rm -f is gone. Per the owner directive the Aspire CLI is now the documented default for both stopping and inspecting runtime state, inspect-first:
aspire ps --format Json --non-interactive --nologo # inspect running AppHosts first
aspire describe --format Json --non-interactive --nologo # inspect resource state
aspire resource <resource> stop # targeted stop
aspire stop --all --non-interactive --nologo # only after confirming every AppHost is yoursNo Docker removal command ships at all. We deliberately declined to ship a "scoped" docker rm: a removal filter we cannot prove is worse than none, so leftover containers are documented as Aspire's to reclaim, with aspire doctor --format Json as the diagnostic. The only surviving docker references in the bundle are read-only docker ps (verified by grep over skills/ and the generated barrel).
Audit findings beyond line 166:
help.md:37-40("dangling AppHost") had the same shape —docker ps -a # then clear exited orphans— and is now Aspire-first too. The never killaspire mcp startrule is preserved there.- This also removed a real self-contradiction:
aspire/SKILL.md:255-257already said "Leave pre-existing containers alone — Aspire reuses persistent ones and deleting them destroys another session's data," whichhelp.mdwas overriding with a blanket wipe. aspire cache clearis documented, but explicitly scoped as CLI disk-cache only — it does not stop AppHosts or remove containers.
Every verb and flag shipped was verified against the installed CLI (aspire 13.4.6+87fe259e); there is no aspire prune/clean/rm, and none is claimed.
| const installed = new Set(manifest.skills); | ||
| const referenced = new Set<string>(); | ||
| const dangling = new Set<string>(); | ||
| for (const skill of manifest.skills) { |
There was a problem hiding this comment.
packages/cli/src/public/features/agent/init/init-agent_test.ts:131-139: This dangling-route check only scans <skill>/SKILL.md, but .claude/skills/help.md is also installed and is likely to contain skill hand-offs. As written, a dangling “use the X skill” reference introduced in help.md would not be caught by this test.
Severity: medium
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
There was a problem hiding this comment.
Confirmed and fixed in 707e8d235. The scan now iterates [...manifest.skills.map((s) => ${s}/SKILL.md), "help.md"] instead of skills only, and dangling entries are keyed by path so a help.md route is attributable.
Two things to stop this regressing, given the test is the evidence for acceptance box 6:
extractSkillReferencesfinds zero references in today'shelp.md, so simply widening the loop would have been a silently vacuous assertion. The test now also asserts the scanned-path set equals the expected list, so the scan cannot quietly narrow back to skills-only.- The
referencedassertion is kept exact rather than relaxed to a subset check.
Proven load-bearing, and re-verified independently by the supervisor rather than taken from the slice report — temporarily appending a ghost-cleaner route to help.md and regenerating the barrel:
agent init installs the diagnostic surface with no dangling skill routes ... FAILED
- "help.md -> ghost-cleaner",
FAILED | 0 passed | 1 failed
and green again after revert (4 passed | 0 failed). The failure is attributed to help.md, which is exactly the hole you identified.
| | Aspire start/stop, dashboard, resource graph, raw traces/logs | `aspire` | | ||
| | Generic Deno runtime / tooling questions | `deno` | | ||
|
|
||
| When the symptom is unclear—something hangs, vanishes, or silently does nothing—start with |
There was a problem hiding this comment.
skills/netscript/SKILL.md:35-36: The router tells agents to start with help.md, but the installed playbook location is .claude/skills/help.md (as referenced in AGENTS_SECTION). This could lead agents to look for a repo-root help.md instead of the installed playbook.
Severity: low
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
There was a problem hiding this comment.
Valid — fixed in 707e8d235. skills/netscript/SKILL.md:36 now reads .claude/skills/help.md, matching how AGENTS_SECTION refers to it.
Swept the rest of the bundle for the same ambiguity rather than fixing only the reported line: skills/netscript-operate/SKILL.md:50 had an identical bare help.md reference and is fixed too. netscript-build turned out not to carry one, so it needed no change. A grep for help.md across the installed bundle now returns only fully-qualified .claude/skills/help.md paths.
|
[PHASE: IMPL-EVAL] [VERDICT: PASS] Independent evaluation summaryEvaluator session: OpenHands run 30715280587, model Surface / archetype: Process verification
Implementation verification
Independent gate verificationAll four scoped gates executed independently by evaluator session:
scaffold.runtime: Explicitly N/A per plan, PR body, and issue scope. No scaffold/plugin/DB/Aspire-helper changes. PR metadata verification
Acceptance evidence (from worklog)
Architecture debtNone created or closed. Minor drift recorded: ConclusionThe implementation satisfies the approved plan and archetype gates. All required gates pass, all symptom routes are present and reachable, AGENTS guidance is complete and idempotent, the generated barrel is fresh and covered by the asset freshness task, and the PR carries correct metadata. OPENHANDS_VERDICT: PASS |
Co-authored-by: openhands <openhands@all-hands.dev>
…ilable to dogfood Closes the gap that forced acceptance box 5 unticked: skills/aspire, skills/deno and skills/help.md landed on main at 67cf596 while this run was in flight. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLqimdJHXcojxbUGBPVfci
* plan(cli): lock agent init discoverability design * eval(plan): PLAN-EVAL PASS for 1026 agent init discoverability Evaluator: Opus 5 supervisor (owner-waived open-model lane, 2026-08-01). No open-model evaluator dispatch is required for this fix train. * chore(harness): record blocked plan evaluator * fix(cli): ship discoverable Aspire and Deno skills * fix(cli): bound Aspire agent initialization * test(cli): prove agent symptom discoverability * eval(impl): IMPL-EVAL PASS for 1026 aspire agent wiring Evaluator: Opus 5 supervisor (owner-waived open-model lane, 2026-08-01). Gates re-run by the evaluator; cold-start behaviour verified live. Held as draft pending a human decision on merge order with #1023. * chore(harness): brief the destructive-cleanup follow-up slice Scopes the Augment high-severity finding 3696483256 (docker ps -aq | xargs -r docker rm -f in skills/help.md) to a single convergent doc fix plus an assets barrel regeneration, aligned with #1034 707e8d2 so the pending rebase cannot drop either side's content. * fix(skills): replace destructive docker cleanup with Aspire CLI guidance Addresses Augment finding 3696483256 by converging the two cleanup sections with sibling #1034 commit 707e8d2. skills.generated.ts was regenerated with gen:assets-barrel rather than hand-edited. * chore(harness): brief the rebase and review-thread slice Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLqimdJHXcojxbUGBPVfci * fix(cli): gate Aspire delegation on the selected agent host * test(cli): validate linked routing targets for dangling routes * chore(harness): brief the supervisor un-reflow correction Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLqimdJHXcojxbUGBPVfci --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2 participants
Summary
netscript agent initinstalled three skills (164 lines) whose own routing told agents to "use the aspire skill" — a skill it never installed. This ships the complete five-skill diagnostic surface plus a symptom-indexed playbook: 6 files, 863 lines, with no dangling routes.The installer itself was never the defect. It already writes every file the manifest lists, so the fix is manifest + content + the generated barrel, and the installer's bundle iteration and hash verification are untouched.
Scope
Why this exists — and what it does not prove
Measurement across five agent build rounds on
0.0.2:netscript plugin doctor0 uses,aspire otel0 uses,aspire export0 uses — despiteplugin doctorbeing named six times in the shipped skills. A mention in a capability table does not change behaviour; the command has to be reachable from the symptom an agent is actually in.So
plugin doctoris now reachable from three symptoms, not just a verb table:help.md— "A plugin install succeeded, but nothing is wired"netscript-build— "install seemed to work, but its boundary never runs"netscript-operate— "a plugin's boundary never runs, or installation succeeded but nothing is wired"This is structural, not behavioural, proof. Whether these commands actually get invoked can only be confirmed by re-measuring in another instrumented build round against 0.0.3. Please do not read this PR as evidence that the zero-invocation metric has moved.
What changed
skills/aspire/SKILL.mdHealthyis not proof,otel logs|spans|traces,describe --format Json,resource restart,exportskills/deno/SKILL.mdcheckon one file,infofor resolution, permissions, cache/--reload, sanitizers, the exit-code/pipe trapskills/help.mdskills/manifest.jsonhelp.md; bundle0.1.0→0.2.0netscript/-build/-operateinit-agent.tsAGENTS_SECTIONnames all five skills andhelp.md, and directs Aspire vs Deno problems explicitlydeno.jsoncheck:assets-barrelnow diffsskills.generated.ts— it did not before, so editingskills/**without regenerating passed CIREADME.md,docs/site/reference/ai/skills.md,docs/site/reference/cli/commands.mdThe supplied drafts carried their own trap:
deno/SKILL.mdrouted todeno-freshandnetscript-deno-toolchain, neither of whichagent initinstalls. Adopting them unedited would have shipped two new dangling routes while claiming to fix dangling routes. Both are rewritten as direct domain guidance.Acceptance evidence (issue #1023 boxes)
aspireand adenoskill alongside the existing ones — freshagent init --host claudeinto a clean temp dir:aspire/SKILL.md,deno/SKILL.md,help.md,netscript/,netscript-build/,netscript-operate/— 6 files, 863 lines (vs 164 onmain).agent initdoes not install — route extractor over installed content reports none;grepfor every unshipped repo skill name acrossskills/returns nothing..claude/skills/help.md.changedFiles: []test still green).plugin doctor,aspire otel,aspire logs,deno infofrom the symptom — structurally; see the caveat above.Validation (re-run independently by the evaluator)
run-deno-check.ts --root packages/cli --ext ts,tsxdeno lint packages/clideno test -A .../agent/init/deno task check:assets-barreldeno task quality:scan/arch:checke2e:cli run scaffold.runtimeReviewer note
The route extractor was tightened in
9fcd1044fto require the literal word "skill". Verified this removes genuine false positives (Use `help.md`, "the Aspire CLI") without weakening detection — it still catches all three pre-fixaspireroutes, as shown above.Harness
.llm/runs/fix-1023-agent-init-skill-surface--skills-discoverability/3ab64720f— the branch point, containing none of this PR's commits — and a green workflow is not an evaluator verdict. The supervisor verdict above is the only evaluation backing this PR.Review round — owner review, addressed in
707e8d235/a5310a19chelp.md:166shippeddocker ps -aq | xargs -r docker rm -f— force-removes every container on the machinedocker pssurvives anywhere in the bundle.<skill>/SKILL.md, missinghelp.mdhelp.md, dangling entries keyed by path, and the scanned-path set is itself asserted so the scan cannot silently narrow again.help.md, installed path is.claude/skills/help.mdnetscript/SKILL.md; the same bare reference innetscript-operate/SKILL.mdwas found by sweep and fixed too.Comment 1 was fixed as a principle, not a line. It also resolved a live self-contradiction:
aspire/SKILL.mdalready said "leave pre-existing containers alone — deleting them destroys another session's data," whichhelp.mdwas overriding with a blanket wipe. The adjacent "dangling AppHost" block (help.md:37-40) had the same shape and is Aspire-first too; the standing never killaspire mcp startrule is preserved.Every Aspire verb and flag now shipped was checked against the installed CLI (
13.4.6+87fe259e). There is noaspire prune/clean/rmand none is claimed;aspire cache clearis documented but explicitly scoped as CLI disk-cache only, not runtime cleanup.The comment-2 test was verified load-bearing by the supervisor independently of the slice's report — appending a
ghost-cleanerroute tohelp.mdturns it red with"help.md -> ghost-cleaner", and it returns green on revert.Drift / Debt
packages/cli/src/main.tsand--project-rootdo not exist on this checkout; the live contributor entrypoint from a temp cwd reproduces exactly three skills, 164 lines, and dangling Aspire routes.