skillprobe

Behavioral test harness for Claude Code skills. Existing tools lint a skill's structure; skillprobe runs a skill in an isolated fixture workspace via headless Claude Code and asserts on observable outcomes: the response text, the tool calls in the transcript, and the filesystem effects.

It supports trap scenarios — fixtures seeded with a known defect the skill is supposed to catch — so a skill that only passes happy-path prompts is not considered tested.

Static structure linting (frontmatter, file layout, script checks) is a non-goal; for that, see the skill-tester skill. skillprobe answers a different question: does the skill still do what it promises?

Requirements

• Node.js 20 or newer.
• skillprobe run spawns the claude CLI (Claude Code). init and validate do not call Claude and need no credentials.

Install

Not yet published to npm. Run from source:

git clone <this-repo> skillprobe
cd skillprobe
npm install
npm run build
npm link        # optional: puts `skillprobe` on your PATH

Without npm link, invoke the built CLI directly as node dist/cli.js <command>.

Quick start

skillprobe login       # once: browser sign-in, token stored locally (no env vars to manage)
skillprobe init        # scaffold skillprobe.config.json, scenarios/example.yaml, a sample skill
skillprobe validate    # schema-check every scenario (no Claude call)
skillprobe run         # execute scenarios against headless Claude Code

Or drive it from inside Claude Code

Copy skills/skillprobe/ into ~/.claude/skills/ and just talk:

/skillprobe ./my-skills/note-taker

Claude reads the skill under test, drafts a happy-path scenario and a trap scenario for you, validates them for free, and asks before spending quota on a live run — then explains the report, including whether a failure means the skill is broken or the scenario's expectation was wrong. No YAML by hand, no commands to memorize.

validate and run take optional scenario paths/globs; with none, they use the scenarios patterns from skillprobe.config.json (default scenarios/*.yaml).

Scenario schema

Scenarios are YAML. The full contract is in docs/SPEC.md; a minimal example:

name: note-taker-saves-a-note
skill:
  path: ./skills/note-taker      # directory containing SKILL.md; copied into the fixture
fixture:
  files:                         # seeded into the throwaway workdir
    "README.md": |
      # Demo project
prompt: "/note-taker remember to water the plants"
assert:
  - output_contains: "Note saved"
  - file_exists: "notes.md"
  - file_contains: { path: "notes.md", pattern: "water the plants" }
  - tool_used: { name: "Write" }
  - no_writes_outside: ["ws:"]
  - max_turns_used: 5

Path prefixes

Assertion paths use ws: for the fixture workdir (the default when no prefix is given) and home: for the throwaway HOME. Globs are allowed.

Assertion types

Type	Argument	Passes when
output_contains	string or string[]	the response includes every value
output_not_contains	string or string[]	the response includes none of the values
output_matches	regex or regex[]	the response matches every pattern
file_exists	path	at least one path matches (globs allowed)
file_contains	{ path, pattern }	a matching file's contents match the regex
tool_used	{ name, input_matches? }	a tool call matches the name (and optional input regex)
tool_not_used	{ name, input_matches? }	no tool call matches
no_writes_outside	path[]	every tracked change (fixture workdir and throwaway HOME) is within an allowed prefix
max_turns_used	number	the transcript used at most that many turns

no_writes_outside judges what the skill did: infrastructure writes made by Claude Code itself (home:.claude/…) and by Windows (home:AppData/…) are excluded from the verdict. The exclusion is by prefix, so a skill that writes into the throwaway HOME's .claude/ (say, a settings.json) is not flagged either — that HOME is discarded after the run, and the JSON report still records every observed change (see docs/report-schema.md).

CLI

skillprobe init                     scaffold config, an example scenario, and a sample skill
skillprobe validate [scenario...]   schema-check scenarios and fixtures (no Claude call)
skillprobe run [scenario...]        execute scenarios against headless Claude Code

Flags for run:

Flag	Default	Meaning
--model <name>	config, else CLI default	model alias or id (a small model is cheaper)
--max-turns <n>	8	max agent turns per scenario
--timeout <seconds>	300	per-scenario timeout
--concurrency <n>	1	scenarios to run in parallel
--keep-workdir	off	keep the throwaway workdir/HOME after each run
--json	off	emit the machine-readable JSON report

run exits non-zero if any scenario fails. The JSON report schema is documented in docs/report-schema.md; the pretty report and the run summary both include token-usage totals so the cost of a run is visible.

Isolation — what it is and is not

Each scenario gets a throwaway workdir and a throwaway HOME under the system temp directory. The skill is copied into <fakeHome>/.claude/skills/<name>/, and skillprobe points Claude Code's own config and credential discovery at the fake HOME by setting HOME, CLAUDE_CONFIG_DIR, and (on Windows) USERPROFILE. On Windows APPDATA/LOCALAPPDATA are still inherited because the OS needs them, so the isolation targets Claude Code's ~/.claude — not every path a program could consult. The child environment is reduced to an allowlist: base system variables plus the authentication variables a live run needs (ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, CLAUDE_CODE_OAUTH_TOKEN) and ANTHROPIC_BASE_URL for a custom endpoint.

This is not a security sandbox. The skill under test runs inside a real Claude Code session with tool access, and a live run must give that session a working credential. The skill — and any hook a scenario plants in the fake HOME's settings.json — can therefore read that credential from the environment (for example through the Bash tool or a PreToolUse hook) before it does anything else. skillprobe redacts the OAuth token from its own reports as a safety net, but a skill that reads and echoes a credential can still leave it in the transcript that --keep-workdir retains on disk. no_writes_outside audits changes under the two tracked roots (the fixture workdir and the throwaway HOME); it does not observe writes elsewhere on the machine.

Run skillprobe on skills you trust enough to execute. If you must probe an untrusted skill, do not hand it your everyday credential: use a scoped or disposable token you can revoke, and wrap the run in an OS-level sandbox or a throwaway machine.

The verified CLI flags, skill discovery, and transcript location are recorded in docs/harness-notes.md.

Live runs

A live run needs the claude CLI and authentication. The easy path:

skillprobe login    # runs `claude setup-token` for you and stores the token in
                    # ~/.skillprobe/credentials.json for `run` to use
skillprobe logout   # deletes the stored token

login redacts the token from the setup-token output it captures (stdout), and the stored file is written with 0600 permissions on POSIX; on Windows it relies on the ACL it inherits from your user profile, not on POSIX permission bits. The token is a long-lived credential — treat ~/.skillprobe/credentials.json like any other secret, and read the isolation note above before pointing a live run at a skill you do not trust.

Subscription users authenticate once this way and never touch an environment variable. For CI or scripted use, environment variables still work and take precedence over the stored token: ANTHROPIC_API_KEY, CLAUDE_CODE_OAUTH_TOKEN, or ANTHROPIC_AUTH_TOKEN. Set SKILLPROBE_CLAUDE_BIN to point at a specific claude executable if it is not on PATH.

The gated live integration test runs only with SKILLPROBE_LIVE=1; it is excluded from CI, which uses no API keys.

Development

npm run build   # tsc
npm test        # vitest; unit tests use no network and never invoke Claude

License

MIT

日本語のREADMEは README.ja.md にあります。