doesitfire v0.1.2

doesitfire now ships a Claude Code plugin.

/plugin install doesitfire --marketplace diegoperezg7/doesitfire

Its test-skill-activation skill finds your skills, writes prompts that should and should not trigger each one, agrees agents, models and budget with you, runs doesitfire and rewrites the descriptions of the skills that miss, then measures them again. The skill tells Claude to ask before any paid run, and --max-cost still caps the spend.

Added

• Claude Code plugin with the test-skill-activation skill.

Changed

• The GitHub Action is named "doesitfire skill activation test" and runs v0.1.2 by default (uses: diegoperezg7/doesitfire@v0.1.2).

The activation results of the recorded demo do not change.

doesitfire v0.1.1

Small follow-up to v0.1.0. The activation results do not change.

Changed

• The HTML report formats numbers, percentages and money with Intl.NumberFormat for the active language
  (EN $0.05, ES 0,05 US$).
• The report lists the price used for each paid agent, with its source and the date it was checked, and
  results.json records the configured price per agent.

Fixed: recorded demo cost

v0.1.0 priced Mistral ministral-8b-latest at an unverified 0.10 USD per million tokens. The verified price is
0.15 USD per million input and output tokens (ministral-8b-latest is ministral-8b-2512, per Mistral's
models API; price from docs.mistral.ai, checked 2026-11-04).

Recomputed from the recorded token usage, without re-running: 361,490 tokens across OpenCode's 72 runs, so
0.054 USD instead of the 0.036 USD stated in v0.1.0. examples/doesitfire.mistral.yaml now uses the verified price.

Install

uvx --from git+https://github.com/diegoperezg7/doesitfire@v0.1.1 doesitfire demo

doesitfire v0.1.0

First release. doesitfire tests whether your SKILL.md skills actually activate in the real agent CLIs.

What it does

• doesitfire run: runs Claude Code 2.1.317, Codex 0.166.0 and OpenCode 1.19.5 headless in Docker on prompts that
  should and should not load each skill, N repeats each (default 3), and reports recall, false-fire rate and
  precision per agent and skill.
• Detection: a Skill tool call in Claude Code, the native skill tool in OpenCode, reads of SKILL.md in Codex
  (experimental: Codex documents no activation event).
• Sandbox: internal Docker network with an egress proxy that only allows the model API, side-effect tools denied,
  2-turn cap, read-only root filesystem, CPU, memory, pids and time limits. Refuses to run without Docker.
• Providers: Ollama (free, local), Anthropic, OpenAI, OpenRouter and Mistral; keys from env vars only; --max-cost
  budget guard (default 1 USD).
• Outputs: terminal matrix, results.json, JUnit XML, static HTML report (EN/ES, light/dark), badge SVG and
  shields.io endpoint JSON. doesitfire init scaffolds the config. Composite GitHub Action included.
• doesitfire demo: a recorded real run, no keys, no Docker.

The recorded demo, exactly

Agent	Model	Prompts	Repeats	Runs	Errored (excluded)
Claude Code	Ollama qwen3:4b (16k context)	12 of 24	1	12	1 (timeout)
OpenCode	Mistral ministral-8b-latest	24	3	72	11 (Mistral HTTP 400)

Codex is not in the demo: a live Ollama run did not finish in time. Its activation detection is covered by the
Docker integration test in CI (real Codex CLI against a scripted mock model). The agents used different models, so
their columns are not comparable with each other. Cost of the recording: 0.036 USD (Mistral), Ollama free.

Known issues

• Ollama's Responses API drops developer messages, where Codex lists skills. For Codex on Ollama, the egress proxy
  rewrites them to system.
• Turning off qwen3's thinking in Ollama makes it describe the tool call in prose and stop, so doesitfire keeps the
  CLI defaults.

Install

uvx --from git+https://github.com/diegoperezg7/doesitfire@v0.1.0 doesitfire demo

PyPI (uvx doesitfire) is coming. Wheel and sdist attached below.