Skip to content

doesitfire v0.1.0

Choose a tag to compare

View all tags
@diegoperezg7 diegoperezg7 released this 04 Nov 05:37
· 10 commits to main since this release
v0.1.0
c2c6fdd

First release. doesitfire tests whether your SKILL.md skills actually activate in the real agent CLIs.

What it does

  • doesitfire run: runs Claude Code 2.1.317, Codex 0.166.0 and OpenCode 1.19.5 headless in Docker on prompts that
    should and should not load each skill, N repeats each (default 3), and reports recall, false-fire rate and
    precision per agent and skill.
  • Detection: a Skill tool call in Claude Code, the native skill tool in OpenCode, reads of SKILL.md in Codex
    (experimental: Codex documents no activation event).
  • Sandbox: internal Docker network with an egress proxy that only allows the model API, side-effect tools denied,
    2-turn cap, read-only root filesystem, CPU, memory, pids and time limits. Refuses to run without Docker.
  • Providers: Ollama (free, local), Anthropic, OpenAI, OpenRouter and Mistral; keys from env vars only; --max-cost
    budget guard (default 1 USD).
  • Outputs: terminal matrix, results.json, JUnit XML, static HTML report (EN/ES, light/dark), badge SVG and
    shields.io endpoint JSON. doesitfire init scaffolds the config. Composite GitHub Action included.
  • doesitfire demo: a recorded real run, no keys, no Docker.

The recorded demo, exactly

Agent Model Prompts Repeats Runs Errored (excluded)
Claude Code Ollama qwen3:4b (16k context) 12 of 24 1 12 1 (timeout)
OpenCode Mistral ministral-8b-latest 24 3 72 11 (Mistral HTTP 400)

Codex is not in the demo: a live Ollama run did not finish in time. Its activation detection is covered by the
Docker integration test in CI (real Codex CLI against a scripted mock model). The agents used different models, so
their columns are not comparable with each other. Cost of the recording: 0.036 USD (Mistral), Ollama free.

Known issues

  • Ollama's Responses API drops developer messages, where Codex lists skills. For Codex on Ollama, the egress proxy
    rewrites them to system.
  • Turning off qwen3's thinking in Ollama makes it describe the tool call in prose and stop, so doesitfire keeps the
    CLI defaults.

Install

uvx --from git+https://github.com/diegoperezg7/doesitfire@v0.1.0 doesitfire demo

PyPI (uvx doesitfire) is coming. Wheel and sdist attached below.

Assets 4
Loading