v0.4.0 — skillprobe is now skilldx

skillprobe is now skilldx

One name everywhere: the GitHub repo, the PyPI package, the command, the GitHub Action and the Docker image. Old GitHub links redirect here.

• GitHub Action — uses: AndreySukhanov/skilldx@v0.4.0: runs a probe in CI, puts the report in the job summary, fails the build below fail-under.
• PyPI — pipx install skilldx (or pipx upgrade skilldx).
• Docker — ghcr.io/andreysukhanov/skilldx (amd64, arm64).

Upgrading from skillprobe or skilldx 0.3.0

• The command and the module are skilldx now: skilldx run probe.yaml, python -m skilldx, from skilldx import .... The skillprobe command is gone.
• The response cache folder is .skilldx_cache/. Rename .skillprobe_cache/ next to your probe.yaml, or set cache_dir: .skillprobe_cache, and cached responses keep hitting.

The routing golden set behind the bundled reports is on Hugging Face: alex7604/agent-skill-routing.

v0.3.0 — GitHub Action, PyPI (skilldx)

What's new

• GitHub Action — evaluate agent skills in CI in one step; the report lands in the job summary and fail-under fails the build when a skill starts losing its requests:

  - uses: AndreySukhanov/skillprobe@v0.3.0
    env:
      OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
    with:
      probe: evals/probe.yaml
      fail-under: "0.9"

• On PyPI as skilldx — pipx install skilldx (skillprobe on PyPI is an unrelated project). The command is skillprobe, also skilldx.

• The routing probes' response cache is in the repo — change a label in cases.yaml and rebuild the reports on 28 public skills with --cache-only, no API key, nothing to pay. The Action's own self-test runs on it.

• Ready Docker image ghcr.io/andreysukhanov/skillprobe (amd64, arm64), now tagged with each release.

v0.2.0 - routing across 28 public skills

Routing between many skills

A real agent never sees one skill - it sees a list, and every description competes for every request. v0.2.0 measures that.

• Routing labels - loads / not_loads per case ("*" = no skill at all); unlabeled skills are neutral and not scored.
• routing_only mode - each trial stops after the model's first turn, skill bodies are never sent. Thirty skills and a hundred cases cost cents.
• Per-skill routing table - loaded when required · loaded when forbidden, plus a collisions table: which skill took a request that belonged to another.
• Golden sets on 28 public skills - 14 from anthropics/skills, 14 from obra/superpowers, 46 labeled cases, 627 trials on five models. Findings in the README: brainstorming ("You MUST use this before any creative work") takes a plain "add a function" request from test-driven-development in 12 of 12 trials.
• A reply cut off by max_tokens before a routing decision is reported as an error, not as "no skill loaded".

Also in this release

• New README layout with a banner and a contributing guide.
• License changed from MIT to Source-Available - free to read, run and learn from; commercial or derivative use requires written permission. Third-party skills under examples/public-skills/ keep their own licenses (Apache-2.0, MIT).

35 tests, no network. Full reports: examples/public-skills/*/results/report.md.