v0.7.0 - LLM-as-Judge, Multi-Provider LLM & Pattern-Based Optimizer

Overview

Adds LLM-as-judge quality review to sklab evaluate — a 9-criterion rubric grounded in the Agent Skills Spec that scores skills across Activation Quality and Instruction Quality axes. Also ships multi-provider LLM support (OpenAI + Gemini alongside Anthropic), per-skill .sklab/config.yaml, eval history persistence with --optimize chaining, execution providers for trigger test isolation (local + Docker), pattern-based optimizer guidance, and 4 new static checks — bringing the total from 33 to 37.

Breaking Changes

• License change: MIT → Apache 2.0
• docker>=7.0.0 is now a required dependency — Docker SDK for container-based trigger test isolation. The Docker daemon is only needed when using --provider docker.
• openai>=1.0.0 and google-generativeai>=0.8.0 are now required dependencies — needed for multi-provider LLM support in generate, optimize, and evaluate.

What's New

LLM-as-Judge Quality Review

sklab evaluate now scores skills on 9 criteria across two axes using a structured LLM rubric. Runs automatically when an API key is available; --skip-review for static-only.

Activation Quality — will the agent find and activate this skill?

Criterion	What it judges
Intent Clarity	User intent framing vs implementation details
Trigger Coverage	Natural language variations, indirect requests
Scope Precision	Not too broad, not too narrow
Distinctiveness	Differentiation from base agent capabilities

Instruction Quality — will the agent produce good output?

Criterion	What it judges
Domain Expertise	Real specific knowledge vs generic filler
Cognitive Efficiency	Coherent scope, moderate detail, no redundancy
Procedural Clarity	Actionable steps vs vague declarations
Error Resilience	Gotchas, failure modes, recovery guidance
Progressive Disclosure	Core instructions in SKILL.md, heavy content externalized

Each criterion scored 0-4, normalized to 0-100% per axis. Verdict bands: 90+ Excellent, 75-89 Good, 50-74 Needs work, <50 Poor.

sklab evaluate ./my-skill                  # Static + LLM review
sklab evaluate ./my-skill --skip-review    # Static only
sklab evaluate ./my-skill --model gpt-4o   # Use OpenAI for review

Multi-Provider LLM Support

generate, optimize, and evaluate now support OpenAI and Gemini models alongside Anthropic. Provider auto-detected from model ID prefix.

Provider	Prefix	API key
Anthropic (default)	claude-*	ANTHROPIC_API_KEY
OpenAI	gpt-*, o3-*	OPENAI_API_KEY
Gemini	gemini-*	GEMINI_API_KEY

sklab generate ./my-skill --model gpt-4o
sklab optimize ./my-skill --model gemini-2.0-flash

Eval History & Optimize Chaining

sklab evaluate persists full results (37 static checks + 9-criterion judge feedback) to .sklab/evals/, capped at 20 files. sklab optimize reads the latest eval instead of re-running evaluation.

sklab evaluate ./my-skill --optimize   # Evaluate then chain into optimize
sklab optimize ./my-skill              # Standalone (reads latest eval)

Execution Providers for Trigger Testing

sklab trigger now runs tests in isolated environments via --provider:

Provider	Isolation	Flag
local (default)	Temp directory per test	--provider local
docker	Container per test	--provider docker

Docker provider uses a "build once, spawn many" pattern with automatic API key redaction from persisted traces.

sklab trigger ./my-skill --provider docker

Pattern-Based Optimizer Guidance

When the judge flags instruction criteria as low-scoring (≤2), the optimizer dynamically loads spec-sourced before/after transformation patterns and injects them into the prompt. 4 pattern files shipped covering cognitive efficiency, procedural clarity, error resilience, and progressive disclosure (~1600 tokens total).

Enhanced Static Checks

4 new checks (33 → 37 total):

Check ID	Severity	What it catches
content.broken-internal-links	MEDIUM	Markdown links to nonexistent files
structure.files-outside-spec-dirs	LOW	Files outside references/, assets/, scripts/, evals/
content.orphaned-files	LOW	Files in spec dirs not referenced from SKILL.md
structure.scripts-help-support	LOW	Scripts without --help support

Other Changes

• .env file support — API keys loaded from .env via python-dotenv
• Per-skill config — .sklab/config.yaml stores evaluation snapshots, model preferences
• Token cost breakdown — sklab info shows discovery vs front-loaded vs on-demand tokens
• .go scripts — added to valid script extensions
• evals/ directory — recognized as a spec directory

Upgrade

pip install --upgrade skill-lab