Skip to content

PromptCanary v0.2.0

Choose a tag to compare

View all tags
@Mattral Mattral released this 01 Jul 05:40
v0.2.0
70388b8

PromptCanary v0.2.0

Detect silent behavioral drift in LLM providers — before it breaks production.

This release responds directly to community feedback on v0.1.0: current-generation
model examples throughout, first-class support for free local models, a new
Tool Use probe category for agent workflows, trend visualization, property-based
tests, and a full documentation site.

✨ Highlights

Tool Use probes (new category) — silent drift in agent/function-calling
behavior is one of the most damaging and hardest-to-debug regressions. Four new
probes (ToolCallPresenceProbe, ToolCallNameProbe, ToolCallArgsProbe,
ToolCallSchemaProbe) parse OpenAI, Anthropic, and generic JSON tool-call
formats natively — no per-provider configuration needed.

Trend visualization — promptcanary.utils.visualization adds score-history
charts, per-probe heatmaps, and drift timelines. Zero-dependency ASCII rendering
works everywhere; pip install promptcanary[viz] upgrades to interactive Plotly
HTML with no loss of functionality either way.

Updated to current-generation models — README, CLI help text, examples, and
notebooks now reference GPT-5.x, Claude 4.x, and Gemini 3.x instead of legacy
2024-era model strings.

Free local models are now first-class — the Supported Providers table leads
with cost and API-key requirements per provider, and explicitly documents Ollama
models (qwen3.6:27b, deepseek-r1:14b, gpt-oss:20b) as zero-cost canaries
suitable for high-frequency CI checks, with a full cost-aware scheduling guide.

Full documentation site — 23-page MkDocs Material site covering getting
started, core concepts, every probe with examples, per-provider guides,
CI/CD patterns, LangChain/FastAPI integration guides, and auto-generated API
reference. Builds clean in --strict mode (zero broken links).

Property-based testing — Hypothesis now proves core invariants hold across
arbitrary inputs: scores always in [0.0, 1.0], baseline round-trips are
lossless, identical runs never produce false-positive drift.

📊 By the Numbers

  • 19 built-in probes across 5 categories (was 15)
  • 235 tests passing, 89% coverage (was 141 tests, 74%)
  • 23-page documentation site (new)
  • 4 Jupyter notebooks (was 2)
  • Full CLI test coverage via typer.testing.CliRunner (new)

🔧 Fixed

  • @probe decorator now correctly satisfies the BaseProbe ABC contract under
    all conditions (rewritten using type() construction)
  • ToolCallArgsProbe now correctly parses OpenAI's JSON-encoded arguments
    string format inside tool_calls[]

📦 Install

pip install --upgrade promptcanary

🚀 Try It

promptcanary init my-suite
cd my-suite
promptcanary run --provider openai/gpt-5.4 --save-baseline
# ...or free and local:
ollama pull qwen3.6:27b
promptcanary run --provider ollama/qwen3.6:27b --save-baseline

Full changelog: CHANGELOG.md
Architecture rationale: DECISION_LOG.md
Docs: https://promptcanary.readthedocs.io

As always — PromptCanary is community-driven. New probes, canary suites, and
storage backends are the highest-value contributions. See CONTRIBUTING.md.

Assets 2
Loading