git diff, but for LLM behavior.
You swapped gpt-4o for gpt-4o-mini to cut costs. You upgraded Llama 3.1 to 3.2 because the benchmarks went up. Did your model's behavior actually get better — or did it just get chattier, start ignoring your formatting rules, and flip its answers when you rephrase a question?
Benchmarks measure capability. llm-diff measures behavior — and shows you exactly what changed, in the format every engineer already speaks: a diff.
- baseline ollama/qwen2.5:0.5b
+ candidate openai/gpt-4o-mini
One command. Three behavioral dimensions. Colored deltas. CI-friendly exit codes.
pip install llm-diffCompare a local Ollama model against an OpenAI model:
export OPENAI_API_KEY=sk-...
llm-diff ollama/qwen2.5:0.5b openai/gpt-4o-miniIllustrative output (baseline vs candidate shown for clarity):
- baseline ollama/baseline-model
+ candidate openai/candidate-model
╭──────────────────────┬──────────────────┬──────────────┬──────────────┬────────╮
│ Dimension │ Metric │ - baseline │ + candidate │ Δ │
├──────────────────────┼──────────────────┼──────────────┼──────────────┼────────┤
│ Instruction Fidelity │ fidelity score │ 1.00 │ 0.50 │ -0.50 │
│ Verbosity Profile │ total words │ 41 │ 118 │ +77 │
│ │ preamble words │ 0 │ 9 │ +9 │
│ │ markdown headers │ 0 │ 3 │ +3 │
│ Reasoning Consistency│ consistency │ 1.00 │ 1.00 │ 0.00 │
│ Latency │ avg latency (s) │ 0.84 │ 1.92 │ +1.08 │
╰──────────────────────┴──────────────────┴──────────────┴──────────────┴────────╯
Δ = candidate − baseline · green improvement · red regression
More recipes:
# Two local models, fully offline
llm-diff ollama/qwen2.5:0.5b ollama/llama3.2:1b
# Any OpenAI-compatible server: vLLM, LM Studio, llama.cpp, TGI, ...
llm-diff openai/my-finetune openai/my-base \
--openai-base-url http://localhost:8000/v1
# Open-weight models via Hugging Face Inference Providers (OpenAI-compatible)
llm-diff "openai/meta-llama/Llama-3.3-70B-Instruct" "openai/Qwen/Qwen3-8B" \
--openai-base-url https://router.huggingface.co/v1 --api-key $HF_TOKEN
# Machine-readable report for dashboards and CI artifacts
llm-diff ollama/qwen2.5:0.5b openai/gpt-4o-mini --json > diff.json
# Audit the raw responses behind every score
llm-diff ollama/qwen2.5:0.5b ollama/llama3.2:1b --show-responsesModel specs are backend/model. Unprefixed specs fall back to --backend (default: ollama), and only the first slash is parsed — so HuggingFace-style ids like meta-llama/Llama-3.3-70B-Instruct pass through intact.
| Dimension | The question it answers | How it's scored |
|---|---|---|
| Instruction Fidelity | Does the model obey strict formatting constraints? | Probe demands exactly 3 bullets, no introduction. +0.5 for exactly 3 bullet lines, +0.5 for a clean start with zero preamble. |
| Verbosity Profile | How much does the model say, and how much of it is filler? | One open-ended explanation, measured three ways: total words, filler-preamble words ("Certainly! Here is…"), and markdown header count. |
| Reasoning Consistency | Does the same logic get the same answer when reworded? | Two isomorphic syllogisms with nonsense words (memorization-resistant). Score 1.0 if the Yes/No verdict survives the reframing, 0.0 if it flips. |
Every diff runs at temperature=0 by default, so results are as reproducible as the backends allow.
llm-diff speaks exit codes: 0 all probes succeeded, 1 fatal config error, 2 usage error, 3 partial diff (some probes failed — shown as ERR rows, never a crash). Transient failures (timeouts, 429s, 5xx) are retried with exponential backoff; hard failures surface immediately with actionable messages.
# .github/workflows/model-swap.yml (sketch)
- run: llm-diff ollama/current-prod ollama/candidate --json > diff.json
- run: python scripts/check_thresholds.py diff.json # gate on deltasNative --fail-if "fidelity<0.8" gating is on the roadmap below.
The MVP battery is deliberately small — a fast smoke test, not a benchmark. The v1.0 goal is seven dimensions with a much larger, community-built probe set:
| # | Dimension | Status |
|---|---|---|
| 1 | Instruction Fidelity | ✅ shipped |
| 2 | Verbosity Profile | ✅ shipped |
| 3 | Reasoning Consistency | ✅ shipped |
| 4 | Hallucination Propensity — does the model invent facts, citations, and APIs under pressure? | 🔜 planned |
| 5 | Refusal Sensitivity — does the model over-refuse benign requests, and did that change between versions? | 🔜 planned |
| 6 | Domain Retention — did niche knowledge (SQL dialects, legal terms, medical vocab) survive the new release? | 🔜 planned |
| 7 | Sycophancy Drift — does the model cave and change correct answers when the user pushes back? | 🔜 planned |
Also planned for v0.2: custom probe packs via YAML (llm-diff a b --probes my_domain.yaml), N-way comparison matrices, a native hf/ backend prefix for Hugging Face Inference Providers, and --fail-if threshold gating for CI.
The most valuable thing you can contribute is a probe set. The three built-in probes prove the mechanism; the community can make it comprehensive. If you know how models should behave in your domain — medicine, law, code review, customer support, your native language — you can encode that knowledge as probes + scorers, no ML expertise required.
Great first contributions:
- Probe packs — propose a set of prompts + expected behaviors for one dimension (open an issue with the
probe-packlabel to discuss the format). - Scorers — every scorer is a pure function of response text in
llm_diff/scorers.py, trivially unit-testable. Sharpen the heuristics or add new ones. - Backends — the transport layer in
llm_diff/core.pyis ~100 lines per backend.
Dev setup:
git clone https://github.com/SiddharthFX/llm-diff
cd llm-diff
pip install -e ".[dev]"
pytest # fully offline — tests run against a bundled mock backendSee CONTRIBUTING.md for guidelines. Adding a dimension end-to-end touches exactly three places: a prompt in probes.py, a score_* function in scorers.py, and one hook in evaluate() — the table, summary, JSON report, and exit codes pick it up automatically.
This tool exists to answer a bigger question: do successive LLM releases trade behavioral fidelity for benchmark scores? Read the study design in WHITEPAPER.md — and help us run it.
MIT © 2026 Pluto AI Research Lab