fpverify — Behavioral Fingerprinting for LLM APIs & Agent IDEs

Checks whether an LLM endpoint — an API relay, or the agent IDE you pay for — actually serves the model it claims.

中文文档 →

The problem: API resellers and relays can swap the flagship model you paid for with a cheaper or quantized one. The API format stays the same and the model field in the response still says the flagship name, so nothing at the protocol level gives it away. This is not hypothetical: a March 2026 CISPA audit of "shadow APIs" found 45.83% of tested endpoints failed model-identity fingerprint verification, with benchmark gaps up to 47 points — and those endpoints had been used as official models in 187 academic papers (Real Money, Fake Models). One layer up, the same question applies to agent IDEs: every request goes through the vendor's backend, and which checkpoint actually answers is invisible from the client — reroutes and silent A/B tests included (April 2026: Cursor 3.0 was found by reverse engineering to ship a rebranded Claude Code agent, fine-tuned checkpoint included; the vendor described it as an A/B test on under 1% of traffic).

The method: LLMs cannot produce random output. Ask a model to "name a random number between 1 and 100" and its answers concentrate heavily — on different values for different models. In July 2026 we sampled 9 frontier models × 11 fresh instances each; the 99 answers contained only 4 distinct values (see Measurements). The answer distributions over a few dozen such questions form a stable per-model signature. fpverify sends one-token probes to an endpoint, compares the observed distribution against a reference fingerprint (Jensen-Shannon divergence), and decides with a sequential betting test (e-process). The error rate is bounded: an honest endpoint is judged FAIL with probability ≤ α = 0.01, valid at any stopping point.

(Real run, not staged: both endpoints live on 127.0.0.1; the cheating one claims claude-sonnet-5 while serving a cheaper model. Regenerate with experiments/make_demo_gif.py.)

Usage

Commands below use python; on Windows use py -3.13 -X utf8 instead.

git clone https://github.com/Mohamed7415/fpverify
cd fpverify
pip install -r requirements.txt

Case 1: what is your agent IDE actually running?

Applies to Cursor, Codex, and any IDE that can fan out subagents. The bundled library ships in-harness baselines for 9 frontier models (July 2026, Cursor subagents, battery protocol) — to our knowledge the only public reference set collected inside an agent harness. Bare-API fingerprint databases cannot answer this question: the in-harness distribution is a different conditional (vendor system prompt, sampling, routing), and comparing across conditions misjudges (measurements: experiments/frontier/PROTOCOL.md).

python -m fpverify.cli library                          # list available baselines
python -m fpverify.cli reproduce --claimed gpt-5.6-sol

Paste the pack's cursor_prompt.md into the IDE with the model under test selected. It fans out N fresh subagents (no shared context), each answering the original ten-question battery verbatim, then tallies answers and modes per question. Reading the tally (thresholds ship inside the pack):

• strong questions (reference share ≥ 90%) mostly line up → consistent with the claimed model's July-2026 baseline;
• several strong questions deviate at once, or the tally lines up with a different model's column in Measurements → not behaving like the claimed baseline. That is the alarm;
• either way, an issue with the tally attached is welcome — baselines are re-enrolled after confirmed model-version bumps.

This is a same-protocol comparison (battery vs. battery), so the tally reads directly against the baseline. It is change detection against a dated snapshot, not notarization: the platform's model label was trusted once, at enrollment time. What it catches is silent change and cross-model mismatch after that point.

Case 2: you only have a relay key

Start the local web console:

python -m webui.server

The browser opens http://127.0.0.1:8765. Fill in three fields:

Field	Value
Base URL	the relay address, ending in /v1
API Key	the key the relay gave you
Model name	click "fetch model list" and pick from what the relay actually offers

Leave "library entry" on its default (auto-match by model name). Click start. Request count = samples per question × questions in the library — a few dozen to ~300 one-token requests with default settings, costing cents, done in minutes.

Verdicts:

Verdict	Meaning
PASS	no evidence of substitution within budget
FAIL	behavior deviates significantly from the claimed model's reference (false-positive probability ≤ 0.01), or response-level caching detected
BEST_MATCH	reports which library model the behavior matches (claim not in the library, or reference and probe use different protocols)
UNKNOWN	matches nothing in the library
INCONCLUSIVE	not enough evidence; re-run with more samples

Hard PASS / FAIL verdicts are only issued when the reference was collected under the same protocol as the probes (api-channel references, cold single-question). The bundled cursor-harness references were collected under a battery protocol — the same model answers differently when asked differently — so against them the tool only reports relative ranking (BEST_MATCH / UNKNOWN) and says so in the result.

Below the verdict the console shows a self-verification table: the claimed model's most deterministic questions, their reference answers, copy-paste prompts and a downloadable script. Verification does not require this tool — see Case 4.

Privacy: probe traffic goes directly from your machine to the relay; the key stays in local process memory, never written to disk or uploaded. The fingerprint library is public data inside the repo; update it with git pull.

CLI equivalent:

python -m fpverify.cli library        # list the fingerprint library
python -m fpverify.cli identify --base-url https://relay.example/v1 --api-key KEY --model gpt-5.6 --samples 8

Identification degrades in three steps: claimed model in the library with a protocol-matched reference → sequential test verdict (PASS / FAIL); otherwise → report the closest behavioral match (BEST_MATCH); nothing close → UNKNOWN. The library ships 9 frontier models measured in July 2026 (cursor-harness channel, battery protocol, ranking only). Cold-protocol references for the api channel are open for community contribution — one enrollment costs cents; anti-poisoning rules are in refs/README.md.

Case 3: you have an official API key

The reference fingerprint is enrolled directly from the official channel. No shared library involved; this is the strongest evidence mode.

# 1. Enroll a reference from the official API (~720 one-token requests, a few cents;
#    re-enroll after model version bumps)
python -m fpverify.cli enroll \
    --base-url https://api.openai.com/v1 --api-key $OFFICIAL_KEY \
    --model gpt-5.6-sol --samples 20 --out ref_gpt56.json

# 2. Audit any OpenAI-compatible endpoint claiming to serve that model
python -m fpverify.cli audit \
    --base-url https://some-relay.example/v1 --api-key $RELAY_KEY \
    --model gpt-5.6-sol --ref ref_gpt56.json --report audit.json

Blatant substitution typically triggers early stopping within 15 queries ($0.002). Reports include the aggregated JSD and reference bands from the underlying paper (0.140 same-source / 0.227 cross-deployment / 0.463 impostor).

Two self-checks, which double as a test of this project's FPR claim (one cheap official key is enough):

# Enroll model A from its official API
python -m fpverify.cli enroll --base-url https://api.deepseek.com/v1 \
    --api-key $KEY --model deepseek-v4-pro --samples 20 --out ref_a.json

# Audit the SAME official endpoint against A's reference: must PASS
python -m fpverify.cli audit --base-url https://api.deepseek.com/v1 \
    --api-key $KEY --model deepseek-v4-pro --ref ref_a.json

# Audit a DIFFERENT model against A's reference: must FAIL
python -m fpverify.cli audit --base-url https://api.deepseek.com/v1 \
    --api-key $KEY --model deepseek-v4-flash --ref ref_a.json

If an official, direct-connection endpoint FAILs against its own freshly enrolled reference, open an issue with the audit JSON; that would refute the FPR claim.

Case 4: verify a verdict without trusting this tool

python -m fpverify.cli reproduce --claimed gpt-5.6-sol

Exports a reproduce pack for that model: the questions where its reference is most deterministic, the expected answers, and the conditions the reference was collected under (channel / protocol / tier; see experiments/frontier/PROTOCOL.md).

Reproduction must match the reference's channel and protocol. A fingerprint is a conditional distribution over (model × channel × protocol × tier), and the same model answers differently when asked differently. Measured on GPT-5.6 sol: coin flip inside the ten-question battery = tails 11/11; the same question cold, one fresh chat, one question = heads 6/6. Neither is wrong — they are different conditions. So the pack picks the method by the reference's origin:

1. cursor-harness references (battery protocol): paste cursor_prompt.md into Cursor or any agent IDE with subagents — it fans out N fresh subagents, each answering the original battery verbatim (same channel, same protocol);
2. api references you enrolled with an official key (cold protocol): official_api.py (stdlib only, zero dependencies) asks each question in an independent fresh request, exactly like enroll/audit, and prints observed vs. reference side by side;
3. codex_loop.sh / codex_loop.ps1 loops codex exec, a fresh session per run (a nearby harness channel).

Asking by hand on the official website is yet another channel (web); a mismatch there refutes nothing, and a match is only directional.

One rule regardless of method: every sample must come from a fresh conversation or instance. Asking ten times in one chat is invalid — the model sees its previous answers and varies them.

Reproduction cuts both ways: it checks a FAIL, and it checks a PASS. With a reference you enrolled yourself, run the same pack under the same protocol against the official API (validates the reference table) and against your relay (validates the verdict). If both line up, the conclusion no longer depends on us — official_api.py --base-url points at either endpoint.

If it's us you don't trust — say, "the tool is paid by relays to always print PASS":

• Verdicts are computed on your machine by open-source code, with no telemetry. We never see your URL, key, or result, so there is no channel for per-run tampering; any rigging would have to live in public code.
• Feed it a case you know is fake: the cross-audit self-check at the end of Case 3 (or the cheating endpoint in the local demo below). The mismatched run must FAIL. A tool rigged to always pass is exposed on the spot; CI pins the same assertion.
• The --report JSON includes every raw answer count from the endpoint under test (observed_counts); together with the reference file you enrolled yourself, anyone can recompute the verdict with independent code.
• No paid "certification" or vendor whitelisting, ever. No vendor ads.

Local demo (no keys required)

This section validates the tool itself; no real service is involved. sim/mock_server.py starts fake endpoints on your machine: --kind honest answers from a built-in simulated distribution, --kind swap simulates a relay that claims claude-sonnet-5 but serves a cheaper model. Expected outcome: the first PASSes, the second FAILs within ~15 queries.

pip install httpx

python sim/mock_server.py --port 18801 --kind honest --model claude-sonnet-5 &
python sim/mock_server.py --port 18802 --kind swap   --model claude-sonnet-5 &

python -m fpverify.cli enroll --base-url http://127.0.0.1:18801/v1 --api-key mock \
    --model claude-sonnet-5 --out ref.json
python -m fpverify.cli audit  --base-url http://127.0.0.1:18801/v1 --api-key mock \
    --model claude-sonnet-5 --ref ref.json     # PASS
python -m fpverify.cli audit  --base-url http://127.0.0.1:18802/v1 --api-key mock \
    --model claude-sonnet-5 --ref ref.json     # FAIL

The mock relay implements nine adversaries (--kind): honest / drift / quantized / swap / pin / filter_en / true_random / cache / partial_mimic; see sim/adversaries.py.

Measurements: frontier models cannot be random

July 2026: 9 frontier models, 11 fresh independent instances each (sampled through Cursor subagents under a battery protocol — one instance answers all ten questions in one reply; model identity is platform-guaranteed). Asked "name a random number between 1 and 100", the 99 instances produced 4 distinct answers: 73, 47, 37, 42 — with 73 at 65.7%. Median per-question entropy across all models: 0.44 bits; uniform random would be 6.64 bits.

The most tangible check: open a fresh chat and ask Claude Fable 5 for a random number between 1 and 100. In our runs, 9 of 11 fresh instances answered 73; asked cold as a single question, the thinking variant still answered 73 in 5 of 6. But not every question is stable across protocols — see the note under the table.

Single answers collide (five models share 73 as their mode); the combination of distributions is what forms the fingerprint:

Model (July 2026)	Rand 1–100 (mode)	Color	Animal	City	Coin flip
Claude Fable 5	73 (82%)	teal	otter	Kyoto	heads (100%)
Claude Fable 5 thinking	73 (100%)	teal	otter	Kyoto	heads (100%)
Claude Sonnet 5 thinking	37 (91%)	blue	elephant	Paris	heads (100%)
Claude Opus 4.8 thinking	73 (100%)	blue	fox	Tokyo	heads (100%)
GPT-5.6 sol	73 (91%)	orange	otter	Lisbon	tails (100%)
GPT-5.6 terra	47 (36%)	teal	otter	Lisbon	tails (100%)
GLM-5.2	73 (91%)	teal	fox	Kyoto	heads (91%)
Composer 2.5	47 (100%)	purple	elephant	Tokyo	heads (91%)
Grok 4.5	73 (100%)	teal	otter	Lisbon	heads (45%)

The numbers above are distributions under the battery protocol. A fingerprint is conditional on (model × channel × protocol × tier), and changing the protocol shifts it: GPT-5.6 sol flips its coin tails 11/11 in the battery but heads 6/6 when asked cold on the official website; Fable 5 thinking flips heads 11/11 in the battery but tails 5/6 cold. Not a contradiction — conditional distributions behaving as such. This is why every comparison in this tool requires matching protocols, and cross-protocol comparisons degrade to relative ranking (protocol text and measured evidence: experiments/frontier/PROTOCOL.md).

Findings relevant to auditing (full analysis: docs/RESEARCH_NOTES.md §7):

• Same weights, different reasoning mode → same fingerprint. Fable 5 vs. its thinking variant: JSD 0.034, inside the self-noise band. Fingerprints bind to weights; a relay silently disabling thinking mode is invisible to them and needs latency side-channels.
• Sibling variants are separable. GPT-5.6 sol vs. terra: JSD 0.295, above the noise band p95 = 0.217 (n=11, preliminary).
• Family clustering fails. Claude-internal mean distance 0.393 vs. cross-family 0.481 — same magnitude. Fingerprints track weights, not vendors.
• Language is an independent axis. The same model answers EN and ZH probes with nearly disjoint distributions (JSD 0.605–1.0); in Chinese, most models switch their most frequent number to 42. Multilingual probes add discriminative power.

Raw data is committed under experiments/frontier/batch_*.json; every number and figure regenerates with fixed seeds:

python -X utf8 experiments/analyze_frontier.py

Detection performance (controlled simulation)

Validated against nine adversary types with known ground truth (α = 0.01, auto-calibrated tolerance, budget 600 queries; experiments/run_evaluation.py):

Adversary	Detection	Mean queries to verdict	Est. cost*
Full model swap	100%	15	$0.002
Pinned answers ("always say 73")	100%	14	$0.002
True-random smokescreen	100%	14	$0.002
Filtering known probe phrasings	100%	17	$0.003
Partial distribution mimicry	100%	17	$0.003
Response cache	100%	89	$0.014
Quantized downgrade	100%	116	$0.018
Honest endpoint / benign drift	0 false alarms in 400 audits	—	—

* at flagship pricing ($2.5/M input, $10/M output). The closer the fraud imitates the real distribution, the more evidence is needed; everything above was caught within a $0.02 budget.

Adversarial analysis

Four rounds of red/blue co-evolution in simulation: the relay proposes an evasion, the detector counters, repeat. Economic constraint: the relay only profits by serving the cheap model on real traffic, so "serve the flagship whenever a test is suspected" counts as zero savings. Full ledger: docs/COEVOLUTION.md.

• Content-based routing (flagship for anything probe-like): defeated by business-native and capability-splitting probes. The relay either routes those to the flagship too, or tightens its threshold until its cheap-traffic share drops from 0.52 to 0.23. Evasion conflicts with profit.
• Account-based routing (flagship for unfamiliar accounts): defeats one-shot external audits. Countermeasure: blend probes into a real account's organic traffic and audit passively over time.
• Random dilution (serve the cheap model to a fraction ε of all traffic, blind to content and account): the convergence point, and a statistical limit. Any test holding FPR ≤ α must leave a tolerance band of width ~δ, and a small enough ε hides inside it. In this simulation the attacker's safe window is ε ≈ 0.20–0.28; tightening δ pushes benign-drift FPR to 0.092. Catching dilution rate ε costs ~1/ε² samples; the only lever is larger enrollment and long-run accumulation.

Summary: structured substitution is caught cheaply; the residual risk is low-rate random dilution, a trade of "ε saved" against "~1/ε² samples spent by the auditor". The anytime-valid design is what allows a continuous audit to keep accumulating that evidence.

How it works

1. Probe: semantically trivial questions with categorical one-token answers ("random number 1–100", "random color", coin flip, …), in multiple phrasings and languages to resist string-matching filters.
2. Normalize: canonicalize answers; map unseen answers to an OTHER bucket (Good-Turing missing-mass handling).
3. Compare: Jensen-Shannon divergence between the endpoint's empirical distribution and the reference, aggregated across probe cells.
4. Decide: a sequential betting e-process accumulates evidence query by query. Anytime-valid: stop whenever, early-stop obvious cases, type-I error stays ≤ α. The benign-drift tolerance δ is auto-calibrated per reference via Dirichlet posterior-predictive simulation.

Based on "One Token Is Enough" (Bruckner, arXiv:2607.10252, 2026), which established single-token distribution fingerprints on 165 models and 326k requests. This project adds the sequential e-process decision layer (early stopping + anytime-valid FPR control), adversarial hardening (multilingual paraphrase probes, cache/latency screening), auto-calibration, and the frontier study above.

Related tools

Nearest neighbors first: three tools build on the same underlying paper's bare-API fingerprints. The paper's reference dataset is public (Zenodo, CC-BY-4.0) and browsable at tosea.ai — if all you need is a quick bare-API check against that snapshot, those are fine choices.

Tool	Approach	Verdict type
tosea.ai fingerprint DB + checker	the paper's snapshot as a browsable database (167 models, bare API via OpenRouter, temp 1.0, one collection date); samples your endpoint in-browser	JSD comparison against the snapshot
llm-fingerprint-detector	zero-dependency TypeScript CLI over the same dataset, 11 bundled references	fixed-sample JSD threshold
ChatHub model fingerprint	commercial web tool; compares a trusted reference endpoint against the endpoint under test (8- or 40-cell battery)	fixed-sample JSD comparison

Different approaches:

Tool	Approach	Verdict type
LLMmap	active fingerprinting, trained classifier (52 models; USENIX Security 2025)	closed-set classification + open-set embeddings
api-relay-audit	security scan: injection, SSE integrity, identity keywords	substitution treated as "signals, not proof"
veridrop	protocol conformance + Claude thinking-signature (cryptographic) + usage-field forensics	strong for Claude; protocol-level elsewhere
RelayRadar (AI45Lab)	adaptive discriminative prompts (AB3IT), TVD + permutation p-values	fixed-sample hypothesis test
relay-radar (AetherCore)	passive style monitoring + LLMmap probes	accuracy-style score
zing	capability/knowledge profiles (context window, tokenizer, cutoff)	profile consistency check
KBF (arXiv:2605.29524)	knowledge-boundary numerical recall	fixed-sample binomial test
cocodot llmprobe	six-probe relay check: self-reported identity, judge-scored capability, latency, context, rate limits, consistency	score card; operated by a relay vendor

What fpverify does differently:

1. Anytime-valid sequential decisions. The e-process keeps FPR ≤ α at any stopping point, which enables early stopping (~15 queries for blatant swaps) and continuous low-rate passive auditing — the only regime that survives account-level adaptive routing (see the adversarial analysis). Re-running fixed-sample tests continuously inflates their real error rates.
2. Condition-matched references, including the agent-harness channel. A fingerprint is conditional on (model × channel × protocol); a bare-API snapshot cannot answer "what is my agent IDE actually running", and comparing across conditions misjudges. fpverify pins protocol as a first-class attribute and ships in-harness baselines for 9 frontier models (July 2026, raw data committed, one-command reproduction) — to our knowledge the only public set of its kind.
3. Auto-calibrated benign-drift tolerance (Dirichlet posterior-predictive), instead of a hand-tuned threshold.
4. Verdicts you can walk away from: every reference exports a reproduce pack that re-runs the evidence under the reference's own conditions, without this tool (Case 4 above).
5. A breaking-point analysis that states where detection fails (random dilution ε ≈ 0.20–0.28; catching ε costs ~1/ε² samples), rather than implying the detector is unbeatable.
6. Vendor neutrality: no paid certification, no vendor ads, verdicts computed locally with no telemetry — relay-operated checkers cannot make that claim.

veridrop's Claude thinking-signature check is cryptographic and complementary; when auditing Claude endpoints, run both. Behavioral fingerprints are the layer that works for every model with no server-side cooperation.

Project layout

fpverify/     reusable library: probes, normalization, JSD, e-process, calibration, nearest-neighbor, library identify, reproduce packs, CLI
refs/         community reference fingerprint library (manifest + per-model distributions, contribution protocol)
webui/        local web console (stdlib server; keys never leave your machine)
sim/          red team: model distributions, adversaries, HTTP mock relay, traffic model, blue-team probes
experiments/  evaluation, frontier study, red/blue co-evolution (FPR, power, budget, distance matrices)
tests/        statistical property tests (fairness, FPR bound, power, end-to-end, co-evolution, library/identify, reproduce)
docs/         research notes (problem, threat model, method, experiments, frontier study, multimodal roadmap) + co-evolution ledger

Roadmap

• Protocol-aligned import of the public reference dataset. The underlying paper's 167-model bare-API fingerprints are CC-BY-4.0 on Zenodo. Where probe phrasing and sampling parameters can be aligned with ours, importing them extends api-channel coverage at zero collection cost; imported entries stay marked with their origin and collection conditions, and cross-protocol comparisons still degrade to ranking as usual.
• Per-audit probe paraphrasing. Probe phrasings currently come from a finite, public template bank, so a relay could in principle hard-code rules for known wordings (costs and failure modes are covered in the adversarial analysis: pinned answers are caught in ~14 queries, and semantically perfect mimicry costs about as much as running the real model). The plan is to generate fresh paraphrases locally at audit time from a per-run seed, removing any stable string for a rules file to match.
• Multimodal extension (v2). The decision core (JSD + sequential e-process + calibration) is modality-agnostic: embed an image/video output, quantize it to a codebook, and the same machinery applies. Extending substitution detection to image/video generation APIs — with fixed-seed reproducibility as an extra signal — is the planned v2. Design in docs/RESEARCH_NOTES.md §8; not implemented yet.

Limitations

• A verdict is statistical evidence, not cryptographic proof. FAIL means the distribution deviates significantly from the reference; causes include model substitution, quantization, version rollback, or caching. Keep the JSON report and re-audit before drawing conclusions.
• Fingerprints are highly sensitive to collection conditions (channel, protocol, tier, and any user system prompt are all part of the conditioning). The tool treats protocol as a first-class attribute and degrades cross-protocol comparisons to relative ranking — the flip side is that a reference is only valid under its own conditions, and changing conditions means re-enrolling.
• The frontier fingerprints were sampled inside the Cursor agent harness under a battery protocol (system prompt present, temperature not controlled). They demonstrate non-randomness and separability but are not directly comparable to bare-API numbers; n=11 per model is small and the self-noise band is wide.
• Same-weights mode changes (thinking on/off) are invisible to the fingerprint; detecting them requires latency/length side-channels.
• An adversary that identifies audit traffic at the account level defeats any one-shot certification; the countermeasure is continuous, low-rate, blended auditing.
• No guarantee against an adversary that perfectly reproduces the target model's full conditional distribution — but doing so costs approximately as much as running the real model.

License

MIT