v0.7.0 — P1: evidence (Turkish, moderation, fair baselines, ablations)

Roadmap P1, evidence. Engine behaviour is unchanged. This release adds the benchmarks that show what Kadı does, plus fixes for the bugs found while producing them.

📊 Full report: benchmarks/results/p1/REPORT.md

Setup

• Datasets: five, in two languages. Banking77, CLINC150, MASSIVE tr-TR, OffensEval-2020 TR, Measuring Hate Speech.
• Grid: 10 seeds, α ∈ {2, 5, 10}%, two reflexes (n-gram and embedding k-NN).
• Baselines: a tuned threshold and a learned deferral router, both fitted on the same 500-label prefix.

Validity: each method checked against its own promise

Method	Promise	Runs above α	Cells with mean above α
Kadı LTT	risk ≤ α w.p. ≥ 95%, at all times	0/220 (CP95 ≤ 1.4%)	0/22
Kadı conformal	E[risk] ≤ α	26/220	0/22
Tuned threshold	none	34/220	3/22
Learned deferral	none	57/220	4/22

The uncalibrated baselines are strong on some datasets and fail silently on others. On MASSIVE-tr with the n-gram reflex, learned deferral exceeded α in 10/10 runs (7.4% error at α = 5%).

Turkish (α = 5%)

Dataset	Policy	Local share	Error
MASSIVE-tr	LTT	45.3%	1.86%
OffensEval-TR	LTT	36.4%	2.35%
OffensEval-TR	conformal	51.3%	4.35%

Ablations

• Calibrating on escalations (the v0.1 selection-bias bug) gives 18% error at α = 5% on binary moderation.
• Adaptive free sampling is worth 0.6–11.9 points of local share (median 7).
• Frozen versions cost 5–14 points, and no violation was observed without them. What they buy is provability.
• With a 3%-noisy oracle, Kadı stays within α of the oracle, and its error against gold sits at the noise floor.

Fixed

• Background checkpoints could pack a model that was still changing when the served version aliased the live one. State is now deep-copied under the lock, with a regression test.
• The k-NN reflex rebuilt its matrix on every update (quadratic). It now grows incrementally, with bit-identical results.
• The benchmark's embedding lookup now handles PII-redacted texts.

Honest caveats

• Conformal's guarantee is in expectation, not per run. Earlier wording ("held in every run") did not separate the two policies; THEORY §8 now does.
• Every number uses gold labels or a simulated noisy oracle. The real-LLM-oracle run is ready but not executed, because it needs an API key. See docs/REAL_ORACLE.md.
• Tasks not covered yet: code-review gating and tool selection.

108 tests; CI is green on Python 3.11–3.13, Docker and the TypeScript client.

Kadı v0.6.0: exact, time-uniform guarantees (roadmap P0)

Kadı v0.6.0: exact, time-uniform guarantees (roadmap P0)

This release completes P0 of the five-engineer roadmap. It removes the statistical caveats the v0.5 review found:

• the LTT certificate held only pointwise in time;
• independence between the model and its calibration data held only approximately;
• the drift breaker tested a different quantity from the one α bounds.

The guarantee is now exact and holds at every point in time. That has a measured cost in local share, reported below.

What changed

Statistics

• Frozen model versions.
  • A live model learns from every oracle label, but only frozen snapshots answer.
  • Each version is calibrated only on samples drawn after its freeze, and nothing is re-scored, so the scored model is independent of its calibration data by construction.
  • A certified candidate replaces the served version when it answers at least as often on the same samples.
• Anytime-valid Learn then Test. Each threshold is tested with a betting e-process, and Ville's inequality makes the certificate hold uniformly over time.
• δ spending across versions. δ_v = δ·6/(π²v²), so all certificates ever issued hold simultaneously, at all times, with probability ≥ 1 − δ.
• Drift e-detector. A Shiryaev–Roberts e-detector watches the deployed risk Pr(local ∧ wrong), with a guaranteed expected time to a false alarm (drift_arl). On an alarm the served version is re-certified on fresh data, the court goes on probation and an alert fires.
• Empty-set rule. An empty conformal set answers the argmax. This is free against α: it is already counted as a miscoverage. It also fixed a hidden non-monotonicity, where a looser α could lower the local share.
• Sampling. The sampling rate depends only on the past, never on the input. Escalated cases are sampled for free, local answers within sample_budget, and calibration labels also train the live model afterwards.

Systems

• Off-lock binary checkpoints, measured at CLINC150 scale:

  	v0.5	v0.6
  Lock held during a checkpoint	5.82 s	0.011 s
  Checkpoint size	411 MB (JSON)	43 MB

  Snapshots are written atomically in the background, and a SHA-256 check refuses tampered files.

• Model version in the ledger. Every local decision records its model version inside the HMAC hash chain. Older chains keep verifying.

• Distribution renamed kadi-cascade, because kadi on PyPI belongs to Kadi4Mat. The import name stays kadi.

• The dashboard and MCP stats show the served and candidate versions and the drift monitor.

Results (Banking77 + CLINC150, full datasets, 3 seeds, gold labels as oracle)

bge-small + k-NN, α = 5%	Banking77 local / error	CLINC150 local / error	Oracle calls removed
conformal (bound in expectation)	70.2% / 2.52%	84.7% / 3.40%	65% / 79%
LTT (bound w.p. ≥ 95%, at all times)	66.2% / 1.74%	78.1% / 1.32%	61% / 73%
naive threshold τ = 0.5 (no guarantee)	85.9% / 18.80%	87.2% / 10.41%	—

• The bound held in 126/126 runs, checked per seed.
• The honest cost. Compared with v0.5, strict validity reduces the local share by a median of 9.3 points across 36 configurations (range −1.5 to +19.7). Strict-α LTT loses the most. A first implementation cost about 30 points; ENGINEERING §5.5 shows how it was engineered down. Roadmap P3 targets the rest.
• Conformal vs LTT. Conformal now answers more often than LTT in 13 of 18 configurations. Its guarantee is weaker; choose by the guarantee you need.

Upgrading

• Existing databases upgrade in place, and learned models are kept.
• Certificates are re-issued, because v0.5 calibration samples predate the frozen versions. Expect a fresh warm-up before local answers resume.
• candidate_rate is replaced by sample_budget.

Correction

The v0.5.1 notes said the dashboard had been checked in a browser under the new CSP. That check most likely ran against an older copy of the code: the preview tool used a stale working directory. The nonce behaviour was covered by an HTTP-level test. In v0.6.0 the dashboard was rendered from this repository: the page served a per-response nonce, the scripts ran and no CSP violations were reported.

Verification

• 102 tests. New tests cover:
  • time-uniform LTT validity (simulation);
  • exact certification time;
  • δ-spending sum;
  • e-detector ARL and detection;
  • frozen versions, promotion and restart;
  • tampered snapshots;
  • checkpoint lock time;
  • drift alarm → re-certification → recovery end to end;
  • the model version inside the chain.
• CI is green on Python 3.11–3.13, the semantic reflex, TypeScript and Docker.

pip install "git+https://github.com/zencefilefendi/kadi@v0.6.0"
pip install "kadi-cascade[embed] @ git+https://github.com/zencefilefendi/kadi@v0.6.0"

Full changelog: v0.5.1...v0.6.0 · Theory · Benchmark results

Kadı v0.5.1: security fixes, roadmap and statistical caveats

Kadı v0.5.1: security fixes, roadmap and statistical caveats

This release fixes security issues and updates the documentation. Calibration and decision logic are unchanged, and existing databases upgrade in place.

Security: 7 findings from a second independent review, all fixed

Each finding has a regression test in TestSecurityRegressionsV2.

ID	Issue	Fix
V1	Rows without a hash could be inserted after the chain began and slip past kadi verify	The chain's start point is recorded once; any unchained row after it is reported as tampering (v0.1/v0.2 rows stay valid)
V2	Prompt delimiters could be forged by nesting them (</in</input>put>)	Untrusted text and context are HTML-escaped instead of having their tags stripped
V3	One deeply nested JSON line crashed the MCP server	Returned as a parse error (-32700) on MCP and a 400 on HTTP
V4	Requests with bad tokens could use up the rate-limit quota of legitimate callers	Failed authentications are rate-limited in a separate bucket
V5	With a clock that goes backwards, pruning could cut the hash chain in the middle	Pruning deletes a contiguous range of the oldest rows (by id)
V6	The dashboard CSP allowed 'unsafe-inline' scripts	A fresh script nonce per response; API responses allow no scripts at all
V7	Text sent by the caller was copied verbatim into errors that go back to LLM agents	Truncated and reduced to a single line

Five-engineer roadmap

ROADMAP.md · ROADMAP.tr.md

Five independent reviews (statistics, ML, systems, security, research/ecosystem) were merged into one plan:

• P0, validity first:
  • anytime-valid certification;
  • certifying frozen model snapshots;
  • taking checkpoints off the hot path (measured: a 365 MB JSON checkpoint holds a lock for 6 s at 150 labels);
  • a new PyPI name, because kadi is taken.
• P1, evidence: a real LLM oracle with cost accounting and an oracle self-consistency (ε) measurement, strong baselines and ablations, and a Turkish benchmark.
• P2, scale: a numpy fast path, an async oracle pipeline with prompt caching and the Batch API, OpenTelemetry and a guarantee SLO, Streamable HTTP for MCP, and Postgres.
• P3, better reflex: a learned selection score, cold-start seeding, prototypes, a multi-tier cascade, clustered conformal and active audits.
• P4, hardening: a Merkle log with externally anchored heads, signed releases with SLSA provenance and an SBOM, fuzzing, OIDC roles, poisoning resistance, and KVKK/GDPR erasure through crypto-shredding.
• P5, research and adoption: a paper plan, framework integrations and 90-day targets.
• Moonshot: proof-carrying decisions.

Honest statistics: THEORY §7c "Known statistical caveats"

The review found subtleties the earlier docs glossed over. They are now documented, and fixing them is roadmap P0:

• the LTT certificate holds pointwise in time, not uniformly over the stream;
• independence between the model and the calibration data is approximate;
• the drift breaker tests Pr(wrong | local), not the quantity that α bounds;
• the finite-sample ACI bound on a thinned stream is loose;
• sampling rates are mixed in the calibration window;
• memory answers are neither calibrated nor audited.

The empirical bound still held in 126/126 benchmark runs.

Verification

• 95 tests (8 new), passing with both the stdlib-only interpreter and the [embed] environment.
• The dashboard was checked in a browser under the new CSP, with no violations.
• CI is green on Python 3.11–3.13, the semantic reflex, the TypeScript client and Docker.

pip install "git+https://github.com/zencefilefendi/kadi@v0.5.1"

Full changelog: v0.5.0...v0.5.1

Kadı v0.5.0: MCP server with agent-as-oracle

Kadı v0.5.0: MCP server with agent-as-oracle

Kadı can now be used as a tool by agents in Claude Code, Claude Desktop or any MCP client, with no API key. The calling agent is already a frontier model, so it becomes Kadı's oracle. Kadı learns from the agent's verdicts and, once it can certify the error target, stops asking about routine cases.

pip install "git+https://github.com/zencefilefendi/kadi@v0.5.0"
claude mcp add kadi -- kadi mcp

How it works

agent ── kadi_decide(question, text) ──► Kadı
                                          ├─ confident & calibrated ──► {"status": "decided", "label": ...}
                                          └─ otherwise ──────────────► {"status": "needs_verdict", "ticket": ...}
agent judges the case itself ── kadi_resolve(ticket, label) ──► Kadı learns + calibrates

• The agent's verdict is the oracle label. It trains the local reflex. A random share is held out for calibration and is never trained on.
• The guarantee is relative to the agent's own judgement. Pr(Kadı ≠ agent) ≤ target_error, and with the default ltt policy it holds with probability ≥ 95%.
• Audits are blind. Audit tickets list all options and don't reveal Kadı's answer, so the measured agreement isn't anchored.
• In tests: across 1,200 simulated cases, Kadı answered more than 60% on its own in the second half, with error ≤ 5%.

Added

• kadi mcp: a stdio MCP server with zero dependencies, no SDK required.

• Dual-era protocol support:

  • legacy clients use initialize, with versions 2024-11-05, 2025-03-26, 2025-06-18 and 2025-11-25;
  • modern clients (2026-07-28) use server/discover and send io.modelcontextprotocol/protocolVersion and clientCapabilities in each request's _meta;
  • an unsupported version returns -32022, and missing required metadata returns -32602.

• Eight tools:

  Tool	Purpose
  kadi_define_question	register a decision type (persists across restarts)
  kadi_list_questions	questions, local share, measured agreement
  kadi_decide	decide a case, or get a ticket
  kadi_resolve	answer a ticket with your verdict
  kadi_teach	add a known-correct label (never calibrates)
  kadi_stats	local share, audits, calibration, probation, alerts
  kadi_recent_decisions	history with filters and pagination
  kadi_verify_ledger	verify the tamper-evident hash chain

  Every tool has annotations (readOnlyHint and related hints). Results come back as structuredContent, and errors are actionable tool errors.

• Engine: a two-phase API, Kadi.decide_or_defer() / Kadi.resolve(), plus the Pending type. decide() now runs on the same route → complete path, with no behaviour change.

• Docs: docs/MCP.md covers setup for Claude Code and Desktop, the tools, an example session and protocol details. The threat model gains T17 (untrusted previews re-entering the agent's context) and T18 (an agent poisoning Kadı with wrong verdicts).

Verification

• 87 tests, 10 of them new. They cover protocol negotiation for both eras, JSON-RPC errors, the learning loop, persistence and conflicts, pagination, and stdio end-to-end through a real subprocess.
• CI is green on Python 3.11–3.13, the semantic reflex, the TypeScript client and Docker.

Known limitations

• Warm-up cost. At α = 5%, Kadı needs about 240 resolved tickets before it answers alone, because LTT needs at least 59 held-out calibration samples. A lower α needs more. This is the price of a certified bound.
• Not yet tried with a real client. The server hasn't been exercised against Claude Code or Desktop on a real machine. Compatibility was verified against the specification and with stdio end-to-end tests.
• Transport. Only stdio for now; Streamable HTTP is on the roadmap.

Full changelog: v0.4.1...v0.5.0 · README · Türkçe

Kadı v0.4.1: documentation and API metadata patch

Kadı v0.4.1: documentation and API metadata patch

A patch release on top of v0.4.0. It does not change calibration, decisions or stored data. Upgrading is safe, and existing kadi.db files work unchanged.

Changed

• GET /api/questions now returns policy ("conformal" or "ltt"), delta and instructions for each question. The TypeScript client's QuestionInfo type has the same new fields.

Documentation

• Status and known limitations (README and README.tr): a table of what has been validated and what hasn't.

  • Validated: the calibration guarantee (126/126 benchmark runs), the reflex models, the service and the security controls.
  • Not yet validated: a real LLM oracle in benchmarks, Turkish data, multi-instance scale-out, production traffic.

• The synthetic demo was re-measured with the current engine and its tables updated in README, README.tr and THEORY:

  	v0.2 measurement	v0.4 measurement
  route_ticket (marginal): local / audited agreement	93.4% / 97.3%	94.3% / 99.1%
  is_urgent (per-label): local / audited agreement	42.6% / 95.3%	62.6% / 97.8%

• Refreshed roadmap. It now opens with a real-oracle (Claude) benchmark, a Turkish intent benchmark and an MCP server.

• kadi[embed] install line added to the quick start.

• CONTRIBUTING: setup for developing the semantic reflex, plus a benchmark rule. Hyper-parameters are tuned only on orderings with seeds ≥ 100; evaluation uses seeds 0–2.

• SECURITY: supported versions are now 0.4.x.

Install

pip install "git+https://github.com/zencefilefendi/kadi@v0.4.1"
pip install "kadi[embed] @ git+https://github.com/zencefilefendi/kadi@v0.4.1"   # + semantic reflex

Full changelog: v0.4.0...v0.4.1

Kadı v0.4.0: semantic reflex (embeddings + k-NN)

Kadı v0.4.0: semantic reflex

Kadı is a calibrated judge that sits in front of your frontier LLM. It answers routine, schema-shaped decisions locally, with a statistical bound on how often they disagree with the oracle, and escalates everything else. This release adds an optional semantic reflex: sentence embeddings plus a k-nearest-neighbour vote over oracle-labelled examples. It lets Kadı answer far more decisions locally, with the same guarantee.

Highlights

LTT, α = 5%	Banking77 (77 intents)	CLINC150 (150 intents)
v0.3 reflex (hashed n-grams), share answered locally	63.9%	79.5%
v0.4 reflex (bge-small + k-NN), share answered locally	79.6%	89.8%
Oracle calls saved compared with v0.3	−36%	−40%
Realised error (target ≤ 5%)	2.14%	3.18%

• The bound held in 126 of 126 benchmark runs, checked per seed: 2 datasets × 3 reflexes × 2 policies × 3 α × 3 seeds, plus Mondrian.
• The core stays zero-dependency. Embeddings live in the optional kadi[embed] extra (fastembed / ONNX, no PyTorch).
• k-NN settings were tuned on separate orderings (seeds 100–102), never on the evaluation runs (seeds 0–2).

Added

• kadi.embed, the optional semantic reflex:
  • FastEmbedder: multilingual MiniLM by default, covering 50+ languages including Turkish.
  • HybridFeaturizer: hashed n-grams plus a dense embedding block.
  • KNNHybridReflex: a k-NN vote mixed with the online logistic model.
• Pluggable Kadi(featurizer=..., reflex=...), or reflex = "embed" in kadi.toml.
• Benchmark: --reflex bge|minilm --knn, with embeddings cached once per dataset.
• CI job that tests the semantic reflex with numpy and a toy embedder, without downloading a model.

Changed

• The novelty (out-of-distribution) guard measures unseen mass only over the sparse n-gram block, so dense features can't hide unfamiliar inputs.

Install

pip install "git+https://github.com/zencefilefendi/kadi@v0.4.0"                     # core, zero dependencies
pip install "kadi[embed] @ git+https://github.com/zencefilefendi/kadi@v0.4.0"       # + semantic reflex

from kadi import Kadi, Choice, AnthropicOracle
from kadi.embed import FastEmbedder, HybridFeaturizer, KNNHybridReflex

kadi = Kadi(AnthropicOracle(), featurizer=HybridFeaturizer(FastEmbedder()), reflex=KNNHybridReflex())
intent = Choice("intent", INTENTS, target_error=0.05, policy="ltt")   # risk ≤ 5% with 95% confidence
d = kadi.decide(intent, text)

Also in the 0.x line (since v0.2)

• v0.3: the Learn then Test policy (a high-probability guarantee), a tamper-evident HMAC-chained ledger (kadi verify, kadi anchor), and the public Banking77 / CLINC150 benchmark with SHA-256-pinned data.
• v0.2: shadow mode, kadi replay, Mondrian (per-label) conformal, the drift breaker, KVKK/GDPR-oriented redaction, the hardened HTTP service (11 issues from an adversarial review fixed, each with a test), the TypeScript client and Docker.

Known limitations

• Benchmarks used gold labels as the oracle. The real-LLM oracles are implemented, but they haven't been benchmarked yet (benchmarks/run.py --oracle anthropic).
• No Turkish benchmark yet, although the default multilingual model supports Turkish.
• Single process with SQLite. Multi-instance deployments need the Postgres backend on the roadmap.

Documentation commits made after this tag (status and limitations section, re-measured synthetic demo, policy/delta in GET /api/questions) are on main.

Docs: README · Türkçe · Theory · Threat model · Benchmark results · Changelog

Kadı v0.3.0: Learn then Test, tamper-evident ledger, public benchmark

Kadı v0.3.0: Learn then Test, tamper-evident ledger, public benchmark

This release gives Kadı a stronger guarantee, an auditable decision history and the first measurements on public data.

Learn then Test policy (policy = "ltt")

The conformal policy bounds disagreement with the oracle in expectation. LTT (Angelopoulos, Bates, Candès, Jordan & Lei, 2021) makes a PAC-style statement instead: with probability ≥ 1 − δ over the calibration sample, the decision risk is ≤ α.

• Decision rule: answer locally when the top-label probability is at least λ̂. The threshold λ̂ is certified with exact binomial p-values and fixed-sequence testing on a grid fixed in advance.
• Validity: checked by simulation (tests/test_core.py::test_ltt_certificate_is_valid). The minimum sample size needed for a certificate is exposed.
• Usage: kadi replay --policy ltt gives an α sweep on your own history. The dashboard shows λ and a policy badge.

Tamper-evident ledger

• Chaining: every decision row carries prev_hash and row_hash = HMAC(K, prev_hash ‖ canonical row), with K derived from KADI_SECRET.
• kadi verify re-derives the chain and names the first edited, deleted or inserted row.
• kadi anchor exports the chain head for safekeeping outside the database, which also protects against someone who holds the key.
• Pruning stays verifiable through a recorded anchor.

Public benchmark (benchmarks/)

• Datasets: Banking77 (13,083 queries, 77 intents) and CLINC150 (22,500 queries, 150 intents). Files are downloaded on demand and pinned by SHA-256.
• Setup: full datasets, 3 seeds, gold labels as the oracle, compared against confidence-threshold baselines.

Share answered locally / error	Banking77	CLINC150
naive threshold τ = 0.5	85.9% / 18.80% ❌	87.2% / 10.41% ❌
Kadı conformal, α = 5%	54.7% / 1.28% ✅	74.7% / 2.16% ✅
Kadı LTT, α = 5%	63.9% / 2.47% ✅	79.5% / 3.19% ✅
Kadı LTT, α = 10%	74.4% / 6.09% ✅	85.4% / 6.94% ✅

• The bound held in all 54 runs (2 datasets × 3 policies × 3 α × 3 seeds), checked per seed.
• LTT kept 3.5–9 points more decisions local than conformal at every α.
• Per-label (Mondrian) guarantees rarely certify with 77–150 labels, as the theory predicts. Use them for questions with few labels.

Performance

• The reflex learner's inner loops are about 3–5× faster.
• Calibration scores are kept incrementally, and full re-scoring is time-budgeted to about 10% of wall time.

Install

pip install "git+https://github.com/zencefilefendi/kadi@v0.3.0"

A newer release exists: v0.4.1. It adds the semantic reflex (embeddings + k-NN), which raises the local share at α = 5% to 79.6% on Banking77 and 89.8% on CLINC150.

Full changelog: v0.2.0 → v0.3.0 · Theory · Results at this tag