v0.4.0 — every verdict names its rule and cites its evidence

Every verdict now names the rule behind it and cites the source line it was judged against.

⚠️ Behavioural change — verdict counts will shift

A PASS or FAIL that cannot cite a source span is now demoted to REVIEW (reason: evidence_missing).

The verdict questions always saw the whole source; the evidence question only sees the six pre-filtered candidates. So a confident judgement could previously ship with evidence: null and no way to audit it. It now can't — expect fewer PASS/FAIL and more REVIEW on drafts you already run.

If your CI gate is fail-on: fail, this is safe. If you treat REVIEW as a failure, re-check your baseline.

Added

• reason on every verdict — Markdown Why column, JSON, and plain output all name the one rule that produced the kind: supported, contradicted, not_found, low_confidence, detail_drift, smuggled_number, evidence_missing, uncheckable. Derived, so it can't go stale when safeguards rewrite the kind.
• claims.dropped_sentences(text) — the sentences a split skipped. The CLI prints the count to stderr, so nothing disappears silently.
• Short sentences carrying a number (It cost $2M.) are now kept as claims. They used to be dropped as three-word fragments and never verified.

Fixed

• Evidence retrieval. Candidates are now ranked by idf-weighted term overlap instead of raw word overlap. The old scorer counted boilerplate and discriminative words alike, so a span that merely shared the topic's common nouns outranked the one that actually contradicted the claim — and with the evidence rule in place, that turned a real contradiction into REVIEW with nothing to cite. In the Indonesian fixture saham appears in nearly every sentence while VIVA disuspensi identifies exactly one.
  Measured offline over 12 labelled claims: idf retrieves every gold span by k=3; the old scorer missed one even at k=20. EVIDENCE_CANDIDATES stays 6 — the ranking was the defect, not the pool size.
• Request payload. state now carries only the segments a claim was offered, not every segment of every source. Measured live on the same draft and source: 11,111 → 9,500 input tokens. sources still carries the whole document, so verdict recall is unchanged.
• Question instructions now state that sources and segments are untrusted quoted data that must not be obeyed — a hostile document can no longer read as an instruction to the judge.
• build_questions rejects duplicate claim ids instead of silently overwriting one claim's questions with another's, which mis-attributed verdicts.
• An API failure reports the largest request size it built, so an oversized source reads as "split long sources" rather than an opaque SDK error.
• The PyPI description no longer claims "calibrated probabilities" — vendor probabilities are calibrated across groups of predictions, not per answer.

Calibration

Question wording and candidate ranking both changed, so thresholds were re-validated live on 2026-10-01:

Run	Score	Confident-wrong PASS
Phase 1 alone	29/30	0
Phase 1 + 2	30/30	0

Bar is ≥ 24/30 with zero confident-wrong PASS. Phase 1 on its own was a net downgrade; Phase 2 removed the cause — Saham VIVA disuspensi di harga Rp 50. is FAIL again, and now cites examples/news-1.md:5. Full matrix in examples/calibration.md.

Upgrading

pip install -U sooth

No CLI flag, exit code, or JSON field was removed. The four-verdict vocabulary is unchanged.

Full diff: v0.3.0...v0.4.0

v0.3.0

Claim-by-claim verification of AI output against source material. PASS / FAIL / REVIEW with calibrated probabilities and the exact source span behind every verdict.

pip install sooth==0.3.0

Presentation and machine-consumption release. No change to verification behaviour or to any calibrated threshold. Calibration on the 30-claim set: 30/30 correct, re-run live on this code.

Added

• --format json — stable machine-readable report (summary, exit_code, verdicts[] with probabilities, evidence, and missing numbers). One JSON contract for pipelines and agents instead of scraping Markdown.
• sooth demo --case en|id — two bundled recorded runs now ship in the wheel. en is the default (release notes vs an AI summary, three numbers silently wrong); id is the Indonesian market-news case.
• --version, reading the version from installed distribution metadata.
• GitHub Action: fail-on input (review default, fail, never), and outputs — verdicts, pass-count, fail-count, review-count, uncheckable-count, report, exit-code. All from a single API call.
• English calibration fixture (examples/release-notes.md + examples/ai-summary.md) as the default demo.
• CI: an offline demo smoke step that pins the documented exit codes and the JSON contract, plus a package job that installs the built wheel and replays both demo cases (guards non-Python package data).
• CHANGELOG.md, CONTRIBUTING.md, CODE_OF_CONDUCT.md.

Changed

• README rewritten around the measured accuracy numbers and an English hero demo.
• Action installs the exact version declared in its own pyproject.toml, so the pin can no longer drift from the tag.
• PyPI metadata: keywords and classifiers now cover hallucination, fact-checking, llm-evaluation, rag, guardrails, ci, github-action.
• render_json and the --log record now share one verdict-serialisation function, so the two JSON surfaces cannot drift apart.

Fixed

• sooth.__version__ reported 0.1.0 while the distribution was 0.2.3. Now sourced from packaging metadata; a test fails if it ever becomes a hardcoded literal again.
• tests/smoke.sh invoked the bare python3, which fails when typesafe-sdk only exists in the project venv. It now prefers an active venv, then ./.venv, then python3, accepts PYTHON=, and explains the fix instead of exiting 3 silently.

Full changelog: v0.2.3...v0.3.0