Repository navigation
Releases
Releases · hemanth1999k/mistrial
mistrial 0.2.2
[0.2.2] - 2026-10-26
Fixed
mistrial report --html nonewrote the HTML to a file literally namednone. It now skips the HTML likerunanddemodo.- Two runs in the same second overwrote each other's saved JSON under
.mistrial/runs/. Same-second runs now get-2,-3, ... suffixes. - Pointing
migrate,gate-risk,escalate,report,compareorsentinel baselineat the wrong kind of file (a rubric-check result, a missing path) crashed with a traceback. They now say what the file is and exit 2.sentinel checkwithout a baseline does the same. mistrial --versionwas hardcoded and had fallen behindpyproject.toml(it said 0.2.0 on 0.2.1). It now reads the installed package metadata, and a test keeps the two in step.
Added
examples/hard.jsonl: 64 cases that pit precision against reasonableness (rounding, units, hedges, extra facts, equivalent code, multi-step answers with a wrong step). Built because gpt-5.5 and the gpt-6 judges never flip on the tricky set; on this one every one of them does. README has the six-judge run.
mistrial 0.2.1
Changed
- The source distribution contains only the package, tests, examples and the top-level docs. The wheel is unchanged.
[0.2.0] - 2026-10-26
First public release. Ten additions over 0.1.0, each answering a documented way LLM judges go wrong in production.
Added
- Rubric sensitivity (
mistrial rubric-check): run the judge under three rewordings of the rubric that mean the same thing, report the share of verdicts that change and Krippendorff's alpha across wordings. Built-in rewordings forcorrectnessandquality; bring your own with--variant. - Rubric reversal (same command): ask with the criterion flipped and check that verdicts invert. Reports the cases where the judge gave the same answer to "is it correct?" and "is it incorrect?".
- Gate false-alarm rate (
--pass-markonrun/report, andmistrial gate-risk): how often your CI gate would go red with nothing changed, by re-sampling each case's observed verdicts; plus a safe mark and the runs per case that bring the rate under 5%. - Judge migration (
mistrial migrate): old judge vs new judge on the same cases, with your pass mark translated to the new judge and a per-case cutoff mapped by quantile. - Sentinel (
mistrial sentinel baseline/check): freeze a run on anchor cases, re-judge fresh on a schedule, exit 2 on JUDGE_DRIFT. Design credit: judge-drift-sentinel. - Parser audit: every LLM verdict keeps the raw reply and how it was read (strict JSON, fenced, embedded in prose, no verdict). The card shows the counts and examples.
- Adaptive re-ask (
--adaptive): two runs everywhere, extra runs only where the two disagreed. The card reports the calls skipped. - Escalation (
mistrial escalate): a CSV of the cases a human should look at, with a column for their verdict;--langsmith-datasetpushes them to LangSmith. - Stabilized evaluator (
--stabilize Kandmistrial.stabilize()): the judge wrapped in a K-sample majority vote with the same evaluator signature, measured next to the raw one. - GitHub Action (
action.yml): runs the card in CI, posts it as a PR comment, uploads the HTML report, fails the job on a gate.--markdownwrites the same card anywhere. examples/tricky.jsonl: 32 labeled cases built to be hard to judge (hedges, rounding, unit conversions, extra details, equivalent code). The bundledexamples/cases.jsonlstays as the easy set.mistrial demosays up front that its three judges are fake and flaky on purpose.docs/img/compare.png, a feature comparison against judgecheck, judgebench, Assay, judge-drift-sentinel and LangSmith/promptfoo, and a "Standing on shoulders" README section with the papers and repos this builds on.- Judges for every OpenAI-compatible provider:
gemini:,groq:,mistral:,deepseek:,xai:,together:,openrouter:,ollama:(local, no key) andcustom:withMISTRIAL_BASE_URL.<PROVIDER>_BASE_URLoverrides any base URL.mistrial providerslists them. - An 88 second animated explainer video (
docs/explainer.mp4, not committed) rendered fromdocs/explainer/explainer.html: the problem, why it slips through, the command, the real numbers, the fix. Narration script and re-render steps indocs/explainer/SCRIPT.md.
Changed
- The project is called mistrial. Package, command, env vars (
MISTRIAL_BASE_URL,MISTRIAL_API_KEY) and the.mistrial/directory all follow. It was briefly called judgeflake before any release. - README, demo GIF and report screenshot now show real, current OpenAI judges (
gpt-5.4-nano,gpt-5.4-mini,gpt-5.5) instead of the fakedemojudges. The numbers in the README come from 480 live calls made on 2026-10-25. - README opens with the problem, why nothing catches it today, and what mistrial does, with three diagrams in
docs/img/. - Saved runs are format version 2 (adds
adaptive,calls_saved, andtext/parseper verdict). Version 1 files still load.
pip install mistrial
pipx run mistrial demo # no key needed