juried 0.4.1

Fixed

• --threshold, --juried-threshold and --tolerance explain that they were removed in 0.4
  and name their replacement, instead of falling through to pytest or argparse as unknown
  flags. They are deleted for good in 0.5.

juried 0.4.0

Breaking changes

• threshold is gone, as 0.3 announced. A juried.toml or a scenario file that still sets
  it, or a JURIED_RUN_THRESHOLD override, stops the run with an error that names the file
  and says to set misses instead, where 0.3 derived misses from it and printed a notice.
  juried run --threshold is no longer a juried flag, so it falls through to pytest, which
  rejects it, and --juried-threshold is gone from the plugin. The JSON report and the JUnit
  user_properties still carry threshold, the interval lower bound the gate is equivalent
  to, as docs/stability.md promises, so schema_version stays at 1.
• [run] gate_on defaults to "auto", a new third value. The gate runs on the
  judge-corrected interval's lower bound whenever reports/juried-calibration.json is
  usable, that is, it is for the configured judge (same provider and model, and the same
  temperature and prompt version where the report records them) and the judge is not too
  weak to correct on the whole set; otherwise it runs on the observed passes, the header
  says gate on observed passes (auto: ...) with the reason, and the summary repeats it as
  a warning naming the command that fixes it. A 0.3 project that already has a calibration
  report for its judge is therefore gated on the corrected rate from this release on, with
  early stopping off because the corrected interval needs every attempt: the run makes
  every planned attempt where 0.3 stopped a scenario once its gate was decided, so it costs
  more, and a scenario that passed on its observed count may now fail, since the corrected
  lower bound has to meet the rate misses implies. gate_on = "observed" restores the
  0.3 behaviour exactly: the gate on the observed passes, early stopping as configured, and
  the corrected rate reported beside it.
• gate_on = "corrected" is strict throughout. A report whose judge is too weak to correct
  on the whole set is refused at startup, as a missing report or one for another judge
  already was, where 0.3 fell back to the observed passes scenario by scenario. A criterion
  whose own cases are enough to use but too weak still falls back for that scenario alone,
  and the gate failure block says so.
• juried compare --tolerance, the deprecated alias for --min-effect, is gone.
• The end of run warning about a missing calibration report is replaced by the gate
  warning above, printed once and for every judge, the stub included; a run with
  gate_on = "observed" and no usable report says no corrected rate once, with the
  reason, instead of two lines.

Added

• The JSON report's defaults carry gate_on_configured, the value in the file, beside
  gate_on, which is now what the run resolved it to, observed or corrected; early_stop
  there follows the resolved value. juried run --gate-on and JURIED_RUN_GATE_ON take
  auto, observed or corrected. juried run --dry-run and juried estimate print a
  gate on line and plan early stopping against the resolved value, so a plan whose gate
  resolves to the corrected rate is the full sample.
• A "Running in CI" section in the README with a complete GitHub Actions job: it installs
  juried, runs every scenario with --no-early-stop against a staging URL held in a
  secret, keeps reports/ as an artifact, fetches juried-report.json from the last
  successful run on the default branch and gates on juried compare against it. The same
  file is at examples/faq-bot/.github-workflow-example.yml, and a test checks that its
  steps only call subcommands juried --help lists.
• A test that fails once the judge price table in pricing.py is more than four months
  old, naming the date and asking for the table and PRICES_DATED to be refreshed.
• Provider plugins. A provider name that is not anthropic, openai or stub is looked
  up in the juried.providers entry point group, for the judge and for generation alike,
  and the class found is instantiated with the arguments the built in providers get
  (model, temperature, max_tokens, base_url, api_key_env). It must subclass
  juried.judge.Provider; a class that does not, or that leaves fingerprint, judge or
  generate undefined, is a config error naming the class and the missing methods, and a
  built in name always wins over a plugin. judge.provider and generate.provider therefore
  accept any non empty string, and an unknown name is reported when the provider is built
  rather than when the config is read. docs/judge.md shows the pyproject.toml a plugin
  needs and docs/stability.md lists Provider and its required methods.

Changed

• The README, docs/runs.md, docs/judge.md, docs/calibration.md, docs/configuration.md,
  .env.example, the example project's juried.toml and the config juried init writes
  describe the three gate_on values and the gate that follows the calibration report.
• docs/releasing.md pulls main and checks the version before tagging, since a tag on the
  commit before the version bump rebuilds the previous release without any error.
• The README's version badge reads the latest GitHub release rather than PyPI, since
  shields.io caches the PyPI figure for twelve hours and the release one for five minutes.

juried 0.3.1

Fixed

• The example line under "Each scenario then reports" in the README and docs/judge.md
  gave a corrected rate of 84% for 18/20 with a 6% false pass and 3% false fail rate; the
  estimator gives 92%, and the interval now matches too.
• The sdist no longer carries the tests or the example project, which had taken it over
  the 200 KB that pyproject.toml promises. The wheel is unchanged.

juried 0.3.0

Breaking changes

• The gate is a count of tolerated misses, not a threshold on the interval. [run] misses
  (default 1) and a per scenario misses say how many attempts may fail, and a scenario
  passes when passes >= runs - misses. The Wilson interval is still computed and shown but
  no longer decides. With the default 20 runs the requirement is unchanged at 19 of 20; a
  scenario that set runs without a threshold now tolerates one miss at any count rather
  than the count the threshold table gave it. threshold is deprecated and is removed in
  0.4: set without misses, it derives misses (the largest count whose lower bound still
  met it) and the run header prints a notice naming the value to set; set alongside a
  disagreeing misses, or at a value no run count could meet, it is a config error rather
  than a warning. A misses at or above runs is also an error, since such a gate could
  never fail.
• Terminal output no longer mentions a threshold. The header prints
  gate needs 19/20 passes (1 miss tolerated), the pytest summary shows
  19/20 (needs 19, interval 0.75 to 0.99), and a gate failure states the gate, the misses
  on the judged attempts and the interval.
• The JSON report's scenario entries and defaults carry misses. required_passes is
  always an integer, and threshold is always populated: the configured value while the
  deprecated key is set, otherwise the interval lower bound the gate is equivalent to. No
  field changed meaning, so schema_version stays at 1.
• The HTML report's scenario table shows "Gate needs" and one "Interval" column in place of
  the threshold, lower bound and upper bound columns.
• juried compare tests a drop instead of subtracting it. For each scenario in both
  reports it runs Fisher's exact test, one sided for a decrease, on the passes and fails,
  and a drop is a regression only when its p-value is below --alpha (default 0.05) and
  the pass rate fell by at least --min-effect (default 0.10). Smaller or less certain
  drops are listed under "drops within noise" with the p-value and the Newcombe 95%
  interval for the difference, and do not fail the step; a drop in the interval's lower
  bound alone is no longer a regression, and a rise is "improved" only on the same test
  turned round. --tolerance is a deprecated alias for --min-effect, removed in 0.4.
  The comparison JSON gains schema_version (1), alpha, min_effect,
  detectable_drop and within_noise at the top level and p_value, diff,
  diff_interval and significant per change; tolerance is gone from it.
• A quoted phrase in expected is enforced before the judge, for every provider: a
  response missing one fails the attempt with the reason and no judge call is made. The
  match now ignores case, runs of whitespace, hyphens and punctuation that ends a word, so
  "14-days," satisfies "14 days"; [judge] strict_quotes = true restores the word for
  word match. The stub judge uses the same rule. Attempts that a check decided carry a
  verdict with model checks, are not written to .juried/verdicts.jsonl, and count as
  failures in the gate as any failed attempt does.
• Provider.generate takes an adversarial flag, so a provider implementation written
  against 0.2 must accept it.

Added

• Every scenario's pass rate is corrected for the judge's own error rate. When
  reports/juried-calibration.json is for the configured judge (same provider and model,
  and the same temperature and prompt version where the report records them), juried takes
  the judge's sensitivity and specificity from it, per criterion when the criterion has at
  least [judge] min_calibration_cases (default 10) labelled cases and from the whole set
  otherwise, and applies the Rogan-Gladen estimator with a bootstrap interval of 2,000
  iterations from a fixed seed, each resampling the calibration cases and drawing the
  observed rate from the Jeffreys posterior Beta(passes + ½, fails + ½), so an observed
  10/10 or 0/10 gets an interval of real width rather than "100% to 100%". The
  pytest summary, the gate failure block, JUnit user_properties, the JSON report
  (corrected_rate, corrected_interval, judge_sensitivity, judge_specificity,
  calibration_cases_used, calibration_scope, bootstrap_seed, corrected_refused, and
  a top level calibration entry naming the report used) and the HTML report (a
  "Corrected" column and a note naming the report) all carry it. A judge whose sensitivity
  plus specificity minus one is below 0.5 is "too weak to correct" and the report says so
  with the calibration accuracy. A report for another judge is warned about, and a run
  with no report says there is no corrected rate and how to get one.
• [run] gate_on = "corrected" gates on the corrected interval's lower bound against the
  rate misses implies, and the header says gate on judge-corrected rate. The default
  stays "observed" in 0.3 so nothing changes silently; the release after 0.3 flips it.
• Sequential stopping. A scenario ends as soon as its gate is decided: lost once the
  failed attempts exceed misses, won once the passes reach runs - misses. [run] early_stop (default true), JURIED_RUN_EARLY_STOP and juried run --no-early-stop
  control it; gate_on = "corrected" forces full sampling. Every scenario records
  attempts_planned, attempts_made and early_stopped in the JSON report, the JUnit
  user_properties and the HTML report. A stopped scenario's rate is a bound, not an
  estimate; the gate verdict is unaffected.
• juried compare refuses a scenario that stopped early in either report, with a message
  naming it, unless --allow-early-stopped is passed, and prints a note when a scenario's
  attempts made differ by more than a factor of two between the reports. The comparison
  JSON gains early_stopped and notes.
• juried run --dry-run and juried estimate add an early stop line: the expected
  attempts, requests, calls and cost under early stopping, from an exact dynamic programme
  over the (passes, fails) states using each scenario's pass rate in the last report;
  scenarios without one are planned in full. --no-early-stop plans the full sample.
• The label queue marks responses from a scenario that stopped early, in the terminal and
  on the HTML page, since the sampled share of such a scenario leans towards the verdict
  that decided it.
• The calibration report's judge entry records temperature and prompt_version, so a
  report can be matched to the judge it measured.
• juried label grows the calibration set from real output. After every run the responses
  a human should look at are written to .juried/label-queue.jsonl, deduplicated by
  content and tagged with why: split (the votes disagreed), boundary (the scenario
  finished within one miss of its gate), check_disagreed (the checks passed but the judge
  failed the response), low_confidence (the judge's reason hedged, by a fixed word list),
  or sampled (a random share, [label] sample_rate, default 2%, of the rest); the queue
  keeps earlier unlabelled entries and is capped at [label] queue_size (default 50),
  newest first. juried label shows each case in the terminal with the judge's verdict
  last and takes p, f, s, n or q, appending labels to
  calibration/from-runs/<criterion>.yaml with a source naming the run; --html writes
  reports/label-queue.html (no scripts) and label-queue.csv, and --import reads
  marks back from either. Calibration cases gain an optional source field, juried calibrate reports how many cases came from hand labelling and how many from runs, and
  every run's summary says how many responses are waiting and the calibration set size
  per criterion.
• juried run --misses N and JURIED_RUN_MISSES, overriding [run] misses. A command
  line --misses or --threshold replaces the file's gate outright, so it cannot disagree
  with it.
• JUnit user_properties misses_tolerated and passes_needed; threshold is still
  written, derived as above, so existing dashboards keep working.
• juried compare reads the passes a gate needed from misses, required_passes or
  threshold, whichever the report carries, and writes it as passes_needed in --json.
• The target side of a run is counted and priced. Every request to the endpoint is recorded
  per attempt with its HTTP status, bytes and latency, and summed per scenario and per run.
  New [target] keys: usage_input_path and usage_output_path, dotted paths to token
  counts in the reply; input_price and output_price in US dollars per million tokens;
  and cost_per_request, a flat price per call for a target that reports no tokens. The
  summary always prints a target usage: line, with tokens and an estimate, a per request
  estimate, or the request count and "cost unknown" with the keys to set, and an
  estimated run cost: line whenever both sides have a figure.
• juried run --dry-run, also juried estimate: collects every scenario, counts the
  requests each side would get allowing for turns and votes, prices them from the
  configured target prices and the judge price table, prints the plan and exits 0 without
  sending anything. It assumes 400 input and 150 output tokens per call and says so, or
  uses the averages from the last reports/juried-report.json when one exists.
• The JSON report's summary.usage and each scenario's usage gain judge, target and
  total_estimate_usd (null when either side is unknown); the existing flat keys are the
  judge's figures and are kept. Attempts carry target_usage and a requests list, and
  the report has a top level target entry with the URL template and prices. The HTML
  spend tile shows both sides, and the report warns when the target's cost is unknown.
• juried compare prints, after the findings, the smallest drop the two...

juried 0.2.1

Added

• A calibration set of 32 labelled responses for the example project, eight per criterion, and its report against claude-haiku-4-5, which agreed with every label. The example's stub calibrate run now writes to reports/stub/ so it does not overwrite that report, and docs/calibration.md describes the set as the pattern to copy.
• Check, Live provider contract and PyPI version badges at the top of the README.

Changed

• The live provider workflow fails when either provider was skipped for want of a key, naming the missing secret, so a green run means both clients were exercised.
• The HTML report's totals sit on one row of tiles, or two even rows when the replay and split verdict tiles apply, and the gate rule is stated once above the first table rather than under every table. The README's screenshot is refreshed to match.
• juried generate no longer prints a usage line under the stub provider, which spends nothing.

Fixed

• An empty judge.model or generate.model, whether from juried.toml or an empty JURIED_JUDGE_MODEL, now stops the run with "judge.model must not be empty" instead of reaching the provider and coming back as an HTTP 400. The live tests treat an empty model override as unset, which is how the workflow exports it.
• The sdist no longer carries the README logo and the report screenshot, which the README references by URL; it is under 200 KB rather than over 1 MB.

Full changelog: v0.2.0...v0.2.1

juried 0.2.0

Breaking changes

• The default runs is 20 rather than 10. At the default threshold of 0.7 a scenario now
  needs 19 of 20 passes, so one miss is tolerated; 10 runs needed all 10. Each scenario
  costs twice as many requests unless runs is set explicitly (#26).
• Transport and judge errors are no longer part of the pass rate. The rate and the Wilson
  interval are computed over attempts that reached a verdict, and any error makes the
  scenario "incomplete", which still fails the pytest item but is reported separately from
  a quality failure (#21).
• Responses are no longer replayed from the cache by default. Every run samples the feature
  afresh; replay is opt in with cache_responses = true or juried run --cache-responses,
  and a run that replayed anything says so loudly (#18).
• The JSON report gained fields. Scenario entries carry judged, judge_errors,
  quality_met, status, required_passes, responses_from_cache, judge_agreement,
  split_verdicts, usage and turns; attempts carry judge_error, votes,
  agreement, usage and transcript; failures carry agreement, votes and
  transcript; summary carries incomplete, judge_errors, responses_from_cache,
  split_verdicts and usage; judge carries votes and prices_usd_per_million;
  defaults carries required_passes. gates_failed no longer counts incomplete
  scenarios (#17, #18, #20, #21, and the cost and multi turn changes).
• Verdicts cached by 0.1.x are not reused. The judge prompt changed (prompt version 2 to 5)
  and the cache key now includes the vote index and the live transcript, so .juried/
  from an earlier release is ignored rather than trusted (#19, #20, #28).
• judge provider implementations must accept a transcript argument on judge and
  return usage from complete_json (#28 and the cost change).

Added

• juried calibrate: judges responses your team has labelled by hand under calibration/
  and reports accuracy, false passes and false fails, with --min-accuracy as a CI gate.
  juried init writes a starter calibration/example.yaml, and a run with a real judge
  and no calibration report ends with a warning (#20, #25).
• juried compare old.json new.json: lists gates lost, scenarios newly incomplete, new
  failing scenarios and drops in pass rate or lower bound, and exits 1 on any regression.
  --tolerance ignores small drops and --json writes the findings to a file.
• [judge] votes: judge each response an odd number of times and take the majority; the
  report shows split verdicts and the pytest summary flags them. Verdicts are not cached
  while votes are above 1, so agreement is re-measured on every run (#20, #27).
• Token usage and estimated spend. Every provider response's usage is recorded on its
  verdict and summed per attempt, scenario and run; the pytest summary, the JSON and HTML
  reports, generate and calibrate all report tokens and an estimate at dated list
  prices, overridable with [judge] input_price and output_price.
• Live multi turn scenarios through a turns field: each turn is answered by the feature
  and its reply becomes context for the next; the transcript is recorded per attempt,
  shown for failing runs and given to the judge in its own section (#28).
• Generated scenarios may carry a short history for follow up messages (#24).
• [judge] concurrency, a separate cap on judge requests in flight (#22).
• [generate] temperature and base_url, so generation no longer borrows the judge's
  temperature (#24).
• ${NAME} expands in [target] url and body as well as headers, and an unset
  variable stops the run before any request (#21, #24).
• The gate requirement is stated everywhere: the pytest header prints
  gate needs 19/20 passes at threshold 0.70 (1 miss tolerated), every gate failure
  repeats it, and the report shows "needs N / M" under the threshold (#17).
• A Check workflow runs make check on Python 3.11, 3.12 and 3.13 for every pull
  request, and a weekly Live provider contract workflow sends one judge and one generate
  request per provider to the real APIs (#23).
• Python 3.11 is supported (#23).

Changed

• All scenarios in a session run together on one background event loop with a shared
  client and provider, bounded by run.concurrency and judge.concurrency, instead of
  one event loop per pytest item. Output order, -x and -k are unchanged (#22).
• The judge prompt wraps every section in tags, treats the response as untrusted output
  whose instructions and claims about the verdict carry no weight, and asks its question
  after the response (#19).
• {{message}} and {{history}} follow one rule: a value that is exactly a placeholder
  keeps its type, and inside longer text each becomes text, the history as JSON (#24).
• The README says plainly that the threshold is a floor on the interval's lower bound and
  not a pass rate, with a table of runs, threshold, passes needed and misses tolerated; that
  votes measure judge stability rather than correctness; that the stub judge is a substring
  matcher; and what multi turn support does and does not do. The roadmap moved from
  CONTRIBUTING.md to the README (#17, #20, #24, and the votes note).

Fixed

• A TargetConfigError on one attempt, such as a bad response_path, stops the scenario
  with one clear message instead of a traceback and cancels its sibling attempts (#21).
• A judge failure on one attempt is recorded as a judge error rather than crashing the
  scenario and discarding the other attempts (#21).
• verdicts.jsonl is appended under a file lock so pytest-xdist workers cannot interleave
  lines (#22).
• The always true field.annotation check in apply_env_overrides (#24).
• The README's claim that nothing but keys is ever read from disk (#24).
• Concurrency tests no longer assert wall clock time, which failed on slow CI runners (#29).

v0.1.1

Patch release. Adds PyPI project URLs, author, keywords and classifiers to the package metadata. No code changes.

v0.1.0

Juried first release