Skip to content

judgment 0.11.0

Choose a tag to compare

@github-actions github-actions released this 08 Oct 11:46
· 176 commits to main since this release

Added

  • jud record, jud eval and jud tune run a rubric over its labelled
    cases from the shell (decision 0021; docs/reference/cli.md). jud record RUBRIC CASES --out DIR asks the configured backend once per case, or once
    per turn for a conversation labelled with from_turn, and writes each
    answer as a .jud recording. It keeps what DIR already answers, so an
    interrupted run resumes, and --refresh asks again. The server a
    recording names is the base URL (TYPESAFE_BASE_URL, or base_url in the
    configuration file) without its userinfo, query and fragment, so a
    credential kept in the URL is never written to a recording, and jud tune
    strips a recording's server the same way for its tuning block. jud eval RUBRIC CASES [--replay DIR] grades the answers against the labels and
    reports, per question, the model's accuracy with its 95 % interval, its
    Brier score and calibration error, how often the policy's gate acts and how
    often its verdict is right, and every miss, as text or as one JSON object
    (--json). --min-accuracy [QUESTION=]ACCURACY holds the model's
    accuracy per question and makes jud eval exit with the new status 3 when
    a bar is not met. jud tune RUBRIC CASES --replay DIR reads recordings
    only, prints each gate's sweep on stderr and proposes the bars as the
    policy and tuning blocks on stdout; --out PATH writes the whole
    rubric instead.

    Contract: exit status 3 (jud eval alone), the keys of jud eval --json
    (rubric, cases, requests, models, questions, min_accuracy) and
    the two places that write files, jud record --out and jud tune --out,
    join the command's contract (docs/reference/stability.md). Neither
    command writes over a rubric or a cases document it was given, and jud tune --out never writes among the recordings. Every other command stays
    read-only. stdout is the result and nothing else; progress, tables and
    failures go to stderr. Statuses 0, 1 and 2 keep their meaning.

Changed

  • The documentation is rebuilt in Diátaxis form under docs/: tutorials
    (start/), guides (guides/), reference (reference/), concepts
    (concepts/) and the project's own pages (project/, where the
    decision, verification and research records moved unchanged). The .jud
    specification is docs/reference/jud-format.md, in BCP 14 wording, and
    the plugin's references/format.md is generated from it; the command
    line's reference is generated from the binary's help; the README is a
    front door. Every Rust block, whole .jud document and marked transcript
    on a page is held to the code by a test, and every internal link is checked
    (mise run docs:links). The rebuild itself changed no behaviour, API or
    format rule.
  • Replay::open reads a directory in name order, so that when two files
    record one request the same one answers on every file system, and a
    .json recording that does not parse is Error::InvalidRecording, which
    names the file, instead of Error::Decode. Breaking for a caller that
    matched Error::Decode there.
  • jud RUBRIC, jud check, jud lower and jud config no longer panic
    (status 101) when stdout is closed (jud check FILE | head -0): a reader
    that has gone is not an error, and the status is the command's own, as it
    already was for jud completion and is for jud record, jud eval and
    jud tune.
  • Messages of the existing paths, each with its status unchanged:
    • A failure of jud RUBRIC names the backend as scheme://host/path,
      without the userinfo, query or fragment of its base URL, so a credential
      kept in the URL is not printed.
    • A misplaced --replay gets a hint. jud --replay DIR eval ... says the
      flag belongs after the subcommand, and jud --replay DIR alone says a
      rubric is required; both are status 2 as before.
    • A mistyped bare subcommand (jud evaluate) reads as a rubric path that is
      not there, and the message now adds (not a subcommand either: ...) with
      the subcommands' names.
    • Under jud RUBRIC --replay DIR, a recording that matches the state and
      the rubric but no longer fits the questions is reported as a recording under DIR matches this state and rubric but cannot answer it, where the
      message said that no recording answers; status 1 as before.
    • jud check points a document that is not jud/v1.3 at
      docs/reference/jud-format.md, where it named the page's old path,
      docs/jud.md.