Repository navigation
judgment 0.11.0
Added
-
jud record,jud evalandjud tunerun a rubric over its labelled
cases from the shell (decision 0021;docs/reference/cli.md).jud record RUBRIC CASES --out DIRasks the configured backend once per case, or once
per turn for a conversation labelled withfrom_turn, and writes each
answer as a.judrecording. It keeps whatDIRalready answers, so an
interrupted run resumes, and--refreshasks again. The server a
recording names is the base URL (TYPESAFE_BASE_URL, orbase_urlin the
configuration file) without its userinfo, query and fragment, so a
credential kept in the URL is never written to a recording, andjud tune
strips a recording's server the same way for itstuningblock.jud eval RUBRIC CASES [--replay DIR]grades the answers against the labels and
reports, per question, the model's accuracy with its 95 % interval, its
Brier score and calibration error, how often the policy's gate acts and how
often its verdict is right, and every miss, as text or as one JSON object
(--json).--min-accuracy [QUESTION=]ACCURACYholds the model's
accuracy per question and makesjud evalexit with the new status 3 when
a bar is not met.jud tune RUBRIC CASES --replay DIRreads recordings
only, prints each gate's sweep on stderr and proposes the bars as the
policyandtuningblocks on stdout;--out PATHwrites the whole
rubric instead.Contract: exit status 3 (
jud evalalone), the keys ofjud eval --json
(rubric,cases,requests,models,questions,min_accuracy) and
the two places that write files,jud record --outandjud tune --out,
join the command's contract (docs/reference/stability.md). Neither
command writes over a rubric or a cases document it was given, andjud tune --outnever writes among the recordings. Every other command stays
read-only. stdout is the result and nothing else; progress, tables and
failures go to stderr. Statuses 0, 1 and 2 keep their meaning.
Changed
- The documentation is rebuilt in Diátaxis form under
docs/: tutorials
(start/), guides (guides/), reference (reference/), concepts
(concepts/) and the project's own pages (project/, where the
decision, verification and research records moved unchanged). The.jud
specification isdocs/reference/jud-format.md, in BCP 14 wording, and
the plugin'sreferences/format.mdis generated from it; the command
line's reference is generated from the binary's help; the README is a
front door. Every Rust block, whole.juddocument and marked transcript
on a page is held to the code by a test, and every internal link is checked
(mise run docs:links). The rebuild itself changed no behaviour, API or
format rule. Replay::openreads a directory in name order, so that when two files
record one request the same one answers on every file system, and a
.jsonrecording that does not parse isError::InvalidRecording, which
names the file, instead ofError::Decode. Breaking for a caller that
matchedError::Decodethere.jud RUBRIC,jud check,jud lowerandjud configno longer panic
(status 101) when stdout is closed (jud check FILE | head -0): a reader
that has gone is not an error, and the status is the command's own, as it
already was forjud completionand is forjud record,jud evaland
jud tune.- Messages of the existing paths, each with its status unchanged:
- A failure of
jud RUBRICnames the backend asscheme://host/path,
without the userinfo, query or fragment of its base URL, so a credential
kept in the URL is not printed. - A misplaced
--replaygets a hint.jud --replay DIR eval ...says the
flag belongs after the subcommand, andjud --replay DIRalone says a
rubric is required; both are status 2 as before. - A mistyped bare subcommand (
jud evaluate) reads as a rubric path that is
not there, and the message now adds(not a subcommand either: ...)with
the subcommands' names. - Under
jud RUBRIC --replay DIR, a recording that matches the state and
the rubric but no longer fits the questions is reported asa recording under DIR matches this state and rubric but cannot answer it, where the
message said that no recording answers; status 1 as before. jud checkpoints a document that is notjud/v1.3at
docs/reference/jud-format.md, where it named the page's old path,
docs/jud.md.
- A failure of