You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
jud eval reports, per question, its answers by outcome (how many labels
name each, how often the model gave it, how often rightly), the majority
label with its share, and five signals: no_better_than_majority, collapsed, never_answered, defers_most and defers_nearly_all, each a warning: line in the text and a code in the new outcomes, majority
and signals keys of --json, none read from fewer than ten answers. The
thresholds they were read with are the new signal_rules key, so a report
says which it used (decision 0022). They are what the tuning skill
computed by hand on the first real-world run (#46).
jud record --dry-run says what a run would do and does nothing: the
requests, how many the directory answers, which are to ask or replace,
the backend and model, whether a key is set, and the time at the median
pace of the recordings already there. No key, no call, no file. It counts
two cases that lower to one request once, and refuses what the run would
refuse before its first call.
jud split CASES [--every N] [--out DIR] writes a tuning set and a
held-out set, every Nth case held out, each case copied as read so the
recordings over the whole set answer both. It writes over nothing and
warns when a label one half has is missing from the other. A from_turn
label counts as the turns it labels, and with --rubric RUBRIC a Score
level written as its index and as its text is one level.
Changed
The Claude Code plugin (now 0.3.1) carries what a first real-world run
taught it (#46): a rubric over Wikimedia's recentchange stream, 93 cases
captured from it, recorded and tuned against tev1:0.8b on Ollama, and a
Rust loop judging the live stream. jud-tune now reports each question
against always giving its commonest answer, names a question that has
collapsed onto one answer, flags a gate that defers nearly every case,
states a local server's cost as time, and allows an announced subset
experiment on a copy to measure a question change. The jud skill and /jud:rubric ask for the field each answer is read from and leave to code
what machine-written fields decide; /jud:cases keeps captured states
verbatim and labels from the best evidence, an outcome known later
included. jud-rust gives the use path for a module mounted from lib.rs and says that every decide is one request a stream caller must
bound. The mock System One server gains collapse and confidence
profile keys, and a new jud-tune scenario uses them.
Fixed
/jud:rust no longer writes a Cargo.lock (or a target/) into the
user's crate when that crate already mounts the module: it runs cargo test there only when a lock file exists, with a target directory outside
the project.
Added
The Claude Code plugin (now 0.3.0) turns a rubric into a typed Rust module:
the jud-rust skill and /jud:rust <rubric>. The module reads the rubric
with include_str! and lowers the request with Rubric::lower, so it sends
what jud lower sends and a retune needs no regeneration. It gives each
Choice and Score an enum and the program one call, decide(backend, model, state), and its own test fails when the rubric's questions drift from it. examples/jud_typed.rs wires the two template modules
(examples/jud/triage.rs, examples/jud/routing.rs) into a program and
tests them; scripts/gen-plugin-rust.sh keeps the plugin's copies equal
to them.
Added
The Claude Code plugin (plugins/jud/, now 0.2.0) measures and tunes
rubrics as well as writing them: a jud-tune skill and three commands. /jud:record says how many calls a run spends, to which backend and model,
before running jud record, and resumes what a directory already holds. /jud:eval grades the recordings and triages every miss into a wrong
label, an ambiguous question, the bar's job, a coverage gap or the model,
proposing each fix without applying it. /jud:tune decides gate by gate
whether to take what jud tune proposes (never a bar of 0 on a handful of
cases, never a bar that absorbs a wrong label), and with apply writes it
into the rubric in place with its comments kept, then shows each gate
before and after; with holdout it tunes on three quarters of the cases and
reports the rest. The commands were battle-tested headless (claude -p --plugin-dir) against a mock System One server that knows the labels
(plugins/jud/skills/jud-tune/evals/); no test calls a real API.