Repository navigation
Releases: chussenot/judgment
Release list
judgment 0.14.0
Added
-
Images in a request, as Jev 1.13 and later take them
(https://docs.typesafe.ai/images):Imageholds a PNG or a JPEG and
reads its format and size from the bytes, refusing what the hosted API
refuses before anything is sent (not a PNG or a JPEG, animated, a side
outside 256 to 1568 pixels, under 78,400 or over 2,458,624 pixels, over
16 MiB) as the newError::InvalidImage;Assetsis the wire's
assetsmap, each image under its own reference (img-and 32 hex
digits of the bytes' SHA-256,Image::reference) or one the caller
chooses, with the reference rules and the 32 MiB total checked
(Error::InvalidAsset,Error::RequestTooLarge).Request::assets
carries them (Request::new(..).with_assets(&assets)), left off the
wire when empty, andSystemOne::answer_with_assetstakes them through
every backend. The client sendsTypeSafe-Danger-Zone-Images: trueon
every request that has an image and on no other, refuses a reference
that appears nowhere in the state or the questions and a body over
32 MiB before sending, reports the images' count and bytes to the
Observer(on_assets) and records the count on the
typesafe.evaluatespan asimages.Error::is_request_too_largeis
also true ofRequestTooLarge. Decision 0024 says why the images are on
the request rather than a call option, why the reference is derived from
the bytes and why the client sets the header. -
canonical::request_fingerprint_withandeval::request_hash_with: the
request fingerprint and hash with the images, each named by the SHA-256
of its bytes under anassetskey that is left out without any, so a
request without images fingerprints as it always has and every committed
recording still replays;RecorderandReplaykey by them, so two
requests that differ only by an image are two recordings. -
Faketakes images, remembers each one's reference and digest in its
Call, and refuses a reference the state does not carry as the hosted
API does. -
The Box primitive, Jev 1.14's fourth question type
(https://docs.typesafe.ai/primitives/box):Questions::r#box(id, instructions)asks where what the instructions describe is in an image
the request carries (no criteria; null instructions are refused, since
they are all the question is), and the client sends the images header
for it even without an image. The answer isAnswer::Box { edges, confidence }, the wire'sboxasBoxEdges(top, bottom, left, right
as fractions of the image) and a confidence the document lets be null;
Response::verifyrefuses a box that is not a region (an edge outside
0..=1, a left not below its right, a top not below its bottom) as
Error::InvalidAnswer, and reads it asBoundingBox, withwidth,
height,center,to_pixels,confidence_at_leastand the geometry
onBoxEdges(iou,padded,area,contains). A Box always answers
with a box, so the typed view's documentation pairs it with a Noul gate.
Fake::r#boxscripts one.Judgment::locatedgrades a Box against a
labelled box by intersection over union ateval::DEFAULT_IOU_AT_LEAST
(0.5) or the label's own bar, with the number onJudgment::iouand
QuestionMetrics::mean_iou; a Box has no Brier score. In a rubric a Box
is gated byconfidenceorbands(neverthreshold,fallbackor
level_at_least) intoVerdict::Locatedor a deferral, and a case labels
it{box: {top, bottom, left, right}, iou_at_least}. -
The
.judformat at minor 4 (jud::MINOR): aboxquestion, the images
a document names underspec.assetson a Rubric andassetson a case
(jud::AssetSource: apathrelative to the document, orformatand
data_base64inline; decision 0025), loaded into a request'sAssetsby
Rubric::load_assetsandCase::load_assets, with a reference that
appears nowhere in the state or the questions refused as the API refuses
it; the Box label{box, iou_at_least}(jud::BoxLabel), graded by
intersection over union at 0.5 unless the label says otherwise (decision
0026); a Box in a gate'sconfidenceorbands;Verdict::in_pixels,
which adds the located box in an image's pixels beside the edges. The
schemas, the format page andexamples/jud/shapes.judwith its forty
labelled synthetic images (examples/jud/images/) move together, and the
format page carries a known fingerprint vector for a request with an
image. A minor 3 reader refuses a minor 4 document by number; a minor 4
reader reads every minor 3 document as it did. -
jud: every subcommand that builds a request loads the documents' images
and keys recordings by them;jud RUBRIC --asset REF=PATH(repeatable)
adds an image from the command line and--pixels WxHnames the size a
located box is also printed in pixels of (the one image's size when there
is exactly one); a located verdict prints aslocatedwithbox, the
confidence, the band andpixels;jud evalreports a Box question's
mean_ioubeside its accuracy, no Brier score, and a miss'siou;jud checkcounts a document's images. -
jud:long_context: truein its configuration file, or--long-context
onjud RUBRIC,jud recordandjud eval, asks for long context on
every call of the run (the switch of decision 0027, passed to
ClientBuilder::long_context); the flag wins over the file, and a replay
ignores both since it asks nothing.jud configreportslong_context
andlong_context_from,jud record --dry-runprints along context:
line, and the line before a run's first call names it when it is on. -
The
multipartfeature:CallOptions::multipartfor one call and
ClientBuilder::multipartfor every call of a client send a request's
images as themultipart/form-databody TypeSafe recommends, a JSON
requestpart in which each asset is{type: image, part}and one part
per image (part-0,part-1, … in reference order, underimage/pngor
image/jpeg), instead of base64 in the JSON body; off by default, and
nothing without an image in the request. The request's fingerprint and
hash, the recordings, the checks before the wire, the images header and
the observer's report are the same under either transport, and the
typesafe.evaluatespan recordsmultipart.ImageFormat::content_type
names the media type. Decision 0028 says why it is a feature and a switch
and not the default; verified against the hosted API, which answers the
form at the same tokens as the JSON body. -
A second Box calibration study, on forty generated document pages with a
round ink stamp a tenth of the page wide among a round logo, a signature
and a barcode (examples/jud/stamps.jud,stamps-cases.judand
images/page-*.png, drawn byscripts/gen-document-pages.pyfrom one
seed), recorded onjev-1.14.0underexamples/recordings/stamps/and
set against the shapes baseline at two sizes
(docs/project/verification/box-calibration-study-pages.md): the misses
are near misses at an overlap of 0.45 to 0.49, twice the pixels turn
every one into a hit at three times the tokens, and the Box's confidence
does not say "nothing here" when a round logo is on the page, which the
Noul beside it does.tests/jud.rsreplays the recordings and pins the
counts. -
Get verdicts on images from the command line,
a guide that walks a shell user from a rubric that names an image to a
verdict withjudalone:--assetand--pixels, the verdicts a Noul
and a Box print and why the Noul is read first, what is refused before
any call, images a rubric always sends, the labelled loop over photos and
what a real run needs and costs; every transcript is replayed from the
stamps recordings and held to the binary bytests/docs_examples.rs. -
Long context (https://docs.typesafe.ai/models#long-context): both
context limits raised from 64k tokens per request and 32k per question
to 262k through theTypeSafe-Danger-Zone-Long-Context: trueheader,
on Jev 1.13 and 1.14.CallOptions::long_context(true)asks for it on
one call andClientBuilder::long_context(true)on every call of a
client (a call'sfalseturns it off again); the header is reserved
(client::DANGER_ZONE_LONG_CONTEXT_HEADER) and recorded on the
typesafe.evaluatespan aslong_context. Off by default, as TypeSafe
says a long request is slower and less accurate, and never read off the
request's size (decision 0027). Verified live: a 74,550-token state the
default budget refuses asmax_tokens_exceededwas answered under the
header byjev-1.13.0in 1.4 s. -
The Claude Code plugin (now 0.4.0) knows Jev 1.14: the
judskill's
rules and guidance for a Box question (what to locate, the Noul beside
it, the reference in the state, the images a document declares, the box
label in fractions and its bar), thejud-tuneskill's reading of a
Box'sjud evalandjud tunenumbers (a mean IoU where a Brier score
would be, the labelled count, the empty images, the triage rows for a box
that is loose, on the wrong thing or over nothing) and what re-encoding
an image costs, and thejud-rustskill's third template,
examples/jud/shapes.rs, a module whosedecidetakes the request's
Assetsand whoseLocatedverdict carries a Box, exercised from the
shapes recordings byexamples/jud_typed.rs; the commands follow, and a
seventh authoring eval asks for an image rubric over five parcel photos
and their annotator's boxes (skills/jud/evals/files/images/), run once
with the skill and once without through the new
skills/jud/evals/run.py, and a ninthjud-rustscenario asks for the
shapes module and replays the recordings with their images through the
module it gets; the plugin guide records both. In a cloud session the
start hook installs the plug...
judgment 0.13.0
Added
-
jud RUBRIC --options JSONand--options-file PATHsupply the options
of each Choice the rubric marksoptions_from: request: the tools an
agent may call, the desks staffed now, whatever changes per request. They
are asked in the order given, before the static options, and refused
before any call when they cannot be asked. Such a rubric used to be
refused byjud RUBRICoutright;jud evalandjud recordalready
took a case's ownoptions, andjud loweralready took--options. -
Jev through OpenRouter's System One API is a backend: base URL
https://openrouter.ai/api, an OpenRouter key.Usage::costholds the
usage.costit sends, in US dollars (Nonefrom a server that sends
none, and left out of a recording then); its generation id (gen-dec-…)
becomesResponse::request_idwhen there is nox-typesafe-request-id;
its 402 for a balance that has run out is the new
Error::PaymentRequired, not retried.mise run live:openrouterruns the
live tests against it withOPENROUTER_API_KEY, under a newopenrouter
profile that expects itsGET /v1/modelsto be OpenRouter's own
catalogue; an unprofiled run againstopenrouter.aistops before it
asks, as one against the hosted API does. Checked against a stand-in
built from OpenRouter's documented shapes; no live run is recorded yet. -
mise run live:pplxruns the ignored live tests against Perplexity's
pplx-decider-v1.1-27b through theautojev-serveits checkpoint ships,
which serves/v1/systemone(PPLX_URL, default
http://127.0.0.1:8010;PPLX_API_KEYalso runs the bearer test). The
model itself needs a GPU with about 49 GiB free; the server's wire was
checked with the checkpoint's ownserver.pyandanswer()over a
stand-in network, where all the live tests pass, with and without a
key. Its departures from the hosted API, and why to name the model
autojev-qwen3.8-27bthough it answers tojev-latest, are in
docs/project/verification/autojev-serve.md. -
The live tests check every body they read as sent, a success, a
refusal or the model list, against the vendored OpenAPI document and
print the verdict, down to the field of the answer kind that fails; a
status the document does not list (the hosted API's 400, laya-serve's
413) is named as such. A success body that does not conform fails the
run against the hosted API and autojev-serve, and is reported for any
other server. A new test,every_body_the_server_sends_holds_to_the_published_document,
sends the request shapes the builders produce and fetches the model
list for it. -
JUDGMENT_LIVE_PROFILE(typesafe,laya,autojev,generic) names
the server the live tests run against, and eachmise run live:*task
sets its own. Where servers differ (a refusal's status, the limits, the
budget) a behaviour is asserted on the profile it was observed on and
printed on the others; laya-serve 0.3.24's statuses, as its verification
record has them, and autojev-serve's are now pinned too. Unset, the
profile isgeneric, except against the hosted API, where it stops the
run.docs/project/verification/autojev-serve.mdrecords what the
autojev profile pins and how it was observed. -
judnames everyTYPESAFE_*orJUD_*variable it does not read
(JUD_BASE_URL,TYPESAFE_MODEL): one line per variable on stderr
before it asks a server and injud record --dry-run, saying what it reads instead and which model and server it is
asking, andignored_environmentinjud config. An unset base URL is
the hosted API, so a misspelt variable used to send billed calls there
without a word. Names only, never values.
Changed
jud recordandjud evalwrite one line to stderr before the first
call to a server, naming the model, the server and where its base URL
came from (asking jev-latest at https://api.typesafe.ai for up to 7 requests (base URL from default)). A server run ofjud evalused to
leave stderr empty on success. Nothing changes under--replay.- The live tests no longer decide from the model name whether the hosted
API's behaviour is asserted: a model whose name starts withjevwas
taken for the hosted API, whichautojev-serve, answering to
jev-latest, is not.mise run live:typesafesets thetypesafe
profile; a hand run against the hosted API sets
JUDGMENT_LIVE_PROFILE=typesafeto keep its pins. - Breaking:
Usagehas a third field,cost: Option<f64>, and is no
longerEq(it staysPartialEq,CopyandDefault). AUsage { .. }
literal addscost: None.
Fixed
- A base URL with a path lost its last segment: the client joined
v1/systemoneonto it as a relative URL, sohttps://openrouter.ai/api
reachedhttps://openrouter.ai/v1/systemone, and any gateway served under
a path got a 404. The path is now kept, with or without the trailing
slash, as the official SDKs keep it. A base that only worked because its
last segment was dropped (https://api.typesafe.ai/v1, which reached
/v1/systemone) now reaches/v1/v1/systemone; drop that segment.
judgment 0.12.0
Added
jud evalreports, per question, its answers by outcome (how many labels
name each, how often the model gave it, how often rightly), the majority
label with its share, and five signals:no_better_than_majority,
collapsed,never_answered,defers_mostanddefers_nearly_all, each a
warning:line in the text and a code in the newoutcomes,majority
andsignalskeys of--json, none read from fewer than ten answers. The
thresholds they were read with are the newsignal_ruleskey, so a report
says which it used (decision 0022). They are what the tuning skill
computed by hand on the first real-world run (#46).jud record --dry-runsays what a run would do and does nothing: the
requests, how many the directory answers, which are to ask or replace,
the backend and model, whether a key is set, and the time at the median
pace of the recordings already there. No key, no call, no file. It counts
two cases that lower to one request once, and refuses what the run would
refuse before its first call.jud split CASES [--every N] [--out DIR]writes a tuning set and a
held-out set, every Nth case held out, each case copied as read so the
recordings over the whole set answer both. It writes over nothing and
warns when a label one half has is missing from the other. Afrom_turn
label counts as the turns it labels, and with--rubric RUBRICa Score
level written as its index and as its text is one level.
Changed
- The Claude Code plugin (now 0.3.1) carries what a first real-world run
taught it (#46): a rubric over Wikimedia's recentchange stream, 93 cases
captured from it, recorded and tuned againsttev1:0.8bon Ollama, and a
Rust loop judging the live stream.jud-tunenow reports each question
against always giving its commonest answer, names a question that has
collapsed onto one answer, flags a gate that defers nearly every case,
states a local server's cost as time, and allows an announced subset
experiment on a copy to measure a question change. Thejudskill and
/jud:rubricask for the field each answer is read from and leave to code
what machine-written fields decide;/jud:caseskeeps captured states
verbatim and labels from the best evidence, an outcome known later
included.jud-rustgives theusepath for a module mounted from
lib.rsand says that everydecideis one request a stream caller must
bound. The mock System One server gainscollapseandconfidence
profile keys, and a newjud-tunescenario uses them.
Fixed
/jud:rustno longer writes aCargo.lock(or atarget/) into the
user's crate when that crate already mounts the module: it runscargo testthere only when a lock file exists, with a target directory outside
the project.
Added
- The Claude Code plugin (now 0.3.0) turns a rubric into a typed Rust module:
thejud-rustskill and/jud:rust <rubric>. The module reads the rubric
withinclude_str!and lowers the request withRubric::lower, so it sends
whatjud lowersends and a retune needs no regeneration. It gives each
Choice and Score an enum and the program one call,decide(backend, model, state), and its own test fails when the rubric's questions drift from it.
examples/jud_typed.rswires the two template modules
(examples/jud/triage.rs,examples/jud/routing.rs) into a program and
tests them;scripts/gen-plugin-rust.shkeeps the plugin's copies equal
to them.
Added
- The Claude Code plugin (
plugins/jud/, now 0.2.0) measures and tunes
rubrics as well as writing them: ajud-tuneskill and three commands.
/jud:recordsays how many calls a run spends, to which backend and model,
before runningjud record, and resumes what a directory already holds.
/jud:evalgrades the recordings and triages every miss into a wrong
label, an ambiguous question, the bar's job, a coverage gap or the model,
proposing each fix without applying it./jud:tunedecides gate by gate
whether to take whatjud tuneproposes (never a bar of 0 on a handful of
cases, never a bar that absorbs a wrong label), and withapplywrites it
into the rubric in place with its comments kept, then shows each gate
before and after; withholdoutit tunes on three quarters of the cases and
reports the rest. The commands were battle-tested headless (claude -p --plugin-dir) against a mock System One server that knows the labels
(plugins/jud/skills/jud-tune/evals/); no test calls a real API.
judgment 0.11.1
Fixed
jud completion zshcompletes a subcommand's own arguments. The script
clap generates declared the optionalRUBRICbefore the subcommand, so in
jud eval <TAB>zsh readevalas the rubric and offered the subcommand
names again;check,lower,record,evalandtunecompleted
neither their files nor their flags. The first word now offers a
subcommand or a file, and each subcommand completes its own arguments.
Bash, fish, elvish and PowerShell were not affected. Regenerate the
installed script (jud completion zsh > ..., ormise run installin a
checkout) to pick it up.
judgment 0.11.0
Added
-
jud record,jud evalandjud tunerun a rubric over its labelled
cases from the shell (decision 0021;docs/reference/cli.md).jud record RUBRIC CASES --out DIRasks the configured backend once per case, or once
per turn for a conversation labelled withfrom_turn, and writes each
answer as a.judrecording. It keeps whatDIRalready answers, so an
interrupted run resumes, and--refreshasks again. The server a
recording names is the base URL (TYPESAFE_BASE_URL, orbase_urlin the
configuration file) without its userinfo, query and fragment, so a
credential kept in the URL is never written to a recording, andjud tune
strips a recording's server the same way for itstuningblock.jud eval RUBRIC CASES [--replay DIR]grades the answers against the labels and
reports, per question, the model's accuracy with its 95 % interval, its
Brier score and calibration error, how often the policy's gate acts and how
often its verdict is right, and every miss, as text or as one JSON object
(--json).--min-accuracy [QUESTION=]ACCURACYholds the model's
accuracy per question and makesjud evalexit with the new status 3 when
a bar is not met.jud tune RUBRIC CASES --replay DIRreads recordings
only, prints each gate's sweep on stderr and proposes the bars as the
policyandtuningblocks on stdout;--out PATHwrites the whole
rubric instead.Contract: exit status 3 (
jud evalalone), the keys ofjud eval --json
(rubric,cases,requests,models,questions,min_accuracy) and
the two places that write files,jud record --outandjud tune --out,
join the command's contract (docs/reference/stability.md). Neither
command writes over a rubric or a cases document it was given, andjud tune --outnever writes among the recordings. Every other command stays
read-only. stdout is the result and nothing else; progress, tables and
failures go to stderr. Statuses 0, 1 and 2 keep their meaning.
Changed
- The documentation is rebuilt in Diátaxis form under
docs/: tutorials
(start/), guides (guides/), reference (reference/), concepts
(concepts/) and the project's own pages (project/, where the
decision, verification and research records moved unchanged). The.jud
specification isdocs/reference/jud-format.md, in BCP 14 wording, and
the plugin'sreferences/format.mdis generated from it; the command
line's reference is generated from the binary's help; the README is a
front door. Every Rust block, whole.juddocument and marked transcript
on a page is held to the code by a test, and every internal link is checked
(mise run docs:links). The rebuild itself changed no behaviour, API or
format rule. Replay::openreads a directory in name order, so that when two files
record one request the same one answers on every file system, and a
.jsonrecording that does not parse isError::InvalidRecording, which
names the file, instead ofError::Decode. Breaking for a caller that
matchedError::Decodethere.jud RUBRIC,jud check,jud lowerandjud configno longer panic
(status 101) when stdout is closed (jud check FILE | head -0): a reader
that has gone is not an error, and the status is the command's own, as it
already was forjud completionand is forjud record,jud evaland
jud tune.- Messages of the existing paths, each with its status unchanged:
- A failure of
jud RUBRICnames the backend asscheme://host/path,
without the userinfo, query or fragment of its base URL, so a credential
kept in the URL is not printed. - A misplaced
--replaygets a hint.jud --replay DIR eval ...says the
flag belongs after the subcommand, andjud --replay DIRalone says a
rubric is required; both are status 2 as before. - A mistyped bare subcommand (
jud evaluate) reads as a rubric path that is
not there, and the message now adds(not a subcommand either: ...)with
the subcommands' names. - Under
jud RUBRIC --replay DIR, a recording that matches the state and
the rubric but no longer fits the questions is reported asa recording under DIR matches this state and rubric but cannot answer it, where the
message said that no recording answers; status 1 as before. jud checkpoints a document that is notjud/v1.3at
docs/reference/jud-format.md, where it named the page's old path,
docs/jud.md.
- A failure of
judgment 0.10.4
Added
- A container image,
ghcr.io/chussenot/jud, forlinux/amd64and
linux/arm64:cat event.json | docker run -i --rm -v "$PWD:/work" ghcr.io/chussenot/jud triage.jud.FROM scratchwith the release
tarball's binary, a CA bundle for the client's TLS and a non-root user,
about 10 MB; taggedX.Y.Z,X.Yandlatest, with a build-provenance
attestation. The release workflow's newimagejob fills the build
context from the tarballs it just built (scripts/image-context.sh),
runs the image on the runner and pushes it after the publish
(The jud command line, Releasing).
judgment 0.10.3
Added
cargo binstall judgmentinstalls the release'sjudtarball instead of
compiling:[package.metadata.binstall]inCargo.tomlnames the asset
and the binary's path inside it, and the release workflow checks, on every
platform it packages, that the templates name the tarball it just built
(The jud command line).- A Homebrew formula for
jud:brew install chussenot/tap/judon Apple
silicon and Linux, from chussenot/homebrew-tap.
The formula installs the release tarballs by their checksums and generates
the shell completions from the binary. It is rendered from each release's
SHA256SUMSbyscripts/homebrew-formula.shand committed to the tap by
the release workflow's newhomebrewjob, so the tap's version is the
release's (The jud command line, Releasing).
judgment 0.10.2
Changed
- A kind's reader (
Rubric::parse,Cases::parse,jud::parse_recording)
handed a document of another known kind refuses it with
jud::Error::OtherKind { found, expected }, "a Cases document, not a
Rubric; this readskind: Rubric", instead ofError::Kind, which said
the kind was not a document kind at all.Error::Kindstill names a kind
the crate does not know.
Fixed
jud lowerandjud checkname the file in every refusal and say what
was expected of it:cannot read rubric PATH,PATH is not a valid Rubric: ...,--state is not JSON, and, for a file that is not a.jud
document at all, the pointer to the envelope thatjud RUBRICalready
gave.jud lower examples/jud/triage-cases.judused to print "kind: Casesis not a document kind" with no path.
judgment 0.10.1
Added
jud --replay DIR RUBRIC, orJUD_REPLAY=DIR, answers from the recordings
under a directory through the crate'sReplaybackend instead of a server:
no key, no network, a state nobody recorded is an error. The README's
command-line transcript runs this way overexamples/recordings/and
tests/jud_cli.rsholds it to what the binary prints.docs/stability.md: what is promised across versions.jud/v1is stable, a
minor version only adds, readers accept everyv1document for at least
twelve months after a later minor is published, fingerprints exclude the
envelope and never change for a document'sspec; the crate's 0.x rule and
the command's contract.- Decision 0020, proposed and not applied: whether a
v1minor stays in the
apiVersionstring or moves to aminorfield underapiVersion: jud/v1,
each option priced. scripts/record_demo.sh(mise run demo) records the command-line demo as
an asciinema cast,docs/demo.cast, from the recordings, with no network;
regenerated by the release bump so the version it shows is the release's.mise run ciis an alias ofmise run check, the gates in CI order.
Changed
- The README opens with the command line for 0.10: a one-line pitch, the mise
install, thecat event.json | jud triage.judtranscript with the verdicts
the binary prints forexamples/jud/triage.jud, then the Rust API; one
comparison table of the four properties where the clients differ, the full
grids linked; the live example's capture trimmed to the command and its
output.
judgment 0.10.0
Added
jud completion <shell>prints a completion script for bash, zsh, fish,
elvish or PowerShell, generated from the command tree the binary parses
with, so it cannot drift from the binary;docs/cli.mdsays where each
shell wants it, andmise run installrefreshes the scripts a shell
already has. The binary's arguments are now parsed byclap:jud --help
and every subcommand's--helpare derived from the same tree, a usage
error names the argument and exits 2 as before, andjud lowerrefuses
--state,--state-fileand--casestogether instead of taking the
last.
Changed
- The
judbinary builds behind its ownclifeature (jud,http,
clap,clap_complete):cargo install judgment --features cli. The
judfeature is the format alone again, so a library build with it
compiles no command-line parser.