Repository navigation
judgment 0.13.0
Added
-
jud RUBRIC --options JSONand--options-file PATHsupply the options
of each Choice the rubric marksoptions_from: request: the tools an
agent may call, the desks staffed now, whatever changes per request. They
are asked in the order given, before the static options, and refused
before any call when they cannot be asked. Such a rubric used to be
refused byjud RUBRICoutright;jud evalandjud recordalready
took a case's ownoptions, andjud loweralready took--options. -
Jev through OpenRouter's System One API is a backend: base URL
https://openrouter.ai/api, an OpenRouter key.Usage::costholds the
usage.costit sends, in US dollars (Nonefrom a server that sends
none, and left out of a recording then); its generation id (gen-dec-…)
becomesResponse::request_idwhen there is nox-typesafe-request-id;
its 402 for a balance that has run out is the new
Error::PaymentRequired, not retried.mise run live:openrouterruns the
live tests against it withOPENROUTER_API_KEY, under a newopenrouter
profile that expects itsGET /v1/modelsto be OpenRouter's own
catalogue; an unprofiled run againstopenrouter.aistops before it
asks, as one against the hosted API does. Checked against a stand-in
built from OpenRouter's documented shapes; no live run is recorded yet. -
mise run live:pplxruns the ignored live tests against Perplexity's
pplx-decider-v1.1-27b through theautojev-serveits checkpoint ships,
which serves/v1/systemone(PPLX_URL, default
http://127.0.0.1:8010;PPLX_API_KEYalso runs the bearer test). The
model itself needs a GPU with about 49 GiB free; the server's wire was
checked with the checkpoint's ownserver.pyandanswer()over a
stand-in network, where all the live tests pass, with and without a
key. Its departures from the hosted API, and why to name the model
autojev-qwen3.8-27bthough it answers tojev-latest, are in
docs/project/verification/autojev-serve.md. -
The live tests check every body they read as sent, a success, a
refusal or the model list, against the vendored OpenAPI document and
print the verdict, down to the field of the answer kind that fails; a
status the document does not list (the hosted API's 400, laya-serve's
413) is named as such. A success body that does not conform fails the
run against the hosted API and autojev-serve, and is reported for any
other server. A new test,every_body_the_server_sends_holds_to_the_published_document,
sends the request shapes the builders produce and fetches the model
list for it. -
JUDGMENT_LIVE_PROFILE(typesafe,laya,autojev,generic) names
the server the live tests run against, and eachmise run live:*task
sets its own. Where servers differ (a refusal's status, the limits, the
budget) a behaviour is asserted on the profile it was observed on and
printed on the others; laya-serve 0.3.24's statuses, as its verification
record has them, and autojev-serve's are now pinned too. Unset, the
profile isgeneric, except against the hosted API, where it stops the
run.docs/project/verification/autojev-serve.mdrecords what the
autojev profile pins and how it was observed. -
judnames everyTYPESAFE_*orJUD_*variable it does not read
(JUD_BASE_URL,TYPESAFE_MODEL): one line per variable on stderr
before it asks a server and injud record --dry-run, saying what it reads instead and which model and server it is
asking, andignored_environmentinjud config. An unset base URL is
the hosted API, so a misspelt variable used to send billed calls there
without a word. Names only, never values.
Changed
jud recordandjud evalwrite one line to stderr before the first
call to a server, naming the model, the server and where its base URL
came from (asking jev-latest at https://api.typesafe.ai for up to 7 requests (base URL from default)). A server run ofjud evalused to
leave stderr empty on success. Nothing changes under--replay.- The live tests no longer decide from the model name whether the hosted
API's behaviour is asserted: a model whose name starts withjevwas
taken for the hosted API, whichautojev-serve, answering to
jev-latest, is not.mise run live:typesafesets thetypesafe
profile; a hand run against the hosted API sets
JUDGMENT_LIVE_PROFILE=typesafeto keep its pins. - Breaking:
Usagehas a third field,cost: Option<f64>, and is no
longerEq(it staysPartialEq,CopyandDefault). AUsage { .. }
literal addscost: None.
Fixed
- A base URL with a path lost its last segment: the client joined
v1/systemoneonto it as a relative URL, sohttps://openrouter.ai/api
reachedhttps://openrouter.ai/v1/systemone, and any gateway served under
a path got a 404. The path is now kept, with or without the trailing
slash, as the official SDKs keep it. A base that only worked because its
last segment was dropped (https://api.typesafe.ai/v1, which reached
/v1/systemone) now reaches/v1/v1/systemone; drop that segment.