Skip to content

judgment 0.13.0

Choose a tag to compare

@github-actions github-actions released this 10 Oct 18:48
· 114 commits to main since this release

Added

  • jud RUBRIC --options JSON and --options-file PATH supply the options
    of each Choice the rubric marks options_from: request: the tools an
    agent may call, the desks staffed now, whatever changes per request. They
    are asked in the order given, before the static options, and refused
    before any call when they cannot be asked. Such a rubric used to be
    refused by jud RUBRIC outright; jud eval and jud record already
    took a case's own options, and jud lower already took --options.

  • Jev through OpenRouter's System One API is a backend: base URL
    https://openrouter.ai/api, an OpenRouter key. Usage::cost holds the
    usage.cost it sends, in US dollars (None from a server that sends
    none, and left out of a recording then); its generation id (gen-dec-…)
    becomes Response::request_id when there is no x-typesafe-request-id;
    its 402 for a balance that has run out is the new
    Error::PaymentRequired, not retried. mise run live:openrouter runs the
    live tests against it with OPENROUTER_API_KEY, under a new openrouter
    profile that expects its GET /v1/models to be OpenRouter's own
    catalogue; an unprofiled run against openrouter.ai stops before it
    asks, as one against the hosted API does. Checked against a stand-in
    built from OpenRouter's documented shapes; no live run is recorded yet.

  • mise run live:pplx runs the ignored live tests against Perplexity's
    pplx-decider-v1.1-27b through the autojev-serve its checkpoint ships,
    which serves /v1/systemone (PPLX_URL, default
    http://127.0.0.1:8010; PPLX_API_KEY also runs the bearer test). The
    model itself needs a GPU with about 49 GiB free; the server's wire was
    checked with the checkpoint's own server.py and answer() over a
    stand-in network, where all the live tests pass, with and without a
    key. Its departures from the hosted API, and why to name the model
    autojev-qwen3.8-27b though it answers to jev-latest, are in
    docs/project/verification/autojev-serve.md.

  • The live tests check every body they read as sent, a success, a
    refusal or the model list, against the vendored OpenAPI document and
    print the verdict, down to the field of the answer kind that fails; a
    status the document does not list (the hosted API's 400, laya-serve's
    413) is named as such. A success body that does not conform fails the
    run against the hosted API and autojev-serve, and is reported for any
    other server. A new test, every_body_the_server_sends_holds_to_the_published_document,
    sends the request shapes the builders produce and fetches the model
    list for it.

  • JUDGMENT_LIVE_PROFILE (typesafe, laya, autojev, generic) names
    the server the live tests run against, and each mise run live:* task
    sets its own. Where servers differ (a refusal's status, the limits, the
    budget) a behaviour is asserted on the profile it was observed on and
    printed on the others; laya-serve 0.3.24's statuses, as its verification
    record has them, and autojev-serve's are now pinned too. Unset, the
    profile is generic, except against the hosted API, where it stops the
    run. docs/project/verification/autojev-serve.md records what the
    autojev profile pins and how it was observed.

  • jud names every TYPESAFE_* or JUD_* variable it does not read
    (JUD_BASE_URL, TYPESAFE_MODEL): one line per variable on stderr
    before it asks a server and in jud record --dry-run, saying what it reads instead and which model and server it is
    asking, and ignored_environment in jud config. An unset base URL is
    the hosted API, so a misspelt variable used to send billed calls there
    without a word. Names only, never values.

Changed

  • jud record and jud eval write one line to stderr before the first
    call to a server, naming the model, the server and where its base URL
    came from (asking jev-latest at https://api.typesafe.ai for up to 7 requests (base URL from default)). A server run of jud eval used to
    leave stderr empty on success. Nothing changes under --replay.
  • The live tests no longer decide from the model name whether the hosted
    API's behaviour is asserted: a model whose name starts with jev was
    taken for the hosted API, which autojev-serve, answering to
    jev-latest, is not. mise run live:typesafe sets the typesafe
    profile; a hand run against the hosted API sets
    JUDGMENT_LIVE_PROFILE=typesafe to keep its pins.
  • Breaking: Usage has a third field, cost: Option<f64>, and is no
    longer Eq (it stays PartialEq, Copy and Default). A Usage { .. }
    literal adds cost: None.

Fixed

  • A base URL with a path lost its last segment: the client joined
    v1/systemone onto it as a relative URL, so https://openrouter.ai/api
    reached https://openrouter.ai/v1/systemone, and any gateway served under
    a path got a 404. The path is now kept, with or without the trailing
    slash, as the official SDKs keep it. A base that only worked because its
    last segment was dropped (https://api.typesafe.ai/v1, which reached
    /v1/systemone) now reaches /v1/v1/systemone; drop that segment.