v0.7.0

What this release is

The Responses surface now speaks OpenResponses — the specification
LM Studio (0.3.39+), llama.cpp, vLLM and SGLang serve at the same /v1/responses path — as well as
OpenAI's own Responses API, and the boundaries between jevper, the official SDK clients and the
caller's own input have been tightened where a body, a header or a field could produce a confident
answer from something the provider never said.

Every fix below is reproduced by a regression test; the server behaviour is measured on all five
local servers and recorded in docs/local-servers.md and the fixture suite.

The OpenResponses dialect

• Input turns carry the item type the specification's union requires. OpenAI accepts the
  {role, content} shorthand, but a server validating against the spec refuses every turn without
  it — and vLLM 0.30.1 renders a parts content list wrongly (the model receives the parts as
  objects), which the type-plus-string form jevper now sends sidesteps.
• Message content is read from every text-bearing part type the two specs allow (output_text,
  text, input_text), and reasoning text from all five part names either spec uses
  (summary_text, reasoning_text, output_text, input_text, text).
• Every logprob token is read from the bytes array beside it where the provider sends one, so a
  server whose tokens are byte-level (ĠA for " A") is read as the label it is rather than dropped
  as a non-label.
• A message item that reports itself incomplete is the truncation it says it is — the partial
  text is not read as an answer, and the error names max_output_tokens; an item still in_progress
  is a provider failure.
• When the spec's phase labels are present, only the final_answer messages are the answer;
  commentary is never concatenated into it.
• A response that streams an event stream where a response object belongs, and a null output, are
  each reported as what they are instead of as an unreadable answer.
• A 200 whose error is carried in the body is ProviderError.embedded, which keeps a body's own
  404 from being read as a missing route by api="auto".
• A refusal of the reasoning.encrypted_content include entry — a server that lists
  message.output_text.logprobs among the values it accepts — now drops only that entry and keeps
  the label readout the caller asked for.

The SDK boundary

• jevper is the only retry loop: a client passed with the official SDKs' max_retries=2 is used
  through a copy with it off, so one logical call is the number of requests RetryPolicy says and
  usage.n_retries counts them all. The caller's own client is untouched.
• The facades refuse a client of the other kind before any request: SystemOneClient with an async
  client, AsyncSystemOneClient with a blocking one, each naming the class to use.
• With strict response validation the SDK refuses a body before jevper sees it; a 200 carrying
  only a provider error is read from the exception's body, so a transient upstream stays retryable.
• extra_headers spelled differently from the client's own (authorization against
  Authorization) replaces the default header instead of joining it on the wire.
• extra_body may not carry model (pass model=) or ask for a stream; logprobs: false switches
  the Responses logprob fields off the way it does on Chat Completions; a caller's own input,
  store or top_logprobs is the value on the wire and is what debug records.

Answers and input that cannot exist

• A label readout whose alternatives are all null — or the sampled token repeated beside them — is
  absence of data, refused rather than normalized into a one-hot answer.
• A log probability that is positive, on any token, is refused instead of being dropped when it
  does not name an option.
• A second JSON object behind a brace of prose is still a second answer, and a valid object behind
  a prose brace is still the answer.
• A sampled token that contradicts an answer text naming a different label is refused whatever
  punctuation follows the label.
• Noul example keys are canonicalized: two spellings of one answer are refused, and a spelling the
  renderer could not find ("True", "yes") is refused before any request instead of raising a
  KeyError mid-prompt.
• A Chat Completions answer whose finish_reason is null or absent is not an answer; a message
  that is null is no choice; a streaming chunk where a response belongs is named as one.
• Text that cannot be encoded as UTF-8 — an unpaired surrogate, which JSON can carry as a \udXXX
  escape — is refused in the state, a question, prompt_cache_key, extra_body or the model id
  before any request, and a provider string that carries one never reaches the public response.

Surfaces and debug

• api="auto" follows the documented order — Responses, Chat Completions, Messages — so a
  hybrid client with no Chat Completions route now reaches Messages after a 404, and a pinned
  logprobs stays a provider error there because Messages carries no logprobs.
• A server that failed every attempt at a pinned method="logprobs" call is a ProviderError, not
  a logprob verdict that would move the call to another surface.
• debug["apis"] and debug["server_limits_by_api"] appear when concurrent questions were answered
  on more than one surface, so the per-question truth is readable next to the scalar api.

Verification

• 638 tests (4 skipped) in the full suite, 55 in the MLflow suite, and ruff check src tests with
  only the three known PYI034 hints.
• A raw-HTTP and end-to-end sweep on ollama 0.34.3, llama.cpp b11139, LM Studio 0.4.1, vLLM 0.30.1
  and SGLang 0.5.20: 31 scenarios per server, the OpenResponses request shapes jevper sends, and
  the recorded bodies replayed as fixtures.

The label readout on the OpenResponses logprob carrier was exercised live on every server that has
one: LM Studio, vLLM and SGLang each returned a real distribution over the options (confidence above
0.9999 for the top label on a clean state; SGLang's 1.1 MB token list including the thinking trace
read correctly), ollama's logprobs: [] and llama.cpp's 400 were reported as the absence they are.
The item shape jevper now sends is accepted by all five, and a caller-set max_output_tokens: 8
produced IncompleteAnswerError naming max_output_tokens on vLLM and SGLang, and the malformed
answer the other three's completed bodies actually are.

jevper 0.6.1

0.6.1 — every shape a provider can send

A six-way review of 0.6.0, a 301-case exerciser that drives the real OpenAI and Anthropic SDKs against
pathological responses on all three surfaces, and a fresh five-server live sweep. Every fix below is
reproduced by a test; each was a wrong answer, a crash outside the JevperError hierarchy, or advice
that could not be followed.

Silent wrong answers — the worst class

• A 200 carrying a provider error beside a readable answer was read as the answer. The error wins
  now, and its code is read as a number or a digit string, so an overloaded upstream stays retryable
  (OpenRouter sends "503" as often as 503).
• A sampled token that contradicts an answer text naming another label returned the token's label. The
  two are two views of one generation; when they disagree, neither is trusted.
• An answer holding two JSON objects was read as the first, with no sign the model had answered twice.
  A second decodable object is malformed now; prose with braces is still tolerated.
• A safety filter (finish_reason: "content_filter", a Responses incomplete_details.reason of the
  same name) arrived as IncompleteAnswerError, telling the caller to raise an output budget that was
  never the problem. It is a refusal in everything but the word, so it is ModelRefusalError now.
• Alternatives that are all unusable (logprob: null) counted as a distribution and produced
  confidence 1.0 from nothing; a positive logprob — probabilities sent where logprobs belong — was
  exponentiated into a confident split. Both are refused with the number that gave them away.

Retries, to the rule the SDKs use

• 409 and every other 5xx were missing from the transient set, so an OpenAI lock timeout or a 501
  killed a call the official client would have retried. x-should-retry now outranks the status in both
  directions, as it does in OpenAI's and Anthropic's SDKs.
• A class whose name merely contains Connection was retried; transport types are matched whole, so a
  caller's ConnectionProgrammingError stays a programming error.
• A Retry-After no runtime can sleep (2**40 seconds raises OverflowError on macOS) reached
  time.sleep; one it can sleep but nobody sane sends could park a call for 136 years. The ceiling is
  now a day — longer than any provider asks, including a quota reset — and past it the curve answers.
  Delta-seconds are read as the 1*DIGIT the grammar defines, and headers may be a mapping, an
  httpx.Headers, or (name, value) pairs.

Crashes and escapes

• A malformed examples container or element, a float("inf") status, an unhashable stop reason, a
  5000-digit integer in an answer, and an exception whose __str__ raises all left the hierarchy.
• A closed SDK client is a ClientCapabilityError; a model_dump that raises no longer discards a
  good answer; a choice whose dump is partial falls back to its attributes; a legacy choice.text is
  read instead of accepted and then dropped.
• A negative token count is reported as unreported, and two of them can no longer cancel into a
  plausible total; a megabyte of gateway HTML is quoted once, briefly.
• A two-step analysis cut off by the output budget was quoted into the answer prompt as a trace.

extra_body, where a caller's field met jevper's

• extra_body={"logprobs": True} dropped top_logprobs, so a caller who asked for a distribution got
  the sampled token alone. logprobs: false still turns both off, because top_logprobs alone is a 400.
• A caller's own thinking.budget_tokens on the Messages surface left max_tokens at 1024 — a request
  Anthropic refuses before generating. jevper sizes for it, leaves the temperature out beside it, and
  refuses locally when the caller's max_tokens cannot hold their budget.

Also

• A truncation error names the knob that opens it: max_output_tokens on the Responses surface, which
  refuses a max_tokens it does not know, and max_tokens elsewhere.
• Example probability keys that collide after normalization ({1: 0.9, "1": 0.1}) and an empty Noul
  mapping are refused before a request instead of silently dropping a number.
• Docs corrected: the retry rule, the truncation and refusal advice, the structured-schema bounds, the
  example-label precedence, the ReasoningConfig signature. uv.lock no longer records 0.5.3.

560 tests pass, 4 skipped; the MLflow suite is 55. Five local servers re-measured on the new code.

jevper 0.6.0

jevper 0.6.0 — parity with the System One contract, and the six ways the wire around it was wrong

The reference adapter TypeSafe ships for this format (system-one-adapter 0.2.1, 2026-09-22) exists to answer
the same questions jevper answers over the same three surfaces, so it is a second source of truth next to the
documented contract. Diffing against it — plus the official Python SDK 0.7.1 and the API docs — closed seven
gaps. Then auditing the result, with the help of five live servers, turned up six more, all of them in the
space between the contract and the SDKs it runs on.

Everything here is pinned by a test (488 pass), and every claim about a server was measured on ollama 0.34.3,
llama.cpp b11139, vLLM 0.30.1, SGLang 0.5.20 and LM Studio, all three surfaces, one box.

Parity with the contract

• A score is an expected value, so it is read off the distribution rescaled to sum 1, as the reference adapter
  does. With normalize_probabilities=False the reported probabilities stay the model's own numbers while
  the score stays on the 0..N-1 line the Jev answer schema documents.
• A generation the provider cut short is no longer read as an answer: IncompleteAnswerError (a length
  finish, a spent max_tokens, a spent context window, an incomplete Responses status) and
  ModelRefusalError are ProviderError subclasses raised before any readout, and neither spends a
  corrective retry. A 200 that carries no answer is a terminal failure, not malformed JSON to be corrected.
• A Responses call whose status is failed or cancelled is a provider failure too. It used to be read as
  an empty answer — one corrective retry spent re-asking a request that was never going to arrive — and a
  failed generation that still carried parseable text was reported as an answer.
• RetryPolicy honors Retry-After and retry-after-ms by default, the way the TypeSafe clients do: in
  seconds or as an HTTP date, matched case-insensitively as HTTP requires, with respect_retry_after=False
  to keep the curve alone. A date already past is a wait of zero, and a delay past what a runtime can sleep
  (MAX_RETRY_AFTER) falls back to the curve rather than answering OverflowError from time.sleep. A date
  in 2099 is still waited out in full — the bound is representability, not a policy cap, so max_delay still
  does not apply to a header jevper honors.
• The Messages surface sends Anthropic's own output_config.format, the counterpart of response_format, and
  keeps the JSON Schema in the system prompt as well: measured across the fleet, vLLM enforces the field
  (a schema naming a constant the prompt never mentions comes back with that constant) while llama.cpp, LM
  Studio and ollama accept it and ignore it, and a server that discards a field it accepted is
  indistinguishable from one that never read it.
• Choice accepts one option. The Jev API documents a 255 maximum and no minimum, and the reference adapter
  answers a one-option Choice with confidence 1.0 — including through a pinned logprobs/grammar readout,
  where there is no rival token to read and the sampled label is the answer. Score keeps its 2–10 levels,
  where the API does ask for two.
• A state handed over as one value is quoted between <document> markers with its angle brackets escaped, and
  every system prompt says the state is untrusted data — the posture the reference adapter takes, and the
  reason a state cannot close its own quote and continue as prompt text. A chat-list state keeps its roles,
  because the turns are already its boundary; a system/developer turn hoisted out of one is quoted on the
  way in, since moving it is a position fix and not a declaration that the caller's state is trusted. A list
  whose elements are not dicts ([1, 2]) is content, not a broken conversation, and is quoted like any other
  JSON value.

What the audit found around it

• output_config now travels in the request body rather than as an SDK keyword. The oldest Anthropic SDK
  jevper supports (anthropic>=0.49) has no such parameter, so every schema-bearing Messages call on it
  raised TypeError: create() got an unexpected keyword argument before a request — and no fallback ladder
  recovers from a local TypeError. Verified end to end against a real 0.49.0 client.
• The schema in that field is rewritten for Anthropic's documented subset. That API rejects numerical
  constraints, and jevper's own schemas bound every probability at minimum: 0 and every Noul answer from 0
  to 1 — so the field added to make native structured output work would have earned a 400 on the one
  provider it exists for, and the ladder would have remembered the field as unsupported for the rest of the
  client's life. Each bound moves into the description of the field it bounded; the prompt keeps
  minimum: 0, where text can say what the wire cannot.
• extra_body reaches the Messages request on every path. It was attached only inside the temperature branch,
  so a caller's own field was silently dropped unless a temperature was sent alongside it — which killed the
  documented chat_template_kwargs thinking-off knob exactly when it is needed. Proved live: the knob
  arrives with no temperature, and so does an 8-token budget, which is how the truncation is observable
  rather than assumed.
• top_logprobs is refused below 0 for every method. The upper bound was enforced everywhere and the lower one
  only for a pinned label readout, so method="auto" passed a negative count to the provider; the docs
  already promised [0, 20].
• model_context_window_exceeded keeps its class and its terminality but stops being told to raise
  max_tokens: the request is already too long to answer in, so the message now names the state and the
  examples as what to shorten.
• debug["server_limits"] is derived from the Limits dataclass, so a new rung cannot be forgotten, and
  the docs now say what the fleet actually does with the Messages schema field rather than what was
  inferred from a field it does not model.

Docs for all of it: api.md,
methods.md,
local-servers.md.

jevper 0.5.3

jevper 0.5.3 — a distribution is credited to the surface that gave it, a refusal is read per surface, and your worker threads inherit your context

Fourteen bugs, found by auditing the request lifecycle, the wire builders, the public inputs and the MLflow integration against their own sources, and confirmed live against ollama 0.34.3, llama.cpp, vLLM 0.30.1, SGLang 0.5.20 and LM Studio on one box plus OpenRouter's free tier.

Tracing

• The questions of one call are answered on a thread pool, and each worker now runs a copy of the caller's context. An enclosing @mlflow.trace span (or any OpenTelemetry span) is the parent of every question's SDK span: one trace per call with one child span per question, instead of one question nesting and the rest landing as roots. This is MLflow's own documented propagation pattern, applied for you. An active run is MLflow thread-local state rather than a context variable, so mlflow.start_run() is still not attached to those spans — docs/mlflow.md says so.
• tests/fixtures/jevper_langchain_model.py can be logged without a model_config. MLflow deliberately refuses the ModelConfig read in that case, which made the fixture impossible to log until something had already been logged.

Reading a provider's refusal

• A Responses server that refuses the logprob carrier by name is read as the carrier refusal, whichever words it uses: OpenRouter's Invalid option: expected one of … and OpenAI's Unsupported parameter: 'include' is not supported with this model. Both move the readout to the surface that carries it, under api="auto" with an explicit method="logprobs" as much as with auto.
• The carrier is recognised per surface. include is the carrier on Responses only; on Chat Completions a message that merely mentions include is about something else, and is no longer mistaken for a refusal of the readout.
• The include ladder drops the reasoning entry first only when the server refuses the field. A server that has listed the includables it offers, and left the logprob one out, gets its verdict at once instead of after a request that would earn the identical refusal.
• A refused second surface is remembered like any other verdict, so a later method="auto" call does not spend a request rediscovering it.

State the questions share

• Concurrent downgrades merge instead of overwriting: one question learning that a field is refused is no longer undone by another question's later write, and the next call stops paying that refusal.
• A readable distribution credits the surface that produced it, not whatever surface the shared context happens to hold when the answer comes back — so a Responses distribution no longer teaches Chat Completions that it can carry one.

The Messages and Responses surfaces

• A temperature is left out when thinking is on: the API refuses a temperature that is not its default alongside a thinking budget, and jevper no longer builds a request it knows will be rejected.
• A caller's max_tokens that cannot hold the thinking budget they asked for is refused locally, naming both numbers, instead of being sent to earn the 400.
• A Responses refusal content part is reported as a refusal, the way the other two surfaces' own shapes already were — previously it read as "no JSON object in the answer".

Inputs the library owns

• A count option has to be an integer. top_logprobs=2.5 and n_retry_malformed=0.5 used to construct cleanly and then raise a bare TypeError from inside the retry loop; top_logprobs="3" raised one from the range check itself. One line naming the option, at construction.
• model has to be a non-empty string, at construction and as a per-call override. It is the key every learned verdict and the derived prompt-cache key hang on, and a number reached that indexing as an AttributeError on .encode(), long after construction and from a frame the caller did not write.
• ChoiceAnswer and ScoreAnswer hold confidence to the bounds the readouts already compute within, and Usage cannot count calls backwards or report a negative or infinite latency. The probabilities beside them stay the provider's own numbers: normalize_probabilities=False promises them verbatim.
• Two documentation claims the code contradicts, corrected: docs/local-servers.md said discrete reads a one-token answer (it asks for a JSON object), and the README called response.model the model actually used (it is the id jevper asked for, which is also what the per-model verdicts are keyed on).

Verification

444 tests pass (up from 418), plus the 55-test MLflow suite on MLflow 3.16.1, a 66-case edge sweep, and a mutation check over the changed paths. Every fix here has a test that fails without it. The per-server facts, live-recorded this round, are in docs/local-servers.md.

jevper 0.5.2

A readout the provider refuses moves where it can still be read, and MLflow is covered end to end

0.5.1 audited every surface. This release carries two fixes for the same shape of problem — what jevper does
when a provider answers, and the answer is not the one that was promised — plus the work that followed it: an
integration suite and a guide for running jevper under MLflow's tracing, its hosting surfaces and its
evaluation. Both fixes were found by running jevper against real servers rather than stubs.

A refused label readout now moves instead of giving up

OpenRouter's Responses API rejects the include entry a logprob request carries, with a 400 that never says
the word "logprob" — it names path: ["include", 0] and the values it would have accepted. With the default
method="auto", jevper already recognised the shape, marked the surface and asked again on Chat Completions,
which answers with a full distribution. With method="logprobs" asked for explicitly, it did not: the call
failed on the provider's own 400, even though the other surface would have produced exactly the readout the
caller asked for.

The surface move is a verdict about the surface, so it now happens however the method was chosen — the same
question, the same method, one request on the surface that can carry it, and the surface is marked so later
calls start there. What an explicit method still does not get is a silent swap: with nowhere to move — a
client that speaks one surface, a route already known to be missing, or grammar, which is a Chat Completions
convention with no counterpart — the provider's refusal is reported rather than answered with a different
readout.

A missing route is remembered, not assumed

With a client that has only messages — no responses, no chat.completions — and a server with no
/v1/messages, the first call under api="auto" reported the provider's 404. The second raised

ProviderError(status=None): AttributeError: 'MessagesOnly' object has no attribute 'responses'

The 404 is remembered per (model, surface) so later calls start where the first one landed, and that is the
optimisation the second call then acted on alone: it moved the call to the other surface without asking
whether the client could speak it. Both halves of the question now go into the decision. A remembered
verdict only ever skips a route, so a client that cannot move stays where it is, pays the 404 again, and
reports the same provider error the first call did. An explicit api="messages" was always unaffected.

MLflow

docs/mlflow.md and tests/test_mlflow.py cover the two ways MLflow and jevper meet, verified against
MLflow 3.16.1:

• Tracing. mlflow.openai.autolog() patches the SDK's resource classes, not the constructors, so a
  jevper client built around an openai.OpenAI is traced without jevper knowing MLflow exists.
  mlflow.anthropic.autolog() does the same for Messages.create. The docs record what each span carries,
  that one SDK call is one trace, and which failures are error spans — a refused request, but not a refusal
  the provider answered with 200, which jevper reads as a malformed or unreadable answer and MLflow sees as
  a successful span.
• Hosting. pyfunc.ResponsesAgent is the current recommendation and the fixture implements
  predict_stream and the Message input shape; pyfunc.ChatModel is deprecated since 3.0.0 but still
  tested, since callers still ship it. LangChain v1 needs models-from-code, with the file calling
  mlflow.models.set_model.
• Gateway. A real gateway in front of a local server: routes served (no /v1/responses), unknown paths
  answered 404 {"detail":"Not Found"} — which jevper reads as a missing route — and responses re-shaped
  into MLflow's own schema, which drops logprobs and reasoning_content while finish_reason, refusal
  and usage details survive.

pip install 'jevper[mlflow]' brings in mlflow[gateway,langchain]>=3.16 for the suite; jevper itself still
imports nothing from MLflow, openai or anthropic, and the caller supplies the client. The suite skips
cleanly without MLflow installed.

Both formats, on one real server

LM Studio answers all three surfaces jevper drives — the OpenAI Chat Completions and Responses formats
and the Anthropic Messages format — so it is where both formats were verified against the same live server,
with a wire capture of what jevper actually sends. Verified live, on a 4B model:

• The whole surface × method grid: auto, structured, discrete and logprobs on each surface, three
  states each, with the labels coming back right (the two misses were the model mislabelling a state, not the
  library); Score and Noul questions on all three; the async driver on all three.
• auto picking the surface for every client shape — chat-only, responses-only, messages-only, and a client
  that speaks all three — and logprobs refused on Messages before any request is sent.
• Reasoning on both formats: Anthropic-shaped thinking blocks with an explicit budget (1542 characters of
  trace), native reasoning on the Responses surface, two_step on Chat Completions and Messages.
• A forced tiny output budget: the error names the budget on every surface (… ran out of output tokens before the answer was complete ('length' | 'max_tokens')).
• The surface-flip fix shipped in this release, against a real server with no /v1/messages route: both
  calls report the provider's 404.
• The MLflow integration itself, in four parts, against LM Studio: tracing with real usage counts, the
  gateway in front of it, an evaluation whose judge was the local model (correctness/mean: 1.0), and a
  jevper-backed model served and invoked through mlflow.models.predict.

Two things LM Studio does that a duck-typed client must know, both now in docs/local-servers.md: its
Responses route accepts a strict text.format schema and ignores it — verified with a schema whose keys the
prompt never named, which came back as the prompt's own shape on Responses and as the schema's on Chat
Completions — and an unknown path answers 200 with an error body rather than 404, so route discovery
cannot catch a typo'd path there.

Verified

Offline: pytest -q 418 passed, 4 skipped (from 408/3); the MLflow suite 53 passed in a Python 3.13
environment with MLflow 3.16.1; ruff check src tests reports the three documented PYI034 hints and nothing
else; a 66-case adversarial sweep fails none; the mutation check's 68 mutations each revert a fix, fail the
test that pins it, and restore the source byte-identical; and the 571-scenario prompt baseline diffs in
zero ways against the pre-fix source.

Live, against the MLflow gateway (no /v1/messages, no /v1/responses) sitting in front of a local server,
for the missing-route fix:

Call	Before	After
1st, api="auto", messages-only client	ProviderError status=404 :: the server has no 'messages' route	same
2nd, same client	ProviderError status=None :: AttributeError: 'MessagesOnly' object has no attribute 'responses'	ProviderError status=404 :: the server has no 'messages' route
api="messages", explicit	the provider's 404 NotFoundError	unchanged

Live, on OpenRouter's free tier, for the refused-readout fix — a provider that serves a real /v1/responses
and refuses the logprob include with 400 Invalid option: expected one of … at path: ["include", 0]:

Configuration	Before	After
api="auto", method="auto"	answered with method="logprobs" on Chat Completions	unchanged
api="auto", explicit method="logprobs"	ProviderError 400 :: Invalid Responses API request	answered with method="logprobs" on Chat Completions, a real distribution
api="responses", explicit method="logprobs"	the provider's 400	unchanged — an explicit surface stays a decision, and grammar still reports its refusal

The five published-wheel smoke scripts pass against a locally built 0.5.2 wheel, and are re-run against PyPI
once the release is published.

Tests

408 → 418 passing (4 skipped). The refused-readout fix is pinned by five tests: the explicit method moving
to the surface that carries its distribution, the move being remembered so the refusal is paid for once, the
readout not being swapped for another method when there is nowhere to move, an explicit api="responses"
still reporting the provider's refusal, and the async driver doing the same. The missing-route fix keeps its
five, and docs/methods.md states both invariants.

The MLflow suite adds 53 tests over the three hosting flavours, autolog tracing on both SDKs, the gateway
contract, evaluation, and the fixture behaviours an adversarial review of the first draft found: a dropped
request state behind a tautological assertion, a missing predict_stream, a dead fallback in the
ModelConfig read, retry ordering under tracing, mid-run store switching, two experiments in one process,
a chat-only duck client, non-ASCII through the gateway, undrained pipes and port races.

jevper 0.5.1

An adversarial pass over every surface: fifteen edge cases, found by three independent audits and by running the fleet

0.5.0 put a third surface in and shipped. This release is what an audit of the whole library — two code
reviews, a test-coverage review, and research into what the SDKs and servers actually do today — turned up,
plus what running the same scenarios against ollama, llama.cpp, vLLM, SGLang, LM Studio and OpenRouter's free
tier afterwards confirmed. Nothing about the public API changed; every fix makes an existing promise true.

Answers that were never read, or read wrong

• A refusal read as malformed JSON. OpenAI reports a safety refusal in a refusal sibling of a null
  content, and the Messages API as stop_reason: "refusal". Both reached the caller as "no JSON object in
  the answer" — true, and useless. The model's own words now travel with the error.
• A spent output budget, unmentioned. _stop_note knew Chat Completions' length and the Responses
  surface's max_output_tokens, but not the Messages API's max_tokens — the surface 0.5.0 added — nor
  model_context_window_exceeded. Worse, two of parse_json_object's three failure paths dropped the note
  entirely, so an answer cut off mid-object — the exact shape a spent budget produces — said only "could not
  parse a JSON object". Live on ollama's and LM Studio's Messages route the message now reads the provider ran
  out of output tokens before the answer was complete ('max_tokens'); raise the limit, where before it named
  the stop reason and nothing else.
• A reasoning item that broke the answer. _as_mapping promises that an unreadable provider object maps
  to nothing; a duck-typed client whose model_dump() answers None went straight into a .get and took an
  otherwise readable answer down with it. Reasoning is decoration, and now behaves like it.
• A choice that is not a choice. A 200 whose first choice is null or {} was read as a blank answer,
  hiding an embedded provider error — which is how OpenRouter reports an overloaded upstream. It is now read
  as a body with no choices, so a transient 503 stays retryable instead of becoming a malformed answer.
• SGLang's reasoning count. SGLang's Chat surface reports usage.reasoning_tokens at the top level rather
  than inside completion_tokens_details, and the count was silently lost. Both places are read now.
• An OverflowError out of a token count. int(float("inf")) raised past _add_count's guard and escaped
  system_one as a raw Python error. An unreadable count is "not reported", as documented.

A caller's own request fields

The SDK merges extra_body into the body after the typed parameters, so a key it names is the value that
reaches the wire — and jevper was building its request as if it were not. Two consequences, both fixed:

• A caller's prompt_cache_key survived the cache-key downgrade, so the "re-ask without it" the ladder
  promises sent the same bytes again and the call failed on a field jevper was supposed to have removed. A
  capability field the server has refused is now dropped from the caller's body too.
• A caller's response_format overrode the strict schema while the builder believed the schema had been sent
  — so the schema reached neither the request nor the prompt, and the model answered whatever shape it liked.
  A format field the caller names is now the effective one, and the schema travels in the prompt beside it.

Verdicts that belonged to the wrong request

• A delayed failure was judged against the wrong surface. Workers share one call context, and a failure
  was classified against whatever that context held when it was handled — so under api="auto" a Responses
  404 that arrived after another question had already moved to Chat was recorded as a Chat 404, and the
  surface that had just answered was written off for the rest of the client's life. Failures are now judged
  against the transport that issued them, the attempt record names that surface, and the same holds in the
  async driver.
• A downgrade inherited the old shape's retry budget. A request that had spent its retries, then been
  downgraded, then hit a 5xx, failed without the retries its new shape was owed. The counter resets at the
  downgrade.
• A refusal of the value was treated as a refusal of the field. budget_tokens: must be at least 1024
  and reasoning_effort must be one of low, medium, high name fields the server knows and numbers it will not
  take; dropping the field there answered the question with the caller's reasoning quietly switched off and
  remembered that as the server's limit. Value complaints now travel back as the provider's own error, and
  nothing is cached. The same distinction now separates a refused logprob include from a refused reasoning
  include, so a Responses request that asked for no logprobs descends the ladder instead of failing.
• A value-only logprob rejection moved the readout anyway, contradicting the rule that a bad number is not
  evidence about the provider. It no longer does.
• A same-surface downgrade undid the method auto had just chosen, because the method was reloaded from
  the client-wide verdict — which deliberately does not cache a first weak absence. Only a real surface move
  reloads it now.
• An exhausted surface fallback raised a private exception. When every route api="auto" could try
  answered 404, the caller got jevper's internal verdict instead of a ProviderError with the status and the
  attempt history. It is a provider failure, and it now reads like one.
• structured_outputs=False spent a schema rung it never sent: the first refusal changed the cached label
  from schema to object while the request already carried json_object, so the second request was
  byte-for-byte the first. The ladder now starts where the request actually is.

Prompts and routing

• A cache key was not method-specific. The derived key hashed the model, the demonstrations and the
  question block, but not the method — whose system prompt and answer shape are part of the prefix a provider
  caches — so a logprobs request and a structured request for one question shared a routing key. Both
  passes of a two-step call still share one, which is the routing that design wants.
• A budget resolved to a path that sends no budget. On the Messages surface ReasoningConfig(mode="auto", budget_tokens=…) resolved to the two-step path, which never sends thinking — leaving the field whose
  documentation promises it silently unused. A budget now selects the surface's own thinking, the way effort
  does on the Responses surface. Chat Completions is unchanged: auto stays two_step there.
• The default max_tokens refused the budget its own documentation recommends. Anthropic requires the
  thinking budget to be strictly below max_tokens, and jevper's Messages default was 1024 — the same number
  the docs call the floor. The default now grows by the budget; the answer keeps the whole 1024 and the
  thinking is paid for out of the extra.
• A state ending on the assistant's turn produced a 400. ollama and LM Studio — both llama.cpp engines —
  answer 400 Failed to initialize samplers: std::exception for a conversation that ends there, and a server
  reading it as a prefill would continue the assistant's turn instead of answering. The question turn follows
  such a state, so the call still ends on a question; every other state shape keeps the documented order that
  makes the prefix reusable.
• Examples were validated inside their own worker, so a multi-question call could spend provider calls
  before discovering a locally invalid example. All of them are checked before the first request.
• Explicit falsy overrides were ignored: method="", api="" and model="" fell back to the
  constructor's values instead of being validated. An unknown api passed per call is now refused too.

Research corrections

• vLLM does not refuse the thinking field, contrary to what 0.5.0's notes say: current vLLM's
  AnthropicMessagesRequest has no such field and pydantic ignores extras, so the field is silently dropped
  and the downgrade ladder never fires there. The live matrix confirms what each server does.
• A system role inside messages is no longer absent from Anthropic's API — mid-conversation system
  messages exist, on some models — but no server here implements them, so jevper still hoists one into the
  top-level system field. The rationale in docs/local-servers.md is now the accurate one.
• The Messages API has a schema field after all (output_config.format), but none of the servers here
  honour it, so the schema still travels in the system prompt, which works everywhere. Documented.
• max_tokens is deprecated in favour of max_completion_tokens on the OpenAI surfaces, and ollama and
  LM Studio document only the former — which is what jevper sends, through extra_body, where the caller
  controls it.

Verified

Offline: pytest -q 408 passed, 3 skipped (from 276); ruff check src tests reports the three
documented PYI034 hints and nothing else; a 66-case adversarial sweep fails none; a 64-mutation check
reverts each fix in turn, confirms a test catches it, and restores every source byte-identical; and the
571-scenario prompt baseline diffs in exactly two intended ways — every derived cache key, because the method
is hashed now, and the assistant-final reordering — with zero non-reorder prompt changes.

Live, against the fleet, on the fixed code:

Server	OpenAI-compatible surfaces	Messages route
ollama 0.14.0	18 edge scenarios, both models: all answer but the two that are meant to fail (a truncated answer, an over-long key)	11/11 on both models
llama.cpp b7187+	same, both models	11/11 on both models
...

jevper 0.5.0

jevper 0.5.0

pip install --upgrade jevper

The Anthropic Messages API

api="messages" speaks POST /v1/messages — the Anthropic-compatible route that every server in the local
fleet now implements (LM Studio 0.4.1+, llama.cpp b7187+, vLLM 0.11.1+, SGLang 0.5.9+, ollama 0.14.0+, and
OpenRouter). Point the anthropic SDK at the same host and port as the OpenAI one and pass it in place of
the OpenAI client:

from anthropic import Anthropic

from jevper import SystemOneClient

client = SystemOneClient(Anthropic(base_url="http://127.0.0.1:1234"), model="qwen3-4b-instruct")

client.system_one(state=record, questions=rubric, api="messages", method="structured")

No new dependency: the client stays duck-typed, exactly as the OpenAI one is, so anthropic is only needed
if you point jevper at it. api="auto" picks this route only when the client has nothing else, because no
server returns logprobs through it.

Four things differ from the OpenAI surfaces, and all four come from the protocol:

• No logprobs exist in it. method="logprobs" and method="grammar" raise UnsupportedMethodError
  before a request is sent; method="auto" answers in JSON without spending a call to discover that.
• max_tokens is required, with no server-side default anywhere that implements the API. jevper sends
  1024 unless extra_body={"max_tokens": n} overrides it.
• Thinking is a budget, not an effort name. ReasoningConfig(mode="native", budget_tokens=2048) sends
  thinking={"type": "enabled", "budget_tokens": 2048}; vLLM's protocol has no thinking field at all, so
  the field is dropped, the call re-asked, and the limit reported in debug["server_limits"]["thinking"].
• Thinking blocks arrive as ordinary response.reasoning parts with their signature kept, and
  usage.cached_tokens is read from cache_read_input_tokens.

The schema now reaches the prompt when nothing else carries it

The Messages API has no schema field — and neither does a server that refuses json_schema, the case jevper
already answered with a plain json_object. In both cases the structured system prompt said "matches the
provided schema exactly" while nothing provided one, which is exactly what a 4B model answers
{"intent": "A"} to: a real answer in the wrong shape, on every structured question. The JSON Schema now
travels in the system prompt whenever the request cannot carry it, and the same fix closes the older
json_object fallback, where the documentation already promised it.

Live before the fix, Qwen3-4B-Instruct-2507 through ollama's Messages route: every structured, auto and
thinking scenario failed with MalformedAnswerError on {"intent": "A"}. After it, all of them answer,
on ollama and llama.cpp alike, and so does OpenRouter's Messages route.

Reasoning and non-reasoning models, on every server

The Qwen3 4B 2507 pair — Instruct (non-reasoning) and Thinking — was run through the whole scenario
matrix on ollama, llama.cpp, vLLM and SGLang, plus a free reasoning model on OpenRouter, with the same
matrix on the Messages route.

The non-reasoning model passes everything the deployment supports: ollama 25 of 27 (the rest are ollama's
Responses route, which cannot carry logprobs, and a bad model id, which is a ProviderError by design).
The thinking model cannot have thinking switched off — its template says so rather than exposing a toggle —
so the label readout reads the reasoning span and answers in prose: structured still answers correctly
everywhere, which is the method to use for an always-thinking model. llama.cpp and SGLang separate their
reasoning and their logprobs cleanly, so logprobs keeps working there.

Three things the fleet showed that no protocol document says:

• A system role inside messages is tolerated, not defined. ollama, llama.cpp, vLLM and SGLang all
  answer 200 for one; the 400 System message must be at the beginning. belongs to the OpenAI surfaces.
  jevper hoists it out either way, so the request says what the API says.
• On vLLM and SGLang the reasoning parser swallows the answer when thinking is on, which is those
  templates' default: the whole generation arrives as a thinking block with no text block, and a structured
  call has nothing to parse — the error now says so. Disable thinking per call, exactly as on the other
  surfaces; with that, every scenario on vLLM's Messages route answers, on both models. SGLang needs the
  server-side decision too: with --reasoning-parser qwen3 and a non-thinking model, whose template never
  emits the marker the parser waits for, the whole generation is classified as reasoning.
  separate_reasoning: false does not change it; dropping the flag does.
• OpenRouter's /v1/messages accepts x-api-key. Its documentation leaves that unconfirmed and shows
  bearer tokens only; the Anthropic SDK's header works, and the Messages route answered billing there with
  the schema in the system prompt.

LM Studio, headless

LM Studio is deployed on the host as well, through llmster (0.0.25-1, the GUI-less daemon) rather than the
desktop app, and both of its surfaces are verified. Two things had to be worked out on the way, and both are
in the runbook above rather than in this library: the installer requires libatomic1, which a box without
root does not have — unpacking the .deb into $HOME and having ldconfig report it gets the install
through — and the daemon's bundled Node needs the same library at runtime, which a patched RPATH supplies
without any environment variable. The daemon also refuses to start while another LM Studio holds the core
services port, which under WSL's mirrored networking means a copy running on the Windows host.

Reached on http://127.0.0.1:1234, the OpenAI-compatible route answers 26 of 27 scenarios on the
non-reasoning model (the one failure is the model's JSON shape on the Responses route) and the
Anthropic-compatible route answers all eleven on both models — including the thinking field, which LM
Studio accepts and keeps separate from the answer, and cache_read_input_tokens, which it reports as 884 on
a warm call and as a plain 0 on a cold one.

Docs

README ·
api ·
methods ·
local servers ·
reasoning ·
few-shot ·
internals

jevper 0.4.0

jevper 0.4.0

pip install --upgrade jevper

A rubric's calls now reuse the provider's prompt cache. The state — the part that changes from call to
call — moved to the end of the message list, after the question block. A provider reuses a cached prefix only
up to the first token that differs, so with the state second every call about a new state reprocessed the
whole prompt. Measured on one 2388-token prompt carrying two examples, second call differing only in the
state, reused prompt tokens went from 40 — the system prompt alone — to 1010 on llama.cpp, 528 on vLLM and
896 on SGLang, and the state-varied second call became the fastest of the four layouts on every one of them
instead of the slowest. On a hosted API this is the difference between caching at all and not: OpenAI's
minimum cacheable prefix is 1024 tokens, so a prompt that only reuses its system message never caches there.

The message list is now [system] + examples + [question block] + state turns, and a demonstration has the
same shape as the real call: the question block first, then the example's state. A chat-list state is still
preserved verbatim, roles included. One exception, measured the hard way: vLLM and SGLang answer
400 System message must be at the beginning. when a system turn follows a user turn, so a state that
carries its own system/developer turn keeps the order you wrote — before the examples and the question
block — and gives up the shared prefix rather than the answer.

prompt_cache_key is sent with every request. Derived per question from the parts of the prompt that do
not change between calls — model, examples, question block, never the state — so every state classified with
one rubric carries one key, and a provider that routes by it keeps them on the same cache. That is exactly
what OpenRouter's sticky routing keys off (it otherwise only activates after a cache hit is detected), and
OpenAI uses the field to optimize routing on models before GPT-5.6. Pass your own on the client
(prompt_cache_key="tenant-42") or per call to group or account for requests your way; a server that refuses
the field gets it dropped and the call re-asked, like the other optional fields, with
debug["server_limits"]["cache_key"] recording it.

usage.cached_tokens reports the reuse. Read from prompt_tokens_details.cached_tokens on Chat
Completions and input_tokens_details.cached_tokens on the Responses surface, summed over the call like the
other token counts. None means the provider said nothing — vLLM needs --enable-prompt-tokens-details, and
SGLang's Chat Completions route needs --enable-cache-report — while a reported 0 stays 0: that is a
cache that is cold or off, which is a different fact.

What the four local servers do with it

All four accept prompt_cache_key, prompt_cache_retention, cache_salt, session_id,
prompt_cache_options and prompt_cache_breakpoint with a 200 and ignore them; prefix caching itself is on
by default everywhere. What they disagree on is reporting it: ollama and llama.cpp always send the count
(ollama also exposes prompt_eval_cached_count natively, llama.cpp timings.cache_n), vLLM and SGLang need
the flags above. cache_salt — the one cache field vLLM and SGLang do implement — stays a deployment
decision and goes through extra_body, because it isolates a cache rather than routing to one.

Smaller

• prompt_cache_retention is deliberately not implemented: OpenAI deprecates it in favour of
  prompt_cache_options, which is GPT-5.6-and-later only, and the default retention is already 24h for most
  organizations.
• A token count the provider sends in a form nobody can read — a word, an object, a list — is reported as
  None instead of raising ValueError out of the client.
• 237 tests pass offline (49 new), ruff is clean apart from the three documented PYI034 hints, the
  66-case adversarial sweep fails none, a 53-mutation check reverts each fix in turn, confirms the test that
  names it fails, and restores every source byte-identical, and the 571-scenario baseline shows the layout
  change is exactly a reorder: every state block moved verbatim, every example turn flipped, every system
  prompt and every recorded answer unchanged.

Live

Ran the full 27-scenario matrix on 0.4.0 against all four servers — ollama 0.34.3, llama.cpp b11139,
vLLM 0.30.1 and SGLang 0.5.20, the last two started with their cache-reporting flags — with the new order in
place. Every scenario passes except the rows that were already documented: ollama's and llama.cpp's Responses
route cannot carry logprobs, a bad model id is a ProviderError by design (ollama, vLLM) or a 200 with an
unreadable answer (llama.cpp, SGLang), and llama.cpp's Responses shim sometimes answers the structured
scenario with the question id instead of the probabilities object, which it also did before this change.

Two of those rows were the change's own doing, and both are fixed rather than shipped. The first was a
chat-list state carrying a system turn: llama.cpp's Qwen template raises System message must be at the beginning. and vLLM and SGLang answer 400 with the same words, because jevper's own system prompt is always
first. That content is now folded into the system prompt — it is an instruction, and dropping it would change
the question — and the scenario passes everywhere. The second is the layout itself, measured in every order
against a real 2388-token prompt before anything changed (table in
docs/local-servers.md).

The captured bodies show the reuse the field exists for — the same prompt twice, second call, of 1989 prompt
tokens: ollama 1985, llama.cpp 1985, vLLM 1584, SGLang 1984. SGLang's first, uncached call reports
prompt_tokens_details: null rather than a zero, which is the distinction Usage.cached_tokens keeps.

Docs: README · api · methods · local servers · reasoning · few-shot · internals

jevper 0.3.0

jevper 0.3.0

pip install --upgrade jevper

A server that refuses one of jevper's request fields no longer fails the call. Structured output, the
reasoning parameters and the Responses include list are each optional — the prompt already asks for one JSON
object and the readout validates it — so jevper drops the refused field, re-asks the same call, and remembers
the limit per surface: json_schema → json_object → no response_format at all, then reasoning, then the
include. The ladder is finite, so a server that refuses everything still ends in a ProviderError, and
debug["server_limits"] reports what it refused.

An answer that never arrived now says why. A reasoning model can spend the whole output budget thinking:
vLLM and SGLang answer status: "incomplete" with an empty message, llama.cpp and ollama finish_reason: "length" with nothing but reasoning tokens. That used to reach the caller as no non-whitespace token in the response — true and useless. The error now names the budget.

An end-of-turn token no longer hides the answer. vLLM and SGLang append their own <|im_end|> to the
logprob stream after the answer text, which broke the tail test the label readout anchors on: with thinking
on, both servers reported first non-whitespace token 'Thinking' is not one of the labels. Up to two trailing
tokens whose text does not occur in the answer are now dropped before the tail is tested — a token the
provider itself excluded from content cannot be part of the answer, and a token that could be is kept, so
this stays a removal rather than a guess.

A structural refusal is remembered; a bounded value is not. llama.cpp's Responses shim answers
top_logprobs requires logprobs to be set to true, which is a statement about the route, so auto writes
logprobs off there and answers in JSON from then on. ollama's top_logprobs must be between 0 and 20 is a
statement about the number, so the next call tries logprobs again.

The four servers, recorded

tests/fixtures/providers/ now holds what ollama 0.34.3, llama.cpp b11139, vLLM 0.30.1 and SGLang 0.5.20
actually answered — 52 raw-HTTP requests each, pruned to the fields jevper reads — and
tests/test_provider_surfaces.py replays every one of them through a real openai client. Among the shapes
they pin: an empty logprob array (ollama's Responses route), a 400 for any request carrying top_logprobs
(llama.cpp's), a model-naming 404 beside a route 404, reasoning versus reasoning_content, {"detail": "Not Found"} beside plain text and a bare JSON string, and two truncations.

docs/local-servers.md gains the matrix those recordings produce, and corrects three claims it had wrong:
llama.cpp does honour reasoning_effort, ollama ignores max_completion_tokens (bound output with
max_tokens), and SGLang rejects the developer role.

Smaller

188 tests pass offline (52 new), ruff is clean apart from the three documented PYI034 hints, the 57-case
adversarial edge sweep fails none, a 42-mutation check reverts each fix in turn, confirms the test that
names it fails, and restores every source byte-identical, and the 571-scenario request/response baseline is unchanged apart from
its one pre-existing row: nothing here changes what a working server is sent.

Live

Re-ran the full 25-scenario matrix on 0.3.0 against all four servers on one RTX 3080: ollama 23 ok,
llama.cpp 23, vLLM 24, SGLang 24. auto resolved a different surface per server — Chat Completions for
ollama and llama.cpp, Responses for vLLM and SGLang — and read a real distribution on all four.
thinking-default, the scenario that leaves thinking ON, now passes everywhere: it failed on vLLM and
SGLang before the end-of-turn tolerance, with first non-whitespace token 'Thinking' is not one of the labels. The non-ok rows are the documented ones — ollama's and llama.cpp's Responses route, which cannot
carry logprobs, and a bad-model control each server answers its own way (ollama and vLLM a 404,
llama.cpp a 200 because it ignores the id, SGLang a 200 and then an unreadable answer) — plus one
model-side flake: llama.cpp's responses-structured twice answered {'intent': 'billing'} instead of the
probabilities object, a scenario two earlier runs on the same code passed.

The route fallback that 0.2.3 could only test offline is proven live. llama.cpp behind a proxy that
answers 404 for /v1/responses: api="auto" re-asked on Chat Completions (retry reason the server has no 'responses' route) and read {billing: 0.999986, sales: 4e-06, technical: 1e-05}; api="responses"
pinned to the same server reported the 404 as a ProviderError rather than routing around it; and a 404
that named the model was reported instead of being mistaken for a missing route.

Docs: README · api · methods · local servers · reasoning · few-shot · internals

jevper 0.2.3

pip install --upgrade jevper

An audit against four local inference servers — ollama, llama.cpp, vLLM and SGLang, each serving
Qwen/Qwen3.5-9B at 4-bit on one 12 GB card — found three defects that only a real server exposes. All three
are in how the library finds the token distribution; none of them could produce a wrong answer, but two
could lose the distribution the caller asked for.

The surface the distribution is on

api="auto" prefers the Responses surface. An openai client object exposes responses.create whether or
not the server behind it implements the route, so against a server without one that call was a 404 wrapped in
a ProviderError. Under auto a 404 that does not name the model is now read as this server has no
Responses route: the call is re-issued on chat_completions and the verdict is remembered for the rest of
the client's life. A 404 that names the model is the model, and an explicit api="responses" is a decision
whose 404 reaches you unchanged.

The same preference has a second, subtler cost, and this one is live: ollama answers /v1/responses with an
empty logprob list, and llama.cpp refuses the logprob fields there outright
(400 top_logprobs requires logprobs to be set to true) — while both carry a full distribution on Chat
Completions. auto used to answer in JSON and never touch the logprobs sitting one route away. A surface that
cannot deliver a distribution now moves the label readout to the other surface once, and is marked so later
calls for that model start where the distribution is; a distribution arriving there later clears the mark. Two
things stop the move: reasoning="native", which exists only on Responses and would silently become a
two-step pass, and a provider failure that survived its retries, which says nothing about the surface.

The first token of the answer

Every one of these servers reports logprobs for every generated token, the thinking span included, while
message.content holds only the answer. The readout took the first non-whitespace token of the completion —
the first token of the reasoning — and raised LabelReadoutError, spending a corrective retry and raising
again. The docs have always said the readout reads the first token of the answer; now it does. When the
provider separates its reasoning and the answer text is exactly the tail of the token stream, the answer's
own tokens start at that offset. The anchor is strict on purpose: no separated trace, or no exact tail, and the
stream is read as it arrives, so a stream that cannot be anchored is reported rather than guessed at.

Smaller

• _switch_surface re-resolves the method for the surface it moves to: the method verdict is keyed by
  (model, surface), so returning to a surface known to withhold logprobs no longer asks for them again.
• docs/local-servers.md is new: per-server base_url/model/thinking-off recipes, what each server
  silently ignores (grammar is llama.cpp-only, strict is ignored by ollama), what fits a 12 GB card
  (--language-model-only for vLLM, a higher --mem-fraction-static for SGLang, which cannot drop the vision
  tower at all), and the one field worth setting on a small GPU: extra_body={"max_tokens": N}, because
  neither builder sends one and a thinking model will spend it on the reasoning first.

Verified

136 tests pass offline against a stdlib HTTP stub driving a real openai SDK client — 17 new, each confirmed
to fail with its fix reverted by a 37-mutation check that restores every source byte-identical. ruff is clean
apart from the three documented PYI034 hints, the 57-case adversarial edge sweep fails none, and the
571-scenario request/response baseline is byte-identical to 0.2.2: every change here is about what the client
does with a response, not what it sends.

Live on one RTX 3080, each server on its own 4-bit copy of Qwen3.5-9B, full 25-scenario matrix per server:
ollama 0.34.3 (qwen3.5:9b, Q4_K_M, 100% GPU), llama.cpp b11139 (34/34 layers offloaded, 5614 MiB,
86-97 tok/s), vLLM 0.30.1 (compressed-tensors AWQ, --language-model-only, 1.88 GiB KV) and SGLang 0.5.20
(same checkpoint, --mem-fraction-static 0.85 with Triton/pytorch attention and sampling backends, because
FlashInfer's own JIT cannot build on this host while SGLang's Triton kernels can). All four landed on 23 of
25 scenarios. The non-ok rows are the deliberate bad-model control — a clean 404 on ollama and vLLM, and on
SGLang a 200 because it does not validate the model id — plus one server-specific fact: ollama's and
llama.cpp's Responses routes, which cannot deliver logprobs (the documented pinned-surface behaviour), or
vLLM's and SGLang's thinking-default, whose reasoning consumed the whole token budget before any answer was
emitted, leaving nothing to anchor to.

The two surface-and-readout fixes show up in single lines. auto-defaults on ollama: 0.2.2 answered from the
Responses surface in JSON (method: "structured", no distribution at all) while 0.2.3 reports
api="chat_completions", methods == {"intent": "logprobs"} and
{billing: 0.999935, technical: 8e-06, sales: 5.7e-05} — the same distribution Chat Completions had all
along. thinking-default — the one scenario that leaves thinking ON — read the answer's own first token out
of 428 tokens of reasoning on ollama and 387 on llama.cpp, where on 0.2.2 nine of ollama's twenty-five
scenarios failed with first non-whitespace token 'Thinking' is not one of the labels and twelve failed
overall.

Docs: README · api · methods · local servers · reasoning · few-shot · internals