You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Fixed
Inference.complete reads a provider's response by shape, not by position (#1333). examples/inference.vera printed 'text' and exited 1 against the Anthropic flagship. The Messages API returns content as a list of TYPED blocks and a reasoning-capable model leads with a thinking block, so _call_inference_provider's data["content"][0]["text"] landed on a block with no "text" key; the resulting KeyError('text') reached the host boundary's blanket except Exception, and str(exc) published the bare missing key as the entire Result::Err payload — a message naming neither the operation, the provider, nor the model. The maintainer's six-provider sweep found five healthy (openai, moonshot, mistral, xai and deepseek all take the OpenAI-style branch, which carries the file's only other response parse) and initially read the failure as a SECOND affected provider: with VERA_INFERENCE_PROVIDER unset, auto-detect takes the first key set to a non-empty value in registry insertion order, so a still-exported VERA_ANTHROPIC_API_KEY won the "xAI run" and the Anthropic parse failed under another provider's name. Three repairs, one per defect. Selection by type: the Anthropic branch collects every block whose type is "text", in order, and joins them, so a leading thinking or tool_use block is skipped rather than mis-read. The OpenAI-compatible branch accepts output_text beside text as a part discriminator, in PREFERENCE order rather than as a union — a gateway that mirrors the same reply under both spellings had every fragment counted twice and returned PositivePositive, a wrong answer delivered as a success, so the first discriminator that yields TEXT wins and the other is not consulted. A hit is a non-empty result, not a non-empty list: treating [""] as one let an empty text part shadow a real output_text part and return Ok("") in either order, while a list whose every part is empty still returns the empty completion it is. Spec 9.5.5 states the preference; the sentence there described joining both, which a review pass caught — Responses-API-shaped gateways spell it that way, and they worked on v0.1.12 because the old code read content positionally and never looked at type at all, so selecting by type alone regressed them — and surfaces message.refusal when the model declined, which is the answer where the shape of the empty content beside it is only the symptom — including when that content is an empty or whitespace-only string, where Ok("") told the caller the model said nothing when it had said why it would not. The rule is symmetric across the branches and covers both signals each spells: an empty or whitespace-only completion is an error when the provider marked the turn a refusal (stop_reason: refusal, or message.refusal present) or a truncation (stop_reason: max_tokens, finish_reason: length) — the latter being #1333's own species, a thinking block exhausting the budget before any text was emitted. The reason is matched case-insensitively — every registered provider emits these lowercase today, but an exact match sent a normalising gateway's MAX_TOKENS straight to Ok(""), losing the answer over a spelling difference — while the diagnostic carries the token exactly as received, so stop_reason=refusal and finish_reason=length are greppable and a consumer is never told the provider said something it did not. An empty completion under any other reason, or none, stays Ok("") exactly as in v0.1.12: a model may legitimately answer with nothing, and the review's wider proposal to treat every empty completion as missing is declined as a behaviour change unrelated to #1333. A non-empty reply is returned unchanged whatever the reason. The blank test is .strip() throughout: the list path tested truthiness while the check downstream tested .strip(), so a whitespace-only text part short-circuited the loop — shadowing a real output_text part and skipping the refusal check that the string form applied, which is why round 9's "string and list forms" held for the list form only when the fragment was exactly "". A selected block's text must itself be a string — the PR review found the same silent-wrong-answer class one level deeper, where {"type": "text", "text": null} was coerced by str() into the successful completion "None", a number into its digits, and an object into a Python repr; a non-string text now names its own type, as does a selected block carrying NO text field. Both refuse, on both branches. (Before this release the key-less block was instead skipped, so [{"type": "text"}, {"type": "text", "text": "Positive"}] returned Ok("Positive") while the same pair with null refused — two malformed shapes treated differently for no reason a caller could see. Blocks of a type other than the selected one, such as thinking or tool_use, are still skipped, by design — but if that leaves no block or part of the selected type at all, the response is an Err naming the types that WERE present and, when the provider sent one, its reason, which spec 9.5.5 now states beside the skip rule rather than leaving to be inferred from it.) Either way the refusal wins over any salvageable block after it, because joining the remainder would return a completion the provider never sent as a whole with no way for the caller to tell it was short. The OpenAI-style branch gets the same treatment, because message.content is a string on an ordinary turn, a list of typed parts on some multimodal ones, and null on a reasoning or tool-call turn — where the old str(...) returned the literal completion "None", a silent wrong answer rather than merely a bad message. Named failures: every shape failure reports the provider, the model that answered, and the block or part types it actually saw — and, for the Anthropic no-text case, the response's stop_reason (content block types: thinking; stop_reason=max_tokens), which separates a reply truncated at the request's token budget from a model that simply said nothing — all of which is what makes the sweep's misattribution class impossible to repeat; a urllibHTTPError — which escaped as the status line HTTP Error 401: Unauthorized, naming neither the provider nor the reason it gave — now reads the error body and quotes the provider's own error.message, falling back to the raw text truncated so a proxy's HTML page cannot become a Vera value. EVERY provider-supplied fragment an Err quotes is redacted before it is surfaced — the rejection body, a 200 body that is not JSON, a stop_reason or finish_reason, a refusal, and the key and type names a shape report lists. A review pass found the non-JSON body building its message with no redaction at all; sweeping every message-building site for provider text that bypassed the rule found six more, so the redaction and the 200-character bound now share one helper and the rule is a property of the module rather than of seven call sites. The configured key is matched literally and unconditionally, with no minimum length: a pathologically short key therefore also redacts incidental text (with the key a, stop_reason=max_tokens renders stop_reason=m[redacted]x_tokens). That is deliberate: a length floor would stop redacting short REAL tokens, which gateways and proxies do issue, and redaction does not trade coverage for tidiness. The configured API key and any credential-shaped token (sk-, sk_, key-, key_, token-, token_, xai-, xai_ followed by eight or more [A-Za-z0-9_-]) become [redacted]. The xai prefix was uncovered until a review pass caught it, so an xai-… token echoed by a gateway — one that is NOT the configured key, and so invisible to the exact-match rule — reached the Err intact. The pattern cannot cover every provider and is not meant to: Mistral issues a bare alphanumeric key with no prefix, which only the configured-key rule can catch, and the two rules are complements rather than alternatives. because providers quote the key they rejected (Incorrect API key provided: sk-…) and an Err is a value the program prints, logs, or ships onward — the fix that surfaced the provider's message is what created the exposure. Bounding is not allowed to become deleting: the truncation window is taken from the first non-space character rather than from position 0, since a window anchored at the front was an implicit bet on how much leading whitespace a body would carry, and 900 spaces before real text filled it entirely and rendered the value as the empty string. A provider-supplied value that is present but renders to nothing now says (blank) rather than trailing off — a whitespace-only stop_reason produced the dangling ; stop_reason=). and a whitespace-only body left the is not JSON: clause with nothing after the colon. "(no keys)" likewise now means there were none, rather than being derived from a rendered string — an object whose only key was 900 spaces was reported as having no keys at all, which is not a truncated statement but a false one. Every other interpolated field is bounded too: stop_reason, finish_reason, the block types and the response keys all pass through the 200-character limit, where a 64 MB stop_reason previously produced a 67 MB Err. Shape messages no longer contradict themselves — absent and present-but-wrong-typed read differently, so {"content": "Positive"} reports 'content' is str, not a list instead of naming, as a key it had, the key it had just called missing; stop_reason is reported on every Anthropic failure branch rather than one; and a choice with no message says so instead of describing the keys of nothing. That read is bounded at 64 KiB rather than unbounded — the 200-character message limit bounded what was printed, never what was held, so a hostile or misconfigured endpoint answering a rejection with megabytes cost that much resident memory to produce a 200-character string. One byte past the cap is requested so an overrun stays distinguishable from an exact fit, and an overrun body is reported as marked raw text rather than parsed: json.loads rejects most cut bodies by itself, but a short envelope padded past the cap with whitespace parses cleanly, and having read part of a body we do not claim to have parsed it. The read is also guarded — a socket already closed raises rather than returning bytes, and losing the detail is a far smaller loss than losing the provider name and status code to a second, unrelated exception; and a 200 whose body is not JSON is named the same way. A boundary that labels what it did not write: an InferenceError — this module's own class, raised at every site that has something to say — passes through verbatim, and every other exception, a plain RuntimeError included, becomes Inference provider '<name>' (<model>) failed: <Type>: <message>, so a future shape surprise can never again surface as a bare quoted key. The verbatim channel belongs to a dedicated InferenceError and nothing else. The rule was first written as "a plain RuntimeError or ValueError, by exact type", which the PR review refuted: those are the types an unforeseen failure raises too, so a RuntimeError("boom") from anywhere below — the transport, a dependency, a later edit — claimed the channel and reached the user as the single word boom, which is this defect one level up. A private class cannot be raised by accident, so every deliberate site in the module now raises it, including the registry's unknown-provider refusal (previously a ValueError, and the one test pinning that type moved with it). isinstance is safe again as a result — the hazard that forced exactness was the standard library subclassing the types being checked for, and nothing outside this module subclasses InferenceError. It extends RuntimeError and deliberately not also ValueError, because the module wraps json.loads in except ValueError twice and a subclass of both could be swallowed by its own handler after a later edit. tests/test_inference_response_shapes_1333.py pins all three across 164 cells, the headline ones END TO END through a compiled program over a mocked transport — the path the report came from — with the six-provider sweep parametrized in registry order and its rows checked equal to the provider registry, so a new provider cannot silently leave the sweep. Mutation-checked one edit at a time, every count re-measured on the tree as it now stands rather than carried forward — a review pass found two of the published figures stale and one of them self-contradicting, the same sentence quoting both 5 and 15 for one mutation. Restoring the by-position read fails 45 cells; the boundary's plain-type rule 18; removing redaction from the shared helper 28; dropping the boundary label 8, reproducing the reported 'text'; restoring the str() coercion 8, every one with "DID NOT RAISE" — that is, by returning a completion; restoring the skip of a text-typed block with no text field 7; disabling the reason clause 21, the same figure whether its function body is emptied or all four call sites are neutralised (an earlier note claimed the two forms differed; they do not); collapsing the output_text preference to a single discriminator 6; removing the empty-completion rule from both branches 10; reverting the blank test to truthiness 4; dropping the reason's case fold 3; removing xai from the credential pattern 3; and restoring the strict .decode("utf-8") 3. The narrower guards fail their own cell or two: the redact-before-truncate order, the exact-key rule beside the pattern, the output_text preference, the truncation window's starting point, the "(no keys)" honesty rule, the credential pattern's eight-character floor — dropping it to one redacts token-based, key-holder and xai-ish alike — and each of the three bounded-read guards. That plain-type figure had read 5, and the explanation offered for the drop — that the cells added since exercise the parse rather than the boundary — was wrong. The real cause was that threading the model into the boundary label made the label a PREFIX of itself: under the mutation the Err reads Inference provider 'anthropic' (claude-opus-5) failed: InferenceError: Inference provider 'anthropic' (claude-opus-5) returned no text block (…), which satisfies startswith("Inference provider 'anthropic' (claude-opus-5)") while being exactly the regression the cell exists to catch. Eleven cells lost their discrimination and nothing went red. They now assert the label's ABSENCE beside the prefix, through one shared helper so the next cell cannot forget it, and the figure went back to 16 — rising with each _assert_deliberate cell added since, which is why the count above is higher. The browser runtime is unaffected: Inference.complete there returns a deliberate, explanatory Err and never calls a provider.