feat(guardian): an OpenRouter provider, so #246 has a third vendor to compare - #412
Merged
Conversation
… compare The two providers this repository speaks to are the two #246 is comparing, and the local mistral key returns 402. Ollama would answer it for nothing but needs a server this project does not have. OpenRouter reaches a third vendor over an OpenAI-shaped API, which is what the comparison was blocked on. Three things in the provider are measurements rather than choices. max_tokens is generous because a first probe capped output at 600 and read three capable reasoning models as incapable: they spend 709 to 2,243 tokens reasoning before the answer, so the truncated reasoning came back where JSON was expected. Hitting the budget now raises instead of returning that prefix, because the limit is this file's choice and attributing it to the model was the error. A 429 raises and carries OpenRouter's body. The free tier gives no warning — /api/v1/auth/key reports no limit and completions carry no rate-limit headers — and the body is what distinguishes an account's daily allowance from one upstream's shared pool, which is why z-ai/glm-5.2:free was rejected as an arm. An error object arriving with HTTP 200 is reported rather than KeyError'd; that is how the account's allowed-providers setting surfaces, and the remedy is in the message. skeptic_arms gains a per-arm model (--arms gemini,openrouter=vendor/model) and a validity gate. The gate is not a tolerance: judge_finding contains provider errors per finding and returns None, so a spent quota, a 429 and unparseable JSON all arrive as 'unruled' — the same row shape as a skeptic that refuted nothing, which is what #246 predicts for a same-vendor skeptic. A broken arm would have confirmed the hypothesis. It fired on the first real run: the free nemotron arm left 96 of 135 findings unruled, all after the first 39, which with today's probes puts the free tier at 50 requests per day. Measured, not recalled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t failed (#246) Two faults, found by running the arm rather than by reading about it. qwen3.7-plus puts its chain of thought in a separate reasoning field and fills content only at the end. On the larger prompts it spent the entire 8,000-token budget thinking and returned content: null — 118 of 135 findings unruled, caught by the validity gate. Raising the budget would have fixed the symptom and broken the experiment: gemini-2.5-flash, the arm it is compared against, is not doing extended thinking, so the two arms would have differed in a dimension nobody chose. The provider now sends reasoning explicitly in both directions, off by default, so a run can state what it did rather than inherit it and be unable to describe itself afterwards. The second fault cost half an hour because judge_finding logged only "Skeptic judgement failed; finding stays unruled" — 118 identical lines with no type and no message. A spent free quota, an HTTP 429, a 402 and unparseable JSON all land in that one except, and they have four different remedies. It now carries the type and the message. Containing the error per finding stays deliberate (#246 3.4); saying nothing about it was not a decision, it was an omission. What the two attempts measured, in benchmarks/guardian/experiments/ 246-cross-vendor/: the free tier allows 50 requests a day (39 answered in the run plus 10 in the day's probes, then a clean wall), and the paid account has $0 credits against $0.159 used, so 402. The comparison itself is still not run, and the gate is why no number was published for it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces an OpenRouter provider to enable skeptic evaluations across a third vendor, addressing issue #246. It also implements a validity gate that refuses to publish results if an arm leaves more than 5% of findings unruled, preventing provider errors from being misinterpreted as leniency. The skeptic runner, fingerprinting, and command-line scripts are updated to support the new provider and model-per-arm specifications, accompanied by extensive unit tests. Feedback from the review highlights two improvement opportunities: first, checking that the choices list returned by OpenRouter is non-empty to prevent an IndexError; second, making the markdown code fence parser in parse_or_raise more robust to handle single-line fences without newlines.
An empty choices list was an IndexError. "choices" not in body passes when
the key is present and holds [], which is what an upstream content filter
returns, and body["choices"][0] then raised IndexError — an exception whose
message says nothing about which provider declined. Now falsy-checked.
A single-line code fence was unparseable. Splitting on a newline returned the
whole string when there was none, so the backticks survived and json.loads
failed — counting a good answer as unruled, which is the reading this whole
file exists to prevent. Regexes now match the fence and its optional language
tag. The suggested fix used lstrip with a character class; lstrip("json")
removes any of j, o, s, n and would eat into the payload, so the shape of the
fix is different from the one proposed while the finding is the same.
SonarCloud separately put build_skeptic_provider at cognitive complexity 18
against a limit of 15, which openrouter pushed it over. Split into one builder
per provider behind a table, matching build_provider. Each arm's requirements
— which key, which default model, whether one exists at all — are per-provider
facts and read better beside that provider than interleaved with the others.
Coverage on the new provider is 100%; the openrouter paths through
build_provider and build_skeptic_provider now have tests of their own rather
than being exercised only through the experiment script.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SonarCloud on #412: an unanchored `\s*```$` is quadratic on a long run of whitespace that never reaches a fence, because every start position retries the \s*. Model output is not input this module chooses, so unbounded cost on hostile shapes is a real property rather than a theoretical one — 200k spaces now parse in under a millisecond. str.endswith answers the same question in one comparison. The opening fence keeps its regex: anchored at ^, it is tried from a single position and cannot backtrack across the input. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The comparison is still not run. What this adds is the third vendor #246 was blocked on, and two guards that stopped two wrong answers from being published.
Why a third provider
The two providers this repository speaks to are the two #246 is comparing, and the local mistral key returns 402. Ollama would answer it for nothing but needs a server this project does not have. OpenRouter reaches a third vendor over an OpenAI-shaped API.
Three things in the provider are measurements, not choices
max_tokensis generous because a first probe capped output at 600 and read three capable reasoning models as incapable — they spend 709 to 2,243 tokens reasoning before the answer, so truncated reasoning came back where JSON was expected. Hitting the budget now raises rather than returning that prefix: the limit was this file's choice, and attributing it to the model was the error.A 429 raises and carries OpenRouter's body. The free tier gives no warning —
/api/v1/auth/keyreports no limit and completions carry no rate-limit headers — and the body is what separates an account's daily allowance from one upstream's shared pool. That distinction is whyz-ai/glm-5.2:freewas rejected as an arm:limit_source: upstream_provider_shared_pool.An error object arriving with HTTP 200 is reported rather than
KeyError'd. That is how the account's allowed-providers setting surfaces, and the remedy is in the message.reasoningis sent explicitly, off by default.qwen3.7-plusputs its thought in a separate field and fillscontentonly at the end; on the larger prompts it spent the whole budget thinking and returnedcontent: null. Raising the budget would have fixed the symptom and broken the experiment —gemini-2.5-flashis not doing extended thinking, so the arms would differ in a dimension nobody chose.The validity gate, which is the real deliverable
judge_findingcontains provider errors per finding and returnsNone, so a spent quota, a 429, a 402 and unparseable JSON all arrive as the same row shape as a skeptic that refused to refute anything — which is exactly what #246 predicts for a same-vendor skeptic. A broken arm would have confirmed the hypothesis.MAX_UNRULED_RATErefuses to print a comparison above 5% unruled, keeps the rows, and exits 2. It fired on both attempts:nemotron-3-super-120b:freeqwen/qwen3.7-plusNeither produced a number. Without the gate, both would have read as "the cross-vendor skeptic refutes nothing".
And a diagnosis that cost half an hour
judge_findinglogged only "Skeptic judgement failed; finding stays unruled" — 118 identical lines, no type, no message. Four different faults land in that oneexceptand they have four different remedies. It now carries both. Containing the error per finding stays deliberate (#246 §3.4); saying nothing about it was an omission, not a decision.To unblock
Top up OpenRouter (~$0.15 for the run with reasoning off), or split the free tier across three days (the input is a frozen recording, so it is deterministic — but the tool has no resume), or a local ollama.
Data and the full write-up:
benchmarks/guardian/experiments/246-cross-vendor/.2031 tests.
🤖 Generated with Claude Code