You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Every eval comparison sets its arms on the same attempts, and an outage scores no arm. Each suite is split
into comparisons of the arms that ran the same tasks. At each task every arm keeps as many attempts as the arm
with fewest measured, all timed in wall time. A missing hosted verdict, a site answering 5xx and an error sent
with HTTP 200 now count as outages, rerun up to five times. The gateway's $0 Jev metering is priced at list for
both Jev-backed arms. The feed moves to schema_version 2.
(#153, #154)
A dense page's step is asked again smaller when Jev sheds it. The gateway answers large Jev requests with
503s far more often than small ones, and a step on a Wikipedia article (160 controls, about 27k tokens) ran out
of retries in six runs of six, ending each one unavailable. When that happens to a large step request, it is
now asked again with the on-screen controls only, then with half of those left; a small request that is still
refused ends the run unavailable as before. The error for a request that split into single questions now
counts every request the call sent, not only the last question's.
A secret that is also part of a site's hostname no longer breaks the addresses a run cites. 0.5.6 kept such
a host readable in reported addresses, but the model's own view of the page was blanked first, so steps, facts
and citations after signing in as practice still linked to https://••••••••.expandtesting.com/secure.
Masking and redaction now share one rule: a value that sits wholly inside the host of an address on an origin
it was typed on is left, wherever the address appears, and every other appearance of it, a password's
included, is still blanked or redacted.
An answer states the value, not the requirement ahead of it. A value read straight off the page was answered
as the requirement it met followed by the value ("Find the latest released version of httpx." then
"httpx 0.28.1"); the answer is now the value alone.
The page a run began on counts as visited. A task saying "Start at" an address could be planned as a
requirement to go there, and a run that opened a deeper page from it was sent back because nothing it was checked
against showed it had been there. The done check and verifier now see every address the run has been on, first
the one it began on, and the planner no longer makes a start address a requirement of its own.