Skip to content

v0.5.7

Choose a tag to compare

@github-actions github-actions released this 25 Sep 21:57
· 71 commits to main since this release
v0.5.7
e265dd1
  • Every eval comparison sets its arms on the same attempts, and an outage scores no arm. Each suite is split
    into comparisons of the arms that ran the same tasks. At each task every arm keeps as many attempts as the arm
    with fewest measured, all timed in wall time. A missing hosted verdict, a site answering 5xx and an error sent
    with HTTP 200 now count as outages, rerun up to five times. The gateway's $0 Jev metering is priced at list for
    both Jev-backed arms. The feed moves to schema_version 2.
    (#153, #154)
  • A dense page's step is asked again smaller when Jev sheds it. The gateway answers large Jev requests with
    503s far more often than small ones, and a step on a Wikipedia article (160 controls, about 27k tokens) ran out
    of retries in six runs of six, ending each one unavailable. When that happens to a large step request, it is
    now asked again with the on-screen controls only, then with half of those left; a small request that is still
    refused ends the run unavailable as before. The error for a request that split into single questions now
    counts every request the call sent, not only the last question's.
  • A secret that is also part of a site's hostname no longer breaks the addresses a run cites. 0.5.6 kept such
    a host readable in reported addresses, but the model's own view of the page was blanked first, so steps, facts
    and citations after signing in as practice still linked to https://••••••••.expandtesting.com/secure.
    Masking and redaction now share one rule: a value that sits wholly inside the host of an address on an origin
    it was typed on is left, wherever the address appears, and every other appearance of it, a password's
    included, is still blanked or redacted.
  • An answer states the value, not the requirement ahead of it. A value read straight off the page was answered
    as the requirement it met followed by the value ("Find the latest released version of httpx." then
    "httpx 0.28.1"); the answer is now the value alone.
  • The page a run began on counts as visited. A task saying "Start at" an address could be planned as a
    requirement to go there, and a run that opened a deeper page from it was sent back because nothing it was checked
    against showed it had been there. The done check and verifier now see every address the run has been on, first
    the one it began on, and the planner no longer makes a start address a requirement of its own.