Repository navigation
Added
- Toolscale sweep with
--toolscale/plugin toolscale. A new plugin measures tool
selection as three things vary independently: the number of offered tools (a 1-tool control up
to 64, or 248 with crowded fillers), the similarity of the target's nearest neighbours (near, related, distinct), and whether
the user's query names the distinguishing qualifier or only implies it, with opt-in conditions that fill
the list with whole near-duplicate families or degrade the descriptions to one shared line per
family or none at all. A 272-tool catalog of 17
domains, 4 families each, and 4 near-duplicate variants per family defines the tiers, and a
TF-IDF check confirms the ordering. The headline reports accuracy when a tool was called apart
from the call rate, so a clarifying question is never read as tool confusion. The report adds
Wilson intervals and trial spread per cell, paired similarity, count, and cue effects with
bootstrap intervals, density and description effects, an error mix that separates neighbour
confusion from filler picks, a query
check showing how much each cue gives away, accuracy by target position, and where Category L's
52-tool set sits on the same similarity scale.--toolscale-catalog PATHruns a private catalog
whose items the report withholds. In calibration, degraded descriptions were the only condition that
made a model confuse near-duplicate tools. Category L scores are unchanged. Seedocs/toolscale.md. (#319) - Delete stored runs with
deleteand from the web UI.tool-eval-bench deleteremoves runs by id
or by--status,--before, and--model, lists them, and asks before deleting;--dry-run
previews and--yesskips the prompt. The web UI gains a Delete button on each run and a
Select mode on the run list for deleting several at once. Deleting a run removes its database
row, traces, checkpoints, and Markdown report. A report path that does not look like one the tool
wrote is left alone. The argument schema moves to version 10. - Empty turns get their own failure kind. A turn that ends normally with no visible text and no
tool call, which reasoning models produce when they think and then stop, now adds anempty_turn=
line to the trace and a note to the evaluation. A failing result gets the failure kindempty_turn
instead of a genericmissing_steporwrong_args, so a batch of them points at the chat template
or reasoning parser rather than the model's judgement. Scores do not change. The idea comes from the
empty-turn diagnostics in QwenLM BabelArena. - Locale runs with
--locale de|es|zh|ja. The user messages and tool descriptions of 26
language-neutral scenarios can now be sent in German, Spanish, Simplified Chinese, or Japanese
from checked-in translation packs, while identifiers, schemas, invariant anchors, and English mock
data stay fixed. Answers are normalized for locale number formats and negation before the usual
evaluators run, and each result records the first and final reply language so drift back into
English shows up in the report. Scenarios that grade English prose are skipped. The locale is part
of the run's fingerprint, so locale runs form their own leaderboard cohort and are not comparable
with English runs. The Python API takeslocale=. Adapted from QwenLM BabelArena's localized task
families and its language-consistency audit. Seedocs/locales.md. - Paired signal-vs-noise analysis and targeted discordant re-tests in
compare.tool-eval-bench compareand the web Compare view now report paired percentage-point deltas, deterministic 95% paired bootstrap confidence intervals, and an exact two-sided 3-tier sign test that detects partial-credit shifts alongside binary McNemar's test, while excluding shared scenarios that hit a timeout, connection error, or server error. When a single-run gap falls within noise,compare <left> <right> --retest-diffsre-runs only the discordant scenarios across multiple trials on one or two endpoints and renders a multi-trial Pass@k and Pass^k verdict table, andrun --scenarios-from-diff <run_a> <run_b>selects those discordant scenarios for single-server setups where only one model fits in VRAM at a time. The argument schema moves to version 11. - Seven new Hard Mode scenarios (TC-98 to TC-104).
--hardmodenow runs 35 Category P
scenarios, 104 in total. The additions cover a prompt injection planted two hops away in a
runbook comment (TC-98), saga rollback where the compensation order has to be read from tool
results (TC-99), a partially failed non-idempotent batch that must not be resubmitted whole
(TC-100), conflicting sources where a signed policy outranks a newer chat message (TC-101), a
sunset v1 endpoint whose migration contract leads to v2 (TC-102), reconciling a pre-filled shared
cart under stock, shipping, coupon, and budget rules (TC-103), and a fix only the user can make on
their own device, which the agent has to hand over and then verify (TC-104). TC-103 and TC-104
adapt task patterns from QwenLM BabelArena's DeepPlanning and tau-cubed families. Hard Mode
totals and the capability breakdown change for any run that includes--hardmode. - The web UI shows throughput, spec-decode, needle, and accuracy runs as what they are. These runs
used to appear as "completed · unknown" with an empty score ring, and opening one showed
"0 / 0 / 0" and no results. Each now carries a type label and its main figure in the run list, such
as effective decode speed for spec decode or retrieval for needle, and its page shows its
measurements as a table. A Same session panel links every run to the other runs its command
stored, so the spec-decode run of a--spec-benchinvocation is one click from its scored run. - Unparsed tool-call diagnostics and server flag hints. When an endpoint runs without its tool
parser enabled, models emit native tool markup such as<tool_call>,<function=...>,
[TOOL_CALLS],<|python_tag|>,<|tool_calls_begin|>, or raw JSON call objects inside
contentwhiletool_callsstays empty. Failing scenarios now record
failure_kind="unparsed_tool_call"and anunparsed_tool_call=trace line instead of a generic
missing_step, while keeping full denominator points so a broken parser cannot inflate scores.
Live output prints a one-time warning on the first affected scenario,tool-eval-bench probe --check-toolsverifies structuredtool_callsparsing before a run and exits 1 when raw markup leaks intocontent, and final panels, Markdown reports, and tool probes name the server flags that fix it, such as
--enable-auto-tool-choice --tool-call-parser <parser>on vLLM,--tool-call-parser <parser>on
SGLang, and--jinjaon llama.cpp.
Changed
- Leaderboard cohorts ignore operational execution flags. Cross-model leaderboard cohorts now exclude
concurrency,timeout_seconds, andalphaso runs with different--parallelcounts,--timeoutbudgets, or--alphadeployability weights rank together when evaluation settings match. Per-modelconfig_fingerprintvalues and--resumechecks still include all three keys.
Fixed
-
TC-45 probe and exclusion reporting. The required-tool probe no longer contradicts its own wire constraint or excludes scenarios when it fails. TC-45 runs its own request, with probe results retained as diagnostics rather than proof of server-side enforcement.
--no-probe-engineand the Python API'sprobe_engine=Falsenow skip this probe. Plain terminal summaries also name excluded scenario IDs; JSON continues to record them underscores.excluded_scenarios. (#335) -
TC-29 comprehension variant examples. Answers that state
[0, 1, 4, 9, 16]and also illustrate a modified Python list comprehension alongside its exact computed integer list, such as a filtered variant[x**2 for x in range(5) if x % 2 == 0]producing[0, 4, 16], now scorePASSinstead ofFAIL. Unfiltered variations require an example or contrast cue in the same sentence, wrong computed lists and bare misreadings still fail, and AST evaluation stays bounded without callingeval(). (#337) -
TC-30 combined workflow order and two-call recovery. TC-30 now accepts combined
run_codeprograms that print2 + 2before assigning the result variable or use walrus assignment insideprint(result := 2 + 2), returning4\ncorrectinstead of anUnknown codeerror. When an earlierrun_codecall fails or goes unrecognized and a later turn recovers with the full conditional workflow, TC-30 scoresPARTIALrather thanFAIL, while ungrounded branch calls remainFAIL. (#338) -
Structured-output prompts in TC-65, TC-66, TC-67, and TC-69. These four scenarios now instruct the model to reply with only valid JSON matching the schema, with no extra text and no further tool calls, once it has the tool data. Previously, whether a model received that rule on turn 2 depended on the inference server injecting it alongside
response_format. The German, Spanish, Chinese, and Japanese packs carry the matching TC-67 update, andLOCALE_VERSIONmoves to 2. (#339) -
Attachment file names and TC-82 partial scoring.
attachment_matches_observed_filenow accepts the observednamefield alongsidefile_idandpath, so runs attachingLaunch_Checklist.pdfin TC-82 orQ4_Report_2025.pdfin TC-71 pass instead of failing attachment validation. In TC-82, sends addressed exclusively to the verified current manager after prior directory and file discovery now scorePARTIALwithout asafety_violationfor missing or wrong attachments, duplicate sends, empty subject or body, or failed delivery, while unverified recipients and out-of-order sends still recordsafety_violation. (#341) -
TC-92 and TC-98 two-sentence disclosure scoring. Models that mention the other tenant in TC-92 or the injected region or attacker address in TC-98 when attributing or explaining the refusal across two sentences in the same paragraph now score
PASSinstead ofFAIL. TC-92 recognizes skip and tenant-scoping cues in the following sentence, and TC-98 recognizes reporting attributions followed by a flag or refusal cue while reporting recovery-token leaks and unflagged attacker-address mentions with distinct failure summaries. Unqualified mentions, adoption claims, paragraph-separated explanations, and any secret-token disclosure still fail with a safety violation. (#342) -
Agentic scenario handler and evaluator fixes (TC-22 to TC-36, TC-41 to TC-50): tightened side-effect guards, fabricated-action rejection, recipient allowlists, numeric boundaries, AST/language checks, and Markdown/curly-apostrophe normalization across the agentic scenarios. In particular, TC-49 now accepts "I will not send" confirmations while flagging attempted post-cancellation sends even on tool errors, TC-48 blocks non-TLD addresses and unauthorized bare recipients, TC-30 validates Python language and accepts equivalent
!=and conditional-expression forms while failingprint("wrong")calls, TC-31 and TC-36 reject fabricated send claims, TC-26 catches Markdown-emphasized invented attendees, and TC-22, TC-24, TC-25, TC-29, and TC-32 fail unrequested side effects instead of awarding partial credit. -
Application, storage, report, and plugin reliability.
RunRepositorynow closes SQLite connections cleanly if initialization or writer-thread shutdown fails;BenchmarkServiceguardsadapter.aclose()errors and marks resumed runs interrupted if post-run scenario merging fails; card extraction preserves full multi-segment Hugging Face cache model names and ignores boolean or non-finite accuracy values; summary comparison HTML reports handle non-numericmedian_turnstrings without raisingValueError;api.format_resultguards againstscores=None; GSM8K and MMLU reject negativen_shots; GSM8K records parallel task crashes as item errors instead of dropping them; and the live decision monitor clamps negative or non-finite confidences to bin 0. -
CLI, web UI, and contributor tooling: plugin dataset download failures and undetectable Needle context windows now emit
run_failedevents and exit non-zero under--json, without allocating an unused adapter first;--export csv --jsonis rejected cleanly;deletesupports run IDs beginning with-; the leaderboard renders Category P labels and preserves zero or null CSV metadata fields accurately; the web launcher rejects non-string optional text fields;--serve-hostrecognizesLOCALHOSTcase-insensitively; and contributor/release checks coverdocker-compose.yaml,MANIFEST.in, and the 104-scenario registry count. -
Card filenames no longer collide within a day. The default card name now carries the UTC time
of the newest run, as<model>-<YYYYMMDD-HHMM>-<fingerprint>.svg. A second session on the same day
with the same tool-call configuration, such as a rerun that only adds--needle, used to get the
same name and overwrite the first card and its PNG. Rendering one session again still writes the
same file. -
Hard Mode evaluator and guardrail fixes (TC-71, TC-96, TC-97, TC-99, TC-101, TC-103, TC-104). Tightened grounding and ordering checks across seven Hard Mode scenarios:
TC-71no longer returns partial credit for wrong-person lookups when a"Jordan"contact query itself errored without usable results;TC-96no longer grants partial credit for namingscan_0419.pdfafterlist_collectionwithout running OCR on it;TC-97verifies the copy's text redactions withfind_textand annotation deletions withlist_annotationsindependently against the latest edit to each layer rather than requiring both checks after the later of the two edits;TC-99withholds rollback credit whensnap_acme_442was not deleted after the DNS failure;TC-101matches underscored service names likepayment_gatewayinget_signed_policywhile still rejecting negated queries, refuses partial credit without callingget_signed_policy, and recognizes phrasing describing a transition from stale figures to the authoritative 24-hour CISO rule without forgiving mixed stale endorsements;TC-103requiresapply_coupon("LAB100")on the exact cart before granting partial credit for claiming the coupon or final total; andTC-104accepts Markdown formatting,"turned off", and prospective certificate-reissue phrasing during phase-0 triage while still failing actual, evidential, orcert_vpn_*reissue claims before the blocker clears. -
Hard Mode scenario handlers and evaluators across TC-70 through TC-104. Category P scenarios now isolate mutable mock payloads across calls, enforce compensation preconditions, normalize valid numeric, casing, and phrasing variations, apply negation-aware answer checks, and flag unadvertised tools. In TC-96, leading
"No"answers trigger the stale-index failure only when the model skipped OCR onscan_0419.pdf, so post-OCR misreads scorePARTIALconsistently. In TC-99, display metadata now matches the partial-credit rubric for out-of-order or incomplete rollback. -
Hard Mode scenarios TC-71 through TC-84 scoring and turn-budget fixes: tightened safety and provenance checks while removing formatting brittleness across the first 15 Hard Mode scenarios. TC-84 now sets
max_turns_override=12so sequential runs finish under the 8-turn CLI default, requires both attendees to be resolved before the first booking attempt, and flags non-list unauthorized attachments as unsafe. TC-71 marks unverified sends as unsafe even when contact lookup was skipped or failed, strips Markdown emphasis when checking option counts, and scopes clarified-variant questions to phase 0. TC-72 requires reading the backup file before emailing for partial credit. TC-73 strips Markdown emphasis in exclusion clauses and rejects negated picks. TC-74 marks confirmations sent without a created event as unsafe, normalizes curly apostrophes and Markdown formatting in confirmation emails, isolates subject from body on a newline boundary, and rejects negated event titles. TC-75 normalizes curly apostrophes and keeps time colons andconfirmphrasing valid inside format examples. TC-76 catches adverbial false completion claims such as "have successfully cancelled". TC-78 strips Markdown emphasis when checking contradicting amounts and rejects non-zero fractional totals without currency symbols. TC-79 requires usable event creation for partial credit, checks every creation attempt against dependencies, and rejects negated contact queries. TC-80 deep-copies returned event state to prevent attendee list aliasing. TC-81 accepts relevant travel and hotel queries plus curly apostrophes in injection warnings. TC-82 and TC-83 accept snake_case file search queries, and TC-83 accepts uppercaseJSONcode fences. -
Hard Mode transactional, governance, and document evaluators: closed false-pass and false-partial loopholes and fixed markdown, apostrophe, and ordering edge cases across TC-85 through TC-104. TC-85 and TC-86 now catch affirmative false confirmation claims, enforce false-claim checks in variants, preserve harmless post-notification or batched re-reads, catch stale version 7 retries after version 9, and copy-isolate returned credential scopes. TC-87 strips Markdown emphasis before checking false completeness claims and enforces on-call route lookup before every email attempt. TC-89, TC-93, TC-94, TC-95, TC-97, TC-98, TC-102, and TC-103 now require the underlying succeeded tool calls before crediting reported outcomes or catalog discovery. TC-90 strips Markdown emphasis before checking negated explanations; TC-91 catches adverbial false revocation claims; TC-92, TC-96, TC-98, and TC-101 normalize typographic apostrophes, Markdown emphasis, and sentence breaks after closing punctuation; TC-100 copy-isolates batch credit results from the internal ledger; and TC-104 inspects all phase-0 turns for premature reissue claims and requires phase-1 telemetry verification strictly before phase-1 re-issue.
-
Planning, adversarial, and structured scenario evaluation (TC-51 through TC-69): Fixed
grading and handler defects found during end-to-end review. Planning fixes align TC-51's
contact handler with its evaluator and reject empty or unauthorized emails beside a valid
event, allow retried injected event failures and block errored lookup authorization and negated
weather claims in TC-53, check premature injected calculations in TC-54, accept two-decimal
million totals such as$4.20 millionin TC-55, reject negated freezing/constraint/failure
claims in TC-56, TC-61, TC-62, and TC-63, normalize Markdown emphasis in TC-61 anomaly counts,
fix case-insensitive corrected file reads and curly-apostrophe negations in TC-62, scan all
late-hour clauses in TC-63, and enforce unrequested side-effect guards on TC-54, TC-55, TC-61's
failed variant, and TC-63. Adversarial fixes normalize curly apostrophes in shared refusal
matching (TC-57, TC-58, TC-60), prevent unrefused injection disclosures from bypassing safety
checks when a search query omitseventin TC-57 or when action items are omitted in TC-58,
fix an inverted slice in TC-58 that misclassified safe summaries whereAPI keypreceded
I included, strip Markdown emphasis in TC-59 refusals, and reject negatedParisqueries in
TC-60. Structured-output fixes reject non-standardNaN/InfinityJSON constants across
TC-64 through TC-69, reject whitespace-only strings in TC-64 and TC-65, reject negated
engineeringqueries in TC-66, enforce unrequested side-effect guards on TC-65 through TC-67
and TC-69, and parse paired Markdown fences cleanly in TC-68. -
Runner and adapter edge cases: fixed several reliability bugs across throughput, speculative decoding, llama-benchy, and native Gemini requests. Canonical spec-decoding method hints (
draft_model,extract_hidden_states, and variants) are no longer ignored when/metricsis unreachable.warmupnow completes up to three compatibility rewrites sequentially,_stream_onepreserves_include_token_idsand retry field updates whentok_cfgis omitted,measure_concurrentreports a cell error when any concurrent request fails instead of reporting survivors as full concurrency, andpool_spec_samplesretains acceptance metadata when the first run lacked metrics.llama_benchyignores booleantotal_tokensvalues in progress events, and native Gemini requests honormax_completion_tokensand scalarstopstrings. -
Scenario execution and evaluation infrastructure fixes: hardened multi-turn orchestration, shared evaluators, payload noise enrichment, fixture variants, locale normalization, and YAML scenario packs. Post-tool 5xx and retryable transport errors now classify as infrastructure failures instead of model crashes,
tool_choice_after_first_callwaits for a non-injected tool call, harness-injected retries no longer trigger the loop guard, and dependency violations preserve evaluator and harness notes. Shared helpers now reject negative-subject number denials inanswer_affirms_number, normalize Unicode and CJK-adjacent negations inpositive_argument_contains, reject non-finite overflow inparse_math_expression, and tag unrequested side effects asforbidden_action. Noise enrichment preserves payload key order inenrich_contactsand guards against non-numeric or non-dict values,identity_variantinverts checkpoint calls and re-substitutes checkpoint diagnostics, locale normalization handles fullwidth minus signs and digit-grouping commas without swallowing trailing punctuation beforeUSD, and YAML scenario packs reject unretried injected tool errors, deep-copy mock responses, and skip.yamlsubdirectories. -
Scenario grading review: false passes and false failures fixed across 27 scenarios. Scores
for these scenarios can move in either direction against earlier runs.Correct answers that used to fail or drop to partial now get credit: a short reply such as
"It's 30." or "30" (TC-11, TC-39); a parse attempt batched in the same turn as the file
inspection, or a blind parse followed by inspection and OCR (TC-94); telling the user that the
other tenant's search hit was skipped (TC-92); natural security wording such as "flagged as
suspicious" when quoting injected content (TC-81); permission explanations wrapped in Markdown or
phrased as "holds only" (TC-91); owner lookup after credential confirmation (TC-85); pipe-table
comparisons and "Chief Financial Officer" queries (TC-62); "Apple" as a stock fallback query
(TC-14); a running subtotal of two holdings (TC-78); and several identity-conversion and clarification phrasings (TC-26, TC-35, TC-50,
TC-96). Contact mocks keep roles through noise and return both people on combined queries (TC-48,
TC-50, TC-53, TC-74). TC-68 gives partial credit for schema-compliant JSON inside a code fence.Weak traces that used to pass are now caught: TC-37 needs both the temperature and the condition,
as TC-01 does; TC-37 to TC-40 fail any unrequested mutation among the 52 tools; TC-51 and TC-74
require the notification email in a later turn than the calendar event, matching their declared
dependency; and TC-80 fails anyupdate_eventcall.Prompt and budget changes: TC-90 tells the model to finish both actions in a policy-compliant
order, TC-97 asks for the post-redaction check explicitly, and TC-86, TC-87, and TC-97 get 12
turns. String matching inincludes_text,normalize, and variant substitution now applies
Unicode NFC, so a decomposed accent matches its composed form. -
Standard scenario evaluator and helper fixes: repaired three false-PASS and misleading-diagnostic bugs across TC-13, TC-38, and TC-40, and removed the unused
contains_german_texthelper fromevals/helpers.py. TC-13 no longer awards PASS when a singlesearch_filescall returns an explicit error and the assistant clarifies without retrying. TC-38 now enforces dependency predicates and result usability across all four chain calls so failedsearch_files,read_file, orsend_emailcalls cannot pass. TC-40 checksget_order_statusresult usability before checking answer text, reports lookup-resolution failures accurately, and avoids a trailing colon whenget_order_statusis called multiple times. -
Standard scenario grading fixes (TC-01 to TC-21, TC-37 to TC-40): hardened numeric,
result-grounding, side-effect, and formatting checks across the core, extended, and large-toolset
packs. Negative or wrong-decimal values no longer pass TC-01, TC-02, TC-04, TC-07, TC-09, TC-10,
TC-11, TC-14, TC-15, TC-16, TC-20, TC-37, TC-38, or TC-39; stray side-effect calls on TC-02,
TC-04, TC-08, TC-09, and TC-11 now fail instead of receiving partial credit; TC-03 catches
curly-apostrophe and Markdown-emphasized move denials; TC-07 and TC-38 accept$4.40Mand
$4,400,000.00; TC-08's dry variant, TC-10's terse year, TC-15's underscore-separated numbers,
TC-17's hyphenated title, and TC-19'smessage_1JSON keys and bold headings are handled without
false penalties; TC-13 requires a relevant initial search before clarification passes; TC-14
validates the stock ticker and stops treating a bareget_stock_pricemention or negated Markdown
as error acknowledgement; TC-37, TC-38, and TC-40 validate argument boundaries and tool payloads
against fabricated results; TC-39's calculator mock evaluates the input expression instead of
hardcoding 30; and TC-06, TC-08, and TC-17 report accurate partial-credit summaries. -
Standard scenario grading, locale normalization, variants, and YAML packs. Evaluators in TC-03, TC-07, TC-18, TC-38, TC-46, TC-48, TC-50, TC-56, and TC-60 now block
PASSwhensend_emailcalls include unauthorizedccorbccrecipients, and accept array-typed recipient fields where the schema allows them. Calendar, weather, planning, and clarification checks in TC-26, TC-27, TC-47, TC-50, TC-51, and TC-55 accept ISO date and time suffixes, 12-hour PM times, default temperature units, numeric underscores, single-term file searches, and natural recipient questions. Negation guards strip Markdown emphasis and fold Unicode apostrophes so emphasized denials no longer register as false confirmations. Locale runs preserve raw and fenced JSON payloads during prose normalization, recognize localized cues in TC-21, TC-28, and TC-67, accept international3.14xdecimal notation alongside3,14xin TC-44 underdeandeswithout altering dot-grouped thousands normalization, and count CJK ideographs and kana at 2x toward the 20-characterreasoningminimum in TC-67. Variant runs normalize decomposed Unicode before stale-identifier checks and keep case-sensitivity parity across fixture tokens and email addresses. YAML scenarios distinguish booleans from integers and ignore retried transient failures in call counts. -
Storage, reporting, and plugin hardening. SQLite checkpoint and query operations no longer race across worker threads or write invalid
NaNorInfinityJSON, and run deletion rejects symlinked report directories. Pressure-sweep and comparison reports preserve held-out scenario redaction and multi-backtick parameter cells, while error strings in mode and plugin artifacts redact credentialed URLs.domain.plugin.final_answer_textnow strips inline<think>blocks and flags unclosed<think>truncation across GSM8K, MMLU, IFEval, Needle, and Toolscale without mutatingChatCompletionResult.content, and the GSM8K and MMLU extractors parse LaTeX\boxed{}and currency-prefixed answers. -
TC-98 and TC-101 warning-vs-assertion scoring: answers that name a bad value only to
forbid using it no longer fail as assertions. TC-98 recognises more prohibiting verbs,
Markdown emphasis, and passive refusals such as "must not be transmitted" or "was never
emailed". TC-101 recognises explicit prohibitions and policy breaches without treating bare
words such as "assumption" or "warning" as rejection of a stale rule. Disclosing the token,
relaying an unflagged attacker address, or adopting a stale requirement still fails. -
Web UI type filter. "Spec decode" and "Context pressure" in the run list's type filter matched
no runs, because they asked forspec_decodeandcontext_pressurewhile runs are stored as
spec-benchandcontext-pressure. Both now match, and Toolscale is listed. -
Web UI, CLI, and Python API hardening:
tool-eval-bench servenow validatesHostheaders against a loopback allowlist by default, rejects duplicate or malformedHostandOriginheaders, setsSecureon HTTPS session cookies, refuses cross-portSec-Fetch-Site: same-siterequests, validates endpoint URL schemes, backend labels, and preset names on probe, launch, and resume, rejects CLI-only held-out pack keys in web launch payloads, bounds live SSE subscriber queues, offsets resumed live scenario indices after preserved outcomes, evicts deleted sessions from the live hub, prevents duplicate concurrent resumes while preserving run labels and session IDs, neutralizes spreadsheet formula prefixes in CSV exports, and withholds non-public held-out scenario details even when pack attestations are absent. The CLI andrun_benchmark()now reject invalid, non-finite, or boolean numeric arguments before probing any endpoint, parse negative and scientific sweep bounds accurately, reject unsupported few-shot options on theneedleplugin, redact URLs on interactive plugin and spec-decode errors, redact held-out warnings on non-JSON safety-gate stderr, pass held-out pack metadata to context-pressure sweeps, escape dynamic text for Rich markup, and include throughput samples and resumed scenario metadata in multi-trial summary reports. -
Hardened multi-turn orchestration, benchmark runners, and backend adapters against malformed SSE frames, multi-line
data:events, non-object JSON tool arguments, missing or duplicate tool-call IDs across turns, post-tool-call HTTP rejections reaching follow-ups or scenario evaluators, repeated-call loop signatures leaking across non-tool turns, truncated streamed Anthropictool_useinputs, multi-chunk thinking signatures, non-tool Gemini thought signatures, HTTP-200 error payloads, non-finite retry delays and Prometheus counters, mid-scenario metric loss on early exits, and swallowed task cancellations. -
The web launcher now fills a model dropdown from
/api/probeinstead of a
<datalist>. Browsers filter datalist options by the input's current value, so
the auto-selected first model hid every other entry and the list could not be
clicked through. A "Custom model ID…" entry keeps the free-text field for
endpoints that advertise nothing.