Repository navigation
v0.10.2
No on-disk format or stage identity changes: IDENTITY_SCHEMA stays 1, every
built-in task's grader digest is the one 0.10.1 computes, and 0.10.0 and 0.10.1
result directories resume. Two configurations that used to run are now refused:
an evaluation that loads a staged dataset holding bytes other than its pinned
revision, and an args.api_base on a model sieval run launches.
Added
- Input-side top-k logprobs. A request that scores its input with
top_logprobs > 0, such asalogprobs(echo=True, logprobs=N), returns the top-N alternatives at every echoed input position inModelOutput.input_top_logprobs(IR:InputScoringResult.top_logprobs, aligned withtoken_logprobs). Supported onsglang_legacyandopenai_completions; a missing or misaligned input top-k is refused.top_logprobsstill holds only the generated tokens'.alogprobs'logprobsdefaults to 0 withecho(what was sent before) and 5 without; an explicit value is now sent withechotoo, so only a caller passingecho=True, logprobs=N>0sends a different request than in 0.10.1. No built-in task does (#178). gatedin the dataset registry.DatasetMeta.gated(frozen wire field; absent reads asfalse, so olderindex.jsonfiles load).hleis gated.sieval dataset listgains a GATED column,dataset showaGated:line, anddataset downloadrefuses a gated source without a Hugging Face token, or the Hub's refusal, with a message naming the gate and what to set (#181, #182).bbh,mgsm,mmlu_reduxandtrivia_qaare registered, each revision-pinned, sosieval dataset downloadfetches them and tasks name them by bare repo id. MGSM points atjuletxara/mgsm, sinceljvmiranda921/mgsmis gone from the Hub. Anhf:source may end in#<glob>, forwarded assnapshot_download(allow_patterns=...);trivia_qastages onlyrc.nocontext/*(about 60 MB of a 35 GB repo) and evaluatesvalidation, because upstreamtestis unlabeled (#180, #184).
Fixed
- Staged datasets are checked against their pin.
hf:sources must carry the download recordsnapshot_downloadwrites naming the pinned commit for every file;url:sources must match their declared checksum.sieval dataset downloadrecords each checksum it verified in<dataset>/.sieval/download/, and probes read that record while size and mtime match, else hash once per process. Probes never write into the data dir, so a read-only or sharedSIEVAL_DATA_DIRworks (#179, #183, #187, #190). - A
sglang_legacymodel with aninfer:block reaches the enginesieval runlaunched; it used to fall back to the OpenAI SDK default base and fail every request with HTTP 404 (#185, #186). - An engine
sieval runlaunched gets the placeholder keylocalwhen the config names none, instead of failing at client creation withoutOPENAI_API_KEYor sending whatever that variable held. An explicit key still wins, and an endpoint the run did not launch still reads the SDK's env var (#188, #191).
Changed
- BREAKING — a stale staged dataset is refused.
sieval dataset downloadrefuses a source whose staged directory holds other bytes, with the reason and a--forcehint, and deletes nothing.--forceon anhf:source clears the repo directory before fetching, sincesnapshot_downloadkeeps files it did not write. Readiness reports such a source as missing data (missing[].stalein JSON,[stale: …]in text).sieval run/sieval evalrefuse a datasetpath:that resolves to a registered source's staging directory holding other bytes; apath:outside staging loads as given. A data dir filled by hand now readsready: no: re-runsieval dataset download <name> --force(#179, #183). - BREAKING —
args.api_basenext topathorinfer.checkpointis refused before any engine starts, bysieval runand--dry-runalike. It used to be ignored by canonical dialects and to bypass the launched engine onsglang_legacy. Remove it to evaluate the launched engine (the #185 workaround is no longer needed), or removeinfer/pathto evaluate the endpoint. A top-levelapi_basestill turns auto-serve off.EvalSessionalso refuses a configuredapi_basethat differs from a realized deployment's (#189, #192).