Repository navigation
v0.10.1
A patch release despite two breaking changes, because neither touches an
on-disk format: meta.json, report.json and the shard records read as
before, IDENTITY_SCHEMA stays 1, and a 0.10.0 result directory still resumes
(one retuned serving plan aside, below). What breaks is configuration (the
embedding endpoint must now be set) and the grader digest; a resume that sees a
changed grader digest regrades on the recorded inference, so no sample is
inferred again.
Added
- Per-rollout generation logprobs at
n > 1.infer_args: {return_logprobs: true, top_logprobs: N}used to be refused aboven: 1; now every rollout's logprobs land inModelOutput.choice_logprobs, oneRolloutLogprobsper rollout intextsorder, and the single-sequence fields stayNone. Supported onsglang_legacy(each sample's ownmeta_info, and a reply with a sample count other thannis refused),openai_completionsandopenai_chat(streaming included).google_genaiandopenai_responsesrefuse it before any I/O. Old records decode withchoice_logprobs=None. For dialect authors: anOutputContractdefined outside SiEval needs achoice_logprobsrule, or building it raises (#171, #175). client_timeout: {connect, read, write, pool}, in seconds, wherevermax_retriesis accepted (a root model, top level orargs, and an inlinegrader/extractor; a derived model is refused). It is a client setting: never sent in the request body and in no stage identity, so changing it resumes without a rerun or regrade. Defaults are 5 s connect and 600 s for the rest; an enginesieval runlaunched defaults to a 30 s connect. A model'sargs.timeoutis still a request argument, not the client timeout (#169, #174).
Fixed
qwen3-30b-a3bSGLangmem_fraction_staticdrops from 0.90 to 0.87 on A100-80G bf16 and H100-80G bf16/fp8. At tp2×dp4, 0.90 left DP0 too little memory for SGLang's piecewise CUDA graph capture and the server OOMed at startup; that H100 configuration never came up at 0.90. The serving plan is part of the inference identity, so an earlier result for this model on these GPUs that used the recipe default (no hand-pinned value) now reads as a different inference:--resumerefuses it, and a leaderboard shows it Pending (#168, #172).- A request timeout's
error_msgnames the phase that timed out, e.g.Request timed out. (timeout phase: connect);error_reasonis unchanged (#169, #174).
Changed
- BREAKING — NL2SH-ALFA and T-Eval no longer default to an internal embedding endpoint. NL2SH-ALFA (both readings) and T-Eval with
eval_thought=Trueneed bothSIEVAL_EMBED_APIandSIEVAL_EMBED_API_KEYset, and the task fails at construction if either is unset. T-Eval withouteval_thoughtneeds neither (#167). - BREAKING — the grader identity is keyed by file path within its code root, not by import name.
identity.grader.moduleskeys become file paths (sieval/tasks/gsm8k_0shot_gen.py, or./…for the task's own plugin package), andidentity.grader.taskbecomes<file key>:<qualname>. Every plugin package the walk reaches is digested whole, whatever its name, namespace packages included. Within the task's own plugin package, absolute self-imports are normalised before hashing and a directory without__init__.pydigests as having an empty one, so a renamed copy (such as sieval-plugins'sieval_plugins__<task>__<variant>) digests the same as the original. Grader digests rotate fleet-wide, built-in tasks included;IDENTITY_SCHEMAstays 1, and a resume of a 0.10.0 directory regrades on the recorded inference (#170, #173, #176). - CI's change detection compares a push to
mainagainst the previous commit, so a re-run no longer fails on a stale default-branch name (#166).