Skip to content

v0.13.0

Choose a tag to compare

@ethan-scitix ethan-scitix released this 25 Oct 08:25
· 14 commits to main since this release
bd51e07

A minor bump: meta.json identity moves to schema 4, where inference
digests what the model is asked, including its context limit, and no longer
how it is served; inference_identity takes context_limit instead of
recipe. RESULT_SCHEMA_VERSION stays 1, so 0.12.x result directories
resume, but a run recorded at a context window that sieval run now raises is
refused on resume. Refusals get their own exit codes and a machine-readable
stderr line, and a task can declare the Python packages it needs.

Migrating from 0.12.x

  • meta.json identity is schema 4 (#233). A reader of meta.json must accept schemas 1 to 4. inference drops the engine and serving recipe and adds the context limit; each runs[] entry records serving instead, never compared. A schema 1 to 3 directory resumes once by the digest it was written with, so its engine and recipe must still match, then is rewritten as schema 4.
  • inference_identity takes context_limit instead of recipe; TaskRunner, MultiTaskRunner.add_task and task_resume_plan take context_limit (#233).
  • New root-model keys: context_limit (an endpoint's window), weights (compared on resume instead of the name when both runs state it), serving (an endpoint's stated engine_version/topology/hardware; refused on a model sieval run launches) (#233), and context (native/max/scaling) (#236).
  • Serving groups (#236): a task needing more than the recipe window raises it up to native, or, up to max, runs on its own scaled engine in serving group <model>@<window>. Groups run one after another; --dry-run lists them under serving, and --serving-group KEY runs one.
  • Breaking for resume: a run recorded at a window sieval run now raises is refused, since the window is part of the inference identity and a resume never reruns inference. Start it fresh, or pin the recorded window with infer.overrides (context_length / max_model_len) (#236).
  • Refusals exit 3 (resume), 4 (hardware) or 5 (environment), not 1, and write one SIEVAL_REFUSAL {json} line per refusal to stderr (#230, #235). A scheduler that treated every non-zero exit alike now sees these codes; the line's v is 2.
  • extra at both placements is refused (models.<m>.extra and models.<m>.args.extra, or both in an inline grader/extractor config). Keep one (#228). A 0.12.0 result directory written from a top-level extra ran without it, so its resume is refused; start it fresh.

Added

  • Refusals exit with their own code. sieval run, eval and infer start exit 3 on a refused resume and 4 on refused hardware (no GPU for an engine, or not the declared GPUs), and print one SIEVAL_REFUSAL {json} line to stderr; both used to exit 1 like a crash. --dry-run, and a batch failing in more than one way, still exit 1. An nvidia-smi timeout now fails instead of reading as "no GPU". StageResumeError and ResumeVersionError gain task; GPU refusals raise LaunchGpuError (a RuntimeError). See Exit codes (#230, closes #229).
  • A task declares the Python packages it needs, and a run without them is refused. Task.python_requires = {import name: PEP 508 requirement} on the class, or a python_requires_for(context) classmethod when the need depends on the task's args. sieval run and sieval eval check it before any engine starts or dataset loads: a package is met when its import name is found (not imported) and, if pinned, the installed distribution satisfies the pin; extras are rejected. A missing package is an environment refusal: exit 5, one SIEVAL_REFUSAL line per task with kind: "environment", reason: "python_requires". --dry-run reports it as a new dependencies check, task list/show as a python-requires entry under missing, and TaskMeta.python_requires is in meta/index.json. sieval infer start does not check it. The SIEVAL_REFUSAL line's v moves to 2; the resume and hardware lines are otherwise unchanged. It is outside every identity, so resumes are unaffected. No built-in task declares one yet (#235, closes #234).
  • How the model is served no longer refuses a resume. meta.json identity moves to schema 4: inference digests what the model is asked, now including the context limit, and drops the engine and serving recipe. Each runs[] entry records serving instead, never compared, so a run on 1 GPU resumes on 8 and a launched engine's run resumes on an equivalent endpoint. New root-model keys: context_limit (an endpoint's window; a launched engine's comes from its plan), weights (compared on resume instead of the name when both runs state it) and serving (engine_version/topology/hardware of an endpoint, recorded in runs[].serving.stated). Readers of meta.json must accept schemas 1 to 4. A schema 1 to 3 directory resumes once by the digest it was written with, so its engine and recipe must still match, then is rewritten as schema 4. inference_identity takes context_limit instead of recipe; TaskRunner, MultiTaskRunner.add_task and task_resume_plan take context_limit (#233, closes #232).
  • sieval run sizes serving to what the tasks need. A task's need is TaskRequirements.min_context (RULER declares one) or its effective max_tokens, overridable with a task's min_context; a root model's new context: block states native / max / scaling. The recipe window is raised up to native; a need up to max gets its own scaled engine in serving group <model>@<window>, run after the default group; anything else, or a need beyond a window pinned by infer.overrides or context_limit, is refused before launch. --dry-run lists groups under serving, --serving-group KEY runs one, and runs[].serving.group records it. sieval infer start <yaml> serves the same raised window. Breaking for resume: a run recorded at a window now raised is refused (inference is never rerun); start fresh or pin it with infer.overrides. An SGLang gen model with no dialect: now defaults to sglang_legacy when a task scores input, instead of refusing (#236, closes #231).

Fixed

  • A model's top-level extra reaches the model. models.<m>.extra passed --dry-run and was then dropped at binding, so Model.extra was {} and the inference identity omitted it. It now binds like args.extra on every dialect and on derived models; args.extra identities are unchanged. A 0.12.0 result directory written from a top-level extra ran without it, so its resume is refused as an inference change; start it fresh (#228, closes #227).
  • Breaking: extra at both placements is refused before any engine starts, on named models and inline grader/extractor configs. Inline configs used to accept both and silently use args.extra. Keep one (#228).