Repository navigation
v0.14.0
A minor bump: new config keys runner_config.stop_after and a model's
lifecycle, a task's grader / extractor may name a models: entry that
sieval run launches, sieval run gains exit code 6 for an engine lost
mid-run, and --dry-run reports new barriers, phases and launches
fields. No identity or on-disk format changes: meta.json identity stays
schema 4, RESULT_SCHEMA_VERSION stays 1, and 0.13.x result directories
resume.
Migrating from 0.13.x
- Exit code 6 (#246):
sieval runexits 6 when an engine it launched exits or fails its health check for 180 s in a row mid-run. It is not a refusal and writes noSIEVAL_REFUSALline; under-o json,data.outcomeisengine_died. A scheduler that treated every non-zero exit alike now sees this code;--resumeretries the failed samples. - New
error_reasonvalues (#246):engine_died(in flight when the engine was lost; a resume retries it and spends a retry) andengine_died_unsent(never sent; retried without spending one). An older SiEval reads both as ordinary retriable failures. stop_aftergoes in the top-levelrunner_config(#247). It is session-wide: a task's ownrunner_config.stop_afteris refused, and it needsrecord_each_stage: true(the default). The stopped run writes noreport.jsonunless it stopped afterreport, and recordsruns[].stopped_after.- New
error_reasonvaluebarrier_iteration(#247): withmax_iterationsunset, a sample asking to iterate past a barrier oninferenceorpostprocessfails with it; a resume without that barrier can retry it. A task withmax_iterationsabove 1 under such a barrier is refused before the run. --dry-runneeds_inferencevslaunches(#247): the top-levelneeds_inferencesays whether a model under test starts; a launched judge or extractor does not set it. The newlauncheslists every engine the run starts, judges included: a caller that readneeds_inferenceto decide whether the run needs GPUs should readlaunches.- Barriers and lifecycles are scheduling only: they enter no identity and never refuse a resume.
runs[]gainsstopped_afterandserving.lifecycles, both additive.
Added
- Stage barriers and
stop_after.runner_config.stop_after: <stage>(preprocess,inference,postprocess,feedbackorreport) ends the run once every sample of every task has finished that stage or failed;--resumecontinues from the next stage with the same identities as an uninterrupted run. The barriers arestop_afterplus every lifecycle boundary, with no key of their own; at each, no sample starts the next stage until all have finished this one.--dry-runlists them underbarriers, with what implied each. From Python:MultiTaskRunner(stop_after=, barriers=, on_barrier=)(#247, closes #245). - Model lifecycles and launched judges. A model
sieval runlaunches may setlifecycle: {from: S, until: T}(either bound optional) to run its engine only between those barriers and then free its GPUs. A task'sgraderorextractormay name amodels:entry, whichsieval runlaunches like the subject; a launched judge that statesweightsis compared by them instead of its name in thefeedbackidentity. With the subjectuntil: inferenceand the judgefrom: inference, the judge takes the subject's GPUs within one run. Models alive at the same time get disjoint GPU blocks (indices into an inheritedCUDA_VISIBLE_DEVICES) and their own ports, and a phase that does not fit is refused before any engine starts. A model starts only when some sample still needs it, so a resume with inference complete starts only the judge.--dry-runreportsphasesandlaunches;runs[].serving.lifecyclesrecords the lifecycles (#247, closes #245).
Fixed
- A run fails fast when an engine it launched dies. An engine lost mid-run (an SGLang out-of-memory exit) used to fail every later request one by one through its client retries until no sample was left.
sieval runnow polls each launched engine's process every second and its health every 10 s; on a loss, every stage still needing that engine fails at once and stages on other engines carry on. Each task still writesreport.json, the engine is stopped (its leftover process group killed so no worker keeps the GPUs), no later serving group starts, and the run exits 6 naming the engine, its exit code or failing health check, the failures per task and its last log lines. An engine stopped on purpose, lost after its last sample, or not launched by SiEval (api_base,sieval infer start) is not a loss (#246, follow-up to #243).