Repository navigation
v0.13.1
A patch release: a max_tokens-derived context need reserves room for the
prompt, without moving a task that fits under the native window into a scaled
group. No on-disk format changes: meta.json identity stays schema 4 and
0.13.0 result directories resume, except where the served window changed
(below).
Migrating from 0.13.0
- A task's served window changes when its need comes from
max_tokens, so its resume is refused. The window is part of the inference identity, and a resume never reruns inference. On a modelsieval runlaunches with a known native window, amax_tokensbelownativethat 0.13.0 served at exactlymax_tokens, or at the recipe window with fewer than 4096 tokens to spare, is now served atnative. The other tasks on that model's default plan share the window, so theirs changes too. Amax_tokensat or abovenativeis served in scaled group<model>@<max_tokens + 4096>, where 0.13.0 served it atmax_tokens(in<model>@<max_tokens>, or unscaled when the recipe window held it). Start such a run fresh, or pin the recorded window withinfer.overrides(context_length/max_model_len). A model whose tasks all fit the recipe window with 4096 to spare, or all setmin_contextor declareTaskRequirements.min_context, keeps its window (#239, #240). - Where 4096 more no longer fits, the task is refused before launch: a
max_tokensabovecontext_limit - 4096on an endpoint, above the plan's window minus 4096 when the native window is unknown, or abovecontext.max - 4096when scaled. Set the task'smin_contextif its prompts are known to be shorter (#239).
Fixed
- A raised serving window leaves room for the prompt. 0.13.0 raised a plan's window to exactly a task's
max_tokens, so every request with a non-empty prompt was refused as longer than the context. A need derived frommax_tokensnow adds 4096 tokens for the prompt;min_contextand a task's declared need already count it and are unchanged. A task with longer prompts setsmin_context(#239, closes #238). - A
max_tokensbelow the native window is served at that window, never scaled. With the 4096 added, a task such asmax_tokens: 38000on a model withnative: 40960would move into a scaled (YaRN) group, and scaling changes outputs. Such a task now raises the window tonative; its prompt getsnative - max_tokenstokens, and a longer prompt is that sample's context overflow, as before. Amax_tokensat or abovenativestill getsmax_tokens + 4096, in a scaled group whencontext.maxandscalingallow (#240, refs #238).