Skip to content

llx 0.4.0

Choose a tag to compare

@rsaryev rsaryev released this 17 Nov 11:01
· 91 commits to main since this release
1ca733e
  • --schema checks the whole of JSON Schema, drafts 4 to 2020-12 as the schema's $schema says: bounds (multipleOf exact in decimal, integers exact past 2^53), lengths in characters, uniqueItems, contains, additionalProperties, patternProperties, propertyNames, const, anyOf/oneOf/allOf/not, if/then/else, dependentRequired and dependentSchemas, unevaluatedProperties and unevaluatedItems, $ref with $id and $anchor; format held to (dates and times, email, uri, uuid, addresses and the rest but idn-*), and pattern as ECMA-262 reads it. Before, only type, required, properties, items and enum were, and data that broke a bound, added a field or had a wrong date was taken. Each message names the field, the rule and what the data has (data.issues[0].line must be >= 1 (is 0)). What is not checked (a lookaround in a pattern, idn-*, a $ref to a document not given) is named on stderr. The checker is its own package (packages/schema, about 230 KB of the binary where the jsonschema crate would add 1.9 MB): it passes every test it runs of the official JSON-Schema-Test-Suite, agrees with the labels of 36,990 LLM answers to real schemas (MaskBench) on 99.98% (jsonschema 99.95%), and is faster than jsonschema on every case of its README. A $ref followed into deep data (a tree a hundred levels down) is no longer cut at 32 levels.
  • Anthropic's models are called over its own Messages API, with the prompt cache marked: a session on claude-sonnet-5-5 paid its whole conversation again at every turn (0 tokens read from the cache, about $0.33 a turn at 150k tokens of context, $2.94 in two minutes), as Anthropic's OpenAI-compatible endpoint caches no prompt and ignores the reasoning level. Now the system prompt and the tools, the context, and the conversation up to the last turn are read from the cache, the level goes out as the model's effort (a budget of thinking tokens for the older models), and the model's signed thinking goes back with its turn. Any provider of the catalog whose model is served over messages is called so; a reply is streamed, so a patch the output limit cut off keeps its whole files (B7).
  • What a request writes to a prompt cache is priced at the catalog's cache-write price (Anthropic's is 1.25 times the input's): --budget and the spend no longer count it as plain input.
  • A session has no time limit by default: a task you watch is stopped by you, not cut off after 30 minutes. llx exec keeps 1800 seconds; --timeout sets one for either. A request the time limit cuts off now ends the task as out of time (exit 2, the limit named) instead of an error of the provider's with no answer (exit 3).
  • A follow-up task no longer rewrites the start of the conversation at its first request: the history's gate measured it as if nothing were known of how the model counts tokens, and an estimate that ran high (a model's sealed reasoning) elided older outputs the prompt cache held, so the task began with a full cache miss. The measure is kept from task to task.
  • apply_patch names every file whose lines it did not find, not only the first, so a patch of many files is mended in one retry; and a long file's nearest lines come with six lines on each side, numbered as a read shows them.
  • The context says when the project has no AGENTS.md or CLAUDE.md, so the model does not search the disk for one (a find .. over a home folder took a minute).
  • OpenAI with an API key is called over its Responses API, as with a ChatGPT sign-in: over chat completions gpt-6.1-sol and gpt-6-astra refused every request with tools at a level (400), and no model's reasoning was kept between turns.
  • --schema keeps the prompt cache to its last request: result is offered with every request, so the tools at the start of every prompt never change. Its data's request offered result alone, and so paid the whole conversation again at every provider. The model may now end the task with result itself, which saves that request; an answer in words is still followed by it, with the same tools. A result beside other calls is not taken (the work goes on), one made while an agent's answer or a job is still to come waits for it, and a result that keeps breaking the schema ends the task without data after two fixes. Over chat completions the call is no longer forced (DeepSeek and Qwen refuse a forced call while they think, Z.ai takes only auto, current Claude models refuse one): it is asked for in words, and asked again after an answer in words.
  • Checked against each provider's documentation:
    • Anthropic: the model's thinking goes back in the order it came, a block cut before its signature is dropped, and thinking the API refused as bound to an earlier history is never sent again (it was stripped and resent at every turn after one elision); a refusal (stop_reason: refusal) or a reply past the window is an error, not an answer; xhigh and max may write 64,000 tokens; the older models think between their tool calls (the interleaved-thinking beta); only Anthropic's own models take the Messages API.
    • A rate limit is waited out as long as the API's Retry-After asks, never past the deadline, at Anthropic, OpenRouter and every chat-completions host; a wait past a minute is said at once, and no pause between attempts grows past a minute. OpenRouter's 402 with a wait (its in-flight budget) is waited out.
    • Z.ai, Kimi K2.x and DeepSeek are turned off with thinking.type, Qwen at DashScope with enable_thinking, as their docs say, where a model names no none level: reasoning_effort: "none" was ignored or refused there.
    • OpenRouter: Gemini is no longer marked for the cache (it caches on its own, and OpenRouter bills each mark as a write); signed reasoning parts with no index are kept apart, not joined into one corrupt value; a reply on the person's own provider key (BYOK) costs what that provider charged too; xhigh and max may write 64,000 tokens, as OpenRouter gives a budgeted model's thinking 95% of the limit there.
    • ChatGPT: a reply that failed for load before any word of it (server_is_overloaded, a rate limit of a moment) is asked again, as Codex does; one past the window says so.
    • Chat completions: a Gemini call's thought signature goes back on the call (Gemini refuses the next request without it), an answer given in chunks (Mistral's reasoning models) is read and goes back whole, a reply the host's filter withheld is an error, and a host that counts cached tokens at the top of its usage (Together) is read; a 403 (a permission or a moderation flag) no longer says the key was refused; OpenRouter's ~vendor/…-latest aliases are marked for the cache as their vendor's models are.
  • A command moved to the background tells the model not to change or check the files it writes until it ends: a formatter still running rewrote files the model was patching.