Skip to content

lmgw 0.6.0

Choose a tag to compare

@github-actions github-actions released this 06 Nov 16:55
· 6 commits to main since this release

A machine check that sets up the graphics card, Find a model, the OpenAI platform as its own upstream, Anthropic's blocks carried natively, image and audio tools in the Chat, and the Chat as a crate other apps can host.

  • The machine check. At start lmgw checks whether this computer can run local models and says what is missing in plain words: the graphics card and its driver (Secure Boot included, amdgpu on AMD), NVIDIA's container toolkit, Podman, the model settings against the card, free space and the engine downloads. It shows as the "This computer" card first in Settings → GPU, as a card on the Overview and above the Chat's empty state, and as a banner in the Chat while local models are involved (a cloud-only setup is not nagged; "Not now" quiets all three). The tray adds Graphics card needs attention… when the verdict leaves ready.
    • Set it up installs the toolkit through a fixed root helper, /usr/libexec/lmgw/lmgw-machine-helper, behind lmgw's own confirmation (every command and file under "Show the commands") and one polkit password prompt. It runs only from the app's window and as a system task (lmgw-machine-setup.service), so closing lmgw doesn't interrupt it. Where the helper can't apply (no dnf, an image-based system) the check lists the commands to type instead.
    • lmgw writes the card's CDI description itself, without root, into its own runtime directory, rewrites it after a driver or package update and passes it to its containers with --cdi-spec-dir; there is no nvidia-ctk cdi generate to run for them.
    • Model settings made for the other vendor's card (AMD Vulkan or NVIDIA CUDA) can be switched in one confirmed step, with every change listed.
    • A start the computer can't make answers 503 gpu_setup with the reason, or the model's fallback answers it. The check and its fix are also the machine_check op and the lmgw__machine_check tool.
  • Engine images arrive with the model. A model download also fetches the image its class runs in, and a start whose image is missing downloads it before admission, showing its bytes, without taking a place in the queue. A failed image download answers engine_download.
  • Find a model. Models → Find a model takes a name as someone said it ("qwen 3.6") or any Hugging Face link, searches the hub live (originals before uploads, no model list of lmgw's own) and answers one card: the original, the upload and the file that fits this machine's card, judged by the listed sizes and then a Range read of the GGUF header, before anything downloads. Get it downloads the file with its companions, plans it, fits its context to the card and creates the row without starting it; Say hello opens a thread. Send to a friend copies the model's name and its public Hugging Face address, and a pasted share text plans that exact file. Add from Hugging Face marks every file by its fit and preselects the recommended one. Self-admin agents get lmgw__model_find, lmgw__model_get and lmgw__model_get_status.
    • Downloads resume from their .part after a restart, are checked by length and sha256 before they are renamed into place, and pause on a metered connection.
    • A fresh install needs no settings to download: it keeps models under ~/.local/share/lmgw/models/, one folder per class, shown and changed under Settings → Runtimes.
  • Binary units everywhere. Every byte figure lmgw shows or says (the machine check, downloads, image pulls, disk, card memory, body sizes, notes, tool sentences) is in KiB, MiB, GiB or TiB, through one formatter.
  • The OpenAI platform gets its own egress. The openai protocol now means api.openai.com only; every other OpenAI-shaped provider is openai_compatible.
    • /v1/responses goes to the platform natively, metered and priced, unless only lmgw's loop can serve the request (an MCP label lmgw serves, a previous_response_id lmgw stored). Hosted tools pass through on the native path. Stored responses belong to the key that made them, and GET/DELETE/input_items on a native id go through its upstream.
    • Chat sends to the platform go over the Responses API unless they set stop, seed, a penalty or carry audio. Encrypted reasoning survives across turns: in the Chat, as a thinking block's signature on /v1/messages and as encrypted_content on /v1/responses. A model's refusal is its own part.
    • Service tiers. lmgw asks for service_tier: auto when the client set none and prices each answer at the tier it reports. Price rows have a tier; traffic at a tier with no price stays unpriced with a note, and the Prices page lists it.
    • Budget floors. A tier with no price, or a stream cut before its usage, still counts against budgets at a floor (the standard price, or a local token count). A budgeted caller is refused with budget_unpriced on a model with no price for what it is billed in, flex without a flex price included.
    • max_completion_tokens for the output cap; llama.cpp samplers are left out and named in x-lmgw-params-ignored; OpenRouter's reasoning object becomes reasoning_effort; OpenAI's error codes, Retry-After and x-ratelimit-* reach the client.
  • One fallback per model and alias. Every model and alias has one Fallback setting, used for every reason it can't answer: the GPU hold, a benchmark, VRAM taken outside lmgw, a deferred background request, a card that isn't set up, an exhausted quota, an unreachable upstream or a disabled row. A fallback is served as if the client had named it, so it can hand on to its own; x-lmgw-fallback-chain lists each hop with its reason. A fallback must be of the same model type and may be a local model; a loop is refused. lmgw remembers an upstream that is out of quota and skips straight to its fallback until a re-check (quota_recheck_seconds, under Usage & cost) or Check now.
  • Anthropic, natively. Between a Messages client and an Anthropic upstream, lmgw now carries what it used to flatten, whole and streamed: thinking blocks with their signatures, blocks it doesn't model (redacted_thinking, server tool blocks, newer types) verbatim, text block citations, compaction blocks and context_management edits, and Files API sources for images and documents. Every other route gets the nearest equivalent (the compaction summary as text, citations dropped) and x-lmgw-params-ignored names what was left out. Signed and verbatim blocks go back only to the upstream that produced them. The Anthropic dialect of /v1/models carries Anthropic's capability blocks.
  • Stalled streams end. An upstream that stays silent past its timeout ends the client's stream with the protocol's own error. Keep-alive comments only hold a stream open while the model may be thinking out of sight or a provider-run tool is running.
  • Structured output, translated everywhere. response_format, text.format and output_config.format are one field that every egress writes in its upstream's form (Anthropic output_config.format, Gemini responseJsonSchema). A model whose capabilities say it can't is refused early with response_format_unsupported; upstream schema refusals arrive as invalid_json_schema or response_format_unsupported with param.
  • Image and audio tools for the Chat. image__generate, image__edit, image__upscale, audio__speak and audio__transcribe are a built-in toolset on /mcp and, in Chat threads, the labels image and audio. A running call shows its stage, step progress and previews and can be cancelled. A generated picture gets a card with Re-roll, Vary, Edit and Upscale that work without the chat model (POST /chat/api/attachments/{id}/media). Default models per tool are set in Settings → Chat, a folder or a thread.
  • The Chat on the Chat plane. The dashboard's Chat reads only /chat/api/* and /v1 routes, scoped to the caller (new reads for its settings, toolsets, tools and knowledge bases), and is a crate of its own, lmgw-chat-ui, with a narrow layout down to 280 px and no inline scripts, so a paired device can host the same Chat. Kai for Firefox shows it in a Firefox sidebar.
    • A message can carry its origin, where the user writes from, kept out of the system prompt.
    • Conversations can be renamed (menu or F2; POST /chat/api/threads/{id}/title), New chat starts in the open thread's folder, the composer is one input bubble, and side panels can be dragged wider or narrower.
  • Device-to-device transfers. A paired device can offer a file to another device of the same app (an app id declared at pairing), relayed through lmgw: a resumable ranged upload checked by sha256 (/transfers/api/transfers), the offer on the Chat feed as transfer.updated, a ranged download, and expiry. The Devices card gets a Transfers section, and enrolment carries the device's own name.
  • API compatibility checks. Every gate run checks each body lmgw sends upstream and each /v1 answer and stream event against OpenAI's OpenAPI spec, Anthropic's SDK schemas and Gemini's discovery document. scripts/compat-live.sh drives multi-turn scenarios through the official SDKs against a running lmgw; its results are published at https://lmgw-dev.github.io/docs/compat/. New on the way: POST /v1/responses/input_tokens, and a sampling parameter a model refuses is retried once without it, left out from then on and named in x-lmgw-params-refused.
  • Background requests. x-lmgw-background: on makes one request a guest on the GPU: it uses a local model only where that disturbs nobody, otherwise the fallback answers or a 503 gpu_deferred says to retry. The Models page shows a model's deferrals over the last 24 hours.
  • The Voices lab. Under Labs: rounds of candidate voice samples from any text-to-speech alias, saved to the voice library with their exact transcript (also lmgw__voice_generate, lmgw__voice_sample_save).
  • Audio. /v1/audio/alignments answers one typed shape whatever model aligned (seconds, the sample rate, each word's character span in the submitted transcript), and every local audio row publishes its languages.
  • Dashboard. Engine builds and images are Engines (Overview's section is Running models); the sidebar nests Labs and Downloads; messages name settings by their labels (Settings → GPU → Admission → Load timeout), not config keys; the window remembers its size.

Breaking / upgrade notes

  • Clients: Kai 0.3.0, Kai mobile 0.2.0 and Kai for Firefox 0.1.0 are the releases built against this one; older Kai builds pin lmgw 0.5.0.
  • openai rows become openai_compatible (migration 0090), whatever their host, and the start notice names each one. Switch an api.openai.com row back to openai on the Upstreams page to get the platform egress. supports_responses is an openai_compatible choice only and is cleared on Anthropic and Gemini rows; audio_cpp rows must be openai_compatible.
  • The global hold fallback and inherit are gone (migration 0095). Each row keeps the fallback it answered with, the start notice names the rows whose fallback came from the global, and a fallback of another model type is refused at the row's next save.
  • Budgets count only what is priced in full: a budgeted caller is refused on models with no price for what they bill (a price of 0 is allowed).
  • Answers changed: an upstream 503 is 502 api_error (only a 529 is overloaded_error); rate and quota refusals carry OpenAI's own code and type, an exhausted quota on /v1/messages is 402 billing_error, and an OpenAI 4xx lmgw has no word for is relayed as sent; languages moved from capabilities.speech to the top of capabilities; /v1/audio/alignments is lmgw's typed shape, no longer audio.cpp's JSON. New codes: gpu_setup, engine_download, budget_unpriced.
  • A silent upstream now ends a stream after its timeout, where keep-alive comments used to hold it open.
  • Migrations 0085 to 0100 run on start. The RPM depends on polkit and ships the root helper and its polkit action (io.github.lmgw-dev.lmgw.machine-setup); building the RPM by hand needs cargo build --release -p lmgw-machine-helper before cargo tauri build, as ci/build.sh does.

Fixes

  • Every answer from the gateway's origin except the dashboard's own sends nosniff and Content-Security-Policy: sandbox, so stored or relayed content can't run as the dashboard.
  • The updater runs pkexec and dnf by absolute path and downloads the RPM into a private directory instead of a fixed /tmp path.
  • The RPM no longer ships group- or world-writable files.
  • A streamed chat stops its model as soon as the client goes, even while the model is prefilling.
  • A split GGUF is charged for every part, not only the first.
  • A tool call with no arguments goes upstream as {}; an MP3's input_audio goes as mp3; the /v1 answers carry the fields OpenAI's and Anthropic's schemas require (logprobs: null and others).
  • Chat attachments, lab files and voice clips answer Range requests with 206.