RustedVINO 0.8.4

RustedVINO 0.8.4 — the fixes from a full test run on an Arc Pro B60, a cheaper sampling default (a few percent faster decode), /tokenize that works for vision models, and hybrid vision models (Qwen3.5 / Qwen3.6) that load on a GPU without --enable-prefix-caching. Versions 0.8.1 to 0.8.3 were internal test builds and were never published; this release contains all of them.

What changed, release by release: CHANGELOG.md.

Download, don't compile

The Linux bundle is self-contained: the OpenVINO 2026.2.1 runtime ships inside it. This release has no prebuilt Windows bundle (the Windows fixes are in the source).

# Linux
tar xzf rustedvino-linux-x86_64-v0.8.4.tar.gz
cd rustedvino-linux-x86_64-v0.8.4
cp config.example.json config.json
./rv start       # then follow README.txt in the bundle: download a model, ./rv add it

Verify the download first if you like: sha256sum -c rustedvino-linux-x86_64-v0.8.4.tar.gz.sha256. The bundle needs glibc 2.39 or newer (Ubuntu 24.04 and later).

How it was tested

An Arc Pro B60 (24 GB) ran the project's QA kit against a release candidate of this bundle (the same code except the memory-warning fix below): 60 of 60 checks pass, no GPU wedge in a 12-step overlap ladder, no out-of-memory kill. The unit suite (1174 library, 117 API and 2 doc tests) passes. Everything beyond that is untested: other GPUs, the Windows build, the NPU, and the 9B hybrid vision model. One change made after the B60 run (the memory warning: it used to fire for correctly sized small models because it compared the weights plus the KV pool with the weights alone) was checked on a laptop iGPU only, not re-run on the B60.

Known issues

• A refusal right after a load is misleading (carried over from 0.8.0). rv load or rv add within 2 seconds of another load says "nothing more can be evicted" or "can never fit". It fits two seconds later (or with --force).
• rv deregister and rv save-defaults without -y and without a terminal print only "aborted"; add -y.
• kill -9 of the server in the middle of GPU inference makes the xe driver log a page fault and a compute-engine reset. The GPU was healthy afterwards. Stop the server with rv stop or SIGTERM.
• The memory ledger is more conservative than the GPU. Six models that fit physically (15.2 GiB) were refused at a 23.5 GB budget. An explicit, smaller vram_gb per model overrides it.
• Hybrid vision models on a GPU have a smaller context (prefix caching is off for them): about 13,900 tokens for the 27B with the default 2 GB KV pool (cache_size_gb). A longer prompt gets a clean HTTP 400.
• gemma-4-e4b-it-int8-ov does not load as a vision model (the OpenVINO paged-attention conversion rejects it). Text-only use is not affected.
• Carried from 0.8.0 and not re-tested: a GPU context can still wedge in rare cases (the server reports server.gpu_poisoned; --supervise restarts the worker), and unloading the reranker can crash the process on some CPU setups (upstream OpenVINO CPU plugin).

What's new since 0.8.0

Faster sampling default

A request that sets only temperature now takes top_p / top_k from the model's own generation_config.json instead of sampling over the whole vocabulary. Measured on the B60: qwen3-1.7b 155.9 → 169.7 tok/s (+8.9 %), qwen3-8b 66.6 → 69.3 tok/s (+4.1 %). temperature: 0 with top_k / top_p used to be a 500 and is now greedy decoding.

API

• GET /v1/models/{id} (OpenAI's retrieve-model route).
• /tokenize and /detokenize work for vision models, and they answer in milliseconds while a long generation runs (they used to wait 21-27 s). /detokenize rejects ids outside the vocabulary with a 400.
• Unknown message roles are a 400 naming messages[i].role; developer is rendered as system.
• Unknown URLs answer a JSON 404 with code: unknown_url.
• INTERFACE.md in the bundle documents the HTTP API.
• gpt-oss models get the harmony reasoning parser automatically (the raw analysis…final channels no longer leak into content).

Memory and models

• Hybrid vision models (Qwen3.5 / 3.6) load on a GPU without --enable-prefix-caching (name the GPU with --device GPU.n; without it they go to the CPU).
• rv add sizes a model by its on-disk size (the 35B-A3B now fits a 24 GB card: 20.5 GiB resident of 22.7).
• A model with a request running is not evicted by a plain load; force is the override (this closes the 0.8.0 known issue about evicting a busy model).
• New default_placement setting. In automatic placement an integrated GPU is now a last resort, and the bundle's example config puts every model on the CPU ("device": "CPU", "default_placement": "device").

Security and packaging

• A server with no keys configured refuses browser requests (an Origin header) on /v1/admin/* and /metrics (403 browser_origin_refused), so a web page open on the same machine can no longer drive an open admin API. curl, rv and Prometheus are unaffected. POST /v1/admin/models refuses a model_id that is not a plain folder name. The bundle README now shows how to create a random admin key.
• The bundle carries its licences: LICENSE, THIRD-PARTY.md, Intel's OpenVINO / oneTBB / oneDNN / hwloc notices under licenses/openvino/, and the licence files of all Rust crates under licenses/rust-crates-licenses.txt (0.8.0 shipped none of these in the archive).
• The bundle is built from the public source tree at the release tag (its own Cargo.lock, the toolchain pinned in the project), so rv version shows a commit hash that exists in the public repository.

Operations

• Stopping the server during a long model load works at once instead of after the compile.
• rv start reports "stopped before it was ready" instead of an error plus a log dump.
• A global device that OpenVINO does not enumerate is named at boot together with the devices that exist.
• Schema errors name the field (invalid value for field `max_tokens` ).

Upgrading from 0.8.0

No config changes are required, but check these:

• A config.json carried over from 0.8.0 has no default_placement and therefore uses "auto": chat, embedding and rerank models go to a discrete GPU when there is one, and an integrated GPU is now the last resort (it used to come before the CPU). On a laptop with only an iGPU and NPU, work that ran there may move to the CPU. Add "default_placement": "device" to put every model on the global device.
• A config.json copied from this release's config.example.json runs everything on the CPU until you set device (for example "GPU.1").
• rv add estimates a model's memory as its size on disk, without the 1.08 margin of 0.8.0.
• A request that sets only temperature samples with the model's own top_k / top_p (set top_k: 0, top_p: 1.0 to get the old full-vocabulary sampling).

Apache-2.0. Third-party notices: THIRD-PARTY.md and licenses/ in the bundle.

RustedVINO 0.8.0

RustedVINO 0.8.0 — a native rv command-line tool, POST /v1/systemone typed decisions, Windows fixes found by a real-hardware re-test, and a long list of "bad input no longer takes the server down" fixes. It ships with known issues, listed below, that you should read before relying on memory-pressure behaviour.

What changed, release by release: CHANGELOG.md.

Download, don't compile

The Linux bundle is self-contained: the OpenVINO 2026.2.1 runtime ships inside it. This release has no prebuilt Windows bundle — the Windows fixes below are in the source, and a packaged Windows build is planned once it has been tried as a zip on a clean machine.

# Linux
tar xzf rustedvino-linux-x86_64-v0.8.0.tar.gz
cd rustedvino-linux-x86_64-v0.8.0
cp config.example.json config.json   # point models_dir at your models
./rustedvino

Verify the download first if you like: sha256sum -c rustedvino-linux-x86_64-v0.8.0.tar.gz.sha256.

Known issues

Read these first. None crashes the server; all are about what happens when memory is short or when the
GPU driver misbehaves. Found by a Windows re-test on a Lunar Lake laptop and by a 6.5-hour torture run.

• Loading under system-RAM pressure can evict more than rv check predicted, and can evict a model
  that is busy. The loader credits an evicted model with its configured size, but the memory the OS
  really gets back is smaller (measured on Windows: a 5.0 GB-configured model returned ~2.0 GB, a 3.5 GB
  one ~0.6 GB). It then evicts the next model. In the worst case it evicts two models and refuses the
  load anyway, or evicts a model that is generating: the eviction waits up to 15 s for its requests to
  finish, then aborts them (clients get a retryable 503 model_unloading, see below). This is most
  visible on small-RAM, shared-memory machines; a 0.8.0 mitigation (a short wait for freed memory) did
  not close it. Workarounds: keep headroom, give models explicit vram_gb close to what they really use,
  avoid loading a large model while others are mid-request, and do not trust rv check under pressure.
  Planned fix: 0.8.1 (credit each model its measured freed memory; never evict into a refusal).
• The system-RAM admission check assumes the full KV cache. The check, and the rv pre-check, price a
  load with the whole configured KV pool; the GPU path would shrink it. So a load can be refused by RAM
  that the GPU gate would have granted a smaller pool for, and the outcome can flip on a few hundred MB.
  rv load --no-check bypasses the pre-check; --kv-cache-gb is not honoured by the pre-check.
• Refusal wording is incomplete. When RAM is the binding limit the message quotes VRAM figures, and
  it does not name the model that is protected or busy. On the RAM path a refusal inside the 2 s
  eviction-grace window says "can never fit", which is wrong; it fits two seconds later or with --force.
• A GPU context can still wedge (sticky CL_OUT_OF_RESOURCES). A 6.5-hour fuzz run hit it twice.
  About 5,600 load/unload cycles with parallel inference, including an NPU model resident and busy,
  did not reproduce it, so the trigger is not known (memory pressure is the main suspect, untested).
  When it happens the server reports it (server.gpu_poisoned in GET /v1/admin/models, a banner in
  rv) and, under --supervise, restarts the worker; without a supervisor you must restart it yourself.
  The Windows build has no supervisor yet.
• Unloading the reranker can crash the process on some CPU setups (a destructor segfault in the
  OpenVINO CPU plugin, upstream). One crash in 1,143 rebuilds; the server no longer rebuilds the
  pipeline after ordinary bad input, which was the usual trigger.
• Windows: no prebuilt bundle yet. The Windows fixes were re-tested on a Lunar Lake laptop (Arc 140V
  iGPU + NPU) from a staged folder, which needed an elevated permissions fix because it was written from
  Linux. A real installer and a Windows service wrapper do not exist yet. Closing the console window ends
  the server immediately, without draining requests (Ctrl+C drains).
• Smaller: max_tokens above max_tokens_cap is silently clamped (documented); rv ps shows an NPU
  model's configured size while rv status and /metrics show the counted footprint; rv --version
  prints the tool's own version (0.1.0).

What's new

rv is a native binary

tools/rv is a separate crate (it links no OpenVINO) and the bundles ship it as rv / rv.exe; no
Python is needed, which matters on a fresh Windows machine. It carries every verb of the old script
(status, list, ps, check, add, load, unload, deregister, reload-config, patch,
preload, save-defaults, logs) and adds resize, completeness, reload-keys, voice-pin,
sessions, session and session-kill. rv status takes the server's PID from the server itself, and
summarises configuration problems as a count and a few names rather than raw JSON.
rv load --force now works on its own.

POST /v1/systemone

Ollama v0.35-compatible typed decisions (choice, yes/no, ordinal score) with probabilities, answered by
one prefill per question on a kind: "decision" model, with no text generation. Validation order, status
codes and error messages follow Ollama, and prompts are byte-identical to Ollama's, pinned by 121 fixtures
generated from Ollama's own code. Decision models need a decision.json sidecar and a last-position
gathered IR (see CONFIG.md), refuse the NPU, and go to the strongest GPU first. Decision requests share
a device's admission budget with chat, wait for their model's turn so they cannot starve chat or
embeddings, and a client that disconnects stops its request before the next question.
Measured honestly: a test on a B70 found the results are not bit-identical to Ollama's on that GPU
(quantisation and device differences), exact on CPU and on the Lunar Lake iGPU; see the test notes in the
repository before treating the two as interchangeable.

Windows (source builds)

• Default config at %LOCALAPPDATA%\RustedVINO\config.json, read by the server and rv; config and keys
  files saved with a UTF-8 byte-order mark (PowerShell 5.1, Notepad) now load.
• The device-tier cache falls back to %LOCALAPPDATA% when there is no HOME; the compile cache and the
  model-hash memo live beside ov_cache_dir when it is set.
• Boot warnings when ov_cache_dir, cache_manifest, the config or the keys file is not writable
  (OpenVINO otherwise skips its cache without a word).
• The admission budget is capped at what the GPU driver reports, with a boot warning, so a config written
  for a bigger machine no longer admits everything and slows every model 3-7x.
• NPU models are charged their real footprint (npu_footprint_factor, default 1.5; set 1.0 for the old
  accounting).
• NPU Whisper: language is detected once on a small CPU Whisper and passed explicitly, so the NPU
  pipeline is built once. Re-tested on Windows: 500 auto-language requests, threads and handles flat,
  500/500 identical transcripts (an earlier approach leaked 3 threads per request, an upstream OpenVINO
  NPU teardown leak). The CPU copy of the weights stays in RAM until the model is evicted.
• Release binaries no longer carry the build machine's paths; packaging scans for them and refuses.

Memory and model switching

• eviction_grace_secs defaults to 2 s (was 300 s, which blocked ordinary model switches for five
  minutes). A model with a running request sorts last among victims and is drained before eviction.
  rv load --force skips only the grace window, never pins or realtime sessions; check?force=true
  previews it.
• A request aborted because its model was evicted (or at shutdown) gets a retryable 503 model_unloading with Retry-After: 5 (streams: an error frame and finish_reason: "error"), not an
  opaque 500. Verified on Windows.
• An admin load that can never fit returns 503 insufficient_memory, not a 500.
• Eviction logs name the binding constraint (vram or system_ram).

Robustness

A 6.5-hour fuzz/torture run (about 5,600 load/unload cycles with parallel inference) found and fixed:

• A WAV declaring sample rate 0 made the audio decoder panic and kill the speech-to-text engine until the
  model was reloaded. Undecodable audio is now 400 invalid_audio.
• NUL bytes, empty documents, or text made only of invisible characters in /v1/rerank, and NUL bytes in
  /v1/embeddings or /v1/audio/speech, are 400 invalid_input, not 500.
• /v1/audio/speech input is capped at 4096 characters (400 input_too_long); 70,000 characters made the
  voice model ask for 78 GB. top_p: 0 no longer answers 500.
• An oversized prompt is 400 context_length_exceeded, not a retryable 429.
• Every rejection on the API routes uses the OpenAI-style JSON error: malformed bodies, wrong content type,
  wrong method (405 with Allow), bad query. Only unknown paths (404) are still plain.

Configuration

• Every shipped config now writes out all 47 global and 20 per-model settings explicitly, at the value
  already in effect. rustedvino --print-default-config and --print-default-model list every setting
  with its default; scripts/fill-config-defaults.py adds whatever your config is missing without
  touching your values. Tests fail if a setting is missing from a shipped config or the reference.
• A boot warning when cache_size_gb: 0.0 is combined with prefix caching (unbounded growth); the starter
  config sets cache_size_gb: 2.0.
• device_budgets guidance: a budget counts every in-flight request and a chat stream holds its slot for
  the whole stream, so size it at or above the chat model's max_num_seqs plus headroom.
• /metrics byte gauges are whole numbers. GET /v1/admin/models server carries pid and
  uptime_secs for valid-key callers.

Upgrading from 0...

RustedVINO 0.7.0

RustedVINO 0.7.0 — Vietnamese speech end to end, and Lunar Lake's NPU as a first-class, fully configurable text-generation target, with several NPU responses that were quietly wrong now right.

Versioning moves to semver from this release (0.6.8 was chosen for luck, not as a semver claim).
What changed, release by release: CHANGELOG.md.

Download, don't compile

rustedvino-linux-x86_64-v0.7.0.tar.gz is self-contained: the OpenVINO 2026.2.1 runtime ships inside it.

tar xzf rustedvino-linux-x86_64-v0.7.0.tar.gz
cd rustedvino-linux-x86_64-v0.7.0
cp config.example.json config.json   # point models_dir at your models
./rustedvino

Verify the download first if you like:

sha256sum -c rustedvino-linux-x86_64-v0.7.0.tar.gz.sha256

Vietnamese speech, end to end

• Speech-to-text: PhoWhisper (VinAI's Vietnamese fine-tune of Whisper, BSD-3-Clause) runs as an
  ordinary Whisper STT model once converted. The README has the conversion recipe (it has traps:
  Python 3.12, an offline re-save, restoring the tokenizer files) and measured error rates: on 5
  synthetic clips (85 words), PhoWhisper-large 1.2% against 3.5–4.7% for stock Whisper.
• Text-to-speech: Piper's vi_VN-vais1000 voice, with a load-time check that espeak-ng
  produces exactly the phonemes the voice expects.
• Voice conversation: PhoWhisper → a chat model → vais1000 over /v1/realtime; about 3.2 s from
  the end of a spoken question to the first spoken reply, measured on one Lunar Lake laptop.

NPU: what changed

Fixed — requests now mean what they say on the NPU

• enable_thinking: false is honoured. The pre-rendered prompt was being chat-templated a second
  time by OpenVINO GenAI (apply_chat_template defaults to true), which dropped the "thinking off"
  prefill; reasoning models then spent their whole budget thinking.
• Sampling settings reach the NPU: temperature, top_p, top_k, seed, presence_penalty,
  frequency_penalty, repetition_penalty and stop. Previously every NPU request ran greedy
  while the response's sampling block reported the requested values. seed is now reproducible.
• Structured output (response_format with a JSON schema) works on the NPU.
• usage.completion_tokens is exact (it counted streamed text chunks, which can hold several tokens).
  Streamed NPU text is sent as soon as each piece is decoded, with the exact count delivered at the
  end of the stream.
• When a long prompt fills the NPU's fixed-size cache and cuts the answer short, the response now
  says finish_reason: "length" instead of "stop".

New — NPU configuration

• min_response_len (per model, NPU only): the answer room the NPU reserves on top of
  max_prompt_len. Prompt + answer can never exceed max_prompt_len + min_response_len
  (OpenVINO default 1024 + 128). Settable in config or via PATCH /v1/admin/models/{id}.
• NPU LLMs now use ov_cache_dir: one weightless compiled blob per model (~0.66 GB for an 8B int4
  model), cached loads in about 6 s instead of ~30 s cold, identical output. Counted against
  ov_cache_max_gb and pruned like every other cached blob. (NPU Whisper still never uses the cache
  dir — it hangs with one.)
• Clearing max_prompt_len / min_response_len via PATCH now restores OpenVINO's default on the
  next load (it used to keep the old value until a restart). The same fix applies to
  max_concurrent_streams on vision models.
• POST /v1/admin/config/reload now picks up file changes to the next-load settings
  (max_prompt_len, min_response_len, kv_cache_gb, max_concurrent_streams, speculative)
  for models that aren't loaded. It also stopped reporting every previously loaded Qwen3 model
  as updated when nothing had changed.
• The NPU over-length error names the right setting (max_prompt_len, not cache_size_gb).
• The NPU prompt-length check follows the length the model was actually compiled with. After a
  PATCH and reload it used to check the old value, so an over-long prompt could reach OpenVINO and
  return its raw error instead of a clean 400.
• An answer that ends exactly at the NPU's cache limit is reported as stop, not length.
• NPU LLMs are no longer charged a GPU-style KV cache (cache_size_gb) when the server checks
  whether a model fits. An 8B int4 NPU model was counted at ~12.5 GB against ~5 GB actually used,
  so it could be refused, or force an eviction, next to a GPU model that fits fine. The admin API
  now reports its real prompt limit (max_prompt_tokens, e.g. 1024) instead of a GPU figure.

Measured on a Core Ultra 7 258V (Arc 140V + NPU): qwen3-8b int4 on the NPU generates ~15–17 tok/s on
mains power (~13 tok/s on battery), first token in ~1.2–2.2 s.

Also in this release

• Piper TTS voices in general (VITS in ONNX; Vietnamese vi_VN-vais1000 above) on /v1/audio/speech and
  /v1/realtime. Phonemes come from espeak-ng, which you install yourself; it runs as an
  external program (GPL-3.0, never linked or shipped), and a load-time self-check refuses the voice
  if it is missing or produces the wrong phonemes.
• tools/livecaptions: live captions from a microphone or any PulseAudio/PipeWire source through
  /v1/audio/transcriptions, optionally translated by a chat model. See its README.
• A CHANGELOG.md in the repository, starting with this release.
• tools/semsearch: semantic search over a folder of Markdown notes using the server's own
  /v1/embeddings (and optional /v1/rerank). See tools/semsearch/README.md.
• The README now credits both authors and states how the project is built (agentic engineering).
• Fix: the GPU text path's rare string fallback no longer risks double chat-templating either.
• Fix: requests without a seed now really sample. Every text engine (GPU, NPU, vision) used to
  fall back to OpenVINO's fixed default seed, so identical requests at temperature > 0 returned
  identical text. Each unseeded request now gets a fresh random seed; an explicit seed is still
  reproducible, with one caveat: with prefix caching on (the default), the first request for a
  prompt can differ from its later repeats, which reuse its cached prompt blocks (measured on GPU and
  vision models; the repeats are identical to each other). Set enable_prefix_caching: false for
  strict reproducibility.

Embeddings

• /v1/embeddings has its own limits instead of sharing the completions array cap (16):
  max_embedding_inputs (default 256 inputs) and max_embedding_batch_tokens (default 32768
  padded tokens — inputs × the longest input). The token budget bounds GPU memory: a batch of 256
  full 512-token chunks held ~7 GB on multilingual-e5-large (Lunar Lake), and the server keeps that
  memory until the model is unloaded.
• The input-length limit is now right for XLM-RoBERTa models (multilingual-e5, bge-reranker-v2-m3):
  512 / 8192 tokens, not 514 / 8194. The extra 1–2 tokens were silently embedded with untrained
  positions instead of being rejected.

Speech translation

• New POST /v1/audio/translations (OpenAI-compatible): speech in any Whisper language → English
  text, with the same request/response formats as transcriptions. It needs a Whisper model trained
  for translation — whisper-large-v3 translates well; large-v3-turbo returns untranslated text.
  English-only (.en) models are refused with 400 model_cannot_translate.

Honest errors and documented limits

• A request over the 50 MiB body limit now gets 413 request_too_large naming the limit. Audio
  and image uploads used to get 400 "could not read field 'file'", and JSON bodies 400
  "malformed request body".
• INTERFACE.md has a new "Request limits and endpoint behaviour" section: the upload cap,
  transcription fields and long-audio behaviour (no 30-second limit; ~27 min of 16 kHz WAV fits),
  the new /v1/audio/translations route, the embedding limits, and /v1/realtime's audio format and
  voice-detection values (it has no transcription-only mode).

Hostname no longer disclosed to unauthenticated callers

• x-ruvi-host used to go out on every response, including the open /health and 401s.
  It is now sent only in reply to a request with a valid inference or admin key. The same gate
  applies to host in image generation_metadata and in GET /v1/admin/models' server block.
  A server with no keys configured never sends it.

Measured device memory

• New /metrics gauges rustedvino_process_gpu_memory_bytes (what the process really holds, per
  driver and region) and rustedvino_model_gpu_memory_estimate_bytes (per model), read from the
  kernel on Linux. GET /v1/admin/models shows gpu_memory_estimate_gb, and the server logs a
  warning when a model's measured memory exceeds its configured vram_gb. See METRICS.md.

Upgrading from 0.6.8

No config changes are required. NPU responses change in visible ways: sampling is now actually
applied (outputs differ from the old accidental greedy decoding unless you send temperature: 0),
token counts are exact, and truncated NPU answers report length.
Tooling that reads x-ruvi-host must send a valid key; a server without keys no longer sends it.
Embedding clients only gain room: any request valid under 0.6.8 (≤ 16 inputs of ≤ 512 tokens =
8192 padded tokens) stays within the new defaults.

Apache-2.0. Third-party notices in THIRD-PARTY.md.

RustedVINO 0.6.8

First public release of RustedVINO — an OpenAI-compatible inference server for Intel hardware, written in Rust.

It runs the whole Intel stack, not just the discrete GPU: Arc dGPUs (B50 / B60 / B70), Lunar Lake's integrated Arc GPU, the NPU, and CPU. Each model lands on a device through capability-based placement rather than a hardcoded device name, so a box without a second GPU boots and serves what it can.

Download, don't compile

rustedvino-linux-x86_64-v0.6.8.tar.gz is self-contained: the OpenVINO 2026.2.1 runtime ships inside it, so there is no separate OpenVINO install, no LD_LIBRARY_PATH, and nothing to set up beyond a model directory and a config file. Built in a clean Ubuntu 24.04 container, so the glibc floor is 2.39 — it runs on genuine 24.04, not only on whatever the build box happened to have.

tar xzf rustedvino-linux-x86_64-v0.6.8.tar.gz
cd rustedvino-linux-x86_64-v0.6.8
cp config.example.json config.json   # point models_dir at your models
./rustedvino

Verify the download first if you like:

sha256sum -c rustedvino-linux-x86_64-v0.6.8.tar.gz.sha256

See README.txt inside the bundle, and CONFIG.md in this repository for every setting.

What it serves

Chat completions (streaming and non-streaming), embeddings, reranking, speech-to-text, text-to-speech, image generation, vision, an admin API for loading and unloading models, Prometheus metrics, and a realtime voice WebSocket. The full route table with methods and descriptions is in INTERFACE.md.

Concurrency goes through OpenVINO GenAI's ContinuousBatchingPipeline, which batches many users' decode steps into a single GPU pass instead of serialising them.

Why it exists

It replaces stormVINO, my own earlier Python server, whose asyncio/SSE stack added a fixed ~7 ms per token — on a 15 ms token cycle, that is nearly half the budget spent scheduling rather than computing.

Measured on an Arc Pro B60, against stormVINO's ~45.6 tok/s:

Load	RustedVINO	vs. stormVINO
1 user	69.0 tok/s	1.5×
8 users (~1k tok)	407 tok/s	8.9×
16 users (128 tok)	814 tok/s	17.8×

These were taken with kv_cache_precision: "u8" — now the server's default, and part of the configuration under test rather than a free variable. A reproducible benchmark harness anyone can run on their own hardware is on the roadmap; until it lands, treat the numbers as illustrative rather than independently verified.

Building from source

Still fully supported and documented under Setup in the README — you need the OpenVINO pip packages, a C++17 compiler, and Rust 1.91 or newer.

Licensing

Apache-2.0. The bundled binary links Symphonia (MPL-2.0) and ships Intel's OpenVINO runtime (Apache-2.0); both are attributed in THIRD-PARTY.md.