offgrid 0.1.0

First release of offgrid, a small native library that adds local, offline AI to an application: chat, tool calling, structured JSON output, embeddings, and image and audio input. Link the library, ship a GGUF model next to your app, and that's it. There's no daemon, no cloud, and no network access at runtime.

It is written in Rust on top of llama.cpp, runs on any GPU through Vulkan, and falls back to the CPU automatically.

What's in 0.1.0

• Chat: streaming, stop strings, sampling options and seeds. Requests use the OpenAI format; the session reuses its KV cache, so follow-up turns only process new tokens.
• Chat templates: each model's own Jinja template, rendered the Hugging Face way, with llama.cpp's built-in formats as a fallback.
• Structured output: response_format (json_object / json_schema) compiled into a grammar, so the output is valid by construction.
• Tool calling: OpenAI tools / tool_choice with each model's native call format (Hermes/Qwen, Llama 3.x, Mistral, Qwen3-Coder, generic JSON), plus a system-prompt fallback for templates without tool support (Gemma 3).
• Reasoning: <think> blocks are split into reasoning_content and streamed separately.
• Vision and audio: images (PNG, JPEG, …) and audio (WAV, MP3, FLAC) as OpenAI content parts, with encoded media reused across turns.
• Embeddings: batched, L2-normalized, float or base64.
• Cancellation from any thread, scoped to a single request.
• offgrid-server: OpenAI-compatible HTTP server (/v1/chat/completions with SSE, /v1/embeddings, /v1/models) with a session pool, an optional API key and graceful shutdown.

Use it from Rust (the offgrid crate), from C/C++ or anything with a C FFI (offgrid.h, JSON in and out), from Deno (bindings/deno), or over HTTP with any OpenAI client.

Downloads

Package	Platform	Contents
offgrid-0.1.0-linux-x86_64.tar.gz	Linux x86_64, glibc 2.28+ (Ubuntu 20.04+, Debian 10+, RHEL 8+)	lib/liboffgrid.so, llama.cpp/ggml libraries, Vulkan and CPU backends, bin/offgrid-server, include/offgrid.h
offgrid-0.1.0-windows-x86_64.zip	Windows x86_64 (tested on Windows 11)	bin/offgrid.dll, llama.cpp/ggml DLLs, Vulkan and CPU backends, bin/offgrid-server.exe, include/offgrid.h, MSVC and MinGW import libraries

Each package is a relocatable folder: keep it together and it runs from anywhere. It includes 14 CPU variants (the best one is picked at load time), and Vulkan is optional: without a GPU or Vulkan driver it runs on the CPU. The only runtime dependencies are the system C library and, for the GPU, the graphics driver's Vulkan loader.

Quick start

tar xzf offgrid-0.1.0-linux-x86_64.tar.gz
offgrid-0.1.0-linux-x86_64/bin/offgrid-server --model Qwen3-4B-Instruct-2507-Q4_K_M.gguf

Then point any OpenAI client at http://127.0.0.1:8080/v1. On Windows, run bin\offgrid-server.exe the same way. Rust projects can depend on the tag directly:

offgrid = { git = "https://github.com/Pinta365/offgrid", tag = "v0.1.0" }

See the README for the C and Deno APIs and for building from source.

Tested

• Linux: RX 9070 XT (Vulkan) and CPU, full test suite.
• Windows 11: RTX 3060 Laptop GPU (Vulkan) and CPU, both the release package and a native MSVC build with the full test suite. Laptops with an AMD iGPU plus a second GPU are handled (AMD's switchable-graphics Vulkan layer is disabled for the process).
• Models: Qwen3-4B-Instruct-2507, Llama 3.2 3B, Gemma 3 4B, Qwen3-0.6B (reasoning), Qwen3-Embedding-0.6B, Qwen2.5-VL-3B, Qwen2.5-Omni-3B and Ultravox 0.5. Qwen3-4B Q4_K_M generates about 150–175 tokens/s on the RX 9070 XT and about 43 tokens/s on the RTX 3060 Laptop.

Known limitations

• Linux and Windows on x86_64 only; macOS and ARM are not built yet.
• The server serves one chat model and one embedding model, with no per-request model switching and no legacy /v1/completions.
• Tool calls stream as one complete fragment each, not argument by argument.
• Embeddings are text-only, and rerankers aren't supported.

The full list is in the README.