ExLlamaSharp 1.2.1-beta

Pre-release: zero-gaps on top of continuous batching. This is not the GitHub Latest download; stable remains 1.1.1.

Install

Download ExLlamaSharp-Setup-win-x64.exe from this pre-release (not the one-liner, which still installs 1.1.1). Run it (UAC), then open http://127.0.0.1:14563

• Admin UI: admin / changeme (change this in Setup)
• API key: sk-exllamasharp-dev

Changes

• A/B, tenants, and LoRA adapter HTTP APIs are real (CRUD + OpenAI X-Ab-Test-Id / X-Adapter-Id)
• EXL3 worker: LoRA load/unload, tools/JSON-schema prompt hints, multi-GPU CUDA_VISIBLE_DEVICES, speculative draft settings
• Embeddings via ONNX when model.onnx is present (local fallback otherwise)
• Quantize/import jobs call real Python convert / folder registration
• SignalR metrics + Charts / A/B UI; tenant filtering when multi-tenancy is enabled
• Worker-only production generate; OpenAI images/audio stay 501 (Media version later)

Notes

Treat this as a beta. Keep a 1.1.1 install if you need the last stable build.

ExLlamaSharp 1.2.0-beta

Pre-release: concurrent EXL3 sequences now share one GPU forward pass (jsonl-v2 worker + .NET admission). This is not the GitHub Latest download; stable remains 1.1.1.

Install

Download ExLlamaSharp-Setup-win-x64.exe from this pre-release (not the one-liner, which still installs 1.1.1). Run it (UAC), then open http://127.0.0.1:14563

• Admin UI: admin / changeme (change this in Setup)
• API key: sk-exllamasharp-dev

Changes

• Real continuous batching: Python ExLlamaV3 Generator.iterate() multiplexes jobs; .NET admits by priority and streams SSE live
• OpenAI /v1/chat/completions and /v1/completions share one runner and SSE writer
• Worker split into locator, jsonl client, admission queue, and protocol types (SOLID)
• Mock and native engines unchanged

Notes

Treat this as a beta. Keep a 1.1.1 install if you need the last stable build.

ExLlamaSharp 1.1.1

Admin Models now shows whether each EXL3 model fits this GPU's VRAM. The installer still bundles the ExLlamaV3 CUDA extension.

Install

Download ExLlamaSharp-Setup-win-x64.exe, run it (UAC), then open http://127.0.0.1:14563

• Admin UI: admin / changeme (change this in Setup)
• API key: sk-exllamasharp-dev

irm https://raw.githubusercontent.com/Kortexio/ExLlamaSharp/main/packaging/install-web.ps1 | iex

Changes

• Models shows a Fits / Tight / Too large badge against this machine's GPU VRAM (estimate from weight size; it does not auto-select a model)
• Removed the Hugging Face Downloads column from the library list
• Setup pip steps no longer fail when pip uninstall has nothing to remove
• Setup.exe bundles the ExLlamaV3 CUDA .pyd, worker deps, Python installer and VC++; PyTorch CUDA still downloads during install

ExLlamaSharp 1.1.0

Reliability release for the Windows Admin UI, service startup, and model downloads.

Install

Download ExLlamaSharp-Setup-win-x64.exe, run it (UAC), then open http://127.0.0.1:14563

• Admin UI: admin / changeme (change this in Setup)
• API key: sk-exllamasharp-dev

irm https://raw.githubusercontent.com/Kortexio/ExLlamaSharp/main/packaging/install-web.ps1 | iex

Fixes

• Admin buttons only become clickable after the Blazor circuit is live (no dead first-run UI)
• Service no longer crashes the host when a GPU model fails to load (ninja/CUDA)
• Auto-load on boot now respects Load model on startup
• Setup wizard actually enqueues Hugging Face downloads and no longer skips a failed admin save
• Hugging Face pulls install huggingface_hub if missing
• OpenAI API returns 502 when inference fails instead of an empty 200
• Backup restore now brings back users, API keys, and model records
• Bind/port from Settings are applied on the next service start
• Native engine jobs call EngineStep so they cannot hang forever

ExLlamaSharp 1.0.0

First public Windows release of ExLlamaSharp — a local LLM server with an OpenAI-compatible API, Blazor admin UI, and EXL3 inference on NVIDIA GPUs.

Install

Download ExLlamaSharp-Setup-win-x64.exe, run it (UAC), then open http://127.0.0.1:14563

• Admin UI: admin / changeme (change this before production)
• API key: sk-exllamasharp-dev

One-liner:

irm https://raw.githubusercontent.com/Kortexio/ExLlamaSharp/main/packaging/install-web.ps1 | iex

Included

• Windows service + system tray
• Chat playground with streaming completions
• Hugging Face EXL3 search and download (Jobs progress)
• API keys, usage audit, setup wizard, diagnostics
• ZIP fallback for scripted installs

See the README for the full feature list and what is still stub/partial.