Skip to content

ExLlamaSharp 1.4.0-beta

Pre-release
Pre-release

Choose a tag to compare

View all tags
@Kortexio Kortexio released this 10 Sep 18:46
v1.4.0-beta
8a6b6ae

ExLlamaSharp 1.4.0-beta

Multi-GPU (highlight)

  • Real pipeline and tensor parallelism via ExLlamaV3 worker (Settings → Multi-GPU)
  • Configurable GpuSplitGb and GpuMemoryUtilization
  • CUDA device remap so cuda:0 is the highest-VRAM card in the visible set
  • Admin UI shows combined VRAM across visible GPUs (Dashboard / Models / Setup fit badges)
  • Vision tower is skipped under multi-GPU (text-only load); use single-GPU mode for VLM
  • On Windows, prefer pipeline over tensor for JSONL worker stability

API / reliability

  • Oversized prompts now fail fast with prompt_too_long (no more silent hang → client 408)
  • Worker submit errors are routed onto the stream channel correctly

Other

  • Smoke helper tools/exl3_worker/smoke_multigpu.py
  • Docs: troubleshooting for TP timeout and VLM + multi-GPU

Install

Download ExLlamaSharp-Setup-win-x64.exe below (Windows x64). Default UI: http://127.0.0.1:14563

Assets 3
Loading