Skip to content
This repository was archived by the owner on Oct 28, 2026. It is now read-only.

Multi GPU

speedyrulz edited this page Oct 26, 2026 · 1 revision

Multi-GPU

With the managed backend the app can use several GPUs in two different ways. Choose on Settings ▸ GPUs.

The tab lists every NVIDIA GPU found by nvidia-smi. Tick the ones to use.

Parallel: one copy of the model per GPU

Each ticked GPU gets its own llama-server with a full copy of the model. Images are handed out to whichever server is free, so two GPUs caption roughly twice as fast as one.

  • Best when: the model plus its context fits on a single card.
  • Mixed cards are fine. A faster card simply finishes more images. In testing, a 4060 Ti generated about 44 tokens/s and a 5060 Ti about 29 tokens/s on the same model, and the work balanced itself.
  • Ports used: First port (default 8090) for the first GPU, +1 for the next, and so on.
  • VRAM per card = model + context + vision projector.

Split: one model across several GPUs

A single llama-server is started with the model spread over all ticked GPUs (--split-mode layer).

  • Best when: the model does not fit on one card, e.g. a 26–31B model at Q5 on two 16 GB cards.
  • Only one server runs, so captions are produced by one model at a time (use Concurrent requests to batch, below).
  • Tensor split: leave blank for automatic, or give ratios per GPU, e.g. 16,16 for equal cards or 3,2 to put more on the first. If one card also drives your monitor or other programs, give it a smaller share.
  • Splitting adds a little overhead from moving data between cards, so don't use it for models that fit on one GPU.

Concurrent requests per server

On the same tab. Each server handles this many images at once.

  • 1 (default) is the lowest VRAM use.
  • 2–4 can increase throughput, especially in Split mode where a single request does not keep both GPUs busy.
  • The context is multiplied by this number, so it costs VRAM. If loading fails or you run out of memory, lower it.

Total workers = (servers) × (concurrent requests).

Choosing a mode

Situation Use
Model fits on one GPU, you have 2+ GPUs Parallel
Model is too big for one GPU Split
One GPU only Either; tick it and leave the mode alone
Two different-size cards, small model Parallel (or tick just the bigger card)
Want the highest-quality large model Split with a bigger model, then raise Concurrent requests

Checking your VRAM

Run nvidia-smi in a terminal (or look at Task Manager ▸ Performance ▸ GPU) while the model loads. Other programs can use several GB already; browsers, games and other AI tools all count. If a model that should fit does not load, close them or lower Context per request and Concurrent requests.

Not using NVIDIA?

If nvidia-smi is not found the GPU list is empty and llama-server picks its own devices. Multi-GPU control in this app relies on CUDA_VISIBLE_DEVICES, so it is only supported for NVIDIA/CUDA. For other hardware run your own server and use an external backend.

Clone this wiki locally