Training Captioner Multi GPU

Multi-GPU

With the managed backend the app can use several GPUs in two different ways. Choose on Settings ▸ GPUs.

The tab lists every NVIDIA GPU found by nvidia-smi. Tick the ones to use. The numbers are the ones nvidia-smi shows, so ticking GPU 1 uses the card nvidia-smi calls GPU 1, even with mixed cards (before 1.3.0, CUDA could number the fastest card first and pick the other one).

With no GPU ticked, one llama-server starts and picks the GPUs itself: it spreads one copy of the model over all of them, much like Split. The tab says so under the list. Before 1.4.0 the tab showed every GPU ticked when none was saved, and pressing OK then saved them all, which quietly switched to one server per GPU in Parallel mode. If you see no GPU ticked after updating, that is what was running all along.

Parallel: one copy of the model per GPU

Each ticked GPU gets its own llama-server with a full copy of the model. Images are handed out to whichever server is free, so two GPUs caption roughly twice as fast as one.

• Best when: the model plus its context fits on a single card.
• Mixed cards are fine. A faster card simply finishes more images. In testing, a 4060 Ti generated about 44 tokens/s and a 5060 Ti about 29 tokens/s on the same model, and the work balanced itself.
• Ports used: First port (default 8090) for the first GPU, +1 for the next, and so on.
• VRAM per card = model + context + vision projector.
• If one server stops responding during a run (for example it ran out of VRAM and closed), the other servers take over its images and the log says which server stopped. If none respond, captioning stops with a message and the remaining images stay uncaptioned rather than being marked as failed.

Split: one model across several GPUs

A single llama-server is started with the model spread over all ticked GPUs (--split-mode layer).

• Best when: the model does not fit on one card, e.g. a 26–31B model at Q5 on two 16 GB cards.
• Only one server runs, so captions are produced by one model at a time (use Concurrent requests to batch, below).
• Tensor split: leave blank for automatic, or give ratios per GPU, e.g. 16,16 for equal cards or 3,2 to put more on the first. If one card also drives your monitor or other programs, give it a smaller share.
• Splitting adds a little overhead from moving data between cards, so don't use it for models that fit on one GPU.

Concurrent requests per server

On the same tab. Each server handles this many images at once.

• 1 (default) is the lowest VRAM use.
• 2–4 can increase throughput, especially in Split mode where a single request does not keep both GPUs busy.
• The context is multiplied by this number, so it costs VRAM. If loading fails or you run out of memory, lower it.

Total workers = (servers) × (concurrent requests).

Choosing a mode

Situation	Use
Model fits on one GPU, you have 2+ GPUs	Parallel
Model is too big for one GPU	Split
One GPU only	Either; tick it and leave the mode alone
Two different-size cards, small model	Parallel (or tick just the bigger card)
Want the highest-quality large model	Split with a bigger model, then raise Concurrent requests

When the servers close

The servers stay loaded between runs and close when you choose Server ▸ Stop Servers or close the app. Since 1.3.0 they also close if the app crashes or is ended from Task Manager, so they never keep holding VRAM. Since 1.4.0 stopping them is almost instant (it used to take most of a second per server), so changing a setting and captioning again restarts the servers sooner.

Checking your VRAM

Run nvidia-smi in a terminal (or look at Task Manager ▸ Performance ▸ GPU) while the model loads. Other programs can use several GB already; browsers, games and other AI tools all count. If a model that should fit does not load, close them or lower Context per request and Concurrent requests.

Not using NVIDIA?

If nvidia-smi is not found the GPU list is empty and llama-server picks its own devices. Multi-GPU control in this app relies on CUDA_VISIBLE_DEVICES, so it is only supported for NVIDIA/CUDA. For other hardware run your own server and use an external backend.