Repository navigation
Multi GPU
With the managed backend the app can use several GPUs in two different ways. Choose on Settings ▸ GPUs.
The tab lists every NVIDIA GPU found by nvidia-smi. Tick the ones to use.
Each ticked GPU gets its own llama-server with a full copy of the model. Images are handed out to whichever server is free, so two GPUs caption roughly twice as fast as one.
- Best when: the model plus its context fits on a single card.
- Mixed cards are fine. A faster card simply finishes more images. In testing, a 4060 Ti generated about 44 tokens/s and a 5060 Ti about 29 tokens/s on the same model, and the work balanced itself.
- Ports used: First port (default 8090) for the first GPU, +1 for the next, and so on.
- VRAM per card = model + context + vision projector.
A single llama-server is started with the model spread over all ticked GPUs (--split-mode layer).
- Best when: the model does not fit on one card, e.g. a 26–31B model at Q5 on two 16 GB cards.
- Only one server runs, so captions are produced by one model at a time (use Concurrent requests to batch, below).
-
Tensor split: leave blank for automatic, or give ratios per GPU, e.g.
16,16for equal cards or3,2to put more on the first. If one card also drives your monitor or other programs, give it a smaller share. - Splitting adds a little overhead from moving data between cards, so don't use it for models that fit on one GPU.
On the same tab. Each server handles this many images at once.
-
1(default) is the lowest VRAM use. -
2–4can increase throughput, especially in Split mode where a single request does not keep both GPUs busy. - The context is multiplied by this number, so it costs VRAM. If loading fails or you run out of memory, lower it.
Total workers = (servers) × (concurrent requests).
| Situation | Use |
|---|---|
| Model fits on one GPU, you have 2+ GPUs | Parallel |
| Model is too big for one GPU | Split |
| One GPU only | Either; tick it and leave the mode alone |
| Two different-size cards, small model | Parallel (or tick just the bigger card) |
| Want the highest-quality large model | Split with a bigger model, then raise Concurrent requests |
Run nvidia-smi in a terminal (or look at Task Manager ▸ Performance ▸ GPU) while the model loads. Other programs can use several GB already; browsers, games and other AI tools all count. If a model that should fit does not load, close them or lower Context per request and Concurrent requests.
If nvidia-smi is not found the GPU list is empty and llama-server picks its own devices. Multi-GPU control in this app relies on CUDA_VISIBLE_DEVICES, so it is only supported for NVIDIA/CUDA. For other hardware run your own server and use an external backend.
Rulz Training Captioner is open source under the MIT License. The RulZ name and logo are excluded from it. · Report an issue
Getting started
Using the app
Models and GPUs
Reference