Training Captioner Models and Backends

Models and Backends

The app talks to a vision-language model through an OpenAI-compatible chat API. It can either start llama.cpp's llama-server for you (managed) or use a server you already run (external). Pick one under Settings ▸ Backend.

Managed backend (llama.cpp)

This is the default and the only mode that can split a model across GPUs for you.

What you need

llama-server.exe: see Installation
A vision-capable GGUF model
Its mmproj (vision projector) GGUF

Without the mmproj file a model cannot see images, even if it is a "VL" model. Download both files from the same model page and keep them in the same folder.

Choosing a model

Families that llama.cpp supports for vision include Qwen-VL, Gemma 3 / Gemma 4, MiniCPM-V, InternVL and Mistral Small vision. Look for a repository that ships a file named something like mmproj-*.gguf next to the main model.

Rules of thumb:

Choice	Guidance
Size	7–12B models caption well and are fast. 25–30B models write more accurate and detailed captions but need two GPUs or a big card.
Quantisation	`Q8_0` is nearly lossless; `Q5_K_M` / `Q4_K_M` are much smaller with small quality loss. Smaller quants leave VRAM for context and parallel requests.
Mixture-of-experts (MoE) models such as "26B-A4B"	Only a fraction of the weights run per token, so they are fast for their size, but all the weights still have to fit in VRAM.
Thinking models	Keep Disable thinking on (see below) unless you specifically want reasoning.
Content	Use whichever models you like; the app does not filter. Choose a model whose own policy suits your dataset.

Settings to fill in

Model (.gguf) and Vision projector (mmproj)
Context per request: the default 4096 is enough for one image plus a caption on most models. An image takes several hundred to a few thousand tokens depending on the model and Resize longest image side. Raise it if the log says the prompt does not fit.
GPU layers (-ngl): 99 puts the whole model on the GPU(s). Lower it to keep some layers on the CPU if you run out of VRAM (much slower).
Extra llama-server args: anything else, e.g. --flash-attn on. Passed straight through.

See Settings Reference for everything.

What the app runs

For each server it launches something like:

llama-server.exe -m <model> --mmproj <mmproj> -ngl 99 -c <context x slots> -np <slots>
                 --host 127.0.0.1 --port <port> --reasoning off

Servers only listen on 127.0.0.1 (your own machine). The exact command line for each server is printed at the top of View ▸ Log.

The servers start the first time you caption, then stay loaded so later batches start instantly. They are restarted automatically if you change any model/GPU setting, and stopped when you choose Server ▸ Stop Servers or close the app. Server ▸ Start Servers loads them in advance.

Thinking models

Many recent models "think" before answering. For captioning that wastes time and, with a small token limit, can use the whole budget and return an empty caption.

Disable thinking (Settings ▸ Generation, on by default) turns thinking off at the server and in each request.
If you want thinking, untick it and set a Thinking limit (default 1024 tokens; -1 = unlimited). The limit is added on top of Max new tokens. It is only enforced for the managed backend.

Using an external server

Choose External OpenAI-compatible server(s) and fill in:

Endpoint URLs: one base URL per line, e.g. http://127.0.0.1:1234/v1. /v1 is added if missing.
Model name: required by LM Studio and Ollama (the name shown in that program)
API key: only if the server asks for one

Works with LM Studio, Ollama (http://127.0.0.1:11434/v1), vLLM, a llama-server you started yourself, or any hosted OpenAI-compatible endpoint.

Parallel captioning: list several URLs and images are shared out between them. The GPUs tab's Concurrent requests per server is then the number of simultaneous requests per URL.

Limits of external mode:

The app cannot start or stop the server, split it across GPUs, or enforce the thinking limit. Configure those in the other program.
The model must accept images in image_url messages (a vision model with its projector loaded).