Add a Local Model

How-to. Register a model so delegate_task can use it. Field details are in Configuration.

A model works as a Claude Code executor if:

• the server speaks the Anthropic Messages API (vLLM ≥ 0.29 and Ollama ≥ 0.14 do);
• tool calling is reliable;
• the context is ≥ 64k tokens.

Ollama

# config/models.yaml
models:
  my-coder:
    backend: ollama
    description: "shown to the planner by list_models"
    ollama_model: some-model:tag        # pulled on first use
    max_agents: 4                       # keep <= OLLAMA_NUM_PARALLEL (compose: 4)

The context length comes from OLLAMA_CONTEXT_LENGTH in docker-compose.yml (65536 by default).

vLLM

models:
  my-vllm-model:
    backend: vllm
    port: 9112                          # unique; must match --port below
    image: vllm/vllm-openai:<tag>       # pin a version
    entrypoint: [vllm]
    ipc_host: true                      # or shm_size: 16g
    memory: 100g
    max_agents: 8                       # keep <= --max-num-seqs
    env: { VLLM_CACHE_ROOT: /cache }
    args: >-
      serve /models/my-model --served-model-name my-vllm-model --host 127.0.0.1 --port 9112
      --max-model-len 131072 --max-num-seqs 8 --gpu-memory-utilization 0.6
      --enable-auto-tool-choice --tool-call-parser <parser-for-this-model>

• Weights go in paths.models_dir/<name>, which is mounted at /models/<name>.
• --served-model-name must equal the model id.
• args is split on whitespace, so write JSON values without spaces: {"cudagraph_mode":"PIECEWISE"}.

Apply and test

docker compose restart agentbox agentbox-launcher
scripts/smoke-test.sh my-coder

A vLLM model that exits during load is reported in the error. Its log is at journalctl -t agentbox-<model-id>.