Configuration

Configuration

.env

Variable	Default (setup.sh)	Meaning
AGENTBOX_UID / AGENTBOX_GID	your ids	Identity agentbox, the launcher and Ollama run as; owns the data dir
DOCKER_GID	getent group docker	Lets the launcher use the Docker socket
AGENTBOX_AGENT_UID / _GID	64000	Agent identity. Should not exist on the host
AGENTBOX_DATA	~/.local/share/agentbox	Holds worktrees/, sockets/, git/ (shadows), run/ (launcher socket) and logs/
AGENTBOX_REPOS_ROOT	~/Projects	Only repos under here can be delegated (mounted read-write into agentbox)
AGENTBOX_VENVS	~/.cache/local-agent-venvs	Venvs for python_env (venvs under the repos root are allowed too)
OLLAMA_MODELS_DIR	$AGENTBOX_DATA/ollama	Ollama model store
AGENTBOX_MCP_TOKEN	random	Bearer token Claude Code must send
AGENTBOX_AGENT_RUNTIME	unset	e.g. runsc to run agents under gVisor
AGENTBOX_PORT	8765	MCP port (127.0.0.1)

Agent resource caps are set in the launcher's environment: AGENTBOX_AGENT_MEM_GB (8), AGENTBOX_AGENT_CPUS (4) and AGENTBOX_AGENT_PIDS (512).

config/models.yaml

Copied from config/models.example.yaml by setup. Apply edits with docker compose restart agentbox agentbox-launcher.

paths:
  models_dir: /srv/models       # host dir mounted at /models in vLLM containers
  vllm_cache: /srv/vllm-cache
default_model: nemotron-3.5-lightning
models:
  my-vllm-model:
    backend: vllm
    description: "shown to the planner by list_models"
    max_agents: 8                # concurrent delegate_task runs; keep <= --max-num-seqs
    port: 9110                   # must match --port in args
    image: vllm/vllm-openai:<tag>
    entrypoint: [vllm]           # optional
    ipc_host: true               # or shm_size: 16g
    memory: 100g                 # container memory cap
    load_timeout_s: 1800
    env: { VLLM_CACHE_ROOT: /cache }
    args: >-
      serve /models/my-model --served-model-name my-vllm-model --host 127.0.0.1 --port 9110
      --enable-auto-tool-choice --tool-call-parser <parser> ...
  my-ollama-model:
    backend: ollama
    ollama_model: some-model:tag # pulled on first use
    max_agents: 4                # keep <= OLLAMA_NUM_PARALLEL

Requirements for a model to work as a Claude Code executor:

• The server must speak the Anthropic Messages API (/v1/messages). vLLM ≥ 0.29 and Ollama ≥ 0.14 do.
• It needs reliable tool calling (--enable-auto-tool-choice with the right --tool-call-parser on vLLM).
• It needs a context of ≥ 64k tokens. Claude Code's system prompt and tools alone are ~20k; the compose file sets OLLAMA_CONTEXT_LENGTH=65536.
• args is split on whitespace, so write JSON arguments without spaces.

GPU arbitration

There is one GPU family at a time:

• Starting a vLLM model unloads Ollama models and stops any idle other vLLM model.
• Using an Ollama model stops idle vLLM backends.
• If agents are still running on the other side, the request waits, retrying every 20 s for up to an hour. It shows up in wait_seconds.

Shipped example (NVIDIA GB10, 128 GB unified memory)

id	backend	notes
nemotron-3.5-lightning (default)	Ollama	30B-A3B, agent-tuned, ~25 GB; no load wait
nemotron-3.5-bolt	vLLM	74B-A9B IQ2 + DFlash drafter; needs a vLLM nightly image built per NVIDIA's Bolt recipe; PIECEWISE CUDA graphs for 8 concurrent agents
ornith-1.5-35b	vLLM	35B-A3B NVFP4 + MTP; serves many sequences, which suits fan-out