GPU Arbitration

Explanation. How agentbox shares one GPU between Ollama and on-demand vLLM servers without pulling a model out from under a running agent.

• One vLLM model at a time. Starting vLLM model A stops any idle vLLM container for model B. If B has running agents, the request waits.
• One GPU family at a time.
  • Before a vLLM model starts, Ollama is asked to unload its models (keep_alive: 0).
  • Before an Ollama model is used, idle vLLM containers are stopped.
  • Either switch waits while agents on the other side are running.
• Waiting is bounded and visible. agentbox retries every 20 s for up to an hour, and reports the time as wait_seconds. Batch tasks per model instead of alternating models.
• Remote models don't count. They use no local GPU and never wait for arbitration.
• Concurrency per model is max_agents in models.yaml. Keep it ≤ vLLM's --max-num-seqs or OLLAMA_NUM_PARALLEL.

Arbitration decides which containers run, not how much memory they take. Size each vLLM entry's --gpu-memory-utilization / --kv-cache-memory-bytes for your GPU. On unified-memory machines (for example NVIDIA GB10), keep a host-level memory guard (earlyoom, or your own watchdog) as a backstop.