Repository navigation
Fusion: a cloud or local planner driving llama-server parallel slots as a pool of agent workers #32211
Replies: 1 comment
|
@sosidudku1 The 3-worker fan-out delay you describe maps directly to VRAM contention in a single llama-server instance. When parallel slots share one GPU, each slot reserves a fixed portion of the context window. If the combined context for all workers exceeds physical memory, the server evicts KV pairs to free space. This eviction forces recomputation during the next forward pass, stalling the loop even if the compute pipeline is idle. Splitting independent tasks helps because each slot consumes a smaller, isolated context window, reducing pressure on the shared cache budget. Chaining dependent tasks, however, forces the system to hold larger intermediate states, often exceeding available memory headroom after accounting for orchestrator overhead. The slowdown stems from this memory pressure where the eviction policy compels the model to re-process context it just generated. |
Uh oh!
There was an error while loading. Please reload this page.
I'm a co-founder of Atomic Agent, an MIT local-first agent that runs on llama.cpp first. Sharing how our multi-agent mode uses llama-server, because the interesting part is mostly on your side of the stack.
The setup. Fusion gives one job to two models. An orchestrator plans, writes a brief for each part and reviews what comes back. It is not allowed to edit files or run shell commands for the whole turn. Workers are short-lived agent sessions that take one brief each, write their files straight to disk and report back. The usual pairing is a cloud orchestrator with local workers, but
/runmode swaptrades the seats so a local model plans for cloud workers.Where llama-server comes in. On a local worker leg the fan-out width is bounded by llama-server's
--parallelslot count. We start the server ourselves, and withlocalModels.managed.parallel: "auto"(the default) the slot count and the context window are fitted together; you can also pin a number. So a 3-worker fan-out on one GPU is three slots of one server sharing the KV cache budget, not three servers.Things we learned the hard way:
/healthfirst, so one crashed server does not burn every remaining brief.Config shape (terminal version,
~/.atomic-agent/config.json):Optional and off by default: a git worktree per worker with a three-way merge back, a fresh-context judge that checks each part against an acceptance list, and automatic replanning of parts that did not close.
How it works in more detail: https://atomicagent.io/fusion/
Code: https://github.com/AtomicBot-ai/atomic-agent (Fusion tools live in
src/tools/fusion/)Curious whether anyone here runs several agents against one llama-server and found a better way to size
--parallelagainst context per slot.All reactions