v0.1.1

Voice agents that can look things up while the caller waits, and a measured answer to "how many callers on one GPU".

pip install fusion-runtime

Tools

A tool is a plain Python function. The model reads its name, docstring and type hints, calls it when it needs to, and answers with what it returned.

from fusion_runtime import Agent, LLM, tool

@tool
async def order_status(order_id: str) -> dict:
    """Look up where an order is and when it will arrive.

    Args:
        order_id: The order number, as the caller reads it out.
    """
    return await orders.lookup(order_id)

agent = Agent(prompt="...", llm=LLM("vllm:hf:Qwen/Qwen2.5-7B-Instruct-AWQ"), tools=[order_status])

What voice needs from tools that text agents don't:

• No dead air. What the model says before a lookup ("Let me check.") is spoken while the tool runs.
• Interruptions win. Talk over the wait and the lookup is cancelled. A tool that fails or times out (10 s by default) tells the model what went wrong instead of ending the call.
• Lookups are remembered. Our first GPU test caught the agent inventing a second order's status: it remembered its own answer but not that it came from a lookup, so it imitated it. Tool calls and results now stay in the conversation.

Tools need a model server that can call them: vLLM, SGLang, llama-server --jinja or a hosted API. The in-process llama.cpp model can't yet, and frun up refuses an agent with tools on it rather than ignoring them. A runnable example is examples/tools_agent.py.

Several callers on one GPU, measured

One RTX 3090, Qwen2.5-7B-Instruct-AWQ, Whisper small and Kokoro on the same card, three turns per caller. Server response, turn end to first audio:

Callers	Model in-process (llama.cpp)	vLLM	SGLang
1	~460 ms	398 ms	410 ms
4	~740 ms	698 ms	848 ms
8	~4600 ms	1086 ms	1054 ms
12	~7500 ms	1083 ms	1264 ms
16	—	2064 ms	1598 ms

With the model on vLLM or SGLang, its first token stayed at 36–72 ms from one caller to sixteen. What grows now is speech, which isn't batched across callers yet: Kokoro's first audio reaches 1.5–2 s at sixteen callers. That's the next thing to fix. With every caller asking about an order, vLLM made the lookup in 122 of 123 turns and SGLang in 109, skipping only questions it had already looked up.

Measure your own with scripts/concurrency_check.py, which prints the median per stage so you can see which one starts to queue.

Also in this release

• Servers by name: LLM("vllm:..."), LLM("sglang:..."), LLM("llama_server:...") find a server you started at its usual address, or at url=.
• Settings that can't be misspelt: each runtime declares what it reads, and a typo fails at startup with "did you mean …" instead of being ignored. New llama.cpp settings: flash_attn, kv_cache_type, use_mlock, main_gpu, tensor_split, seed, llama_kwargs.
• FUSION_LLM_URL alone moves an agent to a server at another address; the model's name comes from the agent file.
• A clearer error when models are missing: it names the agent file and the command that pulls exactly its models.
• scripts/pod_start.sh brings a GPU pod (Runpod or any other) from any state to a running agent on vLLM or SGLang, checking the driver's CUDA version and the disk before installing anything.

Not a fit yet

• Tools on the in-process llama.cpp model
• More than about twelve callers on one GPU: the speech stages queue after that
• Telephony: not built
• Long calls: unmeasured

Links

• Docs: https://fusion-runtime.dev/docs
• PyPI: https://pypi.org/project/fusion-runtime/

Apache-2.0.