Resource-aware scheduling for local LLMs across multiple agents?

[Parikalp-Bhardwaj]

I've been exploring ZeroClaw and was curious about how it handles resource usage when multiple agents share a local inference provider like Ollama or llama.cpp.

I came across Discussion #8671, where someone ran into model switching because summarization was using a different local model.

It got me thinking about what happens when multiple agents and background tasks are running on a machine with limited RAM/VRAM.

For example:

Agent A -> Model A loaded -> Response completed
                              |
                              v
                    Background summarization
                              |
                              v
                         Model B loaded
                              |
                              v
                    Model A may be unloaded
                              |
                              v
                    Next request to Agent A
                              |
                              v
                    Model A loads again

Depending on the provider's configuration and available memory, repeated model switching could add noticeable latency, especially on smaller machines.

I was wondering whether ZeroClaw could benefit from an optional lightweight scheduling policy for local inference requests.

A few things that came to mind:

• Limit concurrent requests to the same local inference endpoint.
• Prioritize interactive agent requests over non-urgent background tasks.
• Avoid unnecessary model switching when possible.
• Expose basic information about which model was used and how long a request waited before execution.

I'm not suggesting that ZeroClaw should manage GPU memory directly. Ollama and llama.cpp already handle model loading and execution.

I'm more curious about whether coordinating requests at the ZeroClaw runtime level would help when multiple agents are sharing limited local resources.

I also noticed #7951, which discusses effort-based local/cloud model routing. This seems slightly different to me: that issue is about choosing a model based on the task, while this is more about coordinating when local model calls run.

A few questions for the maintainers:

1. Is there already something in ZeroClaw that handles this kind of coordination?
2. Would a per-endpoint concurrency limit be useful, or is this better left to Ollama/llama.cpp?
3. Would it make sense to collect some basic metrics around model switching and request latency before considering a scheduling layer?

I'm interested in understanding how this fits with ZeroClaw's lightweight design. If there's a useful gap here, I'd be happy to explore it further and work on a small implementation.

[Audacity88]

@Parikalp-Bhardwaj Thanks for checking in on this — your distinction from #7951 is right. That issue is about choosing a model; this is about coordinating access to a shared inference endpoint.

1. ZeroClaw currently has bounded channel ingress, per-conversation serialization, and channel-level concurrency limits, but those controls do not form a shared scheduler for calls that happen to use the same provider endpoint. So there is not currently equivalent coordination across agents and background tasks sharing one Ollama or llama.cpp server.

2. A small, opt-in per-endpoint concurrency cap could be useful. Ollama or llama.cpp should still own model loading, memory, and execution; ZeroClaw would only coordinate admission around a configured endpoint. The important design questions are how endpoint identity is normalized and where to put the admission point so every relevant provider call is covered.

3. Yes, collecting evidence first makes sense. A useful initial slice would record the selected provider/model, a coarse call purpose such as an interactive turn or summarization, queue delay, and execution duration, without recording prompt contents. The provider-call path already has some model/provider observability and total call timing, but not shared-endpoint queue timing.

I would not start with priority queues or model-affinity scheduling. Instrumentation plus an opt-in concurrency cap is a much smaller experiment, and the resulting data can show whether priority or model-switch avoidance is actually needed.

There is also a directly related RFC, #10970, covering shared admission and resource bounds. Its current review is asking for a daemon-owned first slice across turn origins, fair queues, a separate tool-execution ceiling, and bounded reserved interactive capacity. Your endpoint-specific proposal adds a useful dimension: different agents can compete for the same inference server even when their own turn budgets are separate. Could you add your proposal to #10970, outlining how to identify shared endpoints, which provider calls the limit would cover, and what to measure? That would help establish how the provider-call limit fits the broader admission design before you start implementation.

[Parikalp-Bhardwaj]

Thanks @Audacity88 for pointing me towards this RFC.

Following up from #10971, I spent some time tracing the provider flow.

HailoOllamaModelProvider already has endpoint-level admission using a shared Semaphore(1), which looks like a useful starting point. The main limitation is that it's process-local, so multiple ZeroClaw processes can still hit the same Ollama/llama.cpp endpoint simultaneously.

One thing I think matters is where we acquire the permit. Since the Reliable provider can retry or fall back to another provider, admission probably belongs around each actual provider attempt rather than the whole logical request.

I also think the resource key should represent the shared endpoint, not just the model name or provider alias, since different models may still compete for the same local inference server.

For a first version, I'd keep it small:

• Per-endpoint concurrency limit.
• Hold the permit for the lifetime of a streaming call.
• Acquire the correct endpoint's permit for retries and fallbacks.
• Basic metrics such as queue wait, execution time, and in-flight requests.

The main design question for me is ownership: should the daemon/host coordinator own the shared cross-process admission state, while providers only expose a stable endpoint/resource identity?

I'd be happy to help with the first implementation once we agree on this boundary. Thanks again for the guidance!

[Audacity88]

@Parikalp-Bhardwaj Thanks for tracing the provider flow and offering to help with the implementation. The Hailo endpoint gate is a useful precedent, though it is specific to that provider and stays within one process.

For the first slice, I support daemon-owned admission shared by the configured agents in that daemon, with providers exposing a stable endpoint identity. That is different from coordinating several independent ZeroClaw processes. Cross-process or multi-daemon arbitration should remain deferred under #10970 rather than becoming a requirement for this endpoint experiment.

Your per-attempt proposal makes sense for retries and fallback. The design should acquire the permit for the endpoint actually contacted, release it before retry backoff or switching endpoints, and retain it until a streaming attempt finishes. Cancellation and permit release need an explicit contract too.

Please add this narrower ownership boundary to #10970, along with endpoint normalization, which provider call paths are covered, and queue-wait, execution-time, and in-flight measurements without prompt contents. The RFC is still awaiting revision, so this would be design input for that revision before an implementation PR. Keep model-affinity scheduling out of this first slice; Ollama and llama.cpp should continue to manage model loading and memory.