Support multi-llama.cpp llm targets #61720
Closed
samuelmattjohnston
started this conversation in
Feature Requests
Replies: 2 comments
|
This would be nice, right now you have to use openai_compatible |
0 replies
|
I am guessing the discussions are not the right place to try and do this. Anyone can take the code idc |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
What are you proposing?
Support for configuring multiple named llama.cpp server instances, instead of just one. Each instance gets its own entry in the model dropdown with full autodiscovery, matching how
openai_compatibleandanthropic_compatiblealready support multiple named instances.Why does this matter?
I run multiple machines, each better suited to different models, and want to use them all from Zed at once. Dynamic reconfiguration (add/remove a server without restarting) is a big part of what makes this useful.
This makes it possible to run multiple locally-hosted llama.cpp servers with autodiscovery, instead of having to fall back to
openai_compatible(manually listing models, no autodiscovery) just to get a second instance.Are there any examples or context?
Worlfow screenshots





Working implementation on my branch:
https://github.com/samuelmattjohnston/zed/tree/feat/multi-llama-support
Related: #33992 (general "support multiple instances of a provider" request, resolved for
openai_compatible/anthropic_compatible, but llama.cpp was added afterward and never got the same treatment).Possible approach
I needed this for myself, so I already implemented it on the branch linked above, happy to open a PR from it, but wanted to check in here first since there wasn't an existing issue for llama.cpp specifically. Also I'm not really a good rust dev, so I had help from AI - qwen/claude/more to do this.
All reactions