Agent Mode for 3rd party Model Providers

[underlines]

The new Agent Mode is great. The ability to add custom Model Providers is great too.
A huge drawback is, that Agent Mode is not supported when using Ollama or other providers. Adding Agent Mode support for third party models would further improve the experience for the users:

• Local inference engines such as Ollama, llama.cpp server, vLLM etc. all support tool calling, if the model loaded supports it. LM Studio even shows which model supports Tool Calling with a small icon in the model list.
• Therefore I guess they either maintain this information for popular models manually, or more likely they detect tool calling support by parsing the gguf file format - not sure.
• Ollama itself through the /api/show/ endpoint shows model information, including a capabilities list. Not sure if tool calling is exposed as a capability there.

TLDR: Allow the user to use Agent Mode even for third party Model Provider like Ollama.
If the loaded model doesn't support tool calling an error can be surfaced. No need for VS Code to check for tool calling capability.

[lramos15]

Jan Badertscher (@underlines) We are getting the capabilities from the endpoint. And it does work, I just find many of the models don't like to call tools very much

Here's it failing

Here's it succeeding

[underlines]

Logan Ramos (@lramos15) Thanks for answering.

1. We are getting the capabilities from the endpoint. Does this mean, GitHub Copilot currently fetches capabilities and only exposes models with tool calling capabilities to be shown in the drop down, when Agent Mode is selected? Or do we need to be on insider build to have this? Stable doesn't show any ollama fetched models for Agent Mode.
2. Insider build would probably allow us to use the "github.copilot.chat.byok.ollamaEndpoint" setting as well, right?
3. Does GitHub Copilot properly set a larger context window using "options": {"num_ctx": 32000}? The context size highly affects VRAM usage and should probably be user-editable. If GitHub Copilot uses the standard value by not setting it, it would be 4000, which would mean the agent starts hallucinating by losing initial context pretty fast. Agent-Workflows need at least 16k context.

For your tool calling issues:
4. What Qwen2.5-Coder model did you load? Anything below 24B Parameters will be miserable for Tool Calling from my experience.
5. According to the Function Calling leaderboard, Mistral Small is one of the top open models to perform well on Tool Calling.

[lramos15]

1. Insiders, but everything that goes into insiders comes to stable the following month
2. Insiders as well
3. We do not, this is something that you will need to do before running Ollama serve / in the model file, this is because we're using the Open AI compatible Ollama api to request chat completions and this doesn't support num_ctx. https://www.ollama.com/blog/openai-compatibility
4. This was 32b I believe
5. I've tried that one as well without a ton of luck, I currently am using an RTX 3090. Maybe I don't have enough Vram.

[underlines]

Logan Ramos (@lramos15) Thanks for answering my previous questions—really helpful!

1. I installed the latest Windows Insiders Build, checked for updates, added one Ollama model and one OpenRouter model. Neither model shows up when switching to Agent Mode. What am I doing wrong?

---

3. Context Window:
   I'm still unclear on how to persistently set num_ctx for specific local models in GitHub Copilot using ollama. As far as I understand, the current options are:

• System variable: OLLAMA_CONTEXT_LENGTH=16000 sets it globally, but it’s inflexible—can’t be changed per model or at runtime.

• CLI ollama run: Using /set num_ctx 16000 works but needs to be run before every model load in Copilot, which is manual and unintuitive.

• Modelfile: Creating a new Modelfile with PARAMETER num_ctx from the original model, seems cumbersome and not ideal for quick changes.

• API: Sending "options": {"num_ctx": 16000} in API requests works well and is how tools like Open WebUI handle it. It looks like num_ctx should be handled by the calling application, such as GitHub Copilot via the options, also storing this in VS Code setting to have persistent model options.

Given these limitations, is there a better way to define per-model num_ctx settings when using ollama locally with Copilot?

[lramos15]

I think you've listed all the possible options. I personally use OLLAMA_CONTEXT_LENGTH, and only load one model at a time as I don't have enough VRAM to load multiple.

We cannot do num_ctx due to the aformentioned Open AI compatible api surface. We would need to switch to the Ollama specific API which can be a bit challenging given that so many of our bring your own key providers are built on top of Open AI compatible specs.

[underlines]

Logan Ramos (@lramos15)

Sorry, I ninja edited my previous answer:

1. I installed the latest Windows Insiders Build, checked for updates, added one Ollama model and one OpenRouter model. Neither model shows up when switching to Agent Mode. What am I doing wrong?

1. Added OpenRouter models
   
2. Shows up in Edit mode
   
3. Switch to Agent Mode, then doesn't show any Ollama or OpenRouter models:
   

[lramos15]

99% of the free Open Router models don't support tool calls so agent mode cannot work without tool calling. But as you see Gemini 2.5 Pro experimental (free) is there.

[underlines]

Oh, I get it now, you said it in the beginning: GitHub Copilot actually respects the capabilities and only allows you to use models with properly set capabilities. If we want to try models that didn't set set the tool calling capability, we would have to recreate an ollama modelfile with this set. Right?

[lramos15]

Jan Badertscher (@underlines) Yes, but you'll likely have a bad experience if the model doesn't actually support tools.

[michael-heinrich]

I stumbled upon the same issue and coded up a little proxy script that logs the requests and changes "max_tokens":4096 to "options":{"num_predict":4096,"num_ctx":70000} on the fly for the v1/chat/completions endpoint. That helps with ollama.

https://github.com/michael-heinrich/llm_proxy/blob/main/ollama_proxy.py

[rizbud]

In my case, Gemini 2.5 Flash Preview doesn't show in agent mode on VS Code Insider. It is shown in Ask and Edit mode.
Do the same thing on the stable release, the model is shown in agent mode.

When adding a custom model to a VS Code stable release, it asks whether the model supports tool calling. I don't see this option when adding a custom model to VS Code Insider.