Default model using llama.cpp?

[mroark1m]

With the latest llama.cpp provider changes, I can't figure out how to automatically load a model at pi startup without running commands. Am I missing something? It seems like defaultModel from settings.json is non-functional because the catalog is not yet loaded, even if we have the configuration of llama.cpp in auth.json. To me it should load the catalog and see if the settings.json defaultModel is available, and load it if it is not loaded. If it's not available at all, then it should warn at that point.

If I'm correct (and I'd love a pointer if I'm wrong), I'm curious why this is the case... I tried a local agent and also deepwiki to discuss it and could not come up yet with an extension to make it work like I describe (they all failed, probably this is fixable somehow, I need to spend more time on it). But why is this the way the llama.cpp provider works? It seems like a contrast to other providers and kind of non-intuitive?

[mroark1m]

Also even if I have auth.json configured (by hand or from earlier use of /login), it seems to ignore that, on first startup it says:

Warning: No models available. Use /login to log into a provider via OAuth or API key. See:
/usr/lib/node_modules/@earendil-works/pi-coding-agent/docs/providers.md
/usr/lib/node_modules/@earendil-works/pi-coding-agent/docs/models.md

If I immediately do /model and select the currently loaded model, it works. If I call /llama I can load and unload models also. However, even if the model I want is loaded and set as the default model, if it I try to talk to it without first calling /model to select it, I get:

Error: No API key found for the selected model.

Use /login to log into a provider via OAuth or API key. See:
/usr/lib/node_modules/@earendil-works/pi-coding-agent/docs/providers.md
/usr/lib/node_modules/@earendil-works/pi-coding-agent/docs/models.md

[mroark1m]

I'm not really in love with the idea that you should turn off autoloading either. What if you want to use multiple models for different sub-agents (for example), but you don't have enough vram to load them all simultaneously? It seems like you would likely just fail at some point due to one of the models not being loaded if you follow those instructions, but llama.cpp is smart enough to unload things and load the right model if autoloading is turned on, if you used the chain method, this would only happen between subagents. The ability to load/unload the models is perfectly fine and reasonable functionality without turning off autoloading. Even if you want pi to manage loading authoritatively, since it doesn't load the catalog at start, it's not even aware of what the initial state of the server is, models may very well already be loaded from earlier runs.

[Denkkar]

It's not just you. I'm using llama.cpp router functionality with Pi as well.

So this is really a bug report here:

• /llama and choose model Gemma31B
• Model load
• Prompt anything: it tries to use the previous model Qwen3.6_27B

Why? Because that's what the /model command reports. So there's two work-arounds I've found: either change model with /llama and exit/re-enter into Pi, or follow up /llama with /model so they match up.

FYI: I love that PI has llama.cpp built-in functionality, it's way better than using the Ollama compatibility layer

[mroark1m]

Are you on the latest version? I need to retest. I noticed that there were some updates around that provider since I wrote that.

[Denkkar]

AFAIK I'm on the latest version.

pi --version
0.82.1

[Denkkar]

I see there's a new version today, 0.83.0, so I updated to that and gave it a shot.
Used /llama to pick one model that wasn't the default one, tried a prompt, and it immediately tried to load the default model, not the one I just picked with /llama

[mroark1m]

I agree, behavior changed from what it was earlier, I got the same as you on 0.83.0

[mroark1m]

I guess a lot of what I said up top is fixed. It loads the default model at startup ok now. The way I understand what it's doing now is that "/llama" and "/model" are acting like two different basically unconnected systems. /llama controls what llama-server has currently loaded, only, and /model selects what model the current agent should use right now, only.

If you're using subagents with different models assigned to each agent and a big enough llama-server to have multiple different models running at the same time, this might even be what you'd expect to happen as you might want more than one model loaded at a time, but only one selected for the current window.

[dnv]

I still have to do the /llama followed by /model dance in the latest release of Pi as of today (v0.84.1), this is quite an annoyance.