Add setting and/or button for unloading local LLMs

[insunaa]

Body

What are you proposing?

I would love to have a button that allows me to manually unload a model for a local LLM provider such as Ollama or llama.cpp, ideally in addition to a global setting that unloads the model automatically after a settable period of inactivity. Models should also unload when the zed editor is being shut down. Right now the behaviour is that zed loads a model and then never frees the resources.

Why does this matter?

Local models that run in VRAM or VRAM+System RAM block these resources for any other type of use. For ollama this can be manually circumvented with ollama stop <tab-complete> but for llama.cpp you have to send a POST request to the <llama.cpp-host>/models/unload API endpoint, with a valid API key. You can hack yourself a shell alias that curls this to the server, but imo this cleanup should be done through the application that causes the "mess". llama.cpp suspends a model when the --sleep-idle-seconds parameter is set, but it still keeps a bunch of resources alive in the host memory and some resources in the VRAM. This improves the situation, but only an actual unload solves it.

What problem does this solve?

Undue resource utilization.

What becomes easier or possible?

Starting a game or other applications that require VRAM

Are there any examples or context?

Example of an unload-model button that comes with the llama.cpp web-ui.

Possible approach

Send the appropriate API commands to the local LLM providers