Feature Request: tool to list and delete cached models

[sultanqasim]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I'd love to have a tool to list and delete cached models (that were fetched automatically when using the -hf option). This would be akin to the ls and rm commands in Ollama.

Motivation

A lot of people (myself included) use the -hf option to automatically fetch models from Hugging Face. This places models in a model cache directory, which can get rather big over time. Each model typically has at least three associated files (manifest, gguf, and etag), and sometimes five (adding mmprog gguf and associated etag). Manually managing files in the cache is a bit cumbersome. It would be nice to have an elegant way to see which models you have cached, how much space they take, and have a single command to delete all cached files associated with a model.

Possible Implementation

I built a Python script to do this for my own convenience: https://gist.github.com/sultanqasim/5b6d9654236e18dea4896d3c9ce2dc1b

The output of the script looks like this:

$ ./llama-cache ls
Name                                                        Size (GB) Modified
--------------------------------------------------------------------------------
ibm-granite/granite-4.0-h-small-GGUF:Q4_K_M                 18.1      2025-10-02 13:34
unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_XL            16.5      2025-08-07 00:32
unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:IQ4_XS     12.7      2025-07-24 12:10
unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_XL           16.5      2025-10-01 17:19
unsloth/Qwen3-14B-GGUF:Q4_K_XL                              8.5       2025-08-01 23:03

$ ./llama-cache rm unsloth/Qwen3-14B-GGUF:Q4_K_XL
Deleted: /Users/sultan/Library/Caches/llama.cpp/unsloth_Qwen3-14B-GGUF_Qwen3-14B-UD-Q4_K_XL.gguf
Deleted: /Users/sultan/Library/Caches/llama.cpp/unsloth_Qwen3-14B-GGUF_Qwen3-14B-UD-Q4_K_XL.gguf.json
Deleted: /Users/sultan/Library/Caches/llama.cpp/manifest=unsloth_Qwen3-14B-GGUF=Q4_K_XL.json

[allozaur]

@ServeurpersoCom don't u think this could be added to #16335?

[ServeurpersoCom]

@ServeurpersoCom don't u think this could be added to #16335?

Yes, the model selector could definitely evolve into a more complete system, but as it stands it would need a different backend.
We could integrate a Hugging Face model browser that, in addition to building the llama-swap configuration file, would also handle cleaning up the model cache used by llama-server.
It's basically like recreating an Ollama but directly connected to Hugging Face (like LM Studio) and fully configurable in YAML.
We could even add a page that generates the llama-swap YAML configuration automatically.
That said, it starts to look like a somewhat heterogeneous architecture, ideally, there should be a single unified backend handling everything, including model downloads and swap operations, while keeping llama-server and its API clean.

[allozaur]

I see, let's see also what @ngxson and @ggerganov think about this.

[ngxson]

At least in my case I think such tool could be quite useful. When designing the new cache format (which have the metadata saved along with model file), I already thought of this usage.

The best way IMO is to merge this tool with llama-cli, something like this:

llama-cli pull -hf ...
llama-cli ls
llama-cli rm ...
llama-cli -hf ...

We just need to expand arg.cpp to handle positional arguments

CC @ericcurtin as you're working on the llama-pull tool #16423

[ericcurtin]

At least in my case I think such tool could be quite useful. When designing the new cache format (which have the metadata saved along with model file), I already thought of this usage.

The best way IMO is to merge this tool with llama-cli, something like this:

llama-cli pull -hf ...
llama-cli ls
llama-cli rm ...
llama-cli -hf ...
We just need to expand arg.cpp to handle positional arguments

CC @ericcurtin as you're working on the llama-pull tool #16423

I think it makes sense, I would suggest:

llama-pull
llama-ls
llama-rm

It encourages users to split up code logically without ending up with 5,000 line plus all in one files. It also follows the Unix philosophy of doing one this well. We also get autocomplete of llama-* "for free".

The other issue with populating llama-cli like this is it leads users into believing that llama-cli is a useful all around executable like ollama. Which it is not currently, for inferencing nothing compares to the llama-server binary in the project in terms of maturity for actual usage.

[ericcurtin]

It's also sometimes debatable whether we should go there in llama.cpp or leave this functionality to higher-level tools like Docker Model Runner which also supports pulling from HuggingFace and DockerHub (open-source and open to contributions btw). It has model swapping and pushing also.

The kind of code for this functionality, doesn't have to be very optimal, golang is fine. It's debatable whether it's worth the effort of doing these things in C++, when higher-level code is easier to maintain.

At this level of an application often golang, python, Rust (debatable) can be a better choice in modern times and it's easier to get contributors for higher level codebase than C++.

[ericcurtin]

@ServeurpersoCom don't u think this could be added to #16335?

Yes, the model selector could definitely evolve into a more complete system, but as it stands it would need a different backend. We could integrate a Hugging Face model browser that, in addition to building the llama-swap configuration file, would also handle cleaning up the model cache used by llama-server. It's basically like recreating an Ollama but directly connected to Hugging Face (like LM Studio) and fully configurable in YAML. We could even add a page that generates the llama-swap YAML configuration automatically. That said, it starts to look like a somewhat heterogeneous architecture, ideally, there should be a single unified backend handling everything, including model downloads and swap operations, while keeping llama-server and its API clean.

And this does sound like Docker Model Runner for what it's worth. Docker Model Runner is like Ollama, but you are not locked in to a single OCI registries for models like the Ollama registry. We are committed to using fully upstream llama.cpp also. As well as downloads from huggingface.

Well this sounds like Docker Model Runner except for the yaml, but I think Docker Model Runner would be open yaml configurations if people found that useful.

[ericcurtin]

We do have an issue with the tommarques56 github account in the llama-pull PR, he's left roughly 30 AI bot comments. Sometimes it's not even clear whether he's speaking as a human or an AI bot.

[ngxson]

It encourages users to split up code logically without ending up with 5,000 line plus all in one files. It also follows the Unix philosophy of doing one this well. We also get autocomplete of llama-* "for free".

I think the main issue of separating to multiple binaries will be a bad UX overall. We will eventually need something like llama-help just to list all available tools. Linux can do this way mainly because they have the man command. And btw we also have a large user base on Windows, which don't perceive the same UX on mac/linux

All CLI out there (docker, ollama, codex, claude cli, etc) regroup everything into one binary, and it should be for a reason. I believe it's a better UX overall, especially for users less familiar with CLI stuff.

The other issue with populating llama-cli like this is it leads users into believing that llama-cli is a useful all around executable like ollama. Which it is not currently, for inferencing nothing compares to the llama-server binary in the project in terms of maturity for actual usage.

For llama-cli, by naming it -cli, we already (accidentally) communicate to the users that it should be a multi-purpose tool. Otherwise, we should have called it llama-inference or llama-chat

I believe it's time for us to progressively improve the UX of llama-cli. At least that's what I want to do in the near future. Will provide more info on this idea soon.

[sultanqasim]

The naming on llama-cli does suggest that it should be a general purpose tool, rather than just a chat utility. I don't have a strong preference on which tool name these utilities should go under, but I do think that ls, rm, and pull should be subcommands of the same utility. Popular command line utilities like Git do the same.

[ericcurtin]

The naming on llama-cli does suggest that it should be a general purpose tool, rather than just a chat utility. I don't have a strong preference on which tool name these utilities should go under, but I do think that ls, rm, and pull should be subcommands of the same utility. Popular command line utilities like Git do the same.

Funnily enough, git is a bad example here from the coding viewpoint, it's an example of the opposite, doing one thing and one thing well:

https://github.com/git/git/tree/master/builtin

in some builds of git, one can build it into many binaries like git-clang-format, git-commit etc. but there is one "git" binary that when called like "git clang-format", it will automatically execute "git-clang-format".

But you can also build it to combine all the main()-like functions into one binary. So it kinda does both, but designing every component like a separate binary.