Plan: Moving llama-cli to llama-completion

[ngxson]

We are planning to improve the UX of llama-cli, more details can be found in this issue.

The plan is to migrate the code base of llama-cli into a llama-server-based client. This will effectively allow CLI to inherit all of the features available on server, including:

• Multimodal support
• Fully jinja support (the old llama-cli fails in certain cases)
• Conversation control (regenerate and roll back messages)
• Speculative decoding
• etc.

The current llama-cli will be moved to a new example called llama-completion, and the code will be kept simple to serve as a learning example. If you are already using llama-cli in a deterministic way in your pipeline, please consider using llama-completion if you encouters any problems.

The new llama-cli will have enhanced features (as mentioned above) and improved user experience.

This discussion is added so that users can discuss issues and workarounds if needed.

[pwilkin]

One question: does the scope of this potentially include making llama-cli a lightweight agentic client (i.e. adding support for tool calls / MCP servers) or is that OOS?

[ngxson]

That can be a far-future plan.

Translating tool call <> MCP should be quite trivial as it doesn't require any code interpretation or sandbox code execution. However, this obviously requires having MCP server, which doesn't allow the agent to do trivial tool that other non-c++ frameworks offer, like simple calculator tool for example

[pwilkin]

Yeah, the reason I'm mentioning this is because what they call an "MCP client and server" with the stdio protocol is basically exactly the old fork/execve/dup IPC scheme all Unix users know and love, so this should be also trivially implementable under C/C++.