Chat

Chat

ML::Chat is a conversational LLM client that works with several providers behind one API. It keeps a running message history so the model remembers context across calls.

Providers

enum class Provider { Ollama, OpenAI, Anthropic, Groq, OpenRouter, DeepSeek };

Provider	Enum	API key	Default host
Ollama (local)	Provider::Ollama	not needed	http://localhost:11434
Groq	Provider::Groq	required	https://api.groq.com/openai
OpenRouter	Provider::OpenRouter	required	https://openrouter.ai/api
DeepSeek	Provider::DeepSeek	required	https://api.deepseek.com
OpenAI	Provider::OpenAI	required	https://api.openai.com
Anthropic	Provider::Anthropic	required	https://api.anthropic.com

Any other OpenAI-compatible service (Together, Fireworks, etc.) works with Provider::OpenAI and a custom host.

Construction

Chat(Provider provider, std::string model, std::string apiKey = "", std::string host = "");

The provider is always explicit. The API key is only used by cloud providers (Ollama ignores it). Leave host empty to use the provider's default endpoint.

Chat ollama(Provider::Ollama, "llama3.2:1b");
Chat groq(Provider::Groq, "llama-3.1-8b-instant", std::getenv("GROQ_API_KEY"));
Chat claude(Provider::Anthropic, "claude-3-5-haiku-latest", std::getenv("ANTHROPIC_API_KEY"));
claude.setMaxTokens(1024);   // required by Anthropic

There is a null-safe overload taking a const char* key, so passing std::getenv(...) directly is safe even if the variable is unset.

Settings

void setSystem(std::string instruction);   // persistent system prompt
void setTemperature(double temperature);   // 0.0 = focused, ~1.0 = creative
void setMaxTokens(int maxTokens);          // required by Anthropic, optional elsewhere

Sending messages

// Returns the reply text; appends both prompt and reply to history.
std::string ask(std::string prompt);

// Lower-level, stateless: returns the full HTTP Response without touching history.
Response raw(std::string prompt);

Chat chat(Provider::Ollama, "llama3.2:1b");
chat.setSystem("You are a terse assistant.");
std::cout << chat.ask("My name is Prisha. What is C++?");
std::cout << chat.ask("What is my name?");   // remembers: "Prisha"

Streaming

stream delivers the answer token-by-token through onToken. It returns the HTTP status code. On a 2xx response the exchange is added to history.

long stream(std::string prompt,
            std::function<void(const std::string&)> onToken,
            std::function<void(const std::string&)> onReasoning = {});

chat.stream("Why is C++ fast?",
    [](const std::string& token) { std::cout << token << std::flush; });

Reasoning tokens

For reasoning models (e.g. deepseek-reasoner, gpt-oss), the model's "thinking" is delivered separately from the answer. Pass the optional onReasoning callback, or read it afterwards with lastReasoning().

chat.stream("If a bat and ball cost $1.10 ...",
    [](const std::string& t)  { std::cout << t; },             // the answer
    [](const std::string& th) { std::cout << "[think] " << th; }); // the reasoning

std::string thinking = chat.lastReasoning();  // also works after ask()

History

void reset();                                   // clear history (keeps the system prompt)
const std::vector<Message>& history() const;    // current conversation
bool saveHistory(const std::string& path) const;// dump to a JSON file
bool loadHistory(const std::string& path);      // restore from a JSON file

Message is { std::string role; std::string content; } where role is "system", "user", or "assistant".

chat.saveHistory("conversation.json");
// ... later, or next run ...
Chat resumed(Provider::Ollama, "llama3.2:1b");
resumed.loadHistory("conversation.json");