Using an LLM gateway

Using an LLM gateway

Use a LiteLLM-style gateway with malac, through the OpenAI or the Anthropic wire format.

An LLM gateway (LiteLLM, OpenRouter, Together and the like) puts many models behind one endpoint and one key. Many gateways speak both the OpenAI chat completions format and the Anthropic Messages format, so malac can use them through either provider kind, or both at once.

This guide uses a LiteLLM-based gateway at lora.example.com that offers the aliases default, smart, fast and vision. Replace the host, the key variable and the model names with yours.

Add the gateway

Through the OpenAI format (/v1/chat/completions):

printf '%s' "$LORA_API_KEY" | malac provider add lora \
  --base-url https://lora.example.com/v1 --model default --api-key-stdin --set-default

Through the Anthropic format (/v1/messages), as a second provider on the same gateway:

printf '%s' "$LORA_API_KEY" | malac provider add lora-anthropic --kind anthropic \
  --base-url https://lora.example.com --model smart --api-key-stdin

Note the base URLs: the OpenAI kind takes the URL up to /v1, the Anthropic kind the URL without it.

Then use either:

malac -m lora/smart review the auth module
malac -m lora-anthropic/smart review the auth module

The same in TOML

model = "lora/default"
small_model = "lora/fast"      # compaction summaries
vision_model = "lora/vision"   # describes images for models without vision

[providers.lora]
kind = "openai"
base_url = "https://lora.example.com/v1"
api_key_env = "LORA_API_KEY"
models = ["default", "smart", "fast", "vision"]

[providers.lora-anthropic]
kind = "anthropic"
base_url = "https://lora.example.com"
api_key_env = "LORA_API_KEY"
models = ["smart"]

[models."lora/default"]
context_window = 200000
vision = false

[models."lora/vision"]
vision = true

malac provider models lora asks the gateway's /v1/models for everything it serves, including the concrete models behind the aliases. List the ones you use in models so they show up in /model and the web UI.

Which format to use

	OpenAI format	Anthropic format
Tool calls	yes	yes
Reasoning	streamed as reasoning_content or reasoning	thinking blocks
Prompt caching	whatever the gateway does	malac marks the system prompt, tools and latest turn for caching
Cost in the footer	when the gateway reports cost in the usage	not reported
Embeddings for memory	/v1/embeddings, if the gateway serves them	not available

The OpenAI format is the more widely supported one and the only one that carries embeddings and cost. The Anthropic format is worth it when the gateway forwards Anthropic features such as prompt caching.

Anthropic extended thinking needs signed thinking blocks to be replayed across tool calls. If a gateway returns thinking blocks without a signature, malac does not replay them.

Per-model settings

Gateways rarely tell clients a model's limits, so set them per alias:

[models."lora/smart"]
context_window = 200000
max_output_tokens = 32000

[models."lora/fast"]
context_window = 128000

malac starts compacting at 80% of context_window and sends max_output_tokens as max_tokens (default 16384).

Images

Screenshots from the control service and images the agent reads are sent to the model. Mark aliases that cannot see images with vision = false, and set a vision_model that can: malac then replaces each image with a description before the request goes out. With the OpenAI format, images returned by tools are re-sent in a user message, because tool messages cannot carry images.

Embeddings

Vector memory needs an /embeddings endpoint. If your gateway serves an embedding model, point [memory.embeddings] at it:

[memory.embeddings]
provider = "lora"
model = "embed"            # the gateway's embedding alias

If it does not, use a second provider for embeddings, for example a local Ollama:

[providers.local]
kind = "openai"
base_url = "http://localhost:11434/v1"

[memory.embeddings]
provider = "local"
model = "nomic-embed-text"

Retries

When a request fails with a retryable error before anything has streamed, malac retries it with backoff, up to four attempts, and shows a notice. Errors after streaming has started are not retried.

Gateway features that help

None are required, but they make malac work better:

• an embedding alias, so memory needs no second provider,
• a separate key or route for malac, so its traffic and spend can be tracked and rate-limited on their own,
• model metadata on /v1/models (output limits, vision and reasoning support),
• signed thinking passthrough and prompt caching passthrough on /v1/messages.