Skip to content

v1.2.0

Choose a tag to compare

@Goldziher Goldziher released this 07 Apr 12:19
52600b8

Local LLM Provider Support

Add 6 local LLM inference engines as first-class providers, all using OpenAI-compatible HTTP APIs:

Provider Chat Completions Embeddings Rerank Audio Images
Ollama
LM Studio
vLLM
llama.cpp
LocalAI
llamafile

Quick Start

from liter_llm import LlmClient

client = LlmClient(api_key="")
response = await client.chat(
    model="ollama/llama3.2",
    messages=[{"role": "user", "content": "Hello!"}],
)

No API key needed. Model routing uses the standard provider/model prefix convention.

What's New

  • 6 local providers with accurate endpoint configurations
  • Docker Compose setup for integration testing (Ollama + qwen2:0.5b + all-minilm)
  • CI workflow for local LLM integration tests
  • Comprehensive documentation at docs/usage/local-llms.md
  • Integration tests for chat, streaming, embeddings, and model listing
  • Removed duplicate lm_studio and ollama_chat entries (now 143 providers)

Fixed

  • PHP onError hook passes Exception not string
  • README template compliance for rumdl
  • Added 404 to POST OpenAPI specs

Full Changelog: v1.1.1...v1.2.0