Can I integrate LiteLLM as a model provider into the agent-famework?

[smbus2kw]

Is it possible to integrate LiteLLM to simplify the project setup and gain broader support for models?

[gabrielschmith]

Yes, it is possible to integrate LiteLLM, and it is actually a very good fit.

LiteLLM provides an OpenAI-compatible API layer that can proxy requests to multiple LLM providers (OpenAI, Azure OpenAI, Anthropic, Gemini, Mistral, etc.).

LiteLLM is exposed as an OpenAI-compatible endpoint (usually under /v1), so the factory only needs to point to LiteLLM instead of a direct OpenAI endpoint.

The key point is that no dedicated LiteLLM client is required. We keep using the official OpenAI client and simply override the endpoint to point to LiteLLM. Since LiteLLM is fully OpenAI-API compatible, the integration is transparent to the application.

This means the existing abstractions (OpenAIClient, IChatClient, ChatClientAgent, etc.) remain unchanged.

Example: Using LiteLLM as the Chat Gateway

How LiteLLM is used

• The application continues to use the OpenAI SDK
• The only change is the Endpoint configuration, which points to LiteLLM instead of OpenAI directly
• The model value is a LiteLLM model identifier (which internally routes to OpenAI, Anthropic, Gemini, etc.)

public async Task<IChatClient> CreateChatClientAsync(
    CancellationToken cancellationToken = default)
{
    // LiteLLM endpoint (OpenAI-compatible)
    string[] endpoints = configuration
        .GetSection("Services:LiteLlm:https")
        .Get<string[]>();

    string host = endpoints?.FirstOrDefault()
        ?? throw new InvalidOperationException("LiteLLM endpoint not configured.");

    var endpoint = new Uri($"https://{host}");

    // LiteLLM API key (master key or per-tenant virtual key)
    string apiKey = await settingManager.GetOrNullForCurrentTenantAsync(ApiKey);
    string model  = await settingManager.GetOrNullForCurrentTenantAsync(ChatModel);

    var openAiClient = new OpenAIClient(
        new ApiKeyCredential(apiKey),
        new OpenAIClientOptions
        {
            Endpoint = endpoint
        });

    return openAiClient
        .GetChatClient(model)
        .AsIChatClient();
}

The usage remains exactly the same:

var chatClient = await chatClientFactory
    .CreateChatClientAsync(CancellationToken);

var agente = new ChatClientAgent(chatClient, new ChatClientAgentOptions
{
    Name = "my-agent",
    Description = """
                  My Description
                  """,
    ChatOptions = new ChatOptions
    {
        ResponseFormat = ChatResponseFormat.ForJsonSchema<MyResult>(),
        Instructions = "My Prompt"
    }
});

var response = await agente.RunAsync<MyResult>("My message");
return response.Result;

Production usage and multi-tenancy

We are already using this approach in production, including:

• Multiple tenants
• A different API key per tenant
• Model selection per tenant
• Centralized routing and provider management handled by LiteLLM

From the application’s point of view, each tenant simply receives its own API key and model name. LiteLLM takes care of routing, authentication, and provider selection behind the scenes.

Overall, this keeps the codebase clean while giving us much more flexibility and control over model providers.