LLMix

Read in other languages: English · 中文 · Deutsch · Español · Français · Русский · 한국어 · 日本語 · हिन्दी

Config-driven LLM calls for Python, TypeScript, and Rust. Keep your SDK. Move model behavior into MDA presets. Put cache, retries, key rotation, and rollout control around the call.

LLMix is the layer between your product and the provider SDK.

It does not ask you to rewrite your OpenAI, Anthropic, Gemini, LiteLLM, AI SDK, or custom client code. It wraps the call. The boring parts go around it: response cache, circuit breaker, key pools, singleflight, retry policy, adaptive concurrency, provider kwargs, and MDA config loading.

The model stops being a hard-coded string buried in application code. It becomes data. Change a preset, publish a registry snapshot, reload the service, and the next request can run a different provider or model. No redeploy for the usual model swap dance.

That is the whole thing. Small layer. Sharp edges filed down.

---

Why It Exists

AI products in 2026 do not usually fail because one SDK call is hard.

They fail in the spaces around the call. A key gets rate limited. A provider gets slow. Two hundred users ask the same thing at once. A model swap needs a deploy. A cache key differs by one invisible parameter. One service is in Python, another is in TypeScript, and the Rust worker has to follow the same contract.

LLMix is for that part of the system. The signal chain between your app and the model.

You still own the prompt. You still own the SDK. LLMix owns the harness.

---

Install

Runtime	Package	Import path
TypeScript	npm install @snoai/llmix	@snoai/llmix
Python	pip install sno-llmix	llmix
Rust	cargo add llmix-rs	llmix_rs

Python uses sno-llmix on PyPI because llmix was already taken. The import path is still llmix.

Provider helpers use optional SDKs. Install only the provider clients you call.

# TypeScript OpenAI-compatible helpers
npm install ai @ai-sdk/openai

# Python Redis cache support
pip install "sno-llmix[redis]"

# Rust OpenAI helper and Redis cache
cargo add llmix-rs --features providers-openai,redis

LLMix uses the MDA config packages for preset loading. They are also published as standalone runtime loaders for apps that need .mda validation, integrity, or trust-policy enforcement outside LLMix.

---

Documentation

• Usage reference
• TypeScript guide
• Python guide
• Rust guide
• Secure LLMix configuration
• Key pool operations
• Standalone MDA config loader docs

---

At a Glance

LLMix wraps one provider call at a time.

It is not a router in the LiteLLM sense. It is closer to the harness you keep rebuilding around every agent, coder tool, extraction service, and internal AI workflow once traffic becomes real.

---

Quick Start

TypeScript

import {
  CallPipeline,
  KeyPool,
  TwoTierCache,
  openaiDispatch,
} from "@snoai/llmix";

const pipeline = new CallPipeline({
  dispatch: openaiDispatch(),
  responseCache: new TwoTierCache("memory"),
});

pipeline.setKeyPool("openai", new KeyPool([process.env.OPENAI_API_KEY!]));

const response = await pipeline.call({
  config: {
    provider: "openai",
    model: "gpt-4o-mini",
    common: { temperature: 0.2, maxOutputTokens: 512 },
    caching: { strategy: "memory" },
  },
  messages: [
    { role: "user", content: "Explain LLMix in one sentence." },
  ],
});

console.log(response.content);
await pipeline.close();

Python

import asyncio
import os

from llmix import (
    CallInput,
    CallPipeline,
    KeyPool,
    PipelineConfig,
    TwoTierCache,
    openai_dispatch,
)


async def main() -> None:
    pipeline = CallPipeline(
        PipelineConfig(
            dispatch=openai_dispatch(),
            response_cache=TwoTierCache("memory"),
        )
    )

    pipeline.set_key_pool("openai", KeyPool([os.environ["OPENAI_API_KEY"]]))

    response = await pipeline.call(
        CallInput(
            config={
                "provider": "openai",
                "model": "gpt-4o-mini",
                "common": {"temperature": 0.2, "max_output_tokens": 512},
                "caching": {"strategy": "memory"},
            },
            messages=[
                {"role": "user", "content": "Explain LLMix in one sentence."}
            ],
        )
    )

    print(response.content)
    await pipeline.close()


asyncio.run(main())

Rust

Rust exposes the same pipeline contract. The OpenAI helper is feature-gated.

[dependencies]
llmix-rs = { version = "2.0.0", features = ["providers-openai"] }
serde_json = "1"
tokio = { version = "1", features = ["macros", "rt"] }

use llmix_rs::{
    load_keys_from_env, CallInput, CallPipeline, OpenAiChatHelper, PipelineConfig,
};
use serde_json::json;

let pipeline = CallPipeline::new(PipelineConfig::new(OpenAiChatHelper::new()))?;
pipeline.set_key_pool("openai", load_keys_from_env("openai")?);

let response = pipeline
    .call(CallInput {
        config: json!({
            "provider": "openai",
            "model": "gpt-4o-mini",
            "common": { "temperature": 0.2, "max_output_tokens": 512 },
            "caching": { "strategy": "memory" }
        }),
        messages: vec![json!({
            "role": "user",
            "content": "Explain LLMix in one sentence."
        })],
        singleflight_key: None,
    })
    .await;

See the Rust guide for full main examples and feature flags.

---

What You Get Around Every Call

Concern	What LLMix does
Response cache	L1 memory plus optional Redis L2, with cross-runtime canonical cache keys
Key pools	Round-robin key selection, 429 rotation, and 401/403 dead-key eviction
Retries	Jittered exponential backoff, with Retry-After honored
Circuit breaker	Scoped by provider and effective base URL
Singleflight	Collapses identical concurrent work into one upstream request
Concurrency	AIMD adaptive semaphore, driven by rate-limit feedback
Provider kwargs	Common config becomes provider-specific request fields
Thinking tokens	Optional <think> extraction into normalized response objects
Registry	Immutable config snapshots with one live current.json pointer

The defaults are meant to be boring. Tune them when real traffic gives you a reason.

---

MDA Presets

LLMix uses MDA Source Mode for config authoring. The human notes and runtime settings live in one file. The runtime only sees the resolved JSON. Python, TypeScript, and Rust can require MDA integrity, requires.network, and verifier-hook based signatures while loading or publishing registry snapshots. Real Rekor transport and Sigstore cryptography are supplied by caller-provided clients/verifiers.

---
name: extraction
description: Entity extraction preset.
metadata:
  snoai-llmix:
    common:
      provider: openai
      model: gpt-4o-mini
      temperature: 0.2
      maxOutputTokens: 512
    caching:
      strategy: redis-or-memory
    providerOptions:
      openai:
        reasoningEffort: medium
---
# extraction

Extract named entities. Return compact JSON.

Load it directly when you are authoring or testing:

import { loadMdaConfig } from "@snoai/llmix";

const config = await loadMdaConfig("./config/llm/search/extraction.mda");

from llmix import load_mda_config

config = load_mda_config("./config/llm/search/extraction.mda")

use llmix_rs::load_config;

let config = load_config("./config/llm/search/extraction.mda")?;

For production services, use the registry.

---

Config Registry

Editable MDA files are good for humans. Running services need something quieter.

The LLMix Config Registry publishes authoring files into immutable, content-addressed snapshots. Runtime code reads the active snapshot, not the mutable source tree.

config/llm/
  authoring/
    search/
      extraction.mda
  snapshots/
    2026-05-09T000000Z-...
  current.json

from llmix import ConfigRegistryManager, ConfigRegistryPublisher, resolve_config_dir

root = resolve_config_dir().config_dir
ConfigRegistryPublisher(root).publish()

manager = ConfigRegistryManager.open(root)
config = manager.get_preset("search", "extraction")

import {
  ConfigRegistryManager,
  ConfigRegistryPublisher,
  resolveConfigDir,
} from "@snoai/llmix";

const { configDir } = resolveConfigDir();
await new ConfigRegistryPublisher(configDir).publish();

const manager = await ConfigRegistryManager.open(configDir);
const config = await manager.getPreset("search", "extraction");

Managers expose the active revision and reload health metadata. That makes it easy to say exactly which config a service is running.

---

Provider Coverage

The public dispatch helpers cover the providers we actually test.

Provider	Python	TypeScript	Notes
OpenAI	openai_dispatch	openaiDispatch	OpenAI Responses and chat-style flows
Anthropic	anthropic_dispatch	anthropicDispatch	Messages API, thinking budget validation
Gemini	gemini_dispatch	geminiDispatch	Google GenAI-compatible params
OpenRouter	openrouter_dispatch	openrouterDispatch	OpenAI-compatible
DeepInfra	deepinfra_dispatch	deepinfraDispatch	OpenAI-compatible
Novita	novita_dispatch	novitaDispatch	OpenAI-compatible
Together	together_dispatch	togetherDispatch	OpenAI-compatible
Sno GPU	sno_gpu_dispatch	snoGpuDispatch	On-prem OpenAI-compatible GPU endpoints

Rust currently ships the neutral pipeline plus feature-gated helpers for OpenAI, Anthropic, Gemini, and Sno GPU. Treat Rust provider helpers as beta. The cache, key-pool, registry, retry, and pipeline contract are aligned with Python and TypeScript.

OpenAI-compatible providers reuse the OpenAI request shape with provider-specific base_url handling. That keeps the contract plain. Plain is useful.

---

Environment Variables

Variable	Purpose
OPENAI_API_KEY / OPENAI_KEYS	OpenAI key or comma-separated key pool
ANTHROPIC_API_KEY / ANTHROPIC_KEYS	Anthropic key or comma-separated key pool
GEMINI_API_KEY / GEMINI_KEYS	Gemini key or comma-separated key pool
OPENROUTER_API_KEY / OPENROUTER_KEYS	OpenRouter key or comma-separated key pool
DEEPINFRA_API_KEY / DEEPINFRA_KEYS	DeepInfra key or comma-separated key pool
TOGETHER_API_KEY / TOGETHER_KEYS	Together key or comma-separated key pool
NOVITA_API_KEY / NOVITA_KEYS	Novita key or comma-separated key pool
SNO_LLM_API_KEY	Sno GPU direct dispatcher fallback
SNO_GPU_API_KEY / SNO_GPU_KEYS	Sno GPU key-pool variables for provider id sno-gpu
GPU_BASE_URL	Sno GPU base URL
REDIS_URL	Redis response-cache URL
LLMIX_STATE_DIR	Lock files, batch metadata, and kill-switch state

load_keys_from_env("provider-name") checks PROVIDER_NAME_KEYS first, then PROVIDER_NAME_API_KEY. Dashes become underscores.

---

What This Is Not

• Not a streaming framework. Streaming stays with your SDK.
• Not a prompt framework. Bring your own prompt layer.
• Not a provider marketplace. One call uses the provider named by its config.
• Not a reason to hide every model decision behind indirection. Some things should stay in code.

LLMix is useful when the same model-call shape keeps showing up across services. If you have one script and one key, you probably do not need it yet.

---

Development

# Install TypeScript workspace dependencies
bun install

# Install Python workspace dependencies
uv sync --project packages/llmix/python --extra dev
uv sync --project packages/mda-config/python --all-groups

# Full monorepo checks
bun run build
bun run check
bun run test

---

License

Apache-2.0

Related

• AI SDK
• Promptix