LLM Providers and BYOK

LLM Providers & BYOK

Maestro is provider-agnostic. Every LLM provider is an adapter class behind a common interface, so adding a provider means adding one file — never modifying existing code. Users bring their own keys (BYOK), which are encrypted at rest and never returned to the browser. The framework lives in backend/app/services/llm_service.py.

The adapter interface

• LLMAdapter — base class. AdapterCapabilities declares what a provider supports (native tool-calling, native JSON schema, streaming).
• _OpenAICompatAdapter — shared implementation for the many OpenAI-compatible providers.
• ChatMessage, ToolCall, ToolDef, LLMResponse, StreamEvent, LLMError — the common data types.
• Retries via Tenacity; _is_retryable treats 5xx + transport errors as transient, but not 4xx or read-timeouts.

Chat-capable providers (12)

All are BYOK-selectable as the "brain" of a task:

ollama, openai, anthropic, gemini, groq, deepseek, mistral, xai, openrouter, together, perplexity, custom.

Concrete adapters: OllamaAdapter, OpenAIAdapter, GeminiAdapter, GroqAdapter, DeepSeekAdapter, MistralAdapter, XAIAdapter, OpenRouterAdapter, TogetherAdapter, PerplexityAdapter, CustomOpenAICompatAdapter (SSRF-guarded), and AnthropicAdapter (the real Messages API: x-api-key, system, max_tokens).

LLM_CHAT_PROVIDERS in constants.py is the authoritative list of the 12.

Service (non-chat) providers

BYOK also stores keys for service connectors used by tools/integrations, not as a chat brain: x, github, instagram, google_maps, slack, notion, discord, telegram.

Wrappers (composition, not inheritance)

Adapters are wrapped in layers, each adding one concern:

• FallbackLLMAdapter — tries Gemini, falls back to Ollama per call. This is what makes the free/local tier resilient.
• TokenMeter — counts tokens for every LLM call (routing, planning, subagent, reviewer, synthesis) via a shared _TokenCounter, with estimation when the provider doesn't report usage. This is what feeds the quota ledger (see Billing-and-Quota).
• TracedAdapter — emits OTel-shaped gen_ai.* spans + cost, sitting under TokenMeter.
• AdapterPool — resolves the model per agent role (ctx.role_adapter(role)) using resolve_model: precedence is explicit override > user preference > provider tier default. All roles share one token counter.

Structured output and tool-calling

• structured_call (agents/structured.py) — a schema-validated LLM call. It uses native json_schema when the provider supports it, otherwise falls back to extract_json + Pydantic validation, retrying on mismatch.
• Native tool loop — used when capabilities.native_tools is true (parses/serializes OpenAI-style tool_calls).
• Directive loop — the provider-agnostic fallback: the model emits a JSON ToolDirective, parsed by tools.parse_directive. This lets even a bare completion endpoint use tools.

See Agent-Orchestration for how subagents drive these loops.

Model routing and tiers

constants.py holds MODEL_TIER_DEFAULTS, PROVIDER_TIER_MODELS, PROVIDER_COST_PER_1K_TOKENS, and MODEL_PRICING. Per-user overrides live in users.model_preferences (JSONB), settable via PATCH /users/me and consumed by AdapterPool.

Embeddings

embed_texts uses nomic-embed-text via Ollama. The endpoint is EMBEDDING_ENDPOINT when set, otherwise falls back to FREE_MODEL_ENDPOINT. Dimension is EMBEDDING_DIM = 768. In production, embeddings hit a small dedicated ollama service (embeddings only). See RAG-and-Memory and Configuration.

BYOK security

• Keys are encrypted with AES-256-GCM (core/security.py encrypt_secret): output is base64(12-byte nonce ‖ ciphertext). The master key (API_KEY_MASTER_KEY) is base64→32-byte and lives only in the environment.
• The same master key encrypts the user's TOTP secret.
• API keys are never returned to the frontend — only provider, label, and a key_hint (****abcd) are exposed. See Security.
• A resumed durable task re-decrypts the key from Postgres at run time; the frozen run payload contains no secret. See Agent-Orchestration.
• Custom endpoints pass through the SSRF guard (url_guard.check_public_url) before any request.

Adding a new provider

1. Add the provider to the LLMProvider enum and LLM_CHAT_PROVIDERS (if chat-capable) in constants.py.
2. Add a concrete adapter class in llm_service.py (subclass _OpenAICompatAdapter if OpenAI-compatible, else implement LLMAdapter).
3. Register it in get_adapter.
4. Add tier/pricing entries if you want cost tracking.

No existing adapter, wrapper, or agent code changes. This is the pattern the whole platform is built on. See Contributing-and-License.