systems llm providers

LLM providers

Active contributors: ferdiiskandar

Purpose

This system resolves which provider and model serve a request, adapts each vendor SDK to one streaming interface, stores user-supplied API keys in encrypted form, and estimates token counts for the UI. Both the Optimizer and the Evaluator call into it; no engine talks to a vendor SDK directly.

Directory layout

lib/llm/
  provider-registry.ts     factory, availability, scoped OpenAI overrides
  provider-readiness.ts    guest/desktop readiness resolution
  user-api-keys.ts         encrypted per-user key storage
  token-counter.ts         character-based token estimation
  types.ts                 LLMProviderAdapter and ProviderConfig
  constants.ts             fallback model slugs
  providers/               one adapter per provider

Key abstractions

Symbol	Kind	Path
LLMProviderAdapter	interface (generate, generateStream, validateApiKey)	lib/llm/types.ts
getProvider	factory function	lib/llm/provider-registry.ts
getAvailableProviders	availability function	lib/llm/provider-registry.ts
getScopedProviderOverrides	model/base-URL resolver	lib/llm/provider-registry.ts
resolveGuestProvider, createProviderReadiness	readiness resolvers	lib/llm/provider-readiness.ts
listProviderKeyStates, upsertUserProviderKey, deleteUserProviderKey, resolveProviderApiKey	key storage	lib/llm/user-api-keys.ts
MissingProviderApiKeyError	error class	lib/llm/user-api-keys.ts
estimateTokens, estimatePromptTokens	token estimation	lib/llm/token-counter.ts
OPENROUTER_LANE_MODEL_FALLBACKS	fallback map	lib/llm/constants.ts

getProvider maps provider names to constructors: CLAUDE to AnthropicProvider, OPENAI to OpenAIProvider, MISTRAL to MistralProvider, QWEN and GROK to OpenAI-compatible subclasses, and LOCAL to LocalProvider. The env key map is ANTHROPIC_API_KEY, OPENAI_API_KEY, MISTRAL_API_KEY, QWEN_API_KEY, and XAI_API_KEY; LOCAL needs no key. Prompt builders supply model and base URL via ProviderConfig.

QwenProvider and GrokProvider extend OpenAIProvider and only override the name, default model, and base URL (PROVIDER_BASE_URLS in lib/constants.ts). LocalProvider posts directly to an Ollama /api/chat endpoint and streams newline-delimited JSON.

How it works

graph TD
    Engine[Optimizer / Evaluator engine] --> Overrides[getScopedProviderOverrides]
    Engine --> Keys[resolveProviderApiKey]
    Keys -->|user row present| Decrypt[decrypt AES-256-GCM]
    Keys -->|no user row| Env[env key map]
    Overrides --> Registry[getProvider]
    Decrypt --> Registry
    Env --> Registry
    Registry --> Adapters[providers/* adapter]
    Adapters --> SDK[vendor SDK or Ollama HTTP]

Loading

getScopedProviderOverrides applies only to OPENAI. Model precedence is lane-scoped, then scope-scoped, then shared: OPTIMIZER_INTERACTIVE_OPENAI_MODEL or OPTIMIZER_DEEP_OPENAI_MODEL, then OPTIMIZER_OPENAI_MODEL or EVALUATOR_OPENAI_MODEL, then OPENAI_MODEL. Base URL follows the same shape through OPENAI_BASE_URL, *_OPENAI_BASE_URL, and the per-lane variants. When no lane or scope model is set, the effective base URL is OpenRouter, and the shared model is a key in OPENROUTER_LANE_MODEL_FALLBACKS, the lane gets a fallback slug (deepseek/deepseek-v4-pro maps to google/gemini-3.1-flash-lite on INTERACTIVE and deepseek/deepseek-v4-flash on DEEP).

resolveGuestProvider keeps a requested provider when it appears in the available list, otherwise falls back to OPENAI when present and then to the first available provider, flagging usedFallback. createProviderReadiness wraps that into a ready or missing result with availableProviders and activeProvider.

resolveProviderApiKey returns the decrypted user key when a UserApiKey row exists, then the environment key, and otherwise throws MissingProviderApiKeyError. Keys are encrypted by encrypt/decrypt in lib/crypto.ts using AES-256-GCM; the ciphertext, 16-byte IV, and auth tag are stored as hex on prisma/schema.prisma. ENCRYPTION_KEY is read as a 64-character hex key or hashed with SHA-256.

Token estimation is character-based, not a real tokenizer: estimateTokens divides length by a per-provider ratio (CLAUDE 3.5, OPENAI 4.0, MISTRAL 3.8, QWEN 3.5, GROK 4.0, LOCAL 3.5). estimatePromptTokens adds an output estimate of twice the input, capped at 4096.

Integration points

The desktop IPC command router resolves a key and provider before dispatching Optimizer and Evaluator runs, and the readiness helpers drive the guest versus authenticated provider picker. See Optimizer and Evaluator. Embeddings bypass this registry and read OPENAI_API_KEY directly via lib/embeddings/generator.ts. Relevant environment variables are listed in Configuration, and key handling is covered in Security.

Entry points for modification

• Add a provider: extend the LLMProvider enum, add a constructor to providerConstructors and an entry to envKeyMap in lib/llm/provider-registry.ts, then add an adapter under lib/llm/providers/.
• Change scoped model resolution or OpenRouter fallbacks: lib/llm/provider-registry.ts and lib/llm/constants.ts.
• Change encryption: lib/crypto.ts.

Key source files

File	What it holds
lib/llm/provider-registry.ts	Provider factory, env key map, scoped OpenAI overrides
lib/llm/provider-readiness.ts	Guest provider resolution and readiness results
lib/llm/user-api-keys.ts	Encrypted user key list, upsert, delete, resolve
lib/llm/token-counter.ts	Character-ratio token estimation
lib/llm/constants.ts	OpenRouter lane fallback slugs
lib/llm/providers/*.ts	One adapter per provider