features providers and models

Providers and models

Active contributors: Mario Zechner, kt, Armin Ronacher

Purpose

Providers and models is how a user authenticates against an LLM provider and selects which model the agent uses. It spans credential storage in packages/coding-agent, the model registry in packages/ai, the provider catalog and model patterns in web/protocol, the provider and model HTTP handlers in web/server, and the model and effort pickers in web/design. The /login flows live in the runtime; the web Settings dialog surfaces the same credential surface and the same model allowlist.

The full provider surface is defined in packages/ai (packages/ai/src/env-api-keys.ts, packages/ai/src/models.ts), while the user-facing credential subset the web UI shows is defined in web/protocol/src/provider-catalog.ts. A provider-catalog integrity test fails if the env-scrub list diverges from packages/ai/src/env-api-keys.ts.

How it works

Credential storage

AuthStorage (packages/coding-agent/src/core/auth-storage.ts) is the credential store for API keys and OAuth tokens. It persists to auth.json under the agent dir (~/.prime/agent/auth.json by default) with 0600 permissions, using file locking (proper-lockfile) so concurrent instances do not clobber token refreshes. FileAuthStorageBackend and InMemoryAuthStorageBackend provide the locking backend. Each provider entry is an api_key or oauth credential.

Credential resolution follows a priority order implemented in getApiKeyWithSourceToken: runtime override (--api-key), then for Prime Inference the environment variable, Prime CLI config, and auth.json, and for other providers auth.json before the environment variable, then a fallback resolver (for custom provider keys from models.json). getAuthStatus reports whether a provider is configured without exposing values, and markAuthStale/isAuthSourceStale track expired OAuth tokens so the UI can prompt for re-login. OAuth credentials are refreshed under lock in refreshOAuthTokenWithLock.

Provider catalog and env vars

PI_PROVIDER_CATALOG (web/protocol/src/provider-catalog.ts) is the user-facing credential list shown in Settings: id, display name, and the single env var that detects whether the provider is configured (for example OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, GITHUB_COPILOT_TOKEN). INFRA_PROVIDER_IDS marks infrastructure/config entries that are not selectable LLMs. CREDENTIAL_UI_PROVIDERS is the catalog minus infra and OAuth-only entries. Custom and OpenAI-compatible instances get prefixed ids (custom+<slug>, openai-chat-completions+<slug>) through toInstanceSlug/toOccInstanceId and related helpers. PROVIDER_ENV_SCRUB_VAR_NAMES lists every env var scrubbed on Vercel so org keys never back chat tools.

The authoritative env var map is packages/ai/src/env-api-keys.ts, which maps each provider to its environment variable and detects credentials (including ambient credentials such as AWS profiles and Google ADC). PI_LLM_RUNTIME_PROVIDER_IDS in web/protocol/src/provider-catalog.ts mirrors this list for the runtime providers that can pick up org credentials.

Model registry

packages/ai/src/models.ts builds an in-memory registry from the generated packages/ai/src/models.generated.ts and exposes getModel, getProviders, getModels, and cost helpers. The model metadata (id, name, api, reasoning, input, context window, cost, thinking levels) is defined there. The runtime-side registry is ModelRegistry (packages/coding-agent/src/core/model-registry.ts), which manages built-in and custom models, resolves API keys, registers API providers and OAuth providers, and reads custom providers from models.json.

Model selection and settings persistence

SettingsManager (packages/coding-agent/src/core/settings-manager.ts) persists per-cwd settings, including default provider/model, thinking level, and the enabled-model allowlist. model-resolver.ts (packages/coding-agent/src/core/model-resolver.ts) resolves model references (exact ids or provider/modelId), scopes models to a session, applies the enabled-model patterns, and provides defaultModelPerProvider for initial selection. The allowlist matching itself is modelMatchesPattern (web/protocol/src/model-patterns.ts), which supports exact ids and globs (*, ?) with an optional :high thinking suffix.

The web Settings dialog reads and writes these through web/server. GET/PATCH /api/chat/settings (web/app/src/routes/api/chat/settings.ts delegating to web/server/src/handlers/chat-settings.ts) exposes ChatPiSettings. handleChatModelsGet (web/server/src/handlers/chat-models.ts) lists models with scope=enabled (only configured-authorized models passing the enabled-model pattern) or scope=all, and returns each model's key, thinking levels, availability, and the currently selected model. handleChatProvidersGet/Post/Delete (web/server/src/handlers/chat-providers.ts) list providers with auth status and let the UI set or remove an api_key credential via authStorage.

Model discovery

For custom and OpenAI-compatible providers, POST /api/chat/models/discover (web/app/src/routes/api/chat/models/discover.ts delegating to web/server/src/handlers/chat-models-discover.ts) probes the provider's /v1/models endpoint with the configured base URL and API key, mapping each returned model into the standardized Model shape. It returns an empty list when no key or base URL is configured, on non-OK responses, or on timeout.

Picker UI

ModelPicker (web/design/src/components/agent-elements/input/model-picker.tsx) is the model dropdown in the composer. It renders a Popover list of ModelOptions, highlights the active model, and scrolls it into view on open. EffortPicker (web/design/src/components/agent-elements/input/effort-picker.tsx) selects a thinking level from the model's supported ChatThinkingLevels, using THINKING_LEVEL_DESCRIPTIONS for the option descriptions and returning nothing when the model exposes no levels. Both are controlled or uncontrolled and forward onChange.

TUI login and selection

The TUI has its own login and selection surfaces: /login is a slash command handled in packages/coding-agent/src/core/slash-commands.ts, and the interactive components include login-dialog.ts, oauth-selector.ts, model-selector.ts, settings-selector.ts, and scoped-models-selector.ts under packages/coding-agent/src/modes/interactive/components/. They drive the same AuthStorage and ModelRegistry, so a provider configured in the web UI is available to the TUI and vice versa.

Integration points

• Runtime credentials and settings: packages/coding-agent (AuthStorage, SettingsManager, ModelRegistry, model-resolver). See coding-agent.
• LLM layer: packages/ai (models.ts, models.generated.ts, env-api-keys.ts). See packages/ai.
• Wire contract: web/protocol (provider-catalog.ts, model-patterns.ts, settings schemas). See web-protocol.
• Server handlers: web/server (chat-providers.ts, chat-models.ts, chat-models-discover.ts, chat-settings.ts). See web-server.
• Picker UI: web/design. See web-design.
• HTTP endpoints: web-api.
• OAuth login flows: oauth.

Entry points for modification

• Add a provider credential entry to the web UI: edit web/protocol/src/provider-catalog.ts and the matching env var map in packages/ai/src/env-api-keys.ts; the integrity test enforces the sync.
• Change model metadata or defaults: edit packages/ai/scripts/generate-models.ts (do not hand-edit models.generated.ts) and packages/coding-agent/src/core/model-resolver.ts (defaultModelPerProvider).
• Change the enabled-model allowlist semantics: edit web/protocol/src/model-patterns.ts.
• Change credential resolution or OAuth refresh: edit packages/coding-agent/src/core/auth-storage.ts.
• Change the model list response: edit web/server/src/handlers/chat-models.ts.
• Change the picker UI: edit web/design/src/components/agent-elements/input/model-picker.tsx and effort-picker.tsx.

Key source files

File	Role
packages/coding-agent/src/core/auth-storage.ts	Credential storage (auth.json), resolution priority, OAuth refresh with locking.
packages/coding-agent/src/core/settings-manager.ts	Per-cwd settings persistence: default provider/model, thinking level, enabled models.
packages/coding-agent/src/core/model-registry.ts	Built-in and custom model management, API key resolution, provider registration.
packages/coding-agent/src/core/model-resolver.ts	Model reference resolution, scoping, and defaultModelPerProvider.
packages/ai/src/models.ts	In-memory model registry over models.generated.ts; getters and cost helpers.
packages/ai/src/models.generated.ts	Generated model catalog (do not hand-edit; regenerate via generate-models.ts).
packages/ai/src/env-api-keys.ts	Provider to env-var map and credential detection.
web/protocol/src/provider-catalog.ts	User-facing provider catalog, custom/OCC id helpers, and env scrub list.
web/protocol/src/model-patterns.ts	Glob/exact matching for the enabled-model allowlist.
web/server/src/handlers/chat-providers.ts	List providers with auth status; set/remove api_key credentials.
web/server/src/handlers/chat-models.ts	List models filtered by scope and the enabled-model allowlist.
web/server/src/handlers/chat-models-discover.ts	Probe a custom provider's /v1/models endpoint for discovery.
web/server/src/handlers/chat-settings.ts	Read and update ChatPiSettings.
web/design/src/components/agent-elements/input/model-picker.tsx	Model dropdown in the composer.
web/design/src/components/agent-elements/input/effort-picker.tsx	Thinking-level dropdown for the selected model.