AI Providers

AI Providers

The contextual detection pass and Hush can use any of 25 providers: a hosted API with your key, a cloud platform with your credentials, a model on your own machine, or a ChatGPT subscription. The model is optional. Without one, pattern detection, manual redaction, search, rules and export all work.

Canonical sources: docs/ai-providers.md · docs/connect-openai.md · code: lib/ai/providers/.

History: one AI Gateway until 1.3.0, when official SDK providers, Azure, Bedrock, Vertex and Ollama arrived (#115). 1.4.0 added catalog paging and list prices. 1.8.0 added OpenAI-compatible profiles, LM Studio, llama.cpp, ChatGPT sign-in and pnpm ai (#126).

---

1. Choosing one

Run pnpm setup (or pnpm ai verify). Choose the provider, enter its credential (not echoed), pick a discovered model, and verify it before saving. Discovery and verification happen only in your CLI and send no documents: verification makes two small synthetic requests (a structured-output check and an image check), which hosted providers may bill.

AI_PROVIDER	Credentials / configuration	Discovery
gateway (default)	AI_GATEWAY_API_KEY or Vercel OIDC	Gateway models, with list prices
openai	OPENAI_API_KEY	account models
anthropic	ANTHROPIC_API_KEY	account models
google	GOOGLE_GENERATIVE_AI_API_KEY	Gemini models
xai	XAI_API_KEY	provider models
mistral	MISTRAL_API_KEY (also used by Mistral OCR)	models with capability metadata
togetherai	TOGETHER_API_KEY	provider models
cohere	COHERE_API_KEY	models and endpoints
fireworks	FIREWORKS_API_KEY, optional FIREWORKS_ACCOUNT_ID	account or public catalog
deepinfra	DEEPINFRA_API_KEY	provider models, with list prices
deepseek	DEEPSEEK_API_KEY	provider models
cerebras	CEREBRAS_API_KEY	provider models
groq	GROQ_API_KEY	provider models
perplexity	PERPLEXITY_API_KEY	typed Sonar model ID
baseten	BASETEN_API_KEY	shared Model APIs catalog
azure	AZURE_API_KEY, AZURE_RESOURCE_NAME (model = deployment name)	actual deployments, with management credentials
amazon-bedrock	AWS_REGION + the AWS credential chain, or AWS_BEARER_TOKEN_BEDROCK	foundation models and inference profiles
google-vertex	GOOGLE_VERTEX_PROJECT, GOOGLE_VERTEX_LOCATION, ADC or express key	Google publisher catalog
ollama	OLLAMA_BASE_URL (default http://localhost:11434)	installed models
openrouter	OPENROUTER_API_KEY	/models, with list prices
synthetic	SYNTHETIC_API_KEY	/models
lm-studio	optional AI_BASE_URL (default http://localhost:1234/v1)	loaded models
llama-cpp	optional AI_BASE_URL (default http://localhost:8080/v1)	the served model
openai-compatible	AI_BASE_URL (required), optional AI_API_KEY	/models at that URL
openai-subscription	pnpm ai login --provider openai; nothing in .env	the plan's models

Model lists are live, not a checked-in shortlist. Known-incompatible models are shown disabled with a reason. A model with no capability metadata is labelled unverified and needs a successful probe before it can be saved.

Analysis mode. Text and images is the full pass. Text (image support optional) also accepts a model that passes the structured-output check but cannot read images. OCR still extracts the text, image-region analysis is skipped, and the usage panel says so. A model that fails structured output cannot be selected in either mode.

Existing installs with no AI_PROVIDER keep using the Gateway with their existing AI_MODEL (default anthropic/claude-haiku-4.5) and are not re-probed. Any other provider needs an explicit AI_MODEL and the capability declaration setup writes. Changing the model or destination invalidates that declaration.

---

2. Verification, and what it writes

pnpm ai verify runs setup's probe on its own for any provider and writes three lines into the existing .env, leaving the rest alone:

AI_PROVIDER=openai
AI_MODEL=<model id>
AI_MODEL_CAPABILITIES='{"target":"[\"openai\",\"<model id>\",...]","structuredOutput":true,"vision":true}'

pnpm ai verify --provider ollama --model qwen3-vl:4b-instruct
pnpm ai verify --text-only                      # accept a model that cannot read images
pnpm ai verify --input-price 1.25 --output-price 10
pnpm ai verify --print                          # print instead of writing .env

When verification fails, it says why

Setup, pnpm ai verify and pnpm ai login name the check that failed (structured output or image) and the reason:

• Credentials rejected (401/403) · not found (404: a model this account cannot use, or a wrong base URL) · refused (400: often no JSON-schema output or image input) · rate-limited / over quota (429) · out of credit (402) · provider failed (5xx);
• unreachable: connection refused, DNS failure, untrusted TLS certificate, or no answer within two minutes;
• answered badly: not JSON, JSON that does not fit the schema, or the wrong answer.

What the provider said is quoted only here, because verification sends synthetic requests and cannot quote a document. It is redacted first (configured credentials, anything shaped like a key, bearer token or JWT) and cut to 300 characters. A failure that would hit every model (a bad key, a server that is down) leaves the model selectable, so the real cause gets fixed rather than hidden.

--yes and non-interactive setup stay offline: existing AI settings are kept, with no discovery and no billable probes.

---

3. Local models

Ollama, LM Studio and llama.cpp keep everything on your machine.

• On a fresh configuration, the first local server that responds is offered as the default (Ollama, then LM Studio, then llama.cpp, each on its usual port). An existing hosted provider is never silently replaced.
• Local rows have zero provider spend (spendStatus: local) and default to one request at a time.
• Ollama goes through the official @ai-sdk/openai-compatible adapter against its structured-output endpoint. Ollama cloud models are disabled, because this integration is for local inference.
• In Docker, Compose maps localhost, 127.0.0.1 and ::1 in OLLAMA_BASE_URL / AI_BASE_URL to host.docker.internal at connection time, so the same .env works with pnpm dev on the host. On Linux, Ollama must listen on an interface Docker can reach (OLLAMA_HOST); restrict it to a trusted network.
• llama.cpp needs a chat template, and a multimodal projector (--mmproj) for image analysis. Without one, choose text-only.
• For vision, Qwen3-VL works through Ollama (since 1.6.2). CI tests the contract against a pinned Qwen3-VL 4B Instruct (.github/workflows/local-model.yml, weekly and on provider-layer PRs).

---

4. OpenAI-compatible endpoints

A vendor that speaks only the chat-completions protocol is one row in lib/ai/providers/compatible.ts (id, label, base URL, key variable), all going through the official @ai-sdk/openai-compatible adapter.

• Hosted profiles (OpenRouter, Synthetic) have fixed URLs. AI_BASE_URL does not move them, so a leftover value cannot send a hosted key somewhere else.
• Local profiles (LM Studio, llama.cpp) default to their usual port, and AI_BASE_URL overrides it.
• openai-compatible is anything else (vLLM, a company proxy) named by AI_BASE_URL, with AI_API_KEY sent as a bearer token when set. Because the URL could belong to anyone, it is never assumed local. Price it under AI_MODEL_PRICES key openai-compatible:<model>, and lower ANONIFY_AI_CONCURRENCY for a single GPU.

AI_BASE_URL is the URL that /chat/completions and /models sit under (usually ending in /v1), with no credentials or query string. The capability declaration is bound to it.

To add a vendor: a row in compatible.ts, a line in .env.example and the Compose pass-through list, and a row in the table above.

---

5. Cloud platforms

• Azure: the resource key plus the deployment name as AI_MODEL. Discovering deployments also needs AZURE_SUBSCRIPTION_ID, AZURE_RESOURCE_GROUP and a management credential (az login, managed identity); a typed deployment can still be verified without them.
• Bedrock: the AWS credential chain for discovery and signed inference (AWS_REGION, optional AWS_PROFILE, env credentials or roles). Discovery needs bedrock:ListFoundationModels and bedrock:ListInferenceProfiles. AWS_BEARER_TOKEN_BEDROCK works for inference but not for listing.
• Vertex: Application Default Credentials and a project/location. The publisher catalog may list models unavailable in your region, so verification still matters. GOOGLE_VERTEX_API_KEY enables express mode for inference.

Compose passes environment credentials but does not mount host credential stores. Mount an AWS profile or a Google credentials file read-only with a Compose override, and use paths as seen inside the container.

---

6. ChatGPT subscription sign-in

pnpm ai login --provider openai                   # sign in, then pick, verify and price a model
pnpm ai verify --provider openai-subscription     # choose another model later
pnpm ai status                                    # account, plan, plan limits, this instance's usage
pnpm ai logout --provider openai                  # delete the stored token

Read this first. OpenAI does not register OAuth clients for other tools to use a subscription for inference. Sign-in uses the public client OpenAI ships with Codex CLI, against the backend Codex uses. Whether a ChatGPT plan may be used this way is for OpenAI's terms to decide, and the backend is not a documented API. For a deployed or shared instance, use an API key (AI_PROVIDER=openai). The command prints this caveat every time.

• PKCE with a loopback callback on http://localhost:1455/auth/callback, bound to 127.0.0.1. On a headless host, open the printed link anywhere and paste the final localhost URL back at the (non-echoing) prompt. A wrong state is refused.
• The token is sealed with ENCRYPTION_KEY in one Setting row (ai.login.openai) and never written to .env. Run login where DATABASE_URL reaches the instance's database; the app in Docker reads the same row. If ENCRYPTION_KEY changes, pnpm ai status says the token cannot be opened, rather than claiming you are signed out.
• Tokens near expiry are refreshed on use and written back. A revoked sign-in reaches the reviewer as authorization and is not retried.
• Anonify never reads another tool's credentials (not Codex's auth.json, not Claude Code's store). Sharing a refresh token between tools logs one of them out at random.
• Usage rows are keyed openai-subscription:<model> and cost $0 unless you record a price, which login offers to do.
• There is no Anthropic equivalent. Anthropic's terms reserve Claude Free, Pro and Max sign-in for Anthropic's own apps. pnpm ai login --provider anthropic explains this and exits. Use ANTHROPIC_API_KEY.

A step-by-step for both OpenAI routes, including headless servers: docs/connect-openai.md.

---

7. Prices, spend and limits

• AI_MODEL_PRICES: JSON keyed by provider:model (or the Gateway's vendor/model), each entry { inputPerMillion, outputPerMillion } in USD. These are operator estimates and the only prices the app enforces. The legacy AI_PRICE_INPUT_PER_MTOK / AI_PRICE_OUTPUT_PER_MTOK pair applies only to the currently selected model.
• List prices in setup come from the provider's own model list when it publishes them in a documented unit: the Gateway and OpenRouter (USD per token) and DeepInfra (cents per token). Choosing a priced model offers to copy its price, with source and age shown; nothing is copied without a yes, and the default is no for tiered, representative, stale or overriding prices.
• Model-list cache: .cache/models/<provider>.json (or ANONIFY_MODEL_CACHE_PATH), with source URL and fetch time and never a credential. Prices go stale after 24 h, capabilities after 7 days. Refresh with pnpm models:warm (--provider, --all, --force, --offline). Local servers and the subscription list are always read live.
• ANONIFY_AI_DAILY_SPEND_USD (default 0 = no cap), summed per model at its own rate: at 80% calls go one at a time; at 100% the contextual pass is skipped for the rest of the UTC day, and Hush stops. Missing rates make the cap unenforceable, and the log says so.
• Pacing and retries (lib/services/throttle.ts): ANONIFY_AI_CONCURRENCY (4 hosted, 1 local), ANONIFY_AI_REQUESTS_PER_MINUTE (0 = unpaced), ANONIFY_AI_MAX_ATTEMPTS (4). One process-wide gate is enough, because an instance does not route between providers or fail over.

---

8. Checking what is in force

pnpm ai status

Shows the provider and model, whether they are verified, the calls and tokens this instance has sent, and, for a ChatGPT sign-in, the account, the plan and the plan's usage limits as OpenAI reports them.

In the app, the usage panel shows per-document tokens, duration and calls (and cost when priced), plus any degradation. Hush's header shows provider badges: hosted API, local model ("Stays on this machine"), ChatGPT plan, or gateway. GET /api/assistant reports { available, provider: { id, label, kind, model } } or a reason (not-configured, unsupported, budget).

---

9. What is tested, and what is not

Credential-free tests use synthetic models and local HTTP fixtures to check wire requests, structured output, image attachments, auth, discovery, disabled choices, capability invalidation, degradation and per-model prices. The subscription flow is tested against fixtures, with no live account in CI. A real Ollama + Qwen3-VL run checks the contract and a full Compose install (pnpm smoke --ai). None of this measures model quality: that is Testing and Benchmarks.

---

See also

• Redaction Engine · Hush Assistant · Configuration · CLI Reference