Local AI and model routing

Local AI & multi-model routing

GenOS is provider-neutral: an agent's genome and state are not bound to one model API. The runtime can route work to a cloud model, a local server, or an ordered fallback chain without changing the snapshot/fork/evaluate/replay lifecycle.

Supported providers

URI	Provider	API key required
ollama://model	Local Ollama	No
lmstudio://model	Local LM Studio	No
vllm://model	Local vLLM	No
openai://model	OpenAI	Yes
anthropic://model	Anthropic	Yes
gemini://model	Gemini	Yes
mistral://model	Mistral	Yes
openai-compatible://model	Compatible endpoint	Server-dependent

Automatic local discovery

Studio queries three local defaults:

Runtime	Generation endpoint	Model catalogue
LM Studio	localhost:1234/v1/chat/completions	/v1/models
Ollama	localhost:11434/v1/chat/completions	/api/tags
vLLM	localhost:8000/v1/chat/completions	/v1/models

Embedding and reranking models are excluded from chat routes. Discovery is cached for five seconds. Override the endpoints with GENOS_LMSTUDIO_ENDPOINT, GENOS_OLLAMA_ENDPOINT, and GENOS_VLLM_ENDPOINT.

When no explicit route exists, GenOS tries the discovered chat-capable local models. For example:

ollama serve
ollama pull qwen2.5-coder:7b
export GENOS_DEFAULT_MODEL='ollama://qwen2.5-coder:7b'

Ordered fallback

export GENOS_DEFAULT_MODEL='ollama://qwen2.5-coder:7b'
export GENOS_MODEL_FALLBACKS='lmstudio://local-coder,openai://gpt-5-mini'
export GENOS_MODEL_ROUTING_MODE='fallback'

GenOS tries routes in order and records failed attempts before the selected route. GENOS_PREFER_LOCAL_MODELS=1 moves Ollama, LM Studio, and vLLM URIs ahead of remote routes.

Parallel review

export GENOS_MODEL_ROUTING_MODE='parallel'
export GENOS_MODEL_PARALLEL_REVIEW='ollama://reviewer-a,vllm://reviewer-b'

All configured models receive the request. The result includes attempts and successful responses as reviews. Selection currently remains deterministic by configuration order; it is not an automatic truth aggregator. Robust decisions still require shared criteria, tests, and evidence.

Organization- and project-scoped policy

Studio can persist a route for a specific agent or the wildcard agent *, scoped by organizationId + projectId. This supports configurations such as:

• a local-only organization;
• a project that permits cloud fallback;
• parallel review for a critical agent;
• inexpensive research agents and a stronger verification agent.

What “local” does and does not guarantee

A local endpoint avoids sending requests to a cloud provider by default and may reduce cost or latency. It does not by itself guarantee whole-system privacy, deterministic inference, freedom from hallucinations, or process isolation. The endpoint, logs, network, and available tools still need to be audited.

GenOS keeps the same control plane around either kind of model: budgets, provenance, isolated worlds, fallback, comparison, and evidence-gated promotion.

See the model router, local discovery service, and Hallucination reduction.