AI Synth

AI Synth is a small, self-hosted AI orchestration gateway. It exposes an OpenAI-compatible chat completions API, uses one reasoning model to plan and merge work, and routes delegated tasks to specialized worker models.

The service is designed for a private homelab or internal network. It has no database, scheduler, message broker, or persistent queue.

Table of Contents

• Features
• Architecture
• Requirements
• Configuration
• Run locally
• Run with Docker
• API
• Project layout
• Documentation
• Security
• Contributing
• License

Features

• OpenAI-compatible chat/completions and model endpoints.
• A reasoning model that plans, verifies, repairs, and merges work.
• Capability-based worker routing with load balancing and cooldowns.
• Concurrent execution of independent delegated tasks.
• Bounded worker retries after verification feedback.
• Optional bearer-token authentication for public API endpoints.
• Health reporting for configured workers.
• Streaming responses using server-sent events.

Architecture

Client
  |
  v
FastAPI gateway
  |
  v
Reasoning model creates a JSON plan
  |
  +-- no delegation --> reasoning model writes the answer
  |
  +-- delegated tasks --> router selects workers
                              |
                              v
                         workers execute
                              |
                              v
                    reasoning model verifies results
                              |
                              v
                    reasoning model merges the answer

Workers are execution-only. They cannot create tasks, select other workers, change configuration, bypass verification, or decide the final response.

For simple greetings and arithmetic, the deterministic guard prevents worker delegation. The reasoning model still plans and produces the final response.

Requirements

• Python 3.12 or newer.
• One OpenAI-compatible reasoning endpoint.
• At least one OpenAI-compatible worker endpoint.
• Network access from the gateway to every configured endpoint.
• Docker, if you prefer containerized deployment.

Each model endpoint must accept POST /chat/completions below the configured base URL and return an OpenAI-compatible response containing choices[0].message.

Configuration

Start from config.example.json:

cp config.example.json config.json

Edit the URLs, model names, capabilities, timeouts, and concurrency limits for your environment. Do not publish config.json if it contains private network details or upstream API keys.

The configuration file contains:

Setting	Description
public_model	Model name exposed by the gateway.
max_worker_attempts	Maximum execution attempts for each delegated task.
max_tasks	Maximum number of tasks accepted from one plan.
orchestrator	Reasoning model used for planning, verification, and merging.
workers	List of execution-only model endpoints.

Each node supports:

Setting	Description
name	Stable display name used in logs and health responses.
base_url	Absolute http or https URL, usually ending in /v1.
model	Model identifier sent to the upstream endpoint.
capabilities	Labels used by the router to select the worker.
timeout_seconds	Upstream request timeout.
max_concurrency	Number of requests allowed concurrently.
max_tokens	Default completion limit for that node.
enabled	Optional flag to disable a node without removing it.
api_key	Optional upstream bearer token.
prompt_prefix	Optional text prepended to worker instructions.

The gateway reads these environment variables:

Variable	Default	Description
ORCHESTRATOR_CONFIG	config.json	Path to the JSON configuration file.
ORCHESTRATOR_API_KEY	unset	If set, requires a bearer token on protected API endpoints.
LOG_LEVEL	INFO	Python logging level.

The API key is optional for local development but should be configured before exposing the service beyond a trusted network.

Run locally

Create or reuse a virtual environment, install dependencies, and start Uvicorn:

if [ -z "${VIRTUAL_ENV:-}" ]; then
  if [ ! -d .venv ]; then
    python3 -m venv .venv
  fi
  . .venv/bin/activate
fi
python -m pip install -r requirements.txt
ORCHESTRATOR_CONFIG=config.json uvicorn src.server:app --host 0.0.0.0 --port 8000

If a virtual environment is already active, the commands reuse it and do not create another one. Otherwise, .venv is created only when it does not exist.

Check that the gateway loaded its configuration:

curl http://localhost:8000/health

Send a chat request:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ai-synth",
    "messages": [
      {"role": "user", "content": "Summarize the role of a reverse proxy."}
    ]
  }'

When ORCHESTRATOR_API_KEY is set, add the bearer token to the request:

--oauth2-bearer "$ORCHESTRATOR_API_KEY"

Run tests

After activating the environment and installing the requirements, run all unit and integration tests with:

python -m unittest discover -s tests -t . -v

The integration tests use scripted upstream responses, so they do not require running model servers or making network requests to real model endpoints.

Run with Docker

Create and edit a local configuration before building:

cp config.example.json config.json
docker build -t ai-synth .
docker run --rm \
  --name ai-synth \
  -p 8000:8000 \
  -v "$PWD/config.json:/app/config.json:ro" \
  ai-synth

Pass the API key and log level as environment variables when needed:

docker run --rm \
  --name ai-synth \
  -p 8000:8000 \
  -e ORCHESTRATOR_API_KEY='replace-with-a-long-random-value' \
  -e LOG_LEVEL=INFO \
  -v "$PWD/config.json:/app/config.json:ro" \
  ai-synth

On Docker Desktop, host.docker.internal can reach services running on the host. On Linux, use an address reachable from the container or add an appropriate host-gateway mapping.

API

Method	Endpoint	Authentication	Description
GET	/health	No	Gateway and worker status.
GET	/v1/models	Optional bearer token	Lists the public model.
GET	/models	Optional bearer token	Alias for /v1/models.
POST	/v1/chat/completions	Optional bearer token	OpenAI-compatible chat completion.
POST	/chat/completions	Optional bearer token	Alias for the chat completion endpoint.

Supported request behavior:

• messages is required and must be a non-empty list.
• model is optional and defaults to public_model.
• temperature defaults to 0.2 and must be between 0 and 2.
• max_tokens and max_completion_tokens are supported; only n=1 is supported.
• stream: true returns a server-sent event response containing the completed answer.

The gateway returns standard chat completion fields, including id, model, choices, and aggregated token usage from the reasoning model, workers, and verification calls.

Project layout

File	Responsibility
src/server.py	FastAPI application, authentication, validation, and responses.
src/orchestrator.py	Planning, task execution, verification, retries, and merging.
src/router.py	Capability matching, load tracking, and worker cooldowns.
src/prompts.py	Planner, worker, verifier, and merge prompts.
src/verification.py	Verification decision parsing and validation.
src/utils.py	Configuration, upstream HTTP calls, shared types, and parsers.
config.example.json	Safe starting point for a local configuration.
Dockerfile	Container image definition.

Documentation

• orchestration.md describes the request lifecycle.
• routing.md describes worker selection and failure backoff.
• verification.md describes verification and repair.
• models-chosen.md records the current homelab model layout.
• CONTRIBUTING.md explains the contribution workflow.
• SECURITY.md explains how to report vulnerabilities.

Security

This service forwards user messages to configured model endpoints. Run it behind a firewall or reverse proxy, configure ORCHESTRATOR_API_KEY, and use TLS when it is reachable from an untrusted network. Treat config.json as private configuration and never commit credentials.

Worker output is treated as untrusted data during verification, but upstream models and network endpoints remain part of your trusted deployment boundary. See SECURITY.md for reporting guidance.

Contributing

Contributions are welcome. Please read CONTRIBUTING.md and follow the project CODE_OF_CONDUCT.md.

License

MIT