There is a chasm between being an LLM consumer and being an enterprise "LLM puppeteer." Enterprises have the talent and capital to stitch together game-changing AI services; small businesses and individual developers simply don't.
Ozwald is an effort to bridge that void and unlock the multiverse of opportunity that AI-enabled services represent for smaller teams/budgets.
Right now, Ozwald is a provisioner for container-based systems, with a primary focus on making the provisioner useful for my own AI experiments first.
The goal is to move the project toward closing the gap between the casual LLM user and the enterprise-scale orchestrator. It's about making it possible to give a machine a well-defined task in a narrow domain and having it simply "do the thing."
Ozwald handles the real-world friction of provisioning LLM containers across mixed hardware (different GPUs or CPU-only) with varying runtime parameters. It focuses on the glue: a small, well-typed config, a provisioner API, and a CLI to make starting and stopping services predictable.
- Varieties and Profiles: A clear model for describing services across different hardware (nvidia, amdgpu, cpu-only) and parameter sets that specify model type, context window size, etc.
- Secrets Management: Securely store and inject encrypted environment
variables into services using decryption tokens and the Docker
env-filemechanism. - CLI and API: A simple CLI tool for iterating on service configurations and an API for changing the set of provisioned services.
Ozwald provides secure, isolated, and versioned storage for services through encrypted host-mounted volumes.
Volumes are defined at the realm level in your settings.yml:
realms:
my-realm:
volumes:
- name: scratch-space
type: tmp-writeable
source: scratch
- name: database-state
type: versioned-read-only
source: my-db
service-definitions:
- name: my-service
type: container
volumes:
- scratch-space:/tmp/scratch
- database-state:/data/db:roThe provisioner uses the following environment variables for managing encrypted storage:
OZWALD_ENCRYPTED_VOLUME_FILE: Path to the file that will be mounted as the encrypted volume containing read-write volumes.OZWALD_SYSTEM_KEY: The system provisioner token used for encryption of temporary storage.
Update your provisioner definition in settings.yml to include the
encrypted_storage_dir:
provisioners:
- name: local
host: localhost
encrypted_storage_dir: /mnt/ozwald-storage- Isolation: Write-access directories for containers are strictly isolated and not shared between services.
- Automatic Cleanup: Ozwald automatically clears any leftover
temporary volumes in the
encrypted_storage_diron provisioner startup and shutdown. - Persistence: Containers can request the provisioner to persist a
named
tmp-writeablevolume as a new encrypted versioned volume via the API.
Ozwald requires permissions to perform mount operations on the host. Ensure
the user running the provisioner has the necessary sudo rights or
CAP_SYS_ADMIN capabilities for mount and umount operations.
Calling raw docker commands in a bash script got to be tough to maintain after a time or two. Docker compose ended up being the sort of thing where I would need a large yaml file to configure a ton of very similar services or I would need a different yaml file for every configuration I might want to use.
Ozwald uses a similar configuration language to that of Docker Compose, but it's designed to be more expressive for defining AI-related services that perform predictably across mixed hardware.
Sometimes calling an external API isn't an option for cost, privacy, or lack of connectivity.
Ozwald is designed to be a standalone service that can run local LLMs and/or other containers in a predictable way that takes into account the available hardware.
With Ozwald, you will be able to build your own LLM-puppeteer apps in a way that can accommodate the variety of hardware and runtime parameters developers are asked to support.
Install Ozwald and/or add ozwald to your project's Python dependencies:
pip install ozwaldOzwald can used as an API by another project (like an orchestrator service) or it can be invoked directly by end users.
- Provide a settings file
Create a YAML configuration (settings.yml) that declares hosts,
provisioners, and services:
---
hosts:
- name: localhost
ip: 127.0.0.1
provisioners:
- name: local
host: localhost
cache:
type: redis
parameters:
host: ozwald-provisioner-redis
port: 6379
db: 0
realms:
default:
service-definitions:
- name: qwen1.5-vllm
type: container
description: DeepSeek Qwen 1.5B via vLLM
varieties:
nvidia:
image: openai-api-vllm.nvidia
environment:
GPU: true
amdgpu:
image: openai-api-vllm.amdgpu
environment:
GPU: true
cpu-only:
image: openai-api-vllm.cpu-only
environment:
GPU: false
environment:
MODEL_NAME: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
profiles:
no-gpu:
environment:
MAX_MODEL_LEN: 45000
fast-gpu:
environment:
GPU_MEMORY_UTILIZATION: 0.9
CPU_OFFLOAD_GB: ""
bridge-connector:
port: 8000
name: api-connector
portals:
- name: public-api
port: 7656
bridge:
realm: default
connector: api-connector- Set the mandatory environment variables
The OZWALD_SYSTEM_KEY environment variable is required for authentication
between the CLI and the Provisioner API.
The OZWALD_PROVISIONER environment variable is required for the CLI to know
which provisioner to use (usually the name of the host or "localhost").
The OZWALD_FOOTPRINT_DATA environment variable is required for the CLI to
know where to store footprint data.
export OZWALD_SYSTEM_KEY="your-long-random-token"
export OZWALD_PROVISIONER="daneel"
export OZWALD_FOOTPRINT_DATA="/tmp/footprint.yml"- Start the provisioner
Use the CLI to spin up the provisioner network and containers:
ozwald start_provisioner
ozwald footprint_services my_service[some_profile][some_variety]
ozwald status- Call the API
Query configured or active services:
curl -H "Authorization: Bearer $OZWALD_SYSTEM_KEY" \
http://127.0.0.1:8000/srv/services/configured/hosts[]: Named machines and their IPs.provisioners[]: Defines where the provisioner runs and its state cache (Redis).realms: Groups fornetworks,service-definitions, andvault.service-definitions[]: Descriptions of services, including hardwarevarieties, runtimeprofiles,lockersrequirements, andbridge-connectorsettings.portals[]: Mapping host ports to bridge connectors in specific realms.
Ozwald provides a secure way to manage environment variables (secrets) using Redis-backed storage and user-provided decryption tokens.
- Define Vault and Lockers in Config
Add a vault section to your realm and specify which lockers a service requires:
realms:
default:
vault:
lockers:
api-keys: {}
service-definitions:
- name: my-service
lockers:
- api-keys- Store Secrets via CLI
Use the ozwald secrets set command to encrypt and store secrets in Redis:
# Create a JSON file with your secrets
echo '{"OPENAI_API_KEY": "sk-..."}' > secrets.json
# Store it in the 'api-keys' locker using a token
ozwald secrets set default api-keys --token my-secret-token --file secrets.json- Inject Secrets at Runtime
When updating active services, provide the decryption tokens for the required lockers:
ozwald update_dynamic_services --token api-keys=my-secret-token "my-service[default][nvidia]"Secrets are decrypted at runtime, written to a temporary env-file on the host,
and injected into the container. The temporary file is deleted immediately after
the container starts.
The ozwald command handles local development and inspection:
start_provisioner: Start local provisioner network and containers.stop_provisioner: Stop provisioner containers.status: Check provisioner health.list_active_services: See what's currently running.list_configured_services: See list of configured services.show_host_resources: Inspect CPU/RAM/GPU/VRAM.update_services: Update the desired set of services.footprint_services: Measure resource consumption of configured services.get_footprint_logs: Show footprint logs.get_service_launch_logs: Show service launch logs.get_service_logs: Show service logs.
Example:
ozwald start_provisioner
ozwald status
ozwald stop_provisionerAll non-health endpoints require a bearer token:
Authorization: Bearer <OZWALD_SYSTEM_KEY>
GET /health: Basic health check (no auth required).GET /srv/services/active/: List services currently active or transitioning.POST /srv/services/active/update/: Update the desired set of services.GET /srv/host/resources: Structured summary of CPU, RAM, GPU, and VRAM.POST /srv/services/footprint: Queue a footprinting request.
Example:
curl -H "Authorization: Bearer $OZWALD_SYSTEM_KEY" \
http://127.0.0.1:8000/srv/services/active/The provisioner is the first building block. The compass needle is pointing toward filling the void between LLM-consumer and LLM-enabled service:
- Multi-host Orchestration: Coordinating multiple provisioners.
- AI Pipelines: Composable services for ingest, chunking, and indexing.
- Declarative Ops: Dry-run planning and explainers for service changes.
Ozwald is open source, designed for adoption while keeping the core free.
- The Core (AGPLv3): The Ozwald engine, orchestrator, and provisioner.
- Your Apps (Apache 2.0): Client SDKs and public interfaces. Build your proprietary apps without fear of being forced to open source them.
- Contributing: Requires a signed CLA to ensure sustainability.
For commercial licensing or questions, contact fred@frameworklabs.us.
Author: Fred McDavid