Let's Infer

Local AI, installed like software.

Let's Infer turns local AI hardware into one reliable, OpenAI-compatible inference service. Tell it which model you want. It detects your hardware, selects the fastest qualified runtime, downloads the exact model and engine, starts the service, and keeps it running.

curl -fsSL https://letsinfer.ai/install.sh | sh

No engine hunting. No model-file plumbing. No hardware-specific install path.

• One-command installs — resolve the exact model, runtime, engine, adapter, and dependencies from a model name.
• Hardware-aware speed — signed benchmarks select the fastest qualified runtime for your target; --runtime pins an exact candidate.
• Automatic lifecycle — start immediately, wait for API readiness, restore after reboot, and recover ordinary engine failures.
• One API, every engine — keep the same OpenAI-compatible endpoint while Let's Infer handles concurrency, backpressure, and memory-aware queueing.
• Replication across your hardware — run one model on compatible main and child nodes, with target-specific runtimes behind one load-balanced API.
• Runtime-owned parallelism — qualify exact multi-GPU or multi-node runtimes while core handles generic allocation, lifecycle, recovery, and one endpoint per complete group.
• Live observability — watch requests, throughput, context, cache, utilization, temperatures, power, network, and lifecycle in one command.
• Built-in protection — Watchdog tracks the exact engine process, unified memory, PSI, swap, cgroup events, and crashes without hiding safety trips.
• Verifiable supply chain — signed catalogs, exact Hugging Face revisions, digest-pinned OCI images, deterministic packs, and bound benchmark evidence.
• Independent updates — update core, upgrade a runtime, or roll back one without silently changing the others.
• Reproducible benchmarks — durable, isolated code-and-prose runs capture TTFT, throughput, cache state, hardware telemetry, and validated JSON.
• Community-qualified runtimes — independent users can verify an exact runtime PR; signed full evidence, transparent consensus, and verifier identities determine qualification without granting upload credentials.
• Secure by default — scoped API keys, audit records, mDNS discovery, private controller mTLS, and optional inference-only public exposure.

Explore every feature →

Install Let's Infer and initialize your local node:

curl -fsSL https://letsinfer.ai/install.sh | sh

Install a model:

letsinfer install MODEL

Create an API key and inspect the live service:

letsinfer key create my-app
letsinfer status

Your stable local endpoint is:

http://<hostname>.local:8000/v1

Use it like any OpenAI-compatible API:

curl http://<hostname>.local:8000/v1/chat/completions \
  -H "Authorization: Bearer $LETSINFER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-27b",
    "messages": [{"role": "user", "content": "Why is the sky blue?"}]
  }'

NVIDIA DGX Spark is the first qualified target. The signed production catalog currently includes qualified Qwen3.8 27B NVFP4 and DeepSeek V4 Flash runtimes. New models, quantizations, engines, kernels, and hardware targets ship as independent runtime candidates—without adding model-specific code to core.

Core and runtime updates are deliberately independent. A catalog change never silently moves a running model.

model name + verified node topology
              │
              ▼
       signed runtime catalog
              │
              ▼
 exact model + runtime pack + Engine OCI
              │
              ▼
 OpenAI-compatible gateway + Watchdog

Each immutable runtime binds an exact model, Engine OCI, hardware target, serving recipe, optional optimizations, and benchmark evidence. Core stays model- and engine-agnostic.

• Features
• Installation
• CLI reference
• Nodes, replication, and trust
• Runtime development
• Updates and rollback
• Watchdog
• Benchmark framework
• macOS controller

Install the official, portable Let’s Infer agent skills with:

npx skills add letsinferlabs/skills

Runtime authors use letsinfer-runtime-authoring, radical or new Engine work uses letsinfer-engine-authoring, and qualification uses letsinfer-benchmark. Run the core tests with:

python3 -m tools.core_regression

The same complete suite and native Watchdog tests are required on every core pull request. See Testing and pull-request gates.

Let's Infer is licensed under AGPL-3.0-only.