OmniInfer

Easy, fast, and private LLM & VLM inference for every device

| Demo | Getting Started | About | Documentation | Architecture |

OmniInfer includes a terminal UI for selecting backends, loading models, and chatting with local models.

Linux x64 CLI:

curl -fsSL https://raw.githubusercontent.com/omnimind-ai/OmniInfer/main/scripts/install.sh | bash

Install a specific release:

curl -fsSL https://raw.githubusercontent.com/omnimind-ai/OmniInfer/main/scripts/install.sh | bash -s -- --version v0.3.5

The lightweight installer downloads the CLI-only GitHub Release archive, verifies checksums.txt, and installs omniinfer into ~/.local/bin by default. It does not clone this repository, install backend runtimes, download models, or use sudo.

Install a prebuilt runtime after the CLI is available:

omniinfer backend list
omniinfer backend install llama.cpp-linux

Desktop integrations can add --state-root <path> and --runtime-root <path> to isolate managed files, and backend install ... --json emits streaming JSONL progress. See CLI Usage for the stable integration contract.

On Windows, official vLLM is available through the managed vllm-wsl2-cuda and vllm-wsl2-rocm backends. They run pinned upstream Linux wheels inside a user WSL2 distribution for NVIDIA CUDA or supported AMD Ryzen AI GPUs; vLLM does not support native Windows. See the Windows vLLM setup.

You can also run omniinfer with no arguments to open the TUI; when a compatible backend is missing, the TUI can install the prebuilt runtime before model loading.

macOS arm64 and Windows x64 CLI-only archives are available from GitHub Releases. Homebrew, Scoop, npm, and platform-native one-line installers are planned.

omniinfer serve --cloudflare automatically downloads and verifies a pinned cloudflared helper when neither a managed copy nor a system installation is available. On macOS, the manual fallback is brew install cloudflared; retry the command afterward or pass --cloudflared-path "$(command -v cloudflared)".

Use the source installer when you want a repository checkout plus backend runtime setup, source builds, and optional model setup.

Linux and macOS:

curl -fsSL https://raw.githubusercontent.com/omnimind-ai/OmniInfer/main/scripts/install-from-source.sh | bash

Windows PowerShell:

irm "https://raw.githubusercontent.com/omnimind-ai/OmniInfer/main/scripts/install.ps1?$(Get-Random)" | iex

The source installer detects your platform and hardware, recommends a backend, and walks you through model setup interactively. Use --model /path/to/model.gguf for explicit model setup or --no-model / -NoModel to skip model setup without prompting. Install summaries are written to .local/install-summary.json; source builds also save logs under tmp/test_results/install/.

If you already cloned this repository, build at least one local runtime backend first.

• Windows: see Build Guide: Windows
• Linux: see Build Guide: Linux
• macOS: see Build Guide: macOS
• Android: see Build Guide: Android

After the runtime is ready, start with the OmniInfer CLI from the repository root.

Linux and macOS:

./omniinfer --help

Windows:

.\omniinfer.ps1 --help

Android:

./omniinfer --help

OmniInfer is a high-performance, cross-platform inference engine for running Large Language Models (LLM) and Vision-Language Models (VLM) locally. It abstracts away model compilation, hardware adaptation, and deployment complexity, enabling efficient local inference with minimal configuration.

OmniInfer powers the inference layer of Omni Studio, a unified model orchestration platform.

OmniInfer is fast with:

• Optimized token generation speed and minimal memory footprint
• Multiple backend engines, including llama.cpp, ik_llama.cpp, MNN, MLX, TurboQuant, LiteRT-LM, ExecuTorch QNN, and OmniInfer Native where supported
• Hardware-aware adaptation and optimization

OmniInfer is flexible and easy to use with:

• Seamless multi-backend switching for the best available engine on each device
• OpenAI-compatible and Anthropic-compatible local API endpoints
• Support for text and vision-language workloads
• Fine-grained parameter control for context length, GPU offloading, KV cache, and backend-native launch options

OmniInfer runs everywhere:

• Linux, macOS, Windows — desktop and server
• Android and iOS — mobile and edge devices
• One codebase across CLI, HTTP gateway, and mobile modules

Recommended docs:

• CLI Guide: end-to-end CLI usage for Linux, macOS, Windows, and Android
• Android App Integration: embed OmniInfer in a third-party Android app
• Android Backend Reference: Android backend options for llama.cpp, MNN, LiteRT-LM, and ExecuTorch QNN
• Android Smoke Tests: adb/curl checks and source-build validation
• Android Troubleshooting: common build, runtime, and backend failures
• Build Guide: build and platform packaging notes
• API Reference: OpenAI-compatible local API usage

If you use OmniInfer in research, please cite this repository. GitHub can automatically generate citation formats from CITATION.cff.

@software{omniinfer,
  author = {{Omnimind AI}},
  title = {OmniInfer},
  url = {https://github.com/omnimind-ai/OmniInfer}
}

We welcome and value any contributions and collaborations. Please check out Contributing to OmniInfer for how to get involved.

This project is licensed under the Apache License 2.0 — see LICENSE for details.