Smart launcher for llama.cpp / ik_llama.cpp — auto-detects GPUs, optimizes MoE placement, crash recovery

MONAI Deploy aims to become the de-facto standard for developing, packaging, testing, deploying and running medical AI applications in clinical production.

Create, List, Update, Delete Amazon EKS clusters. Deploy and manage software on EKS. Run distributed model training and inference examples.

run ollama & gguf easily with a single command

[ARCHIVED 2026-04-20 — stampby retired; see bong-water-water-bong] no longer actively developed

Community benchmark database for running LLMs on Apple Silicon Macs

Edge Insights for Vision (eiv) is a package that helps to auto install Intel® GPU drivers and setup environment for Inference application development using OpenVINO™ toolkit

Docker image for a self-hosted Whisper speech-to-text server with an OpenAI-compatible transcription API. Powered by faster-whisper. Supports all Whisper models, JSON/SRT/VTT output, SSE streaming, offline mode, and multi-arch (amd64, arm64).

DFlash vLLM for DGX Spark — Plug & Play Block-Diffusion Speculative Decoding

The definitive Strix Halo LLM guide — 65 t/s on a $2,999 mini PC. Live benchmarks, tested optimizations, and everything that doesn't work.

Dockerized inference server and benchmarks for quantized Gemma 4 family on the NVIDIA DGX Spark (GB10). Features ARM64 CUDA 13 builds using llama.cpp.

Set up and run OpenVINO in Docker Ubuntu Environment on Intel CPU with Integrated Graphics

🚀 The Ultimate Curated List of LLMOps Tools, Frameworks, and Resources - A comprehensive collection of the best tools for Large Language Model Operations

The Private AI Setup Dream Guide for Demos automates the installation of the software needed for a local private AI setup, utilizing AI models (LLMs and diffusion models) for use cases such as general assistance, business ideas, coding, image generation, systems administration, marketing, planning, and more.

This project demonstrates a real-time AI "Meeting Coach" showcasing the use of Confluent Cloud for Apache Flink AI Inference functions to build a real-time Retrieval-Augmented Generation (RAG) pipeline. The demo uses both a static knowledge base of sales documents and real-time simulated meeting data.

Running Llama 3.1 8B and other LLMs on RK3588 NPU - benchmarks and setup guides

Self-hosted AI gateway. 82 models across 12 providers, free-tier-first routing with automatic fallback. Local CPU inference, transcription, and TTS. Models autonomously browse the web (stealth), run agentic Claude Code, and manage object storage via MCP. Expose publicly through Cloudflare Tunnel.

rust onnxruntime inference , yolo and other models

K3ai plugins Repo is the place where we host all the optional capabilites of k3ai. The main goal of the repo is to mantainer k3ai simple and lightweight while adding capabilites in the form of manifests or helm charts.

🦄 Distributed Inference on Kubernetes with DRA and MIG