Production ready LLM model compression/quantization toolkit with hw accelerated inference support for both cpu/gpu via HF, vLLM, and SGLang.

基于SparkTTS模型，提供高质量中文语音合成与声音克隆服务。

☸️ Easy, advanced inference platform for large language models on Kubernetes. 🌟 Star to support our work!

AI-based search engine done right

A guide to structured generation using constrained decoding

Controllable Language Model Interactions in TypeScript

Kernel Library Wheel for SGLang

Experiments with LLMs in clouds (powered by SGLang)

DeepSeek-V3, R1 671B on 8xH100 Throughput Benchmarks

Reading notes on the open source code of AI infrastructure --- AI Infra 的开源代码阅读笔记 (LLM, DL, HPC, CUDA, SIMD...)

SgLang vs vLLM Comparison

Examples of serving LLM on Modal.

llmd is a LLMs daemonset, it provide model manager and get up and running large language models, it can use llama.cpp or vllm or sglang to running large language models.