DeepInfer: Detailed Introduction

[ikaijua]

DeepInfer: Detailed Introduction

DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. DeepSeek-AI open-sourced it on 2026-10-20 under Apache-2.0. The release provides offline inference plus the core features needed to serve models at scale.

• Flagship capability: serve DeepSeek models with high throughput and low latency, integrating DeepSeek's own optimized operators (FlashMLA, DeepGEMM, DeepEP, TileLang) and production-grade features such as DSpark confidence-scheduled speculative decoding, SWA bounded replay, async scheduling, batch-invariant inference, deterministic sampling, and structured outputs.
• Traction: 13k+ GitHub stars and 1.2k+ forks within five days of release; 133 open issues and active maintenance.
• Pricing: free, Apache-2.0, self-hostable.
• Caveat: this is the offline-inference first release — a full KV cache system and Prefill/Decode disaggregation are not yet packaged; only DeepSeek-V4.1-Flash is supported, and it requires NVIDIA SM100 (Blackwell) or Huawei Ascend 950 hardware.

Table of Contents

• What it is: positioning & origins
• Core capabilities
• Hardware & environment
• Quick start: build & offline inference
• Performance & benchmarks
• Current limits & roadmap
• Why it's worth including

What it is: positioning & origins

DeepInfer is the open-source edition of DeepSeek-AI's internal production inference engine — the same engine that drives all of DeepSeek's large-scale online inference (including the high-throughput, low-latency serving of DeepSeek-V4.1-Flash).

• Lineage: the official acknowledgments state DeepInfer was "originally built primarily on vLLM," with inspiration from SGLang and TensorRT-LLM. It is not built from scratch; it is a deeply customized vLLM-based production engine with DeepSeek-specific optimizations.
• Cadence: repo created 2026-10-16, first runnable build pushed 2026-10-20 alongside the announcement. As a companion, DeepSeek also maintains deepseek-recipe (Rust, MIT, 1.6k★), an open-source Token codec aligned with the official API's token handling.
• Scope: aimed at teams that want to run DeepSeek models at production-grade throughput/latency, not a general multi-model serving gateway.

Core capabilities

• Optimized kernels & runtime: model and sampler runtime integrating FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels.
• Parallelism: Attention DP / balanced CP, MoE EP + EPLB load balancing.
• DSpark confidence-scheduled speculative decoding: accelerate decoding with draft tokens and schedule verification by confidence (DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE sets max draft length, default 5).
• Decoder SWA bounded replay: replay only a bounded suffix to cut decoder prefill work.
• Asynchronous scheduling: overlap CPU batch scheduling with device execution (zero-overhead batch scheduler, on by default).
• Batch-invariant inference / deterministic sampling: identical inputs and sampling params reproduce identical outputs (NVIDIA only, DEEPINFER_BATCH_INVARIANT=1).
• Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar via so_args in SamplingParams.

Hardware & environment

• Supported hardware: NVIDIA SM100 (Blackwell) GPUs; Huawei Ascend 950 NPUs.
• Base environment (Linux): GCC ≥ 13.0, Python 3.12, CUDA 13 (NVIDIA) / CANN 9.2 (Ascend).
• Build: create and activate an empty Python 3.12 env, set export DEEPINFER_ACCELERATOR=nvidia (or ascend), then ./build.sh.
• Note: hardware bar is high — currently only SM100 / Ascend 950; ordinary consumer GPUs are not supported yet.

Quick start: build & offline inference

export DEEPINFER_ACCELERATOR=nvidia   # use ascend for Ascend
./build.sh
python examples/offline_inference.py   # first run auto-downloads DeepSeek-V4.1-Flash-1017 from HF

• Use a local checkpoint with DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017.
• Runnable examples for structured output, DSpark, and SWA replay live under examples/.

Performance & benchmarks

• Official benchmarks measured on 4 SM100 GPUs (prefill) and 64 SM100 GPUs (decode, within one scale-up domain), shown as throughput-vs-latency Pareto curves; reproduction via examples/sweep/run_sweep.sh.
• Evaluation benchmarks: see eval/README.md.
• As only SM100 / Ascend 950 are supported, broad multi-hardware comparisons are still limited; treat numbers as vendor-reported.

Current limits & roadmap

What this release does not yet include (per the official README):

• A KV cache system is not yet included;
• VE / Prefill / Decode disaggregation is not packaged (most required engine functionality is reportedly implemented, but not shipped in this release);
• Load balancing across data-parallel replicas for both prefill and decode is not provided.

Also: only DeepSeek-V4.1-Flash is supported, and only SM100 / Ascend 950 hardware. It is currently best for teams studying/reproducing DeepSeek's production inference stack; it is not yet an out-of-the-box multi-model production gateway. The repo is active, so these gaps are likely to close in later releases.

Why it's worth including

• Who it's for: teams self-hosting DeepSeek-V4.1-Flash at production scale; engineers studying inference engines, MoE parallelism, speculative decoding, and TileLang kernels; deployments with NVIDIA Blackwell or Huawei Ascend 950 hardware.
• Why in Awesome-AITools: it meets the four inclusion criteria — ① unique (a top-lab, real-traffic-validated production engine); ② broad (anyone deploying DeepSeek models at scale); ③ usable (open repo, build & offline-inference examples, clear capabilities and free Apache-2.0 pricing); ④ alive (13k★ in days, active issues). It sits in ### LLM Inference & Deployment, complementing DwarfStar (consumer single-card engine) and AirLLM (memory-saving lib) without overlap.
• Watch: still early (offline first release, single model, high-end hardware) — track later releases for the missing production-engine pieces.