InferX

A high-performance LLM inference server in C++20, aiming at throughput and latency comparable to vLLM and SGLang — without Python or libtorch in the serving path.

Status: M11 done. inferx-serve answers OpenAI-compatible completions over HTTP with SSE streaming, running Qwen2.5-3B-Instruct on a paged KV cache with FlashInfer attention, CUDA graphs, and on-device temperature/top-p sampling. Measured against vLLM 0.26 on the same box, same model, same client: bf16 matches it on decode (92.5 vs 95.5 tok/s at batch 1, within this box's run-to-run spread) and FP8 beats it by 1.37x (131.2 tok/s). The scheduler does continuous batching with chunked prefill, recompute preemption, and a radix prefix cache. The CPU is already off the critical path — 1.9% of a step — so M6's overlap pipeline was measured and deliberately not built.

M9's MoE path is now checkpoint-validated and serves gpt-oss-20b directly from MXFP4 weights. Its grouped MoE projections consume device-resident dispatch offsets, removing the per-layer host synchronization; fixed decode shapes now replay as CUDA graphs. Tensor-parallel arithmetic is validated through HostSim; the optional two-GPU NCCL runtime is implemented and awaits rented-hardware validation. Still open: a W4A16 GEMM that beats bf16 (M8), and wiring MLA into a served checkpoint (M9).

Why another engine

Python inference engines pay a structural tax that C++ does not, and InferX is shaped to collect the refund rather than to imitate their architecture:

• No GIL, so no process-per-rank. vLLM and SGLang run one OS process per tensor-parallel rank and serialize each step's batch metadata between them. InferX runs one process with one thread per rank, sharing the scheduler's output by pointer — no IPC, no serialization, no multi-process restart dance.
• No interpreter, so the CPU can run far ahead. Per-step scheduling costs microseconds instead of milliseconds, which keeps the GPU fed at large batch sizes.
• Real ownership semantics. KV blocks, request state, and weight buffers get RAII lifetimes and single-owner rules checkable at compile time.

Design

docs/ARCHITECTURE.md is the design of record — components, data flow, execution model, the decision register, and the risks. Start there.

The load-bearing split: the scheduler holds all the policy and never touches CUDA; the executor touches all the CUDA and holds no policy. That is what makes continuous batching, the paged KV manager, and the radix prefix cache testable on a machine with no GPU.

Getting started

git clone --recurse-submodules <url> && cd inferx
./scripts/install-cuda.sh     # needs sudo; CUDA 13.x is required
./scripts/bootstrap.sh        # submodules, configure, build, test

No GPU, or not upgraded yet:

./scripts/bootstrap.sh --host-only

See docs/DEVELOPMENT.md for toolchain details, conventions, and troubleshooting.

Roadmap

	Milestone	Status
M0	Toolchain, build system, core memory layer	done
M1	Quantized GEMM vs cuBLASLt FP16 baseline	done
M2	Safetensors loader + Llama forward pass	done
M3	Paged KV + FlashInfer attention + naive scheduler	done
M4	HTTP server, tokenizer, OpenAI-compatible streaming	done
M5	Continuous batching, chunked prefill, prefix cache, preemption	done
M6	Overlap pipeline + CUDA graphs for decode	done (graphs; overlap measured, not built)
M7	Tensor parallelism	NCCL runtime done; two-GPU validation pending
M8	W4A16 weights + FP8 KV cache	serving paths done; W4 tuning open
M9	MoE and MLA	layers done (TP=1, no checkpoint yet)
M10	Benchmarks vs vLLM/SGLang	done (vLLM; SGLang unmeasured)
M11	Serve gpt-oss-20b from MXFP4 weights	done

M1 deliberately preceded the model layer: torch-free quantized GEMM was the largest unknown in the design, and finding out it was intractable after building everything on top of it would have been expensive. It landed on FP8 e4m3 via cuBLASLt rather than CUTLASS — sm_89 implements FP8 matmul natively, and the row-major y = x·wᵀ mapping already produces the TN layout it requires. CUTLASS moves to M8/M9, where mixed-input W4A16 and grouped MoE GEMM have no alternative.

M4's original server used cpp-httplib but has since migrated to the planned Boost.Beast/Asio event-driven transport. Its tokenizer was delivered as our own byte-level BPE rather than FFI to the Rust tokenizers — asserted against a 140-string corpus generated by HF's own tokenizer, with exact id equality required. docs/ARCHITECTURE.md §14 and §15 carry the reasoning.

Measured

Against vLLM

Qwen2.5-3B-Instruct, RTX 4080 SUPER, vLLM 0.26.0, both engines driven by the same client over HTTP with matched settings (bf16, chunked prefill and prefix caching on, 8 concurrent sequences, greedy). Prompts are unique per request, so neither engine's prefix cache answers what the other computed.

	inferx bf16	inferx fp8	vLLM bf16
decode tok/s, batch 1	92.5	131.2	95.5
TTFT, batch 1	16.6 ms	16.5 ms	17.8 ms
output tok/s, 8 in flight	670	843	720
prefill tok/s, 2k prompt	10509	13973	12254

Decode tok/s moves ~6% run to run on this box for both engines, which is more than the bf16 gap — so bf16 is a tie on decode, a 1.2 ms win on TTFT, and a real 0.86× on prefill. FP8 is the actual result: 1.37× vLLM on decode, 1.17× on throughput under load.

Writing the benchmark found two defects nothing else could see: TCP_NODELAY was off, putting a flat ~43 ms Nagle floor under every TTFT, and FP8 activation quantization ran in one CUDA block, which cost prefill 3.9× under --quantization fp8. Both are fixed; ARCHITECTURE.md §14 has the diagnosis. ./scripts/bench_serve.sh reproduces the table.

The kernels underneath

SM clocks locked at 2400 MHz. Decode is batch 1 at context 256; the weight-bandwidth floor is quoted because the step is bound by it.

	launch-by-launch	with CUDA graphs	bandwidth floor	% of floor
bf16	11.58 ms	10.90 ms	8.39 ms	77%
FP8 e4m3	9.28 ms	8.29 ms	4.62 ms	56%

Prefill through FlashInfer's paged kernel holds above 9.2k tok/s from 128 to 16384 prompt tokens — 176 ms for 2k, 774 ms for 8k.

The M5 scheduler work is measured against the two things it trades between. Chunked prefill cuts p99 inter-token latency 10.6× (183 → 17 ms) when a 2k prompt arrives beside eight decoding sequences, at the cost of 1.6× on that prompt's own time-to-first-token. The prefix cache cuts warm TTFT 5.9× (97 → 16 ms) behind a shared system prompt, and holds multi-turn chat flat at ~15 ms where recomputing the conversation each turn climbs past 50 ms.

M8's FP8 KV cache halves KV memory: the same 2048-block pool is 1.21 GB in bf16, 0.60 GB in fp8 e4m3, so twice the concurrency (or twice the context per sequence) fits in the same VRAM. Decode tok/s is unchanged by it — at context 256 the step is weight-bandwidth-bound, not KV-bound, so the cache format does not move batch-1 latency (11.02 ms bf16 vs 11.13 ms fp8 per token, within run noise). Per-layer K/V scales freeze from warmup and the fp8 decode path is captured into the same CUDA graphs. (bench/decode_step_bench --fp8-kv.)

Numbers, method, and the re-measurement of every historical claim are in ARCHITECTURE.md §14.

Target hardware

Primary development target is a single sm_89 (Ada) GPU with 16 GB, which is why quantization is v1 scope rather than a later optimization. Tensor parallelism is designed in from the start but cannot be performance-tested on that box; see ARCHITECTURE.md §7.4 for how correctness is validated without multi-GPU hardware.