1Cat-vLLM 1.3.0

1Cat-vLLM 1.3.0

1Cat-vLLM 1.3.0 is a V100/SM70 release centered on long-context Qwen prefill.

The key improvement is not a chunk-size adjustment or simply enabling CUDA Graphs. This release rebuilds the dataflow of D=256 causal long-prefill Attention on V100.

D=256 long-prefill Attention: 17.92 -> 46.63-47.1 TFLOP/s

For representative Qwen long-context causal-Attention shapes, effective Attention compute per V100 and per tensor-parallel rank increased from 17.92 TFLOP/s on the previous production path to 46.63-47.1 TFLOP/s with the new D=256 path: approximately 2.6x.

Metric	Previous production path	1Cat-vLLM 1.3.0
Effective causal Attention compute	17.92 TFLOP/s	46.63-47.1 TFLOP/s

This is an Attention-operator result, not a claim of 2.6x whole-model throughput. It counts useful causal QK and PV FLOPs only. The retained before/after captures use representative nearby long-context Q/K dimensions.

What changed

• Native D256 Split-D N32 kernel. D=256 is split into four D64 slices. Paired warps share the QK probability result, so PV work scales without recomputing QK.
• Rebuilt Volta operand flow. QK/TN and PV/TT use layouts tailored to their Tensor Core operations. A phase-swizzled K layout plus vectorized shared-memory stores and loads reduces replay and keeps the Tensor Core pipeline supplied.
• Software pipeline for V100. The next K tile is prefetched while the current tile is computed, and PV operands are double-buffered to overlap memory movement with HMMA execution.
• Numerical contract preserved. The path retains causal online softmax and FP32 output accumulation; exact-dense A/B validation retains matching output hashes.
• E5M2 FP8 KV bridge. FP8 KV pages are dequantized once into reusable FP16 workspace before entering the high-throughput D256 engine, avoiding generic per-element FP8 overhead in the hot attention path.

End-to-end release curve

The supplied image is the public end-to-end benchmark for this release. It is a continuous API-side Qwen3.8-27B-FP8 sweep from 1,024 to 246,784 prompt tokens, not a separate historical A/B benchmark.

• Prefill begins at approximately 2.82K tok/s at 1,024 tokens, peaks at approximately 3.95K tok/s around 21.5K tokens, and declines smoothly to approximately 2.02K tok/s at 246,784 tokens.
• Decode is approximately 63 tok/s at short context, remains around 52 tok/s through the mid-long range, and is approximately 42-43 tok/s at 246,784 tokens.

The figure is deliberately included as the complete source of the public curve. These values replace the previous, mismatched FP16-KV exact-dense A/B table.

Also included

• E5M2 FP8 KV-cache routing and long-context decode refinements.
• Bundled Flash-V100, paged-KV utilities, TurboMind SM70 kernels, and FlashQLA SM70 native payload in the wheel.
• Release quality and long-context boundary validation, including the 256K context limit.

Wheel and build target

• Asset: 1cat_vllm-1.3.0-cp312-cp312-linux_x86_64.whl
• SHA256: 2bdb14a9c44f83ee6a766d88ed0d85b11390d6f5d65747e8dbe80a8e2d5d63e0
• Build target: Python 3.12, CUDA 12.8, Torch 2.10, SM70/V100.