vLLM Mach 0.1.0a10

v0.1.0a10 ports Mach's checkpoint-hybrid runtime to vLLM 0.29.0 and patched ExLlamaV3 1.4.9. It retains the a9 execution paths, including Temporal M24, fused communication and prefill, opt-in FP16 recurrent state and BA overlap. PyTorch remains at 2.13.0 and FlashInfer at 0.6.18.

The port fixes vLLM 0.29's cached plain-weight embedding dispatch when Mach installs online EXL3 embedding, moves MXFP6 workspace planning before memory profiling, and includes the local FlashInfer IPC allocation patch previously carried by the validated image. The GDN overlay preserves the new upstream behavior while retaining Mach's guarded optimized paths.

On the same dual RTX 5090 pair, Qwen3.8-27B K5/K6 TP2 throughput at 3000 input / 1000 output tokens remained within 0.2% of a9 at c4/c16/c24/c32. Each concurrency ran twice per version; all 640 measured requests completed. This dependency upgrade preserves performance rather than adding a new speedup.

Validation includes 206 passing CPU tests, 11 native BF16 tests, four upstream sliced-MGEMM cases, 52 real-checkpoint eager/Graph cases, 40/40 service task checks and 2,592 serial-versus-overlapped state/output byte comparisons. GPU validation used the a10 development build; the release changes only version metadata and documentation.

Use the attached source archive for the matching runtime patches and native build instructions. Rebuild the ExLlamaV3 BF16, M32 and Temporal extensions against 1.4.9. The Python wheel does not install native extensions or runtime patches. Existing a9 environments can stay on vLLM 0.28. This is a GitHub prerelease, with no PyPI publication.

See the installation profile and regression results.

vLLM Mach 0.1.0a9

v0.1.0a9 adds optional FP16 recurrent state and M16/M24 BA overlap to the Qwen3.8-27B K5/K6 checkpoint profile. Base defaults and dependency pins are unchanged.

The new profile keeps convolution state and attention KV in BF16, uses EXL3 QKV at M16/M24, and retains merged MXFP6 QKV at M32. Enable it with qwen38-checkpoint-fp16-ssm.env and --mamba-ssm-cache-dtype float16. FP16 recurrent state changes numerical precision.

The README now includes throughput, numerical fidelity and combined tradeoff plots for six configurations, with raw data and verification scripts. On the 3k/1k source-stack comparison, K5/K6 averages 35.6% more output throughput than official vLLM 0.29 FP8 across c4/c16/c24/c32. Its gold-token logprob MAE is 46.2% lower than the tested locally calibrated NVFP4, while retaining 88.8% to 97.4% of its throughput. These curves describe the optimization source stack; Mach integration acceptance is reported separately.

The port passed 169 CPU tests and TP2 service acceptance: 2,592 output/conv/SSM byte comparisons between serial and overlapping execution, 40/40 task checks, and 96 requests at 3000 input / 128 output tokens. Exact stored-reference text agreement was 37/40.

Install the wheel and reapply the matching GDN overlay from the attached source archive. The overlay installer supports upgrading the a8 overlay. Existing native dependencies and runtime patches are still required; the Python wheel does not install them. This is a GitHub prerelease, with no PyPI publication.

See configuration and validation and benchmark details.

v0.1.0a8

v0.1.0a8

Extend the existing Qwen3.8-27B EXL3 K5/K6 checkpoint profile with 32 observed prefill shapes and complete Mach GPU acceptance of M32 BA overlap. Dependency versions and base defaults are unchanged.

• Add an opt-in lossless BF16 collective for 32 prefill row counts at H5120, TP2 and SM120. The separate M4096 direct SUM path is retained. Native package 0.1.0a4 builds all four collective modules.
• Add qwen38-checkpoint-long.env, which enables the accepted prefill paths and BA32 with diagnostics disabled. Temporal M24 and existing weight loading are unchanged.
• Add optional Graph-replayed serial-reference assertions for BA/QKV verification. Rejected BA24 and attention-compaction experiments are excluded.

Validation: 133 package tests passed, along with 384 synthetic boundary cases and 64 changing-input Graph checks. The diagnostic service verified 8192 real-model residual/norm boundaries bitwise against installed FlashInfer and checked BA/QKV producers at 48 layers per rank during replay. A separate verifier-off service passed 40/40 tasks and completed all 640 fixed-token requests across short c32 and long c16/c24/c32. Exact stored-reference text agreement was 38/40; the existing batch-dependent routing remains. This is integration acceptance, not full-model bitwise equivalence or a new paired performance result.

Upgrade the Python wheel, native collective build, GDN overlay and caller patches together. The wheel alone does not install native dependencies or apply patches. The attached full source archive contains the matching profiles and build instructions; native dependency wheels are not included. This is a GitHub prerelease, not a PyPI publication.

See installation and validation details.

vLLM Mach 0.1.0a7

Optional lossless BF16 prefill collectives and experimental M32 BA/QKV stream overlap for the existing vLLM 0.28.0 / FlashInfer 0.6.18 profile. Dependencies and default serving behavior are unchanged.

• Add input-only, input+SUM and direct SUM collective paths for TP2 SM120, BF16, M4096×H5120. Each restores the tested BF16 communication values exactly; direct SUM also avoids the producing rank's local SUM write and reread. Native package 0.1.0a3 builds all three modes with CUDA 13.0.
• Add experimental BA/QKV branch overlap for the checkpoint profile's physical M32 packed-decode fallback. It uses Mach's active merged MXFP6 route and joins the auxiliary stream before recurrent decode. VLLM_MACH_BA_OVERLAP defaults to off.
• Allow the GDN installer to upgrade the known previous Mach overlay while continuing to reject unrelated edits.

Validation: 131 package tests passed. Wheel construction, overlay installation and repeated-install checks passed. Before the BA increment, each collective port passed double-GPU boundary, changing-input Graph, alias and mixed-workspace checks. The direct SUM service passed 40/40 tasks with zero pass/fail regressions; each rank verified 128 full residual/norm instances bitwise against installed FlashInfer, and all 256 c32 fixed-token requests completed.

BA overlap has not completed GPU/Graph/service acceptance on Mach. Leave it disabled for a validated deployment. Its source experiment's performance and tensor checks are supporting evidence, not acceptance of this port. The combined a7 stack has not had a fresh GPU service run.

No new Mach end-to-end speedup is claimed. Historical codec and overlap gains were measured on a separate 1024-input/256-output runtime and must not be added together or extrapolated to 3000/1000. The model's existing batch-dependent EXL3/MXFP6 routing remains; lossless collective checks do not establish full-model bitwise or business-quality equivalence.

The wheel does not install native extensions or apply dependency patches. Use the attached source archive for native builds and the matching runtime profiles. All new features are opt-in. The Python wheel, full source archive and SHA-256 checksums are attached; native dependency wheels are not included. This is a GitHub prerelease, not a PyPI publication.

Collective installation · Collective validation · BA overlap and limitations

v0.1.0a6

Complete checkpoint/Temporal runtime profile for Qwen3.8-27B Dense EXL3 K5/K6 with paired original MXFP6 weights, vLLM 0.28.0 and SM120 TP2.

• Add a version-locked FlashInfer 0.6.18 specialized GDN overlay with a hash-checked, idempotent installer. The framework gate defaults to off.
• Add SM120 TP2 AllReduce capacity settings and a vLLM 0.28 backport of fused QK norm/MRoPE from upstream PR #52676.
• Provide one opt-in environment file covering checkpoint loading, BF16, Temporal M24, B12X, fused reconstruction, K6 embedding, sampling and fused collectives. Base provider defaults are unchanged.
• Fix the package's reported version to match its wheel metadata.

Validation: 122 package tests passed; the release wheel passed installation, version and no-CUDA-initialization import checks. The aligned TP2 service passed 40/40 tasks with zero pass/fail regressions and 37/40 exact stored-reference outputs. Both workers used specialized GDN; both Temporal bundle shapes and seven CUDA Graph sizes were observed.

On the fixed-token 1024-input/256-output workload, the aligned configuration recovered the earlier deployment gap. After a full 128-request c16 warmup, two measured repetitions averaged 983.46 output tokens/s for Mach and 979.93 for the frozen reference. The +0.36% difference is not an established speedup. Mach's first full round still showed extra latency; it did not recur in the warmed repetitions. These checks do not establish broad model quality, bitwise equivalence or 3k/1k performance.

Upgrading the wheel alone does not enable the aligned configuration. The attached source archive contains runtime patches, the GDN installer, the environment file and native build sources. Build native dependencies in the serving environment and apply the documented profiles before starting vLLM. Prebuilt FlashInfer communication modules that bypass the MXFP8 patch remain unsupported.

Complete installation profile · Alignment results · Validation

The Python wheel, full source archive and SHA-256 checksums are attached. Native dependency wheels are not included. This is a GitHub prerelease, not a PyPI publication.

vLLM Mach 0.1.0a5

Original MXFP6 checkpoint loading and optional Temporal M24 execution for the Qwen3.8-27B Dense, vLLM 0.28, SM120 TP2 profile.

• Load packed MXFP6 weights without requantizing EXL3 weights; merge QKV/QKVZ at physical M32 and prefill M>=128.
• Add a separately built Temporal M24 K6 kernel for the two validated QKV/QKVZ bundle shapes. Enable with EXL3_TEMPORAL_QKV_M24=1 alongside BF16 I/O and M24 support. Defaults to off; other shapes keep their existing routes.
• Reject prebuilt FlashInfer communication modules that bypass the MXFP8 layout patch. Use the matching source-build profile.
• Include the ExLlamaV3 1.4.8 BF16 source profile. Default dependency pins remain unchanged.

Validation: 115 package tests passed, 3 skipped. The M24 port matched the frozen Temporal implementation bitwise in real-weight eager and changing-input Graph checks on both ranks. The combined TP2 service captured seven Graph sizes and passed 40/40 tasks at concurrency 24, with zero regressions and 33/40 exact reference outputs.

Temporal changes the EXL3 reduction order. This release does not claim bitwise equivalence to the older EXL3 kernel, full-model losslessness, or a new E2E performance gain. Private FLA tuning is not included.

Configuration · Temporal build · FlashInfer profile · Validation

The Python wheel and SHA-256 checksum are attached. Build native dependencies against the serving environment; their wheels are not included.

vLLM Mach 0.1.0a4

Compatibility fixes for the public EXL3 installation path.

• Pass CPU Hadamard group IDs to the public ExLlamaV3 BF16 API while retaining GPU IDs for the independent M32 module.
• Add EXL3_BF16_IO_LEGACY_CUDA_GROUP_IDS=1 for older experimental BF16 wheels.
• Keep B12X optional when its route was not explicitly selected.

Upgrade from 0.1.0a3 before using public ExLlamaV3 commit d0094bc. Build native dependencies against the serving environment.

Validation: 68 package tests passed, 3 skipped. Public ExLlamaV3 and M32 source builds succeeded. A Qwen3.8-27B TP2 service without B12X captured all seven configured graph sizes and passed 40/40 tasks with no new failures; exact output agreement was 39/40. This is a functional compatibility check, not a performance gain or validation of a public MXFP6/FlashInfer dependency rebuild.

Installation · Validation

The wheel and SHA-256 checksum are attached. Native dependency wheels are not included.

vLLM Mach 0.1.0a3

Alpha preview.

• Add opt-in BF16 grouped QKV/QKVZ decode for M24 and M32 using M16-sized chunks.
• Add a separately built SM120 M32 extension, with public source dependencies and a split-path fallback when the module is absent.
• Add an optional vLLM 0.28.0 patch that stages Gumbel sampling metadata on the device.
• Document build requirements, numerical differences, and validation coverage.

Existing configurations keep their previous dispatch. The new paths require explicit configuration; see experimental decode paths. The Python wheel does not bundle the optional native extension or apply runtime patches.

Validation: 60 package tests passed, 3 skipped; real-weight operator and changing-input Graph checks passed; the combined EXL3/MXFP6 TP2 service passed 40/40 regression tasks. These results are not a performance benchmark or a claim of full-model bitwise equivalence.

vLLM Mach 0.1.0a2

This alpha adds an opt-in FlashInfer TP2 path that combines AllReduce, residual addition, Gemma RMSNorm, and packed MXFP8 quantization for the selective Qwen3.8-27B EXL3/MXFP6 profile. The packed activation is consumed directly by the MXFP6 gate/up projection.

The release includes version-locked vLLM 0.28 and FlashInfer runtime patches, fail-closed header validation, updated compatibility and validation documentation, and the project logo in the README. The fused collective remains disabled by default.

Validation completed in a clean vLLM 0.28 container with 34 tests passing and 3 optional tests skipped. The release wheel also completed a TP2 service smoke with all configured FULL_DECODE_ONLY graph sizes captured and a successful completion request.

Wheel SHA256: 3254b6a83aa4f03ea98f116a9e1536f061ee1145697676121d64d400a7f9d1f0.

vLLM Mach 0.1.0a1

Initial alpha release of vLLM Mach.

This release provides an out-of-tree EXL3 provider for vLLM 0.28.0, an optional SM120 MXFP6 Dense bridge, and an opt-in Qwen3.8-27B TP2 profile that can route selected EXL3 projections through MXFP6. The Dense hybrid MLP path includes fused SiLU/mul and packed MXFP8 activation quantization before the MXFP6 down projection.

The tested configuration is Qwen3.8-27B Dense with K5/K6 EXL3 weights, TP2/PP1, and SM120. Native operation-boundary checks, the Python test suite, model loading, FULL_DECODE_ONLY CUDA Graph capture, and a short serving request passed with the release wheel. Other models, parallel layouts, vLLM versions, GPUs, routed MoE, and GDN kernel replacement remain unverified. No end-to-end performance claim is attached to this alpha release.

Wheel SHA256: 628833b56aaef9fe02c0289b7aa42cdf17d489417216613f0aef40f2a663ae59