Skip to content

v0.10.1.dev65

@garrett361 garrett361 tagged this 21 Oct 10:59
This PR keeps DeepSeek-V4's mHC write-back gates (`comb` and `post`) in
fp32 instead of rounding them to bf16 before the fused kernel, which
matches how vLLM computes the same step at inference.

- vLLM 0.32's
[`mhc_post_torch`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/model_executor/kernels/mhc/torch.py#L135-L147)
takes `comb` and `post` as fp32 and accumulates in fp32. It rounds once,
when it writes the bf16 result.
- The Triton kernel already computed in fp32, so the early bf16 cast of
the two gates was the only extra rounding.
- `update_states` now casts the gates with `.float()`, and the kernel
output and its fake follow the residual's dtype.
- One gap remains: vLLM keeps the mHC `fn`, `base` and `scale` weights
in fp32, but `fully_shard` gathers them in bf16 here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Assets 2
Loading