Is Qwen3 Next 80B-A3B actually correct?

[Thor-x86]

Dear Mr. Raschka,

Sorry for questioning your knowledge, but I saw difference between your explanation and Qwen's article, exactly on DeltaNet architecture diagram. However, source code from huggingface somehow reflects more into your explanation? I have no idea which one is correct. It is a blessing if you could tell us the source of your explanation.

Thank you!

[Medamineelkhattabi]

Not the author, so I can't say where the drawing came from, but I compared both figures against the code. They draw the same layer and simplify different parts, and the code is what actually runs. I checked transformers at the SHA you linked (Qwen3_5GatedDeltaNet; Qwen3NextGatedDeltaNet computes the same thing with fused projections).

The README's Gated DeltaNet figure (L128) is basically a redraw of Qwen's. The overview figure at the top (L7) is the one that differs:

Part	HF code	Qwen blog	README overview
q, k, v	Linear, causal conv1d, SiLU (L577-L583), L2 norm on q, k (L328-L330)	same	no activation on q/k, σ on v (legend says sigmoid)
β	sigmoid(in_proj_b(x)), one per head (L604)	bare "Lin."	sigmoid
α	exp(-exp(A_log) * softplus(in_proj_a(x) + dt_bias)) (L606)	bare "Lin."	softplus
output	weight * rmsnorm(o) * silu(z), then out_proj (L210-L219, L645-L648)	"Zero-Centered RMSNorm", SiLU gate	RMSNorm, SiLU gate

So the README overview matches the code for α and β, and Qwen's figure is the accurate one for the conv activation.

Qwen's bare α/β is a simplification. Their drawing has the same layout as the block design in Figure 1 of the Gated DeltaNet paper (Yang et al., 2024), whose Section 3.4 says α and β "use linear projection only", with a footnote that α uses Mamba2's parameterization, omitted for brevity. Same topic as #902.

One real difference in the README's simplified code: W_beta outputs d_out (one β per channel), while Qwen and the paper's Eq. 10 use one β per head. With a per-head β, the README loop matches HF's torch_recurrent_gated_delta_rule: I ran both on random inputs, max difference 6e-8.

[rasbt]

Hi there, and thanks for the feedback. Let me know in case there is an issue in my figure that doesn't match the code. The code, tbh, is usually the gold standard because that one is basically the "true" model. It could be that the Qwen blog simplifies a few things.

[Thor-x86]

Your code and diagram are match. If you don't mind, I will pull request to put a notice that exponent in alpha part was omitted for brevity reason. So, anyone who having issue with it will know much earlier.

[Medamineelkhattabi]

Two small things I noticed while comparing, in case they help:

• In the overview figure, q and k have no activation after the short conv, while the code applies SiLU to q, k and v after it (L577-L583). Only v has the σ icon.
• In the README code, W_beta = nn.Linear(d_in, d_out) gives one β per channel, while Qwen3-Next uses one per head (L604). With a per-head β the loop matches HF's recurrent kernel.

[rasbt]

oh yes pls feel free to open an issue/PR

[Medamineelkhattabi]

Opened #1119 for the per-head β in the README code. The q/k activation is in the figure itself, so that one is yours.