update gated delta rule state layout#4636
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds support for a transposed recurrent-state memory layout in the CUDA fused gated-delta-rule kernel and updates Qwen3.5/Qwen3-Next integration to use that layout (intended to be more hardware-friendly than the previous default layout).
Changes:
- Add
transpose_state_layoutsupport to the TileLang fused recurrent gated-delta-rule kernel, including shape validation and a specialized direct load/store path for[V, K]state layout. - Update Qwen3.5/Qwen3-Next recurrent state shapes and the
GatedDeltacall sites to use the transposed layout. - Extend kernel tests to cover transposed state, transposed state +
state_indices, and transposed circular-buffer behavior; adjust one circular-buffer tolerance.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/pytorch/kernel/test_gated_delta_rule.py | Adds coverage for transposed-state layout (including state_indices and circular buffer) and adjusts tolerance. |
| lmdeploy/pytorch/nn/gated_delta.py | Always enables transposed-state layout when invoking gated-delta kernels. |
| lmdeploy/pytorch/kernels/cuda/gated_delta_rule.py | Implements transposed-state layout path and input shape validation in the fused recurrent kernel. |
| lmdeploy/pytorch/configurations/qwen3_next.py | Swaps recurrent state shape to [V, K] for Qwen3-Next. |
| lmdeploy/pytorch/configurations/qwen3_5.py | Swaps recurrent state shape to [V, K] for Qwen3.5 (including spec-decoding shape). |
| lmdeploy/pytorch/backends/gated_delta_rule.py | Extends backend API to accept transpose_state_layout (and cache_seqlens for fused recurrent). |
| lmdeploy/pytorch/backends/cuda/gated_delta_rule.py | Threads transpose_state_layout through CUDA backend calls. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
188
to
200
| core_attn_out, last_state = self.chunk_func( | ||
| q, | ||
| k, | ||
| v, | ||
| g=g, | ||
| beta=beta, | ||
| scale=scale, | ||
| initial_state=init_state, | ||
| output_final_state=output_final_state, | ||
| use_qk_l2norm_in_kernel=False, | ||
| cu_seqlens=cu_seqlens, | ||
| transpose_state_layout=transpose_state_layout, | ||
| ) |
Collaborator
|
may need to merge latest main to fix ut |
Collaborator
Author
It is already done. Main branch has same problem. |
CUHKSZzxy
approved these changes
Jun 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Transposed state is more friendly than default layout.