Add Glm4MoeForCausalLM Support#619
Conversation
quic-shagun
requested review from
ochougul,
quic-amitraj,
quic-hemagnih and
quic-rishinr
as code owners
|
@shagsood do we have approval for this model? also do add this model under validated model list |
|
@vbaddi can you please review this PR? |
Yes we have legal approval for this model. |
|
|
||
| class QEffGlm4MoeMoE(Glm4MoeMoE): | ||
| """ | ||
| MoE Block |
There was a problem hiding this comment.
nit: We can start using our optimized moe block for prefill/decode usecase here?
There was a problem hiding this comment.
Is there a specific PR I need to refer for this?
| key_states, | ||
| value_states, | ||
| attention_mask, | ||
| dropout=0.0 if not self.training else self.attention_dropout, |
There was a problem hiding this comment.
Can we remove this dropout, since we are not using it in eager_attention_forward
| value: torch.Tensor, | ||
| attention_mask: Optional[torch.Tensor], | ||
| scaling: float, | ||
| dropout: float = 0.0, |
There was a problem hiding this comment.
Please remove, since its not used.
| topk_weights = router_scores.gather(1, topk_indices) # [T, 8] | ||
|
|
||
| if self.norm_topk_prob: | ||
| topk_weights = topk_weights / (topk_weights.sum(dim=-1, keepdim=True) + 1e-20) |
There was a problem hiding this comment.
nit: is subfunction verified? I feel this. sum() needs to be replaced w/einsum.
|
@shagsood can you rebase and bring this to main branch? |
Signed-off-by: shagsood <shagsood@qti.qualcomm.com>
Signed-off-by: shagsood <shagsood@qti.qualcomm.com>
Signed-off-by: shagsood <shagsood@qti.qualcomm.com>
Signed-off-by: shagsood <shagsood@qti.qualcomm.com>
Signed-off-by: Onkar Chougule <ochougul@qti.qualcomm.com>
Signed-off-by: Onkar Chougule <ochougul@qti.qualcomm.com>
Signed-off-by: Onkar Chougule <ochougul@qti.qualcomm.com>
Signed-off-by: Onkar Chougule <ochougul@qti.qualcomm.com>
Signed-off-by: Onkar Chougule <ochougul@qti.qualcomm.com>
Signed-off-by: Onkar Chougule <ochougul@qti.qualcomm.com>
Signed-off-by: Onkar Chougule <ochougul@qti.qualcomm.com>
quic-rishinr
requested review from
quic-mamta
and removed request for
ochougul,
quic-amitraj,
quic-hemagnih and
quic-rishinr
|
@ochougul please fix the Lint DCO and other failures |
|
@ochougul Can you please fix the LINT errors |
Signed-off-by: Onkar Chougule <ochougul@qti.qualcomm.com>
|
New PR is raised with the following features. Will close this PR and will proceed with PR #988 |
9 participants
This PR adds support for zai-org/GLM-4.5-Air model.
Open source MoE model with performance and accuracy better than many closed source models: