[bugfix] fix peft_format qwen3_5_moe#43
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the _get_hf_experts_attr method in gpt_bridge.py to include a check for _peft_format when processing qwen3_5_moe models, ensuring correct expert attribute handling for PEFT weights. Feedback suggests extending this logic to other models in the same function to avoid potential errors during LoRA weight conversion.
| def _get_hf_experts_attr(self, is_mtp: bool = False): | ||
| # return hf_grouped, is_gate_up | ||
| if not is_mtp and not self.config.fp8_param and self.model_type == 'qwen3_5_moe': | ||
| if not is_mtp and not self.config.fp8_param and not self._peft_format and self.model_type == 'qwen3_5_moe': |
There was a problem hiding this comment.
The addition of not self._peft_format correctly ensures that qwen3_5_moe uses non-grouped expert attributes when processing PEFT weights. This is necessary because LoRA adapters are typically stored per-expert in the HuggingFace state dict, whereas the base model might use a grouped layout.
However, note that other models returning True, True at line 733 (such as qwen3_vl_moe, llama4, and gpt_oss) currently lack this check. If these models are used with PEFT, they will incorrectly report hf_grouped=True, which will trigger a ValueError at line 936 during LoRA weight conversion. You should consider applying a similar check to line 733 to ensure PEFT support for those models as well.
2 participants
No description provided.