[model] support bailing#55
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for the bailing_moe model by adding it to the model constants, exporting it in the GPTs module, and implementing the BailingMoeBridge class. Additionally, the .gitignore was updated and the configuration parser now includes score_function in its mapping. Feedback indicates that the bailing_moe model type should be explicitly handled in the configuration conversion logic to ensure that qk_layernorm is enabled and the router score function is correctly set to sigmoid.
| 'q_lora_rank': ['q_lora_rank'], | ||
| 'kv_lora_rank': ['kv_lora_rank'], | ||
| 'moe_router_score_function': ['scoring_func', 'moe_router_use_sigmoid'], | ||
| 'moe_router_score_function': ['scoring_func', 'moe_router_use_sigmoid', 'score_function'], |
There was a problem hiding this comment.
While adding score_function to the config_mapping is correct, the bailing_moe model type should also be explicitly handled in the hf_to_mcore_config function (around line 120 and 164) to ensure qk_layernorm is enabled and the router score function is set to sigmoid. The bridge definition in bailing_moe.py includes QK normalization keys and expert bias, which strongly suggests these configurations are required for the model to function correctly in Megatron-Core.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces support for the Bailing MoE model, including a specialized attention implementation to handle its specific QKV weight layout, updates to the configuration parser for MoE parameters, and the addition of integration tests. The review feedback identifies missing bias handling in the Bailing MoE attention layer and state dictionary mapping, which could lead to incorrect model behavior. It also points out that an existing test case was accidentally disabled.
2 participants
No description provided.