[BUG]Error when I was trying to run Mixtral-8x7B-v0.1

[Eeeeclipse]

Before you submit an issue, please search for existing issues to avoid duplicates.

Issue description:

When I was trying to run , I got TypeError: _init_ffn() missing 1 required positional argument: 'weights'``. Looks like the _init_ffn function in lightllm/lightllm/models/mixtral/layer_weights/transformer_layer_weight.py requires "weights" as input but the reference in lightllm/lightllm/models/llama/layer_weights/transformer_layer_weight.py` doesn't provide such input.

Steps to reproduce:

Please list the steps to reproduce the issue, such as:

python -m lightllm.server.api_server \ --model_dir /home/user/.cache/huggingface/hub/models--mistralai--Mixtral-8x7B-v0.1/snapshots/fc7ac94680e38d7348cfa806e51218e6273104b0 \ --host 0.0.0.0 \ --port 9000

Expected behavior:

The server is expected to be launched.

Error logging:

ERROR 09-20 22:55:09 [model_rpc.py:97] File "/home/user/lightllm/lightllm/models/mixtral/model.py", line 33, in __init__ ERROR 09-20 22:55:09 [model_rpc.py:97] super().__init__(kvargs) ERROR 09-20 22:55:09 [model_rpc.py:97] File "/home/user/lightllm/lightllm/common/basemodel/basemodel.py", line 94, in __init__ ERROR 09-20 22:55:09 [model_rpc.py:97] self._init_weights() ERROR 09-20 22:55:09 [model_rpc.py:97] File "/home/user/lightllm/lightllm/common/basemodel/basemodel.py", line 147, in _init_weights ERROR 09-20 22:55:09 [model_rpc.py:97] self.trans_layers_weight = [ ERROR 09-20 22:55:09 [model_rpc.py:97] File "/home/user/lightllm/lightllm/common/basemodel/basemodel.py", line 148, in <listcomp> ERROR 09-20 22:55:09 [model_rpc.py:97] self.transformer_weight_class( ERROR 09-20 22:55:09 [model_rpc.py:97] File "/home/user/lightllm/lightllm/models/mixtral/layer_weights/transformer_layer_weight.py", line 15, in __init__ ERROR 09-20 22:55:09 [model_rpc.py:97] super().__init__( ERROR 09-20 22:55:09 [model_rpc.py:97] File "/home/user/lightllm/lightllm/models/llama/layer_weights/transformer_layer_weight.py", line 17, in __init__ ERROR 09-20 22:55:09 [model_rpc.py:97] super().__init__(layer_num, data_type, network_config, mode, quant_cfg) ERROR 09-20 22:55:09 [model_rpc.py:97] File "/home/user/lightllm/lightllm/common/basemodel/layer_weights/transformer_layer_weight.py", line 21, in __init__ ERROR 09-20 22:55:09 [model_rpc.py:97] self._init_weight() ERROR 09-20 22:55:09 [model_rpc.py:97] File "/home/user/lightllm/lightllm/models/llama/layer_weights/transformer_layer_weight.py", line 23, in _init_weight ERROR 09-20 22:55:09 [model_rpc.py:97] self._init_ffn() ERROR 09-20 22:55:09 [model_rpc.py:97] TypeError: _init_ffn() missing 1 required positional argument: 'weights'

Environment:

Please provide information about your environment, such as:

• Using container

• OS: (Ubuntu 14.04, CentOS7)

• GPU info:

  • nvidia-smi (e.g. NVIDIA-SMI 525.116.04 Driver Version: 525.116.04 CUDA Version: 12.0)
  • Graphics cards: (e.g. 4090x8)

• Python: (e.g. CPython3.9)

  • currently, only python>=3.9 is supported

• LightLLm: (git commit-hash)

  • for container: docker run --entrypoint cat --rm ghcr.io/modeltc/lightllm:main /lightllm/.git/refs/heads/main

• openai-triton: pip show triton

Additional context:

Please add any other context or screenshots about the issue here.

Language:

Please use English as much as possible for better communication.

[hiworldwzj]

@Eeeeclipse thanks, we will fix this soon.

[hiworldwzj]

@Eeeeclipse fixed #1064

[Eeeeclipse]

Today I tried to run Mixtral-8x7B-v0.1 but got

ERROR 10-01 21:20:27 [basemodel.py:789] fused_experts_impl() missing 2 required positional arguments: 'w1_bias' and 'w2_bias'

I fixed this issue by changing the def fused_experts_impl in lightllm/lightllm/common/fused_moe/grouped_fused_moe.py to

def fused_experts_impl(
    hidden_states: torch.Tensor,
    w1: torch.Tensor,
    w2: torch.Tensor,
    w1_bias: Optional[torch.Tensor] = None,
    w2_bias: Optional[torch.Tensor] = None,
    topk_weights: torch.Tensor = None,
    topk_ids: torch.Tensor = None,
    inplace: bool = False,
    use_fp8_w8a8: bool = False,
    use_int8_w8a16: bool = False,
    w1_scale: Optional[torch.Tensor] = None,
    w2_scale: Optional[torch.Tensor] = None,
    a1_scale: Optional[torch.Tensor] = None,
    a2_scale: Optional[torch.Tensor] = None,
    alloc_tensor_func=torch.empty,
    run_config: Optional[dict] = None,
    layout="blocked",
    limit=None,
    alpha=None,
):

[flyinglandlord]

Thanks for your help. We will fix it immediately.