infer llama 报错

[JoshonSmith]

1、Describe the bug

instructKGC/infer_scripts/infer_llama.bash 执行报错
脚本参数：
CUDA_VISIBLE_DEVICES=3 python src/inference.py
--stage sft
--model_name_or_path '/data/.cache/modelscope/hub/modelscope/Llama-2-13b-chat-ms/'
--checkpoint_dir '/data/WEIGHTS/llama2-13b-iepile-lora'
--model_name 'llama'
--template 'llama2'
--do_predict
--input_file '/data/IEPile/test-RE-zh.json'
--output_file 'results/llama2-13b-chat-lora.json'
--finetuning_type lora
--output_dir 'lora/test'
--predict_with_generate
--cutoff_len 512
--bf16
--max_new_tokens 300
--bits 4

3、Screenshots

报错信息：
inputs:
[INST]
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<>
{"instruction": "你是专门进行关系抽取的专家。请从input中抽取出符合schema定义的关系三元组，不存在的关系返回空列表。请按照JSON字符串的格式回答。", "schema": ["传播途径", "鉴别诊断", "辅助治疗", "药物治疗"], "input": "口腔黏膜炎@## 并发症 ### 查看全部   并发症 table 并发症 | 时间表 | 可能性 ---|---|--- ### 口腔念珠菌病 | 短期 | 中 接受头颈部放化疗的患者，尤其是唾液分泌功能严重受损的患者，常并发 真菌感染 。 口腔黏膜炎@需要使用抗真菌药物治疗。"} [/INST]
Traceback (most recent call last):
File "/data//InstructKGC/src/inference.py", line 122, in
main()
File "/data//InstructKGC/src/inference.py", line 116, in main
inference(model_args, data_args, training_args, finetuning_args, generating_args, inference_args)
File "/data//InstructKGC/src/inference.py", line 105, in inference
result = evaluate(model_inputs, generating_args)
File "/data//InstructKGC/src/inference.py", line 81, in evaluate
generation_output = model.generate(
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/peft/peft_model.py", line 977, in generate
outputs = self.base_model.generate(**kwargs)
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/transformers/generation/utils.py", line 1602, in generate
return self.greedy_search(
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/transformers/generation/utils.py", line 2450, in greedy_search
outputs = self(
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 820, in forward
outputs = self.model(
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 708, in forward
layer_outputs = decoder_layer(
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 424, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 311, in forward
query_states = [F.linear(hidden_states, query_slices[i]) for i in range(self.config.pretraining_tp)]
File "/data/condaenvs/deepke-llm/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 311, in
query_states = [F.linear(hidden_states, query_slices[i]) for i in range(self.config.pretraining_tp)]
RuntimeError: mat1 and mat2 shapes cannot be multiplied (457x5120 and 1x2560)

[guihonghao]

检查llama-2-13b-chat-hf的config.json 文件中的pretraining_tp值是否为1，若不为1，请修改为1.

[JoshonSmith]

检查llama-2-13b-chat-hf的config.json 文件中的pretraining_tp值是否为1，若不为1，请修改为1.

ok，解决了