vllm应该已经支持lora了吧

[SilentMoebuta]

Reminder

• I have read the README and searched the existing issues.

Reproduction

Traceback (most recent call last):
  File "/hy-tmp/llama-factory-1/LLaMA-Factory-main/src/api_demo.py", line 16, in <module>
    main()
  File "/hy-tmp/llama-factory-1/LLaMA-Factory-main/src/api_demo.py", line 9, in main
    chat_model = ChatModel()
  File "/hy-tmp/llama-factory-1/LLaMA-Factory-main/src/llmtuner/chat/chat_model.py", line 21, in __init__
    model_args, data_args, finetuning_args, generating_args = get_infer_args(args)
  File "/hy-tmp/llama-factory-1/LLaMA-Factory-main/src/llmtuner/hparams/parser.py", line 265, in get_infer_args
    raise ValueError("vLLM engine does not support LoRA adapters. Merge them first.")

在参数解析的时候直接把lora给挡住了

Expected behavior

No response

System Info

No response

Others

No response

[hiyouga]

没有对所有模型支持

[JieShenAI]

没有对所有模型支持

我想知道，如何做这个 merge 操作呢

[JieShenAI]

合并模型权重，可以参考这篇文章：https://zhuanlan.zhihu.com/p/689333581

[hiyouga]

@JieShenAI https://github.com/hiyouga/LLaMA-Factory/blob/main/examples/merge_lora/merge.sh

[JieShenAI]

@JieShenAI https://github.com/hiyouga/LLaMA-Factory/blob/main/examples/merge_lora/merge.sh

Thank you, it work.