vllm can't run peft model?

[Stealthwriter]

Hi, does vllm runs peft models? I'm trying to run them but im getting an error that the config.json is not available in the model main forlder

[viktor-ferenczi]

vLLM does not support PEFT/LoRA yet, but it is on the Development Roadmap

Related: #289, #815

[Stealthwriter]

vLLM does not support PEFT/LoRA yet, but it is on the Development Roadmap

Related: #289, #815

ok thanks!

[hllj]

You have to merge adapter from peft with their base model first.
This is my codebase for merging you can see here: https://github.com/Reasoning-Lab/Elementary-Math-Solving-Zalo-AI-2023/blob/79ed4742d91755b4a00fadd0079279394222928b/merge_peft_adapter.py
After that you will have your merged model and you can serve with vLLM from output_dir.

[rsong0606]

@hllj Hi there, does this mean the merged model either saving on local or pushing to the hugging face will have the same structure as the base model which means will have similar files and versions as the base model in hugging face?