ValueError: Unknown quantization method: bitsandbytes. Must be one of ['aqlm', 'awq', 'fp8', 'gptq', 'squeezellm', 'gptq_marlin', 'marlin'].

[chjoo7]

I met this error during the call LLM with the Mixtral8x7B fine-tuned LoRA Merged Model. It worked well in torch 2.1.2 and vlllm 0.3.3 but torch 2,3 and vlllm 0.4.2 met this valueError. What is missing here? Should I use the different quantization method like awq or gptq than bitsandbytes?

[llm172]

we meet the same promblem

[jctux]

same here

[sajadn]

same here!

[fersebas]

Same any news? :)

[chjoo7]

I attempted to try the Mixtral 8x7B instruct GPTQ model and fine-tuned it to a custom dataset. I applied to the latest LlamaIndex Vllm call but met an OSError: does not appear to have a file named config.json. config.json is located in the base model so, see if there's any way to point to the tokenizer path into the Vlllm.... but not. It's interesting since the previous Vllm accepted the tokenizer path input.