Combine multi-LoRA and quantization

[Yard1]

There is no fundamental reason why multi-LoRA cannot work with quantized models. We will most likely want to keep LoRA's unquantized and dequantize the base model output before applying LoRAs with punica kernels. That seems to be the pattern present in other projects too.

Originally posted by @Yard1 in #1804 (comment)

[jacob-hansen]

Has there been any progress with this? Or has anyone tested multi-LoRA with different quantization to see what might work?

[thincal]

@Yard1 is there any plan for this support? really depends on this wonderful feature and also needs to know the real effect. thanks.

[whyiug]

@thincal #4012