Any plans for QLora?

[asmith26]

Hi there! keras-nlp supports Lora, e.g. from https://ai.google.dev/gemma/docs/lora_tuning

...
gemma_lm.backbone.enable_lora(rank=4)
...

Just wondering are there any plans to implement QLora finetuning, to help make finetuning even more accessible on smaller GPUs?
https://arxiv.org/abs/2305.14314
https://huggingface.co/blog/4bit-transformers-bitsandbytes

Many thanks for any help, and this lib!

[mattdangerw]

Yes actually! We are currently working or rolling out some support for quantization generally and quantize lora.

We will work on some actual docs for this soon, but for now you can check out the qlora.py in the description of this PR to see what we are working on -> keras-team/keras#19356

I believe the 4 bit normal float from the qlora paper is not currently available, but int8 quantized weights layered in with lora should be good to go.

Early days on the feature, so any feedback appreciated as we build it out.

[github-actions]

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[mattdangerw]

Guide on this incoming...

keras-team/keras-io#1889

[Gopi-Uppari]

Hi @asmith26,

Could you please confirm if this issue is resolved for you with the above comment ? Please feel free to close the issue if it is resolved ?

Thank you.

[asmith26]

Have got this working https://keras.io/examples/keras_recipes/parameter_efficient_finetuning_of_gemma_with_lora_and_qlora/#qlora-finetuning

Many thanks again!