llama : make loras compatible with repacking

[ggerganov]

fix #12587
ref #12181 (comment)

When a LoRA adapter requires to be loaded to an extra buffer type (such as repacked bufts), load in to the default CPU buffer type instead.

[ggerganov]

@giladgd @amakropoulos Could you test if this change fixes the LoRA usage with Q4_K?

[amakropoulos]

@ggerganov yes that fixed it!

I get these warnings but otherwise works as expected:

llama_adapter_lora_init_impl: loading lora adapter from '/home/benuix/.config/LLMUnity/models/Qwen2-0.5B-Instruct-ru-lora.gguf' ...
llama_adapter_lora_init_impl: lora for 'blk.11.ffn_down.weight' cannot use buft 'CPU_AARCH64'
llama_adapter_lora_init_impl: lora for 'blk.12.ffn_down.weight' cannot use buft 'CPU_AARCH64'
llama_adapter_lora_init_impl: lora for 'blk.14.ffn_down.weight' cannot use buft 'CPU_AARCH64'
llama_adapter_lora_init_impl: lora for 'blk.15.ffn_down.weight' cannot use buft 'CPU_AARCH64'
llama_adapter_lora_init_impl: lora for 'blk.17.ffn_down.weight' cannot use buft 'CPU_AARCH64'
llama_adapter_lora_init_impl: lora for 'blk.18.ffn_down.weight' cannot use buft 'CPU_AARCH64'
llama_adapter_lora_init_impl: lora for 'blk.2.ffn_down.weight' cannot use buft 'CPU_AARCH64'
llama_adapter_lora_init_impl: lora for 'blk.20.ffn_down.weight' cannot use buft 'CPU_AARCH64'
llama_adapter_lora_init_impl: lora for 'blk.22.ffn_down.weight' cannot use buft 'CPU_AARCH64'
llama_adapter_lora_init_impl: lora for 'blk.23.ffn_down.weight' cannot use buft 'CPU_AARCH64'
llama_adapter_lora_init_impl: lora for 'blk.4.ffn_down.weight' cannot use buft 'CPU_AARCH64'
llama_adapter_lora_init_impl: lora for 'blk.5.ffn_down.weight' cannot use buft 'CPU_AARCH64'

[ggerganov]

Yes, the warnings are expected.

[giladgd]

@ggerganov I've run some tests and this seems to solve the issue.

[slaren]

This should work for now. Ideally we would use the same process as the model weights to determine the buffer type. I think this could be done by extracting to a function the relevant parts of the create_tensor lambda in llama_model::load_tensors, and keeping a mapping of each weight to its LLM_TN_IMPL object in llama_model.

[amakropoulos]

Thank you for the quick fix!