Add device parameter for PEFT load_adapter

[uwu-420]

Feature request

Hello,

the function load_adapter of the PeftAdapterMixin class currently doesn't have a device parameter but a device_map parameter which isn't being used by the load_peft_weights function. Instead, the load_peft_weights function will use infer_device to figure out some device.

See
https://github.com/huggingface/transformers/blob/30b453206d224ee5f747afa33ff216671558e6a0/src/transformers/integrations/peft.py#L193C13-L193C97
and
https://github.com/huggingface/peft/blob/5a4b9cade64bac8afdff5006ee9dd815c90b5469/src/peft/utils/save_and_load.py#L392C9-L392C32

Can it be changed to use the device of the model we want to add the adapter to by default?

Motivation

I stumbled over this when I had a model on CPU and the loaded adapter was loaded onto the MPS device which caused other issues for me. I think it would be more intuitive if the adapter is directly loaded to the same device as the model.

Your contribution

Sorry, nothing here. But happy to provide more context.

[amyeroberts]

cc @younesbelkada

[younesbelkada]

Hi @uwu-420
Thanks for the feature request !
Currently you can pass the device through adapter_kwargs={"device": device} but I agree we should use the same device as the model if possible. I will work on that and provide a fix