How to reconstruct delta_w from A & B in MergedLinear?

[JamesDConley]

It's straightfoward for the regular Linear lora layer, but I'm struggling to understand how the merged layer is reconstructed into the weights delta

[JamesDConley]

For reference- here is the code from peft v0.1 that i'm struggling with* (shape mismatch error) and the code from this repo that (I assume) works.

The code From V0.1 of huggingface peft

def eval(self):
        nn.Linear.eval(self)
        self.lora_A.eval()
        self.lora_B.eval()
        if self.merge_weights and not self.merged:
            # Merge the weights and mark it
            if self.r > 0:
                self.weight.data += (
                    transpose(self.lora_B.weight @ self.lora_A.weight, self.fan_in_fan_out) * self.scaling
                )
            self.merged = True

The comparative code in this repo is

# Merge the weights and mark it
  if self.r > 0 and any(self.enable_lora):
      delta_w = F.conv1d(
          self.lora_A.data.unsqueeze(0), 
          self.lora_B.data.unsqueeze(-1), 
          groups=sum(self.enable_lora)
      ).squeeze(0)
      self.weight.data += self.zero_pad(T(delta_w * self.scaling))
  self.merged = True        

It may be that there is a bug in the peft code I used to train my model that is preventing me from calculating the delta_w correctly. Either way though, I definitely am not fully grasping what's going on with the reshaping and other operations here for the full qkv linear layer

[ShengleiH]

Hi @JamesDConley, I have the same issue when merging weights for BLOOM finetuned with LoRA in peft==0.2.0. And here is my solution, you need to update peft/tuners/lora.py as follows:

1. update line 363 in your case to self.lora_B.weight.data
2. update line 366 in your case to self.weight.data += self.zero_pad(transpose(delta_w * self.scaling, True)).T

In my case it works and the generation also looks good. Hope it helps.

[JamesDConley]

Hi @JamesDConley, I have the same issue when merging weights for BLOOM finetuned with LoRA in peft==0.2.0. And here is my solution, you need to update peft/tuners/lora.py as follows:

1. update line 363 in your case to self.lora_B.weight.data
2. update line 366 in your case to self.weight.data += self.zero_pad(transpose(delta_w * self.scaling, True)).T

In my case it works and the generation also looks good. Hope it helps.

Thanks! Didn't work for me but I really appreciate you taking the time.

[JamesDConley]

Oh woah! actually it might work. Made a different function rather than rebuilding the repo and need to move in another function. Thanks!

[JamesDConley]

Hi @JamesDConley, I have the same issue when merging weights for BLOOM finetuned with LoRA in peft==0.2.0. And here is my solution, you need to update peft/tuners/lora.py as follows:

1. update line 363 in your case to self.lora_B.weight.data
2. update line 366 in your case to self.weight.data += self.zero_pad(transpose(delta_w * self.scaling, True)).T

In my case it works and the generation also looks good. Hope it helps.

This worked!!!! Thank you so much, you've saved me so much time!