AdaLoRA: where does the Rankallocator work

[louislau1129]

Hi, I have used adalora to do parameter-efficient fine-tuning for a while. It works very well and significantly better than the vanilla lora in my case. But recently I just found two issues.

1. adalora did not use the orthogonal regularization since its forward function in https://github.com/huggingface/peft/blob/main/src/peft/tuners/adalora.py#L216 can't be called. This happened when I used PeftModel to call adalora as below:

config = AdaLoraConfig(target_r=args.rank, init_r=init_r, lora_alpha=64, target_modules=args.target_modules, lora_dropout=0.05, bias="none")
model = get_peft_model(model, config)

I found the reason is that the self.get_base_model() in forward() function of PeftModel did not point to the AdaLoraModel as base model. (https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py#L300)

The original code is as follows:

    def get_base_model(self):
        """
        Returns the base model.
        """
        return self.base_model if isinstance(self.active_peft_config, PromptLearningConfig) else self.base_model.model

After modifying it as the following, it can use the AdaLoraModel forward function as expected.

    def get_base_model(self):
        """
        Returns the base model.
        """
        return self.base_model if isinstance(self.active_peft_config, (PromptLearningConfig, AdaLoraConfig)) else self.base_mode.model

2. The second issue is that I cannot find where the Rankallocator in adalora is called. If I understand correctly, this function is implemented in def update_and_allocate(self, global_step): in https://github.com/huggingface/peft/blob/main/src/peft/tuners/adalora.py#L284. However, when I add a breakpoint here, the program will not stop. Maybe for this reason, I cannot find those masked rank (elements on some rank positions = 0) in lora_E of the peft saved model.

Anyone has idea on this issue? I really appreciate any help you can provide.

[louislau1129]

I have double-checked the second issue, rankallocator indeed did not call in the current peft version. I manually call it to perform the adaptive rank allocation.

[moritzunseld]

Thanks for pointing this out. I've also encountered some weird behavior when benchmarking for my Bachelor's Thesis.

[moritzunseld]

I have double-checked the second issue, rankallocator indeed did not call in the current peft version. I manually call it to perform the adaptive rank allocation.

Where/how do you manually call it?

[louislau1129]

Where/how do you manually call it?

I add the following code in https://github.com/huggingface/transformers/blob/v4.29.1/src/transformers/trainer.py#L2013 to explicitly call this allocate function .

   from peft import PeftModel
   if isinstance(model, PeftModel):
        if getattr(model.base_model, "update_and_allocate", None) is not None:
               model.base_model.update_and_allocate(total_batched_samples)

Besides this, you should also set corresponding tinit, tfinal, deltaT, and total_steps in AdaLoraConfig.
I am not sure if there is a more elegant way to do that, but it works.
Also I did not find too much difference in terms of fine-tuning performance after fixing these two issues using my way. I will further investigate them.
Hope the developer could give some ideas/comments about these issues.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[PluszZh]

Where/how do you manually call it?

I add the following code in https://github.com/huggingface/transformers/blob/v4.29.1/src/transformers/trainer.py#L2013 to explicitly call this allocate function .

   from peft import PeftModel
   if isinstance(model, PeftModel):
        if getattr(model.base_model, "update_and_allocate", None) is not None:
               model.base_model.update_and_allocate(total_batched_samples)

Besides this, you should also set corresponding tinit, tfinal, deltaT, and total_steps in AdaLoraConfig. I am not sure if there is a more elegant way to do that, but it works. Also I did not find too much difference in terms of fine-tuning performance after fixing these two issues using my way. I will further investigate them. Hope the developer could give some ideas/comments about these issues.

I found that this issue still exists in the current version. Can you share your code? Thank you very much!

[geoffvdr]

This issue still exists. Any suggestion on how to make rankallocator / update_and_allocate work with peft?

@QingruZhang maybe?

[BenjaminBossan]

Note that PEFT does not contain training code, as such calling update_and_allocate is out of scope for PEFT. When using Trainer, this could maybe be solved with a callback, but I have only little experience with Trainer. When running a custom training loop, call this method manually, as e.g. shown in this AdaLoRA training script.