DoRA

[Tengal-Teemo]

Hi everyone. I've been testing out mergekit recently, and I'm amazed by it. I was investigating the performance discrepancy between LoRA and Finetune, and came across this paper, proposing DoRA. Would it be possible for mergekit, alongside LoRA extraction, to implement DoRA extraction?

[thomasgauthier]

@Tengal-Teemo DoRA is beneficial during training as it more closely resembles the updates done during full finetuning (see Figure 2 of the paper). In other words it is more likely than LoRA to converge at the same rate as a full FT. I don't think however that it would be useful in a post-training context like LoRA extraction as all the weights updates and convergence have already happened. Closing this for now as DoRA extraction is not a planned feature.

Happy to keep the discussion going however if anyone disagrees with the above points.