Skip to content

v0.20.0

Latest
Latest

Choose a tag to compare

View all tags
@BenjaminBossan BenjaminBossan released this 28 Jul 13:49
v0.20.0
a5526d2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

peft-v0 20 0

This release adds no less than nine new PEFT methods and puts a lot of work into the surrounding infrastructure, for example adding a new image generation benchmark for the method comparison suite and greatly improving the documentation structure.

New Methods

HiRA

@hqsiswiliam added "HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models" to PEFT (#2668). Instead of adding the low-rank product BA to the base weight, HiRA multiplies it elementwise (Hadamard product) with the frozen base weight. Because the base weight itself is full rank, the resulting update is no longer constrained to be low rank, while the trainable parameter count stays the same as LoRA's.

GLoRA

@not-lain contributed GLoRA: "One-for-All: Generalized LoRA for Parameter-Efficient Fine-Tuning" in #3098. It is a flexible PEFT method that extends LoRA with configurable weight, activation, and bias adaptation, delivering richer fine-tuning with no extra inference cost. Use it when you need per-layer flexibility or stronger adaptation than vanilla LoRA. Skip it for non-Linear layers (e.g. Conv/Embedding) or when standard LoRA is already sufficient and simplicity matters.

BEFT

@whubaichuan added "BEFT: Bias-Efficient Fine-Tuning of Language Models" in #3195. BEFT builds on the observation that fine-tuning bias terms alone can be competitive in low-data regimes, but goes further: rather than training all biases, it targets the value projection by default, as the authors found this to be most efficient. This brings the trainable parameter count down to roughly 0.01% of the total parameters.

MonteCLoRA

@victor7246 integrated MonteCLoRA, "Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation" in #2943. LoRA is known to be sensitive to hyperparameters like learning rate and batch size. This new LoRA variant addresses this by treating the low-rank parameters as a distribution rather than a point estimate, using Monte Carlo estimation to obtain a low-variance posterior estimate. This should make training noticeably more robust, at the cost of only O(rank) additional parameters.

VeLoRA

@roymiles added VeLoRA: "Memory Efficient Training using Rank-1 Sub-Token Projections" in #3159. Unlike most PEFT methods, this LoRA variant targets activation memory rather than parameter count: intermediate activations are split into sub-tokens and compressed with a fixed rank-1 projection before being cached for the backward pass, then reconstructed during backpropagation. Conceptually, it is similar to gradient checkpointing, with a lower memory saving but running faster.

Uni-LoRA

@KaiyangLi1992 contributed Uni-LoRA: "One Vector is All You Need" in #3257. The paper shows that parameter-efficient LoRA variants like VeRA and VB-LoRA can all be described as projecting the full LoRA parameter space down to a much smaller subspace, differing only in the choice of projection. Uni-LoRA uses a single global projection across the whole model instead of layer-wise ones, which allows cross-layer parameter sharing and thus very low parameter counts.

FRoD

@Bane-Elvin added FRoD, "Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergence", in #3270. Instead of adding low-rank deltas like LoRA, it reconstructs selected weights with shared rotational subspaces and sparse trainable coefficients. It is especially useful when fast convergence and a higher full-rank capacity ceiling are important, and its large sparse rotational subspace may also be promising for model merging. The main tradeoffs are the costly joint-decomposition initialization and slightly slower forward/backward passes than LoRA, so it may be less attractive for a one-off single-task fine-tune.

MiCA

@sr-networks contributed MiCA, "MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning", in #3260. MiCA is a LoRA variant that initializes B from the SVD of the base weight, taking the left singular vectors belonging to the smallest singular values, and then trains only A. The idea is that adapting these underused directions integrates new knowledge more effectively than adapting the dominant subspace, with less interference with existing capabilities.

DEFT

@MAXNORM8650 added DEFT, "Decompositional Efficient Fine-Tuning for Text-to-Image Models", in #3342. This PEFT method splits a weight update into two learned low-rank parts: a projection that removes a sub-space of the frozen weight, and a low-rank update that injects new content in its place. Use DEFT for personalizing a text-to-image model from a few images while retaining the base model's instruction-following/editability with minimal forgetting. It is less of a fit if you don't need to preserve the base model's other capabilities (a plain additive adapter is simpler) or for layers beyond Linear/Conv1D. The PaRa method is also supported as a special case of DEFT.

Enhancements

Automatic LoRA target selection

@oswaldoludwig added KappaTuneSelector in #3106 based on "The Condition Number as a Scale-Invariant Proxy for Information Encoding in Neural Units". The PR adds a new function, find_kappa_target_modules, which automatically identifies which modules best to target based on the condition number of the base weight matrices. This removes some of the guesswork from choosing target_modules when configuring LoRA or other PEFT methods.

Broader quantization support

#3117 adds generic quantization support to BOFT, MiSS, and VeRA, and #3321 does the same for SHiRA. Previously, quantization support had to be implemented per method and per backend; more methods now work with the common quantization backends out of the box. Note that as a consequence, VeRA now uses the generic quantization backend instead of its previous bitsandbytes quantization backend. If you encounter issues because of that, please let us know.

Multiple adapters with target_parameters

target_parameters, which allows targeting nn.Parameters directly (useful for MoE models that pack experts into a single parameter), now supports adding and loading multiple adapters (#3350).

AdaLoRA for Conv2d

@Anai-Guo added SVDConv2d in #3196, so AdaLoRA can now target torch.nn.Conv2d layers.

Documentation restructure

The PEFT documentation was reorganized in #3300 and #3325, which should make it considerably easier to navigate now that PEFT supports so many methods. Each PEFT method also provides an overview of how well it fares in the PEFT benchmarks. There is also a long tail of documentation improvements from many contributors this release, thanks to everyone who worked on those.

Changes

AutoGPTQ deprecation

Following the switch to GPT-QModel as the backend in the previous release, AutoGPTQ is now formally deprecated (#3190) thanks to @Qubitium.

Benchmarks

Image generation benchmark

The method comparison suite now has an image generation benchmark (#3082). This complements the existing MetaMathQA benchmark and allows to compare PEFT methods on a completely different task. This gives us a more complete picture of the capabilities of different PEFT methods. To check the results, visit the Gradio Space and on the top left, select the task "image-gen".

Screenshot from 2026-07-28 15-48-53

A look at the results from the image generation benchmark

MetaMathQA

Since the last release, we also made several improvements to the MetaMathQA benchmark. You should check the updated results in the Gradio Space. With the help of the community, we now cover a much broader range of experiments. Moreover, we discovered a bug in the forgetting metric -- the new numbers now accurately reflect how much fine-tuning retains of the original knowledge of the base model.

PEFT Shop

#3317 adds a new Gradio Space with the goal of making it very easy to compare different PEFT methods not only with regard to performance but also capabilities (e.g. if quantization is supported). Give this Space a visit to find out which PEFT method is the best fit for your use case.

Screenshot from 2026-07-28 15-30-21

Browse the PEFT shop for the best PEFT method for your use case

All Changes

New Contributors

Full Changelog: v0.19.0...v0.20.0

Contributors

githubnemo, Qubitium, and 54 other contributors
Assets 2
Loading