Skip to content

v0.22.0

Latest

Choose a tag to compare

@BenjaminBossan BenjaminBossan released this 19 Nov 15:30
0ea6d96

Highlights

peft-v0 22

New methods

PaCA (Partial Connection Adaptation)

PaCA documentation (Partial Connection Adaptation, ICLR 2025), added in #3927 by @WooSunghyeon, is a PEFT method that fine-tunes randomly selected input columns ("partial connections") of the pretrained weights instead of adding adapter layers. This eliminates the sequential computation of adapter layers and stores only partial activations for the backward pass. Therefore, PaCA trains faster and uses less memory than LoRA with comparable accuracy (from the paper: 22% less training time, 16% less memory, 23% longer sequences). Use it when training time or activation memory is the bottleneck, also with quantized models (QPaCA). It is less effective when there are few tokens per step or only light modules such as q/v are adapted, and it supports only nn.Linear layers.

Astra initialization for LoRA

Astra (Activation-Space Tail-Eigenvector Low-Rank Adaptation of Large Language Models), added in by @LyoAI in #3725, uses a small task-specific calibration dataset to estimate each target layer's output-activation covariance. It initializes LoRA in the subspace of the covariance matrix's lowest-variance directions, giving the adapter a task-aware starting point. Set init_lora_weights="astra" and run preprocess_astra before injecting the adapter.

NoRA initialization for LoRA

@Joluck implemented Normalized Low-Rank Adaptation initialization for LoRA in #3908. Like normal LoRA, the B weight is initalized to zero so the adapter starts as a no-op, but each column of the randomly initialized LoRA A matrix is normalized along the rank axis. Experimentally, this resulted in faster and better conversion compared to standard LoRA initialization. Set the lora_alpha value equal to the rank r when passing init_lora_weights="nora".

Tensor Parallelism (TP) and FSDP2

One focus of the v0.22.0 release of PEFT was to improve the support for TP and FSDP2, which came through numerous changes. Please install the latest Transformers version (at least v5.17) to benefit from these improvements.

Tensor Parallelism with LoRA

LoRA now supports the embedding_colwise TP plan. Replicated LoRA weights also start from synchronized values across TP ranks, including with init_lora_weights=False and on output heads that gather their output (#4112, #3965). Please also check the updated LoRA TP guide.

FSDP2 training, checkpointing, and merging

  • LoRA now infers the full layer shape on FSDP2 and HSDP meshes with custom dimension names. Switching adapters or temporarily disabling them for a reference pass preserves trainability on FSDP2 shards. Exiting disable_adapter() reshards only when needed (#3838, #3839, #3895, #3938).
  • Adapter exports gather FSDP2 and HSDP adapter DTensors and correctly find adapter weights under nested FSDP or activation-checkpointing wrappers. With sharded adapter weights, call save_pretrained() on every participating rank and set is_main_process=True only on the rank that writes the files (#4065, #4108).
  • merge_adapter() and unmerge_adapter() now update the FSDP2 parameter shards even when reshard_after_forward=False. LoRA also preserves registered MultiheadAttention parameter identities during FSDP2 setup (#4077, #4083).
  • The FSDP guide and FSDP2 example now include an FSDP2 workflow covering training, adapter export, and resuming from a checkpoint (#4107).

Big thanks to @hivaze, @rupeshpoojary9, @Levius-Fubuki, @RerankerGuo, and @DaoyuanLi2816.

Enhancements

PEFT model health check

Call model.print_healthcheck() for a summary of adapter state, dtypes, device, etc. (or model.healthcheck() for a JSON-serializable report). It can flag issues that PEFT detects with the PEFT model. Since it also reports the number of trainable and total parameters, it should be used in place of model.print_trainable_parameters() (#3798, #3944, #4089 thanks to @eSVeeF).

Example output for a LoRA on Llama-3.2-1B (no errors were found):

Model:
  ID: meta-llama/Llama-3.2-1B, type: LlamaForCausalLM, adapters: default (LORA)
Environment:
  peft: 0.21.0, transformers: 5.18.0.dev0, torch: 2.14.0+cu130, Python: 3.13.9
Runtime:
  training: True, devices: cpu, dtypes: bfloat16=1,235,814,400, float32=851,968, gradient checkpointing: False
Parameters:
  trainable: 851,968, total: 1,236,666,368, percent trainable: 0.0689%, adapter layers: 32
State:
  adapter is enabled: True, active: default, merged: none
Layer types:
  default:
    lora.Linear=32

LoRA hotswapping across different target layers

PEFT already supports hotswapping LoRA adapters for some time, but the first loaded adapter must be able to accomodate all following adapters. peft.helpers.create_dummy_lora_config helps with this by building a dummy starting adapter with enough rank and layer coverage for a set of compatible LoRA adapters (#3739). PEFT also now checks the adapter being replaced and resolves its rank and alpha patterns during hotswapping (#3865, #3752, thanks to @DaoyuanLi2816).

More adapters can be converted to LoRA

LoRA conversion, i.e. the conversion of other PEFT methods to LoRA adapters, now supports OFT and BOFT linear adapters, DEFT, and standard and mini MiSS modes (#3360).

Use this feature to make your PEFT checkpoints compatible with frameworks that only support LoRA.

More efficient adapter merging

LoRA and several other methods scale their merge deltas in place, avoiding an extra full-size allocation per layer. This should save a little bit of memory and should speed up merging (#4147, thanks to @Hang-get).

Changes

Python support

PEFT now requires Python 3.11 or newer, Python 3.10 has reached its end of life.

Adapter loading and target validation

  • PeftModel.from_pretrained() and load_adapter() now warn by default about both missing and unexpected adapter keys. Use strict_adapter_check="error" to fail on a mismatch or "ignore" to suppress these checks (#3832 by @rupeshpoojary9).
  • Adapter injection validates all targets before modifying the model (#3981 by @eSVeeF).

Activation checkpointing for k-bit training

prepare_model_for_kbit_training() now uses the Transformers default for activation checkpointing (aka "gradient checkpointing") when no kwargs are supplied, selecting non-reentrant checkpointing on current Transformers versions. Pass activation_checkpointing_kwargs={“use_reentrant”: True} in the checkpointing kwargs to opt into reentrant checkpointing (#4146).

low_cpu_mem_usage

low_cpu_mem_usage=True now keeps dtypes, i.e. doesn’t restore the dtypes in the model to the ones from the checkpoint (which would most likely result in a type mismatch error, see #3509 by @vineethsaivs)

OFT

In #4157, @MaiAditya fixed the forward code of OFT Conv2d layers, which previously could produce incorrect results. This means that existing OFT checkpoints that target Conv2d layers may produce different outputs when used without merging.

modules_to_save suffix matching logic

modules_to_save used to match purely by suffix, but now the whole sub-module name must match. This is best understood with an example: if you pass modules_to_save=["classifier"], it used to match model.classifier and also model.other_classifer. Now it only matches model.classifier. If you want to keep the previous behavior, please pass modules_to_save=["classifier", "other_classifier"] or edit the adapter_config.json accordingly if you have an existing checkpoint (#3824 by @AHSharan).

Outlook: PEFT v1.0

The next PEFT release will be the 1.0 release! For this, we have planned a couple of sweet additions that will extend the scope of what PEFT is currently capable of doing, so stay tuned.

We will also take the opportunity to clean up PEFT and to streamline the API. This will mostly concern PEFT internals and we will take great care of ensuring backwards compatibility in the places where it matters, as well as to keep existing checkpoints working. Still, expect a few smaller breaking changes for the more uncommon workflows. Just as an example: Right now, if you have a PEFT model with multiple active adapters, and then you merge a single of those adapters, all other active adapters are completely ignored. This is very surprising and will raise an error in the future. For most PEFT users, however, everything will keep working as expected. We will have a release candidate so that you can check if anything breaks and give us feedback.

All changes

New Contributors

Full Changelog: v0.21.3...v0.22.0