EoRA

EoRA (Eigenspace Low-Rank Adapter)

Documentation home · Quickstart · Lifecycle

On this page

• Overview
• Workflow at a glance
• Concepts and mathematics
• Usage
• Adapter configuration reference
• Save, load, and correction lifecycle
• Processor and package internals
• PEFT training and loading
• Repository examples and evaluation
• Pitfalls and edge cases
• Source files
• See also

Overview

EoRA recovers part of the output error introduced by weight quantization using a low-rank correction. It measures activation statistics on calibration data, factors the difference between the original and dequantized weights in an activation-weighted eigenspace, and saves the resulting factors as a LoRA-compatible adapter.

EoRA is training-free: adapter generation uses covariance accumulation, an eigendecomposition, and an SVD, rather than a gradient-based fine-tuning loop. It still requires calibration forward passes and access to the original floating-point weights. It can run during quantization or generate an adapter afterward for an existing compatible quantized checkpoint.

The quantized base and the adapter remain separate at inference. EoRA does not change the base checkpoint's quantization method or bit width, and the low-rank factors are floating-point tensors rather than packed low-bit weights.

This page documents source revision c1976918cfe3703fbcf6d1ce2440a82b26394375. The remote-loading restrictions and PEFT integration notes below apply to that revision.

Workflow at a glance

Reading the diagram: Generation requires calibration and matching original weights. The post-hoc API is limited to the GPTQ TorchLinear path. Training-free does not mean calibration-free or merged weights.

Text version of the workflow

• Training-free adapter generation: Capture activation stats → Weight the residual → Factor low-rank A and B → Save a separate adapter.
• Adapter-aware inference: Load the quantized base → Load matching factors → Add the low-rank output.

Open full-size SVG

Concepts and mathematics

The correction being approximated

For a linear layer with weight $W\in\mathbb{R}^{d_{out}\times d_{in}}$, let $W_q$ be the dequantized packed weight and $\Delta=W-W_q$. EoRA constructs

$$B\in\mathbb{R}^{d_{out}\times r},\qquad A\in\mathbb{R}^{r\times d_{in}}$$

so that $BA$ approximates the important part of $\Delta$ for the observed activations. For row-vector input $X$, inference is conceptually

$$Y=XW_q^T+(XA^T)B^T+b.$$

Ordinary truncated SVD of $\Delta$ minimizes an unweighted weight-space error. EoRA instead scales the residual by an eigenspace transform derived from calibration inputs, emphasizing directions that those inputs exercise.

Activation-statistic accumulation

eora_process_input receives the forward-hook input tuple. It takes input[0], converts it to FP32, promotes a two-dimensional input to a batch of one, and computes:

$$C_t=\frac{1}{N}\sum_{b=1}^{B_t}X_b^TX_b, \qquad s_t=\frac{N}{N+B_t}.$$

Here $N$ is the processor's number of calibration batches (len(calibration_dataset) after preparation), not the number of tokens. There is no sequence-length normalization in this helper. The processor accumulates with the recurrence

$$T_t=s_tT_{t-1}+C_t.$$

This is the implementation's weighted accumulation, not a claim that the stored matrix is the usual sample covariance centered around a mean. Inputs are not mean-centered here.

Accumulation stays on the originating device. Each device record stores its accumulated total, the product of its scale factors, count, and batch-index range. Finalization orders device segments by starting index, moves them to the module's target device, and calls merge_eora_segments. The merge updates result = result * segment_scale_product + segment_total; it mutates the first segment tensor. An empty segment list is an error.

Eigenspace scaling, numerical rank, and SVD

eora_compute_lora receives the FP32 weight residual and the accumulated scaling matrix $C$:

1. Convert $C$ to FP64 and compute $C=Q\Lambda Q^T$ with torch.linalg.eigh.
2. Reject non-finite eigenvalues.
3. Retain eigenvalues satisfying $\lambda_i>\lambda_{max}\tau$, with $\tau=\min(1,\max(1,d_{in}),\epsilon_{FP32})$ and $\lambda_{max}$ clamped to at least zero.
4. Set the square roots and reciprocal square roots of discarded eigenvalues to zero. Negative, zero, and numerically insignificant directions are discarded; they are not inverted or shifted to an arbitrary positive floor.
5. Form $S_C=Q\sqrt{\Lambda}$ and $S_C^+=\Lambda^{-1/2}Q^T$, with discarded directions zeroed, and convert both transforms to FP32.
6. Compute the reduced SVD of $\Delta S_C=U\Sigma V^T$.
7. Keep the first requested $r$ singular components and split their singular values between the factors:

$$B=U_r\sqrt{\Sigma_r},\qquad A=\sqrt{\Sigma_r}V_r^TS_C^+.$$

The returned factors are cast to the requested module dtype. Thus the eigensolve is FP64, while the scaled residual and SVD are FP32. A warning reports discarded eigenvalues and the cutoff when the calibration matrix is numerically rank deficient.

If covariance has unsupported/null directions, this is a truncated pseudoinverse, not an exact inverse on the entire feature space. A requested rank beyond the available SVD dimension is sliced down by Torch; the function does not create additional independent components or pad the factors to the requested rank.

The exact factorization API is:

def eora_compute_lora(
        w_wq_delta: Tensor,
        name: str,
        eigen_scaling_diag_matrix: torch.Tensor,
        rank: int,
        dtype: torch.dtype,
        device: torch.device,
) -> tuple[Tensor, Tensor]:

The returned order is (A, B), even though the weight correction is B @ A.

Usage

Generate an adapter while quantizing

Set QuantizeConfig.adapter to a Lora descriptor. The same calibration data is used for the quantizer and EoRA unless adapter_calibration_dataset is supplied separately.

This complete example writes the base and adapter to separate local directories and reloads with an explicit adapter. The selected model may require access permission from its publisher.

from pathlib import Path
from datasets import load_dataset
from gptqmodel import BACKEND, GPTQModel, QuantizeConfig
from gptqmodel.adapter.adapter import Lora

source_model = "meta-llama/Llama-3.2-3B"
base_dir = str(Path("llama-gptq").resolve())
adapter_dir = str(Path("llama-eora").resolve())

calibration = load_dataset(
    "allenai/c4",
    data_files="en/c4-train.00001-of-01024.json.gz",
    split="train",
).select(range(128))["text"]

qcfg = QuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=True,
    adapter=Lora(rank=64, path=adapter_dir),
)
model = GPTQModel.load(source_model, quantize_config=qcfg)
model.quantize(
    calibration=calibration,
    batch_size=1,
    calibration_concat_size=0,
)
model.save(base_dir)
del model

model = GPTQModel.load(
    base_dir,
    backend=BACKEND.GPTQ_TORCH,
    adapter=Lora(rank=64, path=adapter_dir),
)
tokens = model.generate("Capital of France is", max_new_tokens=32)[0]
print(model.tokenizer.decode(tokens))

The absolute adapter location in this example is intentionally loaded with adapter=.... It is not a portable checkpoint-relative manifest. Do not expect another machine to resolve it automatically from saved metadata.

To use a different EoRA distribution:

model.quantize(
    calibration=quantizer_calibration,
    adapter_calibration_dataset=eora_calibration,
    batch_size=1,
    calibration_concat_size=0,
)

These are input datasets, not labels or gradient-training batches. Make them representative of intended inference prompts.

Generate after quantization

GPTQModel.adapter.generate requires the original model, the quantized model, an output adapter path, and calibration data:

from gptqmodel import GPTQModel
from gptqmodel.adapter.adapter import Lora

GPTQModel.adapter.generate(
    adapter=Lora(rank=64, path="./post-quant-eora"),
    model_id_or_path="meta-llama/Llama-3.2-3B",
    quantized_model_id_or_path="./existing-gptq-base",
    calibration_dataset=calibration,
    calibration_dataset_concat_size=0,
    batch_size=1,
)

Here existing-gptq-base is a previously quantized base checkpoint without an active adapter descriptor. It is an alternative workflow to generating the adapter during quantization above. This method generates and saves the adapter; it returns without returning a model. At this revision it explicitly loads the quantized base using BACKEND.GPTQ_TORCH and finds TorchLinear modules. It is not a backend-independent post-hoc adapter generator for every format in GPT-QModel. The original and quantized checkpoints must represent corresponding modules and compatible dimensions.

The exact classmethod signature from gptqmodel/models/auto.py is:

def generate(
    cls,
    adapter: Adapter,
    model_id_or_path: str,
    quantized_model_id_or_path: str,
    calibration_dataset: Union[List[Dict[str, Union[List[int], torch.LongTensor]]], List[str], List[int]],
    calibration_dataset_concat_size: Optional[int] = None,
    calibration_dataset_sort: Optional[str] = None,
    batch_size: Optional[int] = 1,
    tokenizer: Optional[PreTrainedTokenizerBase] = None,
    calibration_concat_separator: Optional[str] = None,
    trust_remote_code: bool = False,
    dtype: Optional[Union[str, torch.dtype]] = None,
    device: Optional[Union[str, torch.device]] = None,
):

It only accepts Lora and validates the output path as local. Dataset concatenation/sorting parameters here use the calibration_dataset_* names shown in this signature, which differ from model.quantize's names.

Per-module rank overrides

EoRA clones the base adapter descriptor for each module and reads dynamic[pattern]["adapter"]["rank"]. A dynamic exclusion skips EoRA for the corresponding module as well.

qcfg = QuantizeConfig(
    bits=4,
    group_size=128,
    adapter=Lora(rank=64, path="./eora-adapter"),
    dynamic={
        r".*self_attn\.(q_proj|k_proj|v_proj)$": {"adapter": {"rank": 32}},
    },
)

See QuantizeConfig reference for dynamic-pattern precedence and exclusion syntax. Saved rank_pattern entries preserve per-module adapter ranks; the base rank remains in LoraConfig.r. When using these exports through external PEFT, also check per-module alpha scaling as described below: the writer does not emit matching alpha_pattern overrides.

Adapter configuration reference

Adapter, Lora, and the serialized descriptor

There is no separate class named AdapterConfig in this revision. The adapter configuration is QuantizeConfig.adapter, declared as Optional[Union[Dict[str, Any], Lora]] with default None, plus the saved PEFT-compatible LoraConfig. Distinguish the runtime descriptor, factor tensors, and adapter JSON rather than treating them as one configuration class.

class Adapter:
    def __init__(self, rank: int = None, path: str = None):
        ...

class Lora(Adapter):
    def __init__(self, rank: int, path: str = None,
                 lora_A: torch.Tensor = None, lora_B: torch.Tensor = None):
        ...

Lora parameter	Default	Meaning
rank	Required argument	Requested generation rank or expected saved base rank. Pass None explicitly when loading to adopt adapter_config.json's r.
path	None	Output directory during generation; existing local directory or explicitly authorized pinned Hub source during loading. Whitespace is stripped from string paths; path-like objects are converted to strings.
lora_A	None	Optional preloaded factor. Runtime apply uses (in_features, rank) orientation; generated/saved factors use PEFT's transposed orientation.
lora_B	None	Optional preloaded factor. Runtime apply uses (rank, out_features) orientation.

Lora.name() returns "lora"; Lora.parameter_keys() returns ["lora_A", "lora_B"]. to_dict() emits only name, path, and rank, not the factor tensors or remote pinning options.

The serialized form is accepted by normalize_adapter:

adapter = {"name": "lora", "path": "./eora-adapter", "rank": 64}
model = GPTQModel.load("./llama-gptq", adapter=adapter)

normalize_adapter accepts None, an existing Adapter, or a dictionary. It copies dictionary inputs, requires a recognized name, and rejects non-tensor values for supplied factor fields. The registered type map contains "lora": Lora.

PEFT-compatible LoraConfig

gptqmodel/adapter/peft.py defines GPT-QModel's own dataclass, separate from peft.LoraConfig. It reads/writes PEFT-compatible metadata; it is not a complete implementation of every PEFT training feature. The runtime Lora.apply only adds its two-matmul correction: it does not implement dropout, DoRA, extra bias terms, layer replication, or alpha scaling from this metadata.

Field	Type	Default	Purpose / limits
base_model_name_or_path	str	""	Associated base checkpoint location.
r	int	8	Saved base rank. This is not the rank-64 choice used in repository EoRA examples.
target_modules	Optional[Union[list[str], str]]	None	Module list or regex; lists normalize to sets.
exclude_modules	Optional[Union[list[str], str]]	None	Exclusions; lists normalize to sets.
lora_alpha	int	8	PEFT scaling metadata; EoRA writer sets it equal to r.
lora_dropout	float	0.0	PEFT dropout metadata.
fan_in_fan_out	bool	False	PEFT layout metadata.
bias	Literal["none", "all", "lora_only"]	"none"	PEFT bias mode metadata.
use_rslora	bool	False	PEFT rank-stabilized scaling option; restrictions below.
modules_to_save	Optional[list[str]]	None	Additional PEFT modules.
init_lora_weights	bool or initialization literal	True	Declared literals: "gaussian", "eva", "olora", "pissa", "pissa_niter_[number of iters]", "loftq"; some combinations are rejected below.
layers_to_transform	Optional[Union[list[int], int]]	None	Layer indices; cannot accompany regex target_modules.
layers_pattern	Optional[Union[list[str], str]]	None	Layer-name pattern; requires truthy layers_to_transform and cannot accompany regex target_modules.
rank_pattern	Optional[dict]	{} (new dictionary)	Per-module rank overrides.
alpha_pattern	Optional[dict]	{} (new dictionary)	PEFT per-module alpha metadata.
megatron_config	Optional[dict]	None	Megatron metadata.
megatron_core	Optional[str]	"megatron.core"	Megatron core module name.
loftq_config	Union[Any, dict]	{} (new dictionary)	Nonempty values are rejected.
eva_config	Optional[Any]	None	Must agree with init_lora_weights="eva"; not an EoRA generation control.
use_dora	bool	False	Combined with nonempty megatron_config, raises an error. Acceptance alone does not provide runtime DoRA execution.
layer_replication	Optional[list[tuple[int, int]]]	None	PEFT layer-replication metadata.
lora_bias	bool	False	True is rejected.
task_type	Optional[str]	"CAUSAL_LM"	Task metadata.
peft_type	Optional[str]	"LORA"	Post-init forces "LORA".
auto_mapping	Optional[dict]	None	Base-model mapping metadata.

Additional constructor validation:

• init_lora_weights="loftq" is rejected even with an empty loftq_config.
• init_lora_weights="eva" without eva_config, or an eva_config with another initialization choice, is rejected.
• use_rslora combined with nonempty rank/alpha patterns and PiSSA or OLoRA initialization is rejected.
• String target_modules cannot be combined with layers_to_transform or layers_pattern.

to_dict() removes None and empty list/dictionary values, converts sets to lists, and leaves ordinary scalar values. save_pretrained(save_dir) creates the directory and writes adapter_config.json. from_pretrained filters unknown and empty fields before constructing the dataclass:

@classmethod
def from_pretrained(
    cls,
    path: str,
    filename: str,
    *,
    allow_remote: bool = False,
    revision: Optional[str] = None,
):

Do not mistake accepted metadata for executable support in the minimal GPT-QModel adapter runtime. For an independently trained PEFT adapter with non-unit alpha scaling or other PEFT features, use the PEFT model-loading path described below rather than assuming Lora.apply reproduces those features.

Save, load, and correction lifecycle

File format and tensor orientation

Artifact	Contents
adapter_model.safetensors	EoRA factor tensors; metadata includes {"format": "pt"}.
adapter_config.json	LoraConfig metadata, base rank/alpha, target-module names, and dynamic rank patterns.
Base quantized checkpoint	Packed weights and its quantization configuration; adapter descriptor is separate metadata, not the factors themselves.

The writer lowercases module keys and uses:

base_model.model.<module_key>.lora_A.weight
base_model.model.<module_key>.lora_B.weight

Saved A has shape (rank, in_features) and saved B has shape (out_features, rank). Runtime loading transposes both to the matrix orientation used by x @ lora_A @ lora_B.

eora_save constructs LoraConfig with r=quantize_config.adapter.rank, lora_alpha=r, collected terminal module names as target_modules, and extracted dynamic rank patterns. Matching the base alpha and rank gives unit conventional LoRA scaling for modules using the base rank; the EoRA factors already encode the required correction.

For dynamically overridden ranks, the writer does not set alpha_pattern. An external PEFT runtime using conventional alpha/rank scaling can therefore rescale those modules differently: base r=64, lora_alpha=64, and a module rank of 32 imply a factor of 2, whereas GPT-QModel's Lora.apply uses unit scaling. Before using a variable-rank export with PEFT, make the per-module alpha equal its effective rank using PEFT-compatible patterns, and verify matching and tensor shapes. Do not assume format compatibility alone guarantees identical outputs.

Saving

EoraProcessor.finalize temporarily attaches generated adapters as model.lora_results. The model writer's save_quantized path calls eora_save, selecting an explicit eora_path override when provided, otherwise quantize_config.adapter.path. Adapter saving consumes and deletes lora_results after writing. It is not a reusable tensor cache for repeated independent saves.

Supply a real output directory. Historical example comments mentioning a default lora.safetensors do not describe the current directory-based writer. The two current standard filenames are the ones listed above.

Runtime loading

Lora.post_init(weight_key, device, ...):

1. Resolves the selected adapter source and validates remote pinning if present.
2. Loads LoraConfig and the safetensors file, caching them by (path, revision, digest).
3. Adopts the saved base rank when rank is None; an explicitly supplied different base rank raises an error.
4. Finds A/B keys by lowercase module-key suffix and transposes their tensors.
5. Removes consumed tensor entries from the cache and drops the cache entry once exhausted.
6. Applies a matching rank_pattern override. Matching checks the exact key, pattern.endswith(weight_key), then PCRE matching against the module key; the first match wins.
7. Moves factors to the module device. FP64 tensors are downcast to FP32; other non-FP16/BF16 dtypes produce a warning.

Passing both lora_A and lora_B directly to post_init is a separate early-return path: it assigns those tensors without file loading, transposition, rank checks, or device/dtype conversion. Such factors must already have runtime orientation and compatible shapes/dtypes/devices.

Missing A/B tensors fail an assertion. On forward, Lora.apply checks A against the input's dtype/device and moves both factors to match when A differs. Keep B compatible with A; there is no independent B-only mismatch check. It adds (x @ lora_A) @ lora_B in place to the output and handles the runtime's batched-output reshaping case. There is no extra multiplication by lora_alpha / r in this method. The optimize hook is currently a no-op.

path must identify a directory/source, not a direct .safetensors file. An explicit local directory is the simplest deployment option:

model = GPTQModel.load(
    "./llama-gptq",
    adapter=Lora(rank=None, path="./llama-eora"),
)

Checkpoint-relative manifests versus explicit external adapters

Adapter paths obtained from quantization metadata are treated as untrusted checkpoint-relative directory paths. They must already exist inside the checkpoint directory. Absolute paths, traversal, backslashes, drive prefixes, and symlinked directory components are rejected. The two standard files cannot link outside checkpoint storage, apart from the narrow allowance for the same Hugging Face repository cache's normal snapshot-to-blob links.

For an adapter outside the checkpoint, pass adapter= explicitly. Remote authorization flags supplied without an explicit adapter are rejected; checkpoint metadata cannot authorize a download. A saved external or absolute path is therefore not a portable automatic-loading manifest. Arrange checkpoint-relative files/metadata deliberately or load the external adapter explicitly.

Remote adapters: explicit opt-in and immutable pinning

Load option	Default	Required behavior
adapter	None	Must be explicitly supplied for remote loading.
allow_remote_adapter	False	Must be True to authorize a remote adapter.
adapter_revision	None	Exactly 40 hexadecimal characters: immutable Hub commit.
adapter_sha256	None	Exactly 64 hexadecimal characters: SHA-256 of adapter_model.safetensors.

model = GPTQModel.load(
    "./llama-gptq",
    adapter=Lora(rank=None, path="organization/adapter-repository"),
    allow_remote_adapter=True,
    adapter_revision=verified_commit_sha,  # 40 hexadecimal characters
    adapter_sha256=verified_weights_sha256,  # 64 hexadecimal characters
)

Obtain the expected digest and revision from a trusted source. The digest covers the weights file; adapter_config.json is loaded from the same pinned revision. Supplying only one pin is rejected. Remote options cannot be used with a local adapter directory.

The resolver supports a Hub repository ID, a nested adapter folder after the repository ID, or an HTTPS Hugging Face blob URL on huggingface.co/hf.co. URL revisions must equal the explicit revision. Arbitrary HTTP hosts and mutable revision names are not accepted. Remote path segments reject empty, ./.., backslashes, null characters, and drive fragments.

Older repository README examples that load a Hub adapter solely by Lora(path="repo", ...) need these explicit authorization and pinning options at this revision.

What “fusing” or “merging” means here

Operation	Behavior
Correction during calibration replay	EoraProcessor.process temporarily sets a floating-point module weight to Wq + B @ A so subsequent calibration sees the corrected output. It retains the uncorrected wq separately and stores A/B for export.
Runtime adapter addition	The quantized projection and low-rank correction are evaluated separately; packed base weights remain packed.
Dense LoRA merge	Add the correction to a dense floating-point base. This is not the same operation as writing into packed integer weights.
PEFT merge_and_unload() on packed GPTQ/AWQ	Unsupported by the documented PEFT wrappers: GPTQ raises ValueError; AWQ raises NotImplementedError.

Temporary corrected dense weights inside calibration do not mean the deployed packed checkpoint has absorbed the adapter. For a standalone merged checkpoint, the documented PEFT workflow merges into the original floating-point model, saves it, and quantizes that merged model again. Requantization can introduce new error; it is not bitwise equivalent to retaining the original quantized base plus its separately evaluated adapter.

Processor and package internals

gptqmodel/eora/__init__.py re-exports eora.py. The package's computational functions are:

Function	Role
eora_process_input(input, name, sample_size, device)	Return (batch_count, FP32 contribution, scale_factor).
merge_eora_segments(segments)	Combine device-local totals and scale products; requires at least one segment.
eora_compute_lora(w_wq_delta, name, eigen_scaling_diag_matrix, rank, dtype, device)	Return (A, B) from eigenspace-weighted residual SVD.

The accumulation APIs are:

def eora_process_input(
        input: Tensor,
        name: str,
        sample_size: int,
        device: torch.device,
) -> tuple[int, torch.Tensor, float]:

def merge_eora_segments(segments: Sequence[tuple[torch.Tensor, float]]) -> torch.Tensor:

Despite the input: Tensor annotation, the processor passes the forward-hook input tuple; the function indexes input[0]. Use a positive sample_size and compatible three-dimensional activation data (or two-dimensional data that the helper promotes to three dimensions). The helper does not validate a zero denominator or normalize arbitrary higher-dimensional inputs.

EoraProcessor is registered/logged as "eora". Its lifecycle is:

1. Record calibration-batch count and initialize per-module/per-device accumulators.
2. Copy the configured Lora, apply dynamic rank overrides, and skip dynamically excluded modules.
3. Collect forward-hook statistics, retaining matrix accumulation on each originating device.
4. Merge segments for one module and consume its FP32 w_wq_diff from the preceding quantization/dequantization processor.
5. Compute A/B, update the floating-point module to its corrected replay weight, and retain the base wq separately. Tensor-parallel padding metadata is used to trim replay weights back to the original columns.
6. Store generated A/B on CPU under module.state["adapter"]; finalize each module into the processor result map.
7. Release accumulation maps and attach lora_results to the model for the writer.

The processor requires collected statistics before processing. Its continuation-state serializer records rank, path, and tensors explicitly rather than pickling adapter instances or runtime kernels. It raises the Torch Dynamo cache-size limit to at least 64 during initialization.

In normal quantization, models/base.py appends EoraProcessor after the selected quantization processor when a Lora adapter is requested. In post-quantization generation, a DequantizeProcessor first makes the quantized residual available, followed by EoraProcessor; configured preprocessors may run before those stages. This ordering is why EoRA needs the matching original model rather than only a low-bit checkpoint.

Device and precision requirements

Area	Requirement / behavior
Factorization	Torch backend must support FP64 eigh and FP32 reduced SVD on the selected device; there is no native MLX implementation in gptqmodel/eora/.
ROCm	Before Torch 2.10, the code temporarily selects the magma preferred linear-algebra backend and restores it after successful factorization. Torch 2.10+ skips that workaround.
Processor replay weights	The implementation asserts corrected weights are FP16 or BF16. Factors are generated in the module dtype.
Runtime	Select a quantized linear backend that advertises Lora support. The adapter does not make unsupported base formats/backends compatible.
Metadata/files	JSON configuration, safetensors tensor storage, PCRE rank-pattern matching, and Hub resolution for authorized remote sources.

Torch GPTQ, Torch AWQ, Marlin, and the ParoQuant Torch/Triton runtime classes declare Lora support at this revision. Their hardware and dtype requirements still apply independently; consult Kernel backends reference. In ParoQuant, the adapter receives the original, unrotated input.

The scaling matrix is quadratic in input width, and its eigendecomposition plus the residual SVD can dominate generation memory/time. EoRA avoids an optimizer loop, not linear-algebra cost. Calibration sequence lengths and per-module widths still matter.

PEFT training and loading

EoRA's generated adapter and a gradient-trained PEFT LoRA share a file convention, but they are different workflows. docs/peft.md documents training on a frozen GPTQ/AWQ base using Transformers and PEFT.

pip install "gptqmodel[hf]" peft

The documented integration requires optimum>=2.3.0 and a current PEFT release. Its FSDP recipe records validation with PEFT 0.21.3.dev0, Optimum 2.3.0, Transformers 5.20.2, and Torch 2.14; these are recorded recipe versions, not universal minimum versions for EoRA.

Select a trainable backend

Base quantization	Transformers-facing backend
GPTQ	"gptq_torch"
AWQ	"torch_awq" (alias for GPT-QModel's AWQ_TORCH)

The PEFT training recipe must not use ordinary inference-only kernels that do not propagate gradients through the quantized layer.

from peft import LoraConfig, get_peft_model
from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained(quant_path)
method = config.quantization_config["quant_method"]
config.quantization_config["backend"] = {
    "gptq": "gptq_torch", "awq": "torch_awq",
}[method]
base = AutoModelForCausalLM.from_pretrained(
    quant_path, config=config, device_map="auto",
)
model = get_peft_model(
    base,
    LoraConfig(
        r=16, lora_alpha=32,
        target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM",
    ),
)
# Run the training loop, then save only the adapter.
model.save_pretrained(adapter_path)

For PEFT-managed inference, including its scaling and wrapper behavior:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained(quant_path, device_map="auto")
model = PeftModel.from_pretrained(base, adapter_path)

For merging, load the original dense base instead, attach the adapter with PEFT, call merge_and_unload(), save the dense checkpoint, then quantize it using the normal workflow. Do not call merge on the packed GPTQ/AWQ base.

FSDP recipe

The checked-in docs/peft/fsdp_config.yaml uses one machine, four processes, BF16 mixed precision, full sharding, transformer-based wrapping, CPU-RAM-efficient loading, synchronized module states, no parameter offload, and fsdp_use_orig_params: false. The documentation records validation on four RTX 4090 GPUs with an 8B GPTQ Llama checkpoint.

Use PEFT's examples/sft dependencies and script, plus the documented integration changes:

• Set the GPTQ backend to gptq_torch before loading the base.
• Set trainer.accelerator.state.fsdp_plugin.auto_wrap_policy using peft.utils.other.fsdp_auto_wrap_policy(trainer.model) after creating SFTTrainer and before training.
• At save time request FULL_STATE_DICT, gather the full state, and on the world-process-zero rank select LoRA A/B tensors, move them to contiguous CPU storage, and replace .default.weight with .weight before writing adapter_model.safetensors.
• Keep --use_4bit_quantization False for an already GPTQ-quantized checkpoint: that flag enables a different bitsandbytes QLoRA path.

The exact launch recipe and save snippet are maintained in docs/peft.md. Lower sequence length when memory is limited; the documented recipe uses batch size one per GPU.

Repository examples and evaluation

File under docs/eora/	Role / actual choices
README.md	Overview, task-specific calibration motivation, generation/inference/evaluation examples, and paper reference.
eora_generation.py	Quantization plus EoRA. CLI defaults: bits 4, group size 128, rank 64. Always uses C4 for base quantization; EoRA can use C4 or MMLU. Sets forward batch size 1, desc_act=True, and concatenation size 0.
post_quant_eora_generation.py	Calls GPTQModel.adapter.generate; positional dataset choices are c4, arc, mmlu. Explicitly supply quantized checkpoint, adapter output path, and rank.
eora_load_and_inference.py	Loads an explicit adapter using BACKEND.TORCH and generates a short completion; requires the adapter argument.
eora_calibration_data_construction.py	C4 text and task-plus-general calibration constructors.
evaluation.py	Compares a base or adapter-equipped checkpoint on ARC Challenge and/or MMLU; imports checkout tooling from tests.eval.

construct_c4() selects 1,024 texts from the named English C4 training shard and forces redownload. construct_ARC() builds question/answer text from 1,024 ARC-Easy and 1,024 ARC-Challenge training examples, then appends 1,024 C4 texts. construct_mmlu() formats the MMLU validation split with answer letters and appends 1,024 C4 texts. These are example sampling policies, not mandatory EoRA defaults or unbiased held-out evaluation sets.

The factorization docstring recommends mixing general and downstream-task input when expanding calibration. Avoid using evaluation answers as calibration data for the same benchmark you intend to report as held-out accuracy.

Example post-quantization command from the checkout:

python docs/eora/post_quant_eora_generation.py meta-llama/Llama-3.2-3B c4 \
  --quantized_model ./existing-gptq-base \
  --eora_save_path ./post-quant-eora \
  --eora_rank 64

Run paired evaluations with otherwise identical settings:

python docs/eora/evaluation.py --quantized_model ./existing-gptq-base --eval_task all
python docs/eora/evaluation.py --quantized_model ./existing-gptq-base \
  --eora_save_path ./post-quant-eora --eora_rank 64 --eval_task all

The evaluation script's task choices are "mmlu", "arc", and "all" (default); "arc" evaluates arc_challenge. It passes the adapter when a path is supplied and adapter=None otherwise. Use a base checkpoint without an adapter descriptor for the unadapted baseline: adapter=None does not disable an adapter that the loader discovers in checkpoint metadata. Use Evaluation for broader evaluation setup and methodology.

Pitfalls and edge cases

• EoRA needs the original and quantized weights' difference. An unrelated floating-point checkpoint is not a valid substitute.
• Post-hoc generation is tied to the GPTQ Torch module discovery path at this revision; adapter application support on another backend does not imply post-hoc generation support for that backend.
• Covariance rank can be much lower than feature width, especially with limited calibration. The implementation discards numerically insignificant directions instead of amplifying them with an unstable inverse.
• Rank is not a percentage and not a quantization group size. It changes the size of floating-point A/B factors and their runtime cost.
• Choose a rank compatible with module dimensions. Oversized ranks do not create additional components, and saved metadata can then misrepresent effective tensor rank.
• When loading, omit rank checking by explicitly passing rank=None, or supply the saved base rank; dynamic rank overrides are applied afterward.
• Generated/saved A/B orientation differs from runtime orientation. Do not pass saved matrices directly to runtime apply without the transpose performed by normal loading.
• EoRA export uses the base lora_alpha=r without per-module alpha overrides. Variable-rank exports need special attention when loading through external PEFT. GPT-QModel's minimal runtime does not reproduce arbitrary PEFT alpha/dropout/DoRA semantics merely because a JSON config contains those fields.
• Saving consumes lora_results; do not assume a second adapter export from the same consumed result state will work.
• Supply a directory, not adapter_model.safetensors, as Lora.path.
• Explicit external adapters and checkpoint-relative manifests have different trust/path rules. Do not disable those restrictions to make a stale path load.
• Remote loading needs caller opt-in, immutable revision, and weights digest. trust_remote_code is not remote-adapter authorization.
• Temporary replay fusion is not packed-weight merging. Keep the adapter with the deployed checkpoint unless you deliberately merge into a dense base and requantize.

Source files

Paths are relative to the repository:

• gptqmodel/eora/__init__.py — package exports.
• gptqmodel/eora/eora.py — statistics, segment merging, eigen/SVD factorization.
• gptqmodel/looper/eora_processor.py — calibration hooks, dynamic rank, correction installation.
• gptqmodel/looper/dequantize_processor.py — post-quantization residual preparation.
• gptqmodel/adapter/adapter.py — adapter interface, Lora, cache, normalization/loading.
• gptqmodel/adapter/peft.py — PEFT-compatible metadata and JSON I/O.
• gptqmodel/adapter/remote.py — checkpoint path confinement, pinning, downloads, digest validation.
• gptqmodel/quantization/config.py — adapter field and dynamic rank-pattern extraction.
• gptqmodel/models/auto.py — public post-quantization adapter-generation API.
• gptqmodel/models/base.py — processor ordering and generation loop.
• gptqmodel/models/loader.py — explicit versus manifest adapter selection.
• gptqmodel/models/writer.py — EoRA factors/config saving.
• gptqmodel/nn_modules/qlinear/torch.py, torch_awq.py, marlin.py — adapter-capable runtimes.
• gptqmodel/nn_modules/qlinear/paroquant.py, paroquant_triton.py — original-input ParoQuant adapter application.
• docs/eora/README.md and its five Python example files listed above — calibration, generation, inference, evaluation.
• docs/peft.md — PEFT training/loading/merge and FSDP integration recipe.
• docs/peft/fsdp_config.yaml — four-process FSDP configuration.

See also

• GPTQ
• AWQ
• ParoQuant
• QuantizeConfig reference
• Quantization lifecycle
• Looper processors
• Loading and inference
• Kernel backends reference
• Hugging Face integration
• Evaluation
• packing formats and compatibility
• Troubleshooting and FAQ