Repository navigation
loraspectra: Visualizing & profiling LoRA singular value spectra, effective rank (erank), and QR-core SVD compression #1120
SalyyS1
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi everyone,
For anyone exploring Appendix E (Parameter-Efficient Fine-Tuning with LoRA) in LLMs-from-scratch and curious about what the learned low-rank matrices
B (d_out x r)andA (r x d_in)actually look like spectrally across transformer layers:We built loraspectra, an open-source, zero-dependency spectral analyzer, Shannon effective rank (
erank) profiler, Eckart-Young SVD rank pruner, and multi-adapter subspace conflict detector for.safetensorsLoRA/DoRA adapters:Mathematical highlights
B @ A:Instead of computing the dense
d_out x d_inproductdelta_W = B @ A(O(d^3)SVD),loraspectracomputes thin QR factorizationsB = Q_B R_BandA^T = Q_A R_Aand runs SVD on the tinyr x rcoreM = R_B R_A^T. This matches full dense SVD singular values to machine precision (1e-13) while running 1,000x to 2,600x faster on CPU.erank) & Energy-Optimal Rank Pruning:Early attention projections (
q_proj,k_proj) frequently concentrate>99%of their Frobenius energy into just 2 to 6 singular directions even when trained atr = 32orr = 64.loraspectra prune --energy 0.99compresses each layer to its exact Eckart-Young rankk_land emits a Hugging Face PEFTrank_pattern.ortho-svd):Measures principal subspace overlap between multiple task adapters and projects secondary adapters orthogonal to primary task directions before merging.
Repository and mathematical benchmarks: https://github.com/SalyyS1/loraspectra
All reactions