Repository navigation
loraspectra: Audit spectral rank, detect dead layers, compress, and orthogonally merge LLaMA-Factory LoRA/DoRA .safetensors in Rust #10969
SalyyS1
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi everyone,
When training LoRA / DoRA adapters with LLaMA-Factory (on Qwen2.5/3, Llama-3, Gemma-3, or multimodal VLMs like Qwen2.5-VL and MiniCPM-o), we often want to:
DEAD_LAYERwhereR_EFF < 1.5) without spinning up a GPU.K_99%or Marchenko-PasturK_MP) and export a PEFT-compatiblerank_pattern/alpha_pattern(adapter_config.json).ortho-svd) instead of destructive linear weight averaging.We built loraspectra, a zero-dependency Rust CLI that runs exact$r imes r$ QR-SVD directly on
.safetensorsfiles in milliseconds on CPU.Quick Commands
Repository: https://github.com/SalyyS1/loraspectra
If this helps you inspect, compress, or merge LLaMA-Factory LoRA checkpoints, please consider starring SalyyS1/loraspectra!
All reactions