A repository for learning and experimenting with Lora (Low-Rank Adaptation) in Python.
LoRA (Low-Rank Adaptation) is a technique for efficiently fine-tuning large language models and other neural networks. It works by freezing the pre-trained model weights and injecting trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.
LoraSetup/
├── data/ # Training and evaluation datasets
├── models/ # Pre-trained models and fine-tuned Lora adapters
├── notebooks/ # Jupyter notebooks for experiments
├── scripts/ # Training and inference scripts
├── src/ # Source code
│ ├── data/ # Data processing utilities
│ ├── models/ # Model definitions
│ ├── training/ # Training utilities
│ └── utils/ # Miscellaneous utilities
├── configs/ # Configuration files
├── requirements.txt # Python dependencies
└── README.md # This file
- Python 3.8+
- PyTorch 1.10+
- Transformers library
- CUDA-capable GPU (recommended)
-
Clone this repository:
git clone https://github.com/KiranChilledOut/LoraSetup.git cd LoraSetup -
Create a virtual environment:
python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate
-
Install dependencies:
pip install -r requirements.txt
Detailed usage instructions will be added as the project develops.
This project is licensed under the MIT License - see the LICENSE file for details.