LoRA and Quantization

LoRA and Quantization (Design-Stage)

The unfinished Phi-2 script configures BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4', bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True) and PEFT LoRA with rank r=16, lora_alpha=32, dropout 0.05, and target modules q_proj, k_proj, v_proj, dense.

These settings are present in the source but not validated. The target module names must be verified against the actual Phi-2 model before running. Quantized training needs a supported PyTorch/CUDA/bitsandbytes environment and sufficient GPU memory. A 4 GB consumer GPU may be insufficient.

Concept

• 4-bit quantization reduces storage for base weights; it does not mean every training tensor uses 4 bits.
• LoRA trains low-rank updates while most base-model weights remain frozen.
• SFT learns from instruction/response examples; DPO is preference optimization using chosen/rejected responses, typically with a reference policy.

This repository does not contain a measured speedup, memory benchmark or published trained LoRA adapter.