Qwen LoRA

Qwen2.5-0.5B-Instruct LoRA Fine-tuning

Qwen mode downloads pretrained weights and tokenizer from Qwen/Qwen2.5-0.5B-Instruct. It is adapter fine-tuning, not pretraining the model from scratch.

flowchart TD
    Text["Training examples"] --> Tok["AutoTokenizer: Qwen2.5-0.5B-Instruct"]
    Tok --> Batch["input_ids, attention_mask, labels"]
    HF["Pretrained Qwen2.5-0.5B-Instruct"] --> PEFT["PEFT get_peft_model"]
    LoRA["LoRA r=8, alpha=16, dropout=0.05, q_proj and v_proj"] --> PEFT
    PEFT --> Model["Qwen + trainable LoRA adapters"]
    Batch --> Loss["Causal LM loss on non-pad tokens"]
    Model --> Loss
    Loss --> DDP["DDP across ranks if world_size > 1"]
    DDP --> Opt["AdamW updates adapter parameters"]
    Opt --> Save["Rank 0: save_pretrained adapter + tokenizer"]

Loading

Implemented settings

Parameter	Value
Base model	Qwen/Qwen2.5-0.5B-Instruct
Task	causal language modeling
PEFT LoRA rank	r=8
LoRA alpha	16
LoRA dropout	0.05
Target modules	q_proj, v_proj
Default optimizer	AdamW, lr 2e-4
Precision in code	float16 when CUDA available, otherwise float32
GradScaler	enabled for Qwen on CUDA

The code sets model.config.use_cache=False, uses right padding, and masks padded labels. It does not implement 4-bit QLoRA, response-only masking, mixed-precision autocast contexts, validation, or evaluation metrics. Test package compatibility and GPU memory before larger runs.

The output directory contains LoRA adapter files and tokenizer assets saved by rank 0, not a merged standalone copy of the full pretrained base model.

See Installation and Checkpoints.