This project focuses on fine-tuning a small language model, Qwen2.5-0.5B-Instruct, for text summarization using a dataset of 5000 French Wikipedia articles. Initial summaries were generated with the larger Qwen2.5-14B-Instruct model, then used to fine-tune the 0.5B model via LoRA on limited resources (Google Colab free tier). The process showcases knowledge distillation from a large to a small model, improving summarization performance efficiently.
Thanks to Yann Millet for the help on the project!
fr_wiki_articles_5000.json: Raw dataset of 5000 French Wikipedia articles, containing titles and full text, used as the input for summarization.fr_wiki_annotated_5000.json: Annotated dataset with article titles, original text, and summaries generated by Qwen2.5-14B-Instruct, serving as the fine-tuning source.fr_wiki_train.json: Training set (4000 articles, 80%) split from the annotated dataset for fine-tuning the 0.5B model.fr_wiki_validation.json: Validation set (500 articles, 10%) used to monitor fine-tuning progress and prevent overfitting.fr_wiki_test.json: Test set (500 articles, 10%) for evaluating the fine-tuned model’s summarization performance.fineTuneLLMSummarization_FINAL.ipynb: Jupyter notebook containing the complete code—summary generation, data splitting, fine-tuning, and evaluation (ROUGE scores).loss.csv: CSV file logging training and validation loss over 3000 steps during fine-tuning, showing model convergence.qwen2.5-0.5b-finetuned-final/: Directory with the fine-tuned Qwen2.5-0.5B-Instruct model and tokenizer, ready for inference.