Home

DistilBERT Fine-Tuning — Project Wiki

**Practical Transformer sequence classification with Hugging Face Transformers and PyTorch. **** This Wiki documents the code in Full_Fine_Tuning_Using_DistilBERT.py and DistilBERT_Colab_Fixed_2026.ipynb.

What the project actually implements

A four-class customer-support text classifier, initialized from distilbert-base-uncased, with two experiments:

1. Full fine-tuning updates DistilBERT and its classification head.
2. Frozen-backbone training freezes the encoder and trains the classification head.

It uses **30 hard-coded sample tickets or examples **, a 24/6 stratified train/validation split, 128-token truncation, and Hugging Face Trainer for both runs. It produces accuracy and weighted F1; the full fine-tuned model also produces a classification report, a confusion matrix, predictions and saved model files.

Scope clarification: Despite the source archive title, the project does not implement pre-training, mid-training, causal language-model training, PPO or DPO. This is supervised fine-tuning of an encoder-based classifier. It is a production-grade demo, not a validated production support-routing system. In the next iteration, a small, GPT-style decoder-only LLM that fits on a consumer-grade GPU will be used, and Pre-training and mid-training will be performed.

Start here

• Overview — capabilities, technology, and boundaries
• High-level system design — data flow and training paths
• Low-level design — functions, framework components, call graph
• Model components — DistilBERT classifier and Trainer dependencies
• Training comparison — parameters, schedules, and accuracy limitations
• Setup — Python/Conda, dependencies, optional GPU
• Run locally — executable commands and outputs
• Notebook guide — notebook filename and Colab steps
• Testing and caveats — verified static checks vs unrun GPU training
• Publishing — code repository versus separate GitHub Wiki

Training flow

flowchart TD
    A["30 labeled support tickets"] --> B["Stratified split: 24 train / 6 validation"]
    B --> C["Hugging Face DatasetDict"]
    C --> D["DistilBERT tokenizer: max 128 tokens"]
    D --> E["Dynamic padding: DataCollatorWithPadding"]
    E --> F{"Experiment"}
    F --> G["Full fine-tune: encoder + head"]
    F --> H["Frozen encoder: train head only"]
    G --> I["Trainer: 4 epochs / 2e-5"]
    H --> J["Trainer: 6 epochs / 1e-3"]
    I --> K["Accuracy + weighted F1"]
    J --> L["Accuracy + weighted F1"]
    I --> M["Best checkpoint and tokenizer"]
    M --> N["classify_query(): label + scores"]
    K --> O["Educational comparison"]
    L --> O

Loading

Project files

• Full_Fine_Tuning_Using_DistilBERT.py: original Python implementation (unchanged)
• DistilBERT_Colab_Fixed_2026.ipynb: original Jupyter notebook (unchanged)
• requirements.txt: original dependency list (unchanged; contains extras for unrelated workflows)
• requirements-classifier.txt: focused dependencies for this example
• original/README.md: unchanged upstream README
• wiki/*.md: these GitHub Wiki pages
• docs/mermaid/*.mmd: editable Mermaid sources

The Wiki is technical documentation; it does not contain a trained model or benchmark results.