Skip to content

Frozen Backbone Training

Sudhanshu edited this page Nov 11, 2026 · 1 revision

Frozen-backbone head training

The comparison model is independently loaded from the same pretrained checkpoint, then the source performs:

for parameter in model_frozen.distilbert.parameters():
    parameter.requires_grad = False

The pre_classifier and final classifier head remain trainable. The script prints the number and fraction of trainable parameters.

Parameter Source value
Epochs 6
Train/eval batch size 8
Learning rate 1e-3
Warmup fraction 0.10
Evaluation Every epoch
Checkpoint saving Disabled (save_strategy="no")
Output directory ./support_classifier_frozen

Because the configurations differ (epochs and learning rate) and the dataset is tiny, comparisons must remain illustrative. The source does not guarantee a controlled production-grade benchmark.

flowchart LR
    M["Pretrained DistilBERT checkpoint"] --> A["Full fine-tune model"]
    M --> B["Frozen-backbone model"]
    A --> A1["distilbert: trainable"]
    A --> A2["pre_classifier + classifier: trainable"]
    B --> B1["distilbert: frozen"]
    B --> B2["pre_classifier + classifier: trainable"]
    A1 --> C["4 epochs, LR 2e-5"]
    A2 --> C
    B1 --> D["6 epochs, LR 1e-3"]
    B2 --> D
    C --> E["Accuracy / weighted F1 comparison"]
    D --> E
Loading

Clone this wiki locally