Repository navigation
Frozen Backbone Training
Sudhanshu edited this page Nov 11, 2026
·
1 revision
The comparison model is independently loaded from the same pretrained checkpoint, then the source performs:
for parameter in model_frozen.distilbert.parameters():
parameter.requires_grad = FalseThe pre_classifier and final classifier head remain trainable. The script prints the number and fraction of trainable parameters.
| Parameter | Source value |
|---|---|
| Epochs | 6 |
| Train/eval batch size | 8 |
| Learning rate | 1e-3 |
| Warmup fraction | 0.10 |
| Evaluation | Every epoch |
| Checkpoint saving | Disabled (save_strategy="no") |
| Output directory | ./support_classifier_frozen |
Because the configurations differ (epochs and learning rate) and the dataset is tiny, comparisons must remain illustrative. The source does not guarantee a controlled production-grade benchmark.
flowchart LR
M["Pretrained DistilBERT checkpoint"] --> A["Full fine-tune model"]
M --> B["Frozen-backbone model"]
A --> A1["distilbert: trainable"]
A --> A2["pre_classifier + classifier: trainable"]
B --> B1["distilbert: frozen"]
B --> B2["pre_classifier + classifier: trainable"]
A1 --> C["4 epochs, LR 2e-5"]
A2 --> C
B1 --> D["6 epochs, LR 1e-3"]
B2 --> D
C --> E["Accuracy / weighted F1 comparison"]
D --> E
Getting started
Architecture
- System Context
- High-Level Design
- Low-Level Design
- Model Components
- Training Sequence
- Data & Labels
- Tokenization
Training & results
Engineering reference