Hands-on Discussion: Chapter 2 - Depth Pruning & Knowledge Recovery

[peremartra]

Welcome to the Chapter 2 Hands-on Lab! 🚀

This is the space to share and discuss your findings from the experiments in Chapter 2. In this lab, we explored how depth pruning affects model performance and how much of that knowledge we can recover through distillation.

Don't just run the code—experiment! Building an intuition for these trade-offs is what separates a practitioner from an expert.

🎯 The Challenges

We encourage you to try at least one of these and post your results below:

1. Aggressive Pruning: What happens if you remove 6 or 8 layers instead of 4? How much RECOVERY_SAMPLES do you need then?
2. Data Efficiency: Can you achieve >90% recovery using only 5,000 samples?
3. Layer Strategy: Does removing the first layers impact the model more than the later ones?
4. Domain Specialization: Have you tried a custom dataset for distillation?

📊 Share Your Results

To keep the discussion organized, please use this template for your comments:

Configuration:

• Layers Removed: [e.g., 6 layers]
• Selection Method: [e.g., Last layers / Specific indices: 2, 5, 8]
• Recovery Samples: [e.g., 15,000]
• Distillation Dataset: [e.g., WikiText / Custom]

Results:

• Accuracy/Loss Recovery: [% vs Teacher model]
• Key Insight: [What did you learn from this specific run?]

Don't hesitate to ask any questions, share your results, or talk about the challenges you faced!