Hands-on Discussion: Chapter 4 - Building smaller and faster LLMs with depth pruning

[peremartra]

Discussion

This is the space to share your findings from the experiments in Chapter 4. In this lab, we transitioned from basic structural modifications to data-driven block selection, analyzing how specific datasets and heuristic protections affect the final performance of the pruned model.

We encourage you to experiment with the notebook and share your concrete metrics. Identifying the exact point where a model degrades or optimizing a PyTorch hook for memory efficiency are the kinds of practical insights that translate directly to production environments.

🎯 The Challenges

Pick one (or more) of the experiments and post your results below:

• Static vs. Data-Driven: How did the static magnitude-based pruning (removing blocks [8, 2, 7]) compare against the activation-based selection for your specific task?
• Heuristic Impact: What exact blocks were selected when you disabled heuristic_protection or adjacent_protection? Did the absence of these safety flags cause a catastrophic drop in metrics, or were they overly cautious for your dataset?
• The Breaking Point: Share your progressive degradation curve. At exactly how many removed blocks did the perplexity stop scaling linearly and begin to degrade exponentially?
• Hook Optimization: How much memory (VRAM) or computation time did you save by calculating the importance scores directly inside the forward hook rather than storing the tensors?
• Scaling Up (1.7B): If you ran the code on Qwen3-1.7B, did the importance patterns change? Were the "passive" blocks still clustered in the same architectural region as they were in the 0.6B model?

📊 Share Your Results

To keep the discussion organized and technically focused, please use this template for your comments:

• Model analyzed: [e.g., Qwen3-0.6B / Qwen3-1.7B]
• Challenge attempted: [e.g., Progressive Degradation / Hook Optimization]
• Blocks Removed: [e.g., 4 blocks, indices: 2, 5, 7, 8]
• Results / Metrics: [e.g., Perplexity increased from X to Y / VRAM usage dropped by Z GB]
• Key Insight: [What specific engineering trade-off did you observe?]

Don't hesitate to ask any questions, share your code modifications, or discuss any out-of-memory (OOM) errors you encountered and how you bypassed them.