Description
Add support for gradually increasing sparsity during training, rather than learning it all at once.
Motivation
Progressive pruning can lead to better final accuracy by allowing the network to adapt gradually to increasing sparsity.
Proposed Implementation
class SparsityScheduler:
def __init__(self, initial_sparsity=0.0, final_sparsity=0.9,
warmup_epochs=10, pruning_epochs=50):
...
def step(self, epoch):
# Update target sparsity for all L0 gates
...
# Usage
scheduler = SparsityScheduler(0.0, 0.9, 10, 50)
for epoch in range(100):
scheduler.step(epoch)
train_epoch()
Features
- Linear/cosine/exponential schedules
- Per-layer sparsity targets
- Magnitude-based guidance
Description
Add support for gradually increasing sparsity during training, rather than learning it all at once.
Motivation
Progressive pruning can lead to better final accuracy by allowing the network to adapt gradually to increasing sparsity.
Proposed Implementation
Features