-
Pruning : Utility to prune low-magnitude weights in a layer.
-
Probalistic Quantization : A demo tool to implement probabilistic quantization of weights to keep the weight statistics unbiased.
-
K-Means Quantization : Idea introduced in Deep Compression paper to reduce number of unique weights to be stored for a NN model.
-
Gradient Traceback : Print all gradients in computation graph via recursive backtracing.
-
Computational Graph Visualization : Visualizes the PyTorch computation graph onto a PDF file. It does not require Tensorboard but requires GraphViz which can be installed with
sudo apt install graphviz. -
Memory Profiling with PyTorch Hooks : Useful in optimizing models, can help visualize memory usage during checkpointed training as well (Note: This is is not working with latest PyTorch update).
-
Unit test to verify parameter behavior during training : Useful while training GANs or during transfer learning where only a certain subset of parameters need to be tuned during training.
-
HDF5 Weights Import Tool : Import .h5 weight files into PyTorch models and export them back.
-
DataLoader with Cache : Implements a caching mechanism in the dataloader, so that items once fetched and tranformed, from the dataset are stored in memory and are not re-processed by the dataloader. If memory limitations remain, an LRU cache (using an OrderedDict) may be used instead of a full array.
-
Knowledge Distillation : Template for knowledge distillation training. Used in Parallel WaveNet training, among other to reduce model size. Useful only for models with softmax outputs. Anneal temperature as training progresses for stable gradients.