Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Relative Weight Change: Understanding how Neural Networks Learn

Made With python 3.7 Maintenance

This repository contains the original PyTorch implementation of the paper Investigating Learning in Deep Neural Networks using Layer-Wise Weight Change by Ayush Manish Agrawal, Atharva Tendle, Harsh Sikka, Sahib Singh and Amr Kayid.


pip install -r requirements.txt

How to run the code ?

Using dataset/architectures provided in the repository


Update the configs.json file to adjust the hyperparameters:

  • epochs : Number of epochs to train the model for
    • Default : 150
  • seed_list : This initializes different weights for each experiment. We average over the results to determine a general learning trend
    • Options : Length of list defines the number of different experiments for a given arch and dataset
    • Default : [0, 42, 123, 1000, 1234]
  • model_name : The architecture to use. Architectures included in the code are
    • Options : AlexNet, VGG19, ResNet18, Xception
    • Default : AlexNet
  • dataset : Dataset to use for training. Datasets included in the code are
    • Options : CIFAR-10, CIFAR-100, FMNIST, MNIST
    • Default : CIFAR-10
  • lr : The learning rate to use for the experiments. We have not used Adaptive learning rate for the simplicity in interpreting the trends
    • Default : 0.001
  • momentum : Used for the optimizer
    • Default : 0.9
  • weight_decay : Used for the optimizer
    • Default : 1e-4
  • batch_size : The number of images per batch in the training dataset
    • Default : 128
  • target_val_acc : Used for early stopping\
    • Default : 94%

Using datasets/architectures that are not included with this repository:

  • Adding a new architecture : - Add a new file for your new_model by going to /models/ - IMPORTANT Make sure that your model class has input_channels and num_classes are added as parameters. - Now, locate to /utils/ and go to line 26 where it says # pick model. Add you model in a conditional as elif model_name == "NewModel": model = new_model(input_channels = configs.input_channels, num_classes = configs.num_classes) - Remember to pass the NewModel in the configs.json file.
  • Adding a new dataset : - Locate to /utils/ and create a new function def load_my_new_model(configs) to load your own dataset. - Now, in the same file, go to line 8 and find the function load_dataset. - Add a conditional to load your model elif configs.dataset == "NewDataset": return load_my_new_model(configs) - Remember to add NewDataset in the configs.json before running


  • This file contains our new proposed Relative Weight Change metric to determine the layer wise learning trends in deep networks.

Repository Structure

├── models
│   ├──
│   │   
│   ├──
│   │  
│   └──
├── utils
|   ├──
│   │   
│   ├──
│   │  
│   └──
├── requirements.txt
└── configs.json


      title={Investigating Learning in Deep Neural Networks using Layer-Wise Weight Change}, 
      author={Ayush Manish Agrawal and Atharva Tendle and Harshvardhan Sikka and Sahib Singh and Amr Kayid},


Issue / Want to Contribute ? :

Open a new issue or do a pull request incase you are facing any difficulty with the code base or if you want to contribute to it.


Want to be a part of our organization? Checkout Manifold Computing


The repository contains the codebase for Relative Weight Change in layers of Deep Neural Networks.



Contributors 4