Skip to content
No description, website, or topics provided.
Python Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
utils @ ef8583c

Norm Matters

See for updated version of this code

This code was was used to implement Norm matters: efficient and accurate normalization schemes in deep networks - Hoffer, Banner, Golan, Soudry (2018):

  title={Norm matters: efficient and accurate normalization schemes in deep networks},
  author={Hoffer, Elad and Banner, Ron and Golan, Itay and Soudry, Daniel},
  booktitle={Advances in Neural Information Processing Systems},

It is based off imagenet example in pytorch with some helpful additions such as:

  • Training on several datasets other than imagenet
  • Complete logging of trained experiment
  • Graph visualization of the training/validation loss and accuracy
  • Definition of preprocessing and optimization regime for each model



  • Configure your dataset path at
  • To get the ILSVRC data, you should register on their site for access:

Model configuration

Network model is defined by writing a .py file in models folder, and selecting it using the model flag. Model function must be registered in models/ The model function must return a trainable network. It can also specify additional training options such optimization regime (either a dictionary or a function), and input transform modifications.

e.g for a model definition:

class Model(nn.Module):

    def __init__(self, num_classes=1000):
        super(Model, self).__init__()
        self.model = nn.Sequential(...)

        self.regime = [
            {'epoch': 0, 'optimizer': 'SGD', 'lr': 1e-2,
                'weight_decay': 5e-4, 'momentum': 0.9},
            {'epoch': 15, 'lr': 1e-3, 'weight_decay': 0}

        self.input_transform = {
            'train': transforms.Compose([...]),
            'eval': transforms.Compose([...])
    def forward(self, inputs):
        return self.model(inputs)
 def model(**kwargs):
        return Model()
You can’t perform that action at this time.