Cannot set per-layer learning rates for ResNet

[varunagrawal]

When I try to set the learning rate of the FC layer in ResNet50 to a different learning rate, I get the following error:

ValueError                                Traceback (most recent call last)
<ipython-input-123-553a220d2682> in <module>()
----> 1 optimizer = get_optimizer(model)

<ipython-input-121-302e147c3398> in get_optimizer(model, lr)
     10 #         {'params': model.classifier.parameters(), 'lr': lr*10}  # the classifier needs to learn weights faster
     11         {'params': model.fc.parameters(), 'lr': lr*10}
---> 12     ], lr=lr, weight_decay=0.0005)

/home/varun/.linuxbrew/Cellar/python3/3.6.0_1/lib/python3.6/site-packages/torch/optim/adam.py in __init__(self, params, lr, betas, eps, weight_decay)
     26         defaults = dict(lr=lr, betas=betas, eps=eps,
     27                         weight_decay=weight_decay)
---> 28         super(Adam, self).__init__(params, defaults)
     29 
     30     def step(self, closure=None):

/home/varun/.linuxbrew/Cellar/python3/3.6.0_1/lib/python3.6/site-packages/torch/optim/optimizer.py in __init__(self, params, defaults)
     37             group_set = set(group['params'])
     38             if not param_set.isdisjoint(group_set):
---> 39                 raise ValueError("some parameters appear in more than one "
     40                                  "parameter group")
     41             param_set.update(group_set)

ValueError: some parameters appear in more than one parameter group

My model code is:

def get_model(pretrained=True, num_classes=5):
    model = models.resnet.resnet50(pretrained=pretrained)

    mod = list(model.children())
    mod.pop()
    mod.append(torch.nn.Linear(4096, 5))
    model.fc = torch.nn.Sequential(*mod)
    
    return model

model = get_model()

and my optimizer code is:

def get_optimizer(model, lr=1e-04):
    return optim.Adam([
        {'params': model.conv1.parameters()},
        {'params': model.layer1.parameters()},
        {'params': model.layer2.parameters()},
        {'params': model.layer3.parameters()},
        {'params': model.layer4.parameters()},
        {'params': model.fc.parameters(), 'lr': lr*10}   # the classifier needs to learn weights faster
    ], lr=lr, weight_decay=0.0005)

If I comment out all the dictionaries except for the last model.fc.parameters one, the code runs, else even for one other layer, I get the above error.

[fmassa]

It seems that you have repeated modules when you try to replace your last classifier? Why don't you just change the last fc, instead of taking all the children, poping the last, and assigning it to the classifier?

[varunagrawal]

Oh I see. I was re-using code from my VGGNet definition and hence the error. Arrgh, why is replacing layers in PyTorch so hard? It would have been so much nicer if the networks followed a consistent format. 😢

[soumith]

@varunagrawal

It would have been so much nicer if the networks followed a consistent format.

Do you mean the networks in the modelzoo? or in general? I could work on making them consistent if you have suggestions on streamlining them.

[varunagrawal]

@soumith I was hoping for the networks in the modelzoo to be consistent initially. The fact that one needs to constantly look up model definition code to understand how to go about updating specific layers and then writing the kind of hacky code that I wrote above to actually update the layers is a major pain point.

I understand that creating the Sequential models in the constructor makes sense since one should be able to leverage parallelism right out of the box as well as ease of model creation. It would be super nice if nn.Sequential implemented the __setitem__ method as it would allow us to index Sequential like an array and update layers accordingly. I don't know if you already have this in the works or how much effort it would be, but I would be willing to put in time for this feature.

One suggestion would be to allow for optional ID arguments to layers and have network weights load according to the specified ID, similar to how Caffe does it. Currently, if I call VGG(pretrained=True, num_classes=200) I get an error due to the network weights being inconsistent with the definition. This can be easily avoided by modifying the ID value after checking if the num_classes parameter is 1000 or something else.