Different learning rate per layer

[Kalkasas]

For tensorflow you can use multiple optimizers to achieve different learning rates per layer.
Is there support for this in tensorpack or do I have to write a custom trainer?

[ppwwyyxx]

Yes, you can specify different learning rate for each parameter, by scaling its gradient. You can overwrite the get_gradient_processor() method in your Model with something like:

    def get_gradient_processor(self):
        return [ScaleGradient([('conv.*/W', 0.5), ('fc.*', 2)])]

ScaleGradient takes a list of (regex, multiplier) tuple. Parameters that are scaled will get printed out so you could check.

// UPDATE: this usage was later deprecated. See comment below.

[Kalkasas]

Exactly what I needed. Thank you!

[coolbay]

I couldn't find any function as get_gradient_processor() in the base class ModelDesc. I tried adding this function into my Model, but it seemed that this function was never called.

[ppwwyyxx]

This usage was deprecated one year ago. Now gradient processors can be added onto optimizer with http://tensorpack.readthedocs.io/en/latest/modules/tfutils.html#tensorpack.tfutils.optimizer.apply_grad_processors .

For example: https://github.com/ppwwyyxx/tensorpack/blob/9972b15036c851ce82e63721cfe0905e68115475/examples/SpatialTransformer/mnist-addition.py#L90-L96