How to change regularization parameters during training?

[alexander-rakhlin]

Hi all,

I am trying to implement flexible regularization scheduler. Instantiate layer like this:
x = Convolution2D(... W_regularizer=l2(10)...)

and later change regularization:
model.layers[1].W_regularizer = l2(0)

I can verify the layer's settings changed:

model.layers[1].W_regularizer.l2
Out[9]: array(0.0, dtype=float32)

but this has no effect during following training whether I compile model anew or not. Where is a caveat?

[bstriner]

Hi Alexander,

The hyperparameters are built-in to the training function when you compile. Editing the model after compilation won't do anything to affect your current training. You will see the same issue if you try to change learning rates or other hyperparameters.

The way to modify hyperparameters during training is to use backend variables in the training function and update those variables during training.

The L1L2Regularizer isn't using variables but it should be.
https://github.com/fchollet/keras/blob/master/keras/regularizers.py
Change:
self.l2 = K.cast_to_floatx(l2)
to:
self.l2 = K.variable(K.cast_to_floatx(l2))

Instantiate but hold a reference to the regularizer.

reg = l2(10)
x = Convolution2D(W_regularizer=reg)

During training, update the variable reg.l2
K.set_value(reg.l2, K.cast_to_floatx(100))

Might want to add a pull request to make l1l2 into variables.

Cheers,
Ben

[alexander-rakhlin]

Hi Ben!

Thank you very much. It works :)

(this is updated message, previously I reported it didn't, but I just drove weights to zero with big initial l2)

I do it to implement MacKay's scheduler if you interested:

https://www.youtube.com/watch?v=vEPQNwxd1Y4&index..
https://www.youtube.com/watch?v=EJfqOAi_rj8&index..

By the way, changing L1L2Regularizer we would need to change get_config method too

    def get_config(self):
        return {'name': self.__class__.__name__,
                'l1': float(K.cast_to_floatx(self.l1.eval())),
                'l2': float(K.cast_to_floatx(self.l2.eval()))}

[bstriner]

Great catch, Alexander. I always forget about get_config. Just put in a PR with a new flag "use_variables" (default=False). If True, it will use variables in L1L2Regularizer instead of constants. I left it as a flag just in case there is some barely-noticeable performance benefit to using a constant.

#4827

[sdahan12]

hi,
bringing it up again,
if I declare variable in my model such as:
Margin_tensor=K.variable(5,dtype='float32')

and then use it as an input to my custom loss function such as:
penalized_loss(margin=K.get_value(Margin_tensor)

how can I access this variable during training to change its value?

[bstriner]

Your custom loss function needs to work on margin_tensor not K.get_value(margin_tensor). You are currently just getting the initial value and using it as a constant for the rest of training.

You can get and change the value using get_value and set_value. Your problem is that you aren't using the variable.

margin_tensor=K.variable(5,dtype='float32')
def custom_loss(ytrue, ypred):
  return binary_crossentropy(ytrue, ypred) + margin_tensor
model.compile(...,loss=custom_loss)

[marioviti]

Hi,

I've been tacking inspiration from this conversation and came up with this solution that works extremely well:

1st: extend the Regularizer with a custom l1l2 regularizer class (do not call it L1L2 as in serialization ,aka when you save and reload your model, shadowing does not work): it should go something like this:

class L1L2_m(Regularizer):
    """Regularizer for L1 and L2 regularization.
    # Arguments
        l1: Float; L1 regularization factor.
        l2: Float; L2 regularization factor.
    """

    def __init__(self, l1=0.0, l2=0.01):
        with K.name_scope(self.__class__.__name__):
            self.l1 = K.variable(l1,name='l1')
            self.l2 = K.variable(l2,name='l2')
            self.val_l1 = l1
            self.val_l2 = l2
            
    def set_l1_l2(self,l1,l2):
        K.set_value(self.l1,l1)
        K.set_value(self.l2,l2)
        self.val_l1 = l1
        self.val_l2 = l2

    def __call__(self, x):
        regularization = 0.
        if self.val_l1 > 0.:
            regularization += K.sum(self.l1 * K.abs(x))
        if self.val_l2 > 0.:
            regularization += K.sum(self.l2 * K.square(x))
        return regularization

    def get_config(self):
        config = {'l1': float(K.get_value(self.l1)),
                  'l2': float(K.get_value(self.l2))}
        return config

2nd: Add your custom object so that when you might want to export your model you won't have any issue in reloading it.

from keras.utils.generic_utils import get_custom_objects
get_custom_objects().update({ L1L2_m.__name__: L1L2_m })

3rd: update your variable using the custom object set_l1_l2 method by accessing the object from the model keras model.

def set_model_l1_l2(model,l1,l2):
    for layer in model.layers:
         if 'kernel_regularizer' in dir(layer) and \
                isinstance(layer.kernel_regularizer, L1L2_m):
                layer.kernel_regularizer.set_l1_l2(l1,l2)

Done.

But wait, can't I access the variables from tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)?
which is btw the same dictionary as model.trainable_variables.

Yes you could but I warmly suggest you not to do so:

Why? because at declaration time your variable scope will be depending on whether you've defined the L1L2_m within a layer (a convolutional layer for example Conv1).
So if you were to look into the graph you'll find your variables scope looks smth like: Conv1/L1L2/l1 ...
or Conv10/L1L2/l1 ...
But the keras deserializer does not works like that and you'll find all L1L2_1/l1 , L1L2_2/l1 .... all grouped together if you save and reload your model (json or h5 format).

Using the object reference method set_l1_l2 gives the same results every time even with a different graph representation.

[philippesamuel]

Hi Ben!

Thank you very much. It works :)

(this is updated message, previously I reported it didn't, but I just drove weights to zero with big initial l2)

I do it to implement MacKay's scheduler if you interested:

https://www.youtube.com/watch?v=vEPQNwxd1Y4&index..
https://www.youtube.com/watch?v=EJfqOAi_rj8&index..

By the way, changing L1L2Regularizer we would need to change get_config method too

    def get_config(self):
        return {'name': self.__class__.__name__,
                'l1': float(K.cast_to_floatx(self.l1.eval())),
                'l2': float(K.cast_to_floatx(self.l2.eval()))}

Hi @alexander-rakhlin,

I'd be highly interested in the MacKay's scheduler you mentioned. Did you manage to implement it?

Best regards.