changeable loss weights for multiple output #2595

xingdi-eric-yuan · 2016-05-03T22:36:39Z

Hi all, what's an easy way to set changeable loss weights for multiple output, for example, is there a way to modify the loss weights in callback?

joelthchao · 2016-05-04T06:29:14Z

You can pass list of K.variable into loss_weights.

alpha = K.variable(0.5)
beta = K.variable(0.5)
model.compile(..., loss_weights=[alpha, beta], ...)

and define your own Callback

class MyCallback(Callback):
    def __init__(self, alpha, beta):
        self.alpha = alpha
        self.beta = beta
    # customize your behavior
    def on_epoch_end(self, epoch, logs={}):
        self.alpha = self.alpha - 0.1
        self.beta = self.beta + 0.1

then pass it into fit

model.fit( ..., callbacks=[MyCallback(alpha, beta)], ...)

P.S. Haven't thoroughly tested it.

xingdi-eric-yuan · 2016-05-05T21:07:50Z

The idea works, thanks!
In callback, I used something like:

def on_epoch_end(self, epoch, logs={}):
        if epoch == 2:
            K.set_value(self.alpha, K.get_value(self.alpha) / 1.5)
            K.set_value(self.beta, K.get_value(self.beta) * 1.5)
        logger.info("epoch %s, alpha = %s, beta = %s" % (epoch, K.get_value(self.alpha), K.get_value(self.beta)))

pGit1 · 2017-01-07T15:21:10Z

Hi guys. I can't find any documentation on what the loss_weights parameter actually does. @joelthchao can you provide an intuitive explanation?

joelthchao · 2017-01-07T20:01:09Z

@pGit1 Please check this example. End of this section provides detail explanations and usages.

pGit1 · 2017-01-07T21:09:09Z

@joelthchao thanks for this. I have read this example actually but I don't see the intuition behind choosing ".2". It says:

We compile the model and assign a weight of 0.2 to the auxiliary loss.

Why was this number chosen? What is the intuition behind this choice? That is what I don't understand.
I mean if I have a classification output and a regression output I would think I would want to have a small loss weight for the regression output so the internal representation of the network would not be dominated by the MSE since those numbers could be huge. Is this understanding accurate? Or should loss_weights be treated as a hyper-parameter in multi output problems (even still an intuition would be nice to understand so as to avoid blindly picking values)?

joelthchao · 2017-01-08T06:11:00Z

It's a hyper-parameter, usually we need to adjust it according to 1) the importance of losses 2) the actual value of losses. We do need experiments to choose a correct number, to prevent one loss dominate others.

pGit1 · 2017-01-08T07:12:46Z

Makes sense. Thank you!

…

On Sun, Jan 8, 2017 at 1:11 AM, Joel ***@***.***> wrote: It's a hyper-parameter, usually we need to adjust it according to 1) the importance of losses 2) the actual value of losses. We do need experiments to choose a correct number, to prevent one loss dominate others. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2595 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANU-St6m_0Q-foL64uK80DDAaggJZwvhks5rQH4AgaJpZM4IWsB_> .

nabsabraham · 2018-10-09T03:35:43Z

@joelthchao I want to update my paramter alpha in the loss in a similar fashion with a callback.

def custom_loss(x, x_pred): 
    loss1 = binary_crossentropy(x, x_pred)
    return K.pow(loss1, alpha)

How can I change the alpha in the callback to actually change the alpha in the loss function? It is not correct for my application to pass alpha as a loss_weight in the model.fit method.
Any help appreciated!

panwarnaveen9 · 2019-03-16T05:55:17Z

You can pass list of K.variable into loss_weights.

alpha = K.variable(0.5)
beta = K.variable(0.5)
model.compile(..., loss_weights=[alpha, beta], ...)

and define your own Callback

class MyCallback(Callback):
    def __init__(self, alpha, beta):
        self.alpha = alpha
        self.beta = beta
    # customize your behavior
    def on_epoch_end(self, epoch, logs={}):
        self.alpha = self.alpha - 0.1
        self.beta = self.beta + 0.1

then pass it into fit

model.fit( ..., callbacks=[MyCallback(alpha, beta)], ...)

P.S. Haven't thoroughly tested it.

@joelthchao don't we need to compile the model again in this case ?

mycal-tucker · 2019-05-06T15:34:16Z

It doesn't seem like we need to recompile as long as we're using the variables.

I can confirm this through running the code myself, and by looking at this comment.

Let me know if you find some different behavior.

panwarnaveen9 · 2019-05-07T14:55:34Z

@mycal-tucker what do you mean by variables ? If I have function like this.. Will it update the params ?

class ChangeLossWeight(Callback):

	def __init__(self, alpha, beta):
		self.alpha = alpha
		self.beta = beta

	# custom behaviour
	def on_epoch_end(self, epoch, logs={}):
		if epoch == 2:
			self.alpha = 0.7
			self.beta = 0.3
		if epoch == 5:
			self.alpha = 0.5
			self.beta = 0.5

mycal-tucker · 2019-05-07T15:34:06Z

I mean a Keras variable. I think your current setup wouldn't work because you're just updating the numbers. Here's something I did that doesn't perfectly match the Callback class model you've got but that does work. Porting my code over to your setup shouldn't be too much trouble.

class MyCustomModel:
    # Define a static variable for now because I don't know how else to get around the callback being static.
    # I am sure that there is a more elegant way, but for now this works.
    beta_var = None
    # Create a new model by stringing together two models.
    def __init__(self, model1, model2):
        model2_output = model2(model1.output)  # Feed model1 into model2
        self.model = Model(inputs=[model1.input],
                           outputs=[model1.output, model2_output ],
                           name='my_custom_model')

    def compile(self, alpha=0, beta=0):
        MyCustomModel.beta_var = K.variable(beta)  # Create variable so it can be updated during training.
        # Two losses to optimize for: mse and binary_crossentropy.
        # Only update one of the weights, to show that we have that flexibility.
        self.model.compile(optimizer=RMSprop(lr=0.001),
                           loss=['mse', 'binary_crossentropy'],
                           loss_weights=[1.0, MyCustomModel.beta_var])

    def train(self, inputs, outputs, epochs, batch_size):
        callbacks = [LambdaCallback(on_epoch_end=LSTMAutoencoder.on_epoch_end)]
        self.model.fit(inputs, outputs, epochs=epochs,
                       batch_size=batch_size,
                       callbacks=callbacks)

    @staticmethod
    def on_epoch_end(epoch, _):
        # Update loss weight parameters, as shown in https://github.com/keras-team/keras/issues/2595
        # Have the weight decrease geometrically, just for fun.
        K.set_value(MyCustomModel.beta_var, K.get_value(MyCustomModel.beta_var) * 0.95)
        print("Beta value", K.get_value(MyCustomModel.beta_var))

Does this make sense to you? As I point out in the comments, my static variable workaround is probably not the most elegant way of getting things working, but it at least works.

tejal567 · 2019-05-16T10:43:40Z

TypeError: ('Not JSON Serializable:')
This error comes while saving the model. Any updates on this?
#9444

Barnonewdm · 2019-05-16T15:46:42Z

TypeError: ('Not JSON Serializable:')
This error comes while saving the model. Any updates on this?
#9444

I've come across the same error.

Barnonewdm · 2019-05-16T17:49:41Z

I have tried the trick from https://github.com/keras-team/keras/issues/9444#issuecomment-395260154.
#9444 works well.

Falcatrua · 2019-08-20T20:29:05Z

Changing the loss_weights by the custom callback as described (@joelthchao and others) doesn't seem to work. The k.variable values are changed, but the total loss calculation doesn't take the new values in consideration. Consider this example where loss_weights were supposed to simply switch from the loss of one output (0, 1) to the loss of other output (1, 0)

import numpy as np
import tensorflow as tf
import keras
from keras.layers import *
from keras.callbacks import Callback
from keras.optimizers import Adam

input_1 = Input(shape=(1,))
hidden_0 = Dense(units=10, activation='tanh')(input_1)
dense_0  = Dense(1, activation='tanh')(hidden_0)
dense_1  = Dense(1, activation='tanh')(hidden_0)

model = keras.Model(inputs=input_1, outputs=[dense_0, dense_1])
model.summary()

alpha = K.variable(0)
beta = K.variable(1)

model.compile(optimizer=Adam(),
              loss=['mean_absolute_error', 'binary_crossentropy'],
              metrics=['accuracy'],
              loss_weights=[alpha, beta])
              

class CustomValidationLoss(Callback):
    def __init__(self, alpha, beta):
        self.alpha = alpha
        self.beta = beta

    def on_epoch_end(self, epoch, logs={}):
        print ("loss_weights used:", K.get_value(self.alpha), K.get_value(self.beta))
        if epoch == 1:
            self.alpha = K.variable(1)
            self.beta = K.variable(0)
           


X = np.random.rand(100,1)
Y1 = np.random.rand(100,1)
Y2 = np.random.rand(100,1)


model.fit(X, [Y1, Y2],
          epochs=4,
          verbose=2,
          callbacks=[CustomValidationLoss(alpha, beta)])

specific values might change, but the total loss doesn't seem to use the updated loss_weights values in its calculation, even though they are printed fine

example of output

 - 1s - loss: 1.2862 - dense_2_loss: 0.4501 - dense_3_loss: 1.2862 - dense_2_acc: 0.0000e+00 - dense_3_acc: 0.0000e+00
loss_weights used: 0.0 1.0
Epoch 2/4
 - 0s - loss: 1.1401 - dense_2_loss: 0.4611 - dense_3_loss: 1.1401 - dense_2_acc: 0.0000e+00 - dense_3_acc: 0.0000e+00
loss_weights used: 0.0 1.0
Epoch 3/4
 - 0s - loss: 1.0499 - dense_2_loss: 0.4716 - dense_3_loss: 1.0499 - dense_2_acc: 0.0000e+00 - dense_3_acc: 0.0000e+00
loss_weights used: 1.0 0.0
Epoch 4/4
 - 0s - loss: 0.9893 - dense_2_loss: 0.4814 - dense_3_loss: 0.9893 - dense_2_acc: 0.0000e+00 - dense_3_acc: 0.0000e+00
loss_weights used: 1.0 0.0

what am I missing?

lorinczszabolcs · 2020-05-07T11:03:29Z

Changing the loss_weights by the custom callback as described (@joelthchao and others) doesn't seem to work. The k.variable values are changed, but the total loss calculation doesn't take the new values in consideration. Consider this example where loss_weights were supposed to simply switch from the loss of one output (0, 1) to the loss of other output (1, 0)

import numpy as np
import tensorflow as tf
import keras
from keras.layers import *
from keras.callbacks import Callback
from keras.optimizers import Adam

input_1 = Input(shape=(1,))
hidden_0 = Dense(units=10, activation='tanh')(input_1)
dense_0  = Dense(1, activation='tanh')(hidden_0)
dense_1  = Dense(1, activation='tanh')(hidden_0)

model = keras.Model(inputs=input_1, outputs=[dense_0, dense_1])
model.summary()

alpha = K.variable(0)
beta = K.variable(1)

model.compile(optimizer=Adam(),
              loss=['mean_absolute_error', 'binary_crossentropy'],
              metrics=['accuracy'],
              loss_weights=[alpha, beta])
              

class CustomValidationLoss(Callback):
    def __init__(self, alpha, beta):
        self.alpha = alpha
        self.beta = beta

    def on_epoch_end(self, epoch, logs={}):
        print ("loss_weights used:", K.get_value(self.alpha), K.get_value(self.beta))
        if epoch == 1:
            self.alpha = K.variable(1)
            self.beta = K.variable(0)
           


X = np.random.rand(100,1)
Y1 = np.random.rand(100,1)
Y2 = np.random.rand(100,1)


model.fit(X, [Y1, Y2],
          epochs=4,
          verbose=2,
          callbacks=[CustomValidationLoss(alpha, beta)])

specific values might change, but the total loss doesn't seem to use the updated loss_weights values in its calculation, even though they are printed fine

example of output

 - 1s - loss: 1.2862 - dense_2_loss: 0.4501 - dense_3_loss: 1.2862 - dense_2_acc: 0.0000e+00 - dense_3_acc: 0.0000e+00
loss_weights used: 0.0 1.0
Epoch 2/4
 - 0s - loss: 1.1401 - dense_2_loss: 0.4611 - dense_3_loss: 1.1401 - dense_2_acc: 0.0000e+00 - dense_3_acc: 0.0000e+00
loss_weights used: 0.0 1.0
Epoch 3/4
 - 0s - loss: 1.0499 - dense_2_loss: 0.4716 - dense_3_loss: 1.0499 - dense_2_acc: 0.0000e+00 - dense_3_acc: 0.0000e+00
loss_weights used: 1.0 0.0
Epoch 4/4
 - 0s - loss: 0.9893 - dense_2_loss: 0.4814 - dense_3_loss: 0.9893 - dense_2_acc: 0.0000e+00 - dense_3_acc: 0.0000e+00
loss_weights used: 1.0 0.0

what am I missing?

I think that you shouldn't reassign the variables as K.variable in:
if epoch == 1: self.alpha = K.variable(1); self.beta = K.variable(0);

Instead just assign values, like this:
if epoch == 1: self.alpha = 1; self.beta = 0;
That's just a tip, I haven't tested it, but it should work, since you are not creating a new K.variable, instead just assigning a new value to the existing one.

dsadulla · 2020-06-14T10:04:58Z

I did this and it was working.. Thanks @joelthchao & @xingdi-eric-yuan :)

class LossWeightAdjust(Callback):
    def __init__(self, alpha, beta, gamma, delta):
        self.alpha = alpha
        self.beta = beta
        self.gamma = gamma
        self.delta = delta
    # customize your behavior
    def on_epoch_end(self, epoch, logs):
        losses = np.array([v for k,v in logs.items() if k in ['val_starts_0_loss', 'val_stops_0_loss', 'val_starts_1_loss', 'val_stops_1_loss']], dtype=np.float64)
        losses = (losses - 0.5*losses.min()) / (losses.max() - 0.5*losses.min())
        losses = losses/np.sum(losses)

        K.set_value(self.alpha, losses[0])
        K.set_value(self.beta, losses[1])
        K.set_value(self.gamma, losses[2])
        K.set_value(self.delta, losses[3])
        
        print("\n Loss weights recalibrated to alpha = %s, beta = %s, gamma = %s, delta = %s " % (np.round(losses[0],2),
                                                                                                  np.round(losses[1],2),
                                                                                                  np.round(losses[2],2),
                                                                                                  np.round(losses[3],2)))
        
        logger.info("Loss weights recalibrated to alpha = %s, beta = %s, gamma = %s, delta = %s " % (K.get_value(self.alpha), K.get_value(self.beta), K.get_value(self.gamma), K.get_value(self.delta)))

span_detection_model = build_model()
    
alpha = K.variable(0.25)
beta = K.variable(0.25)
gamma = K.variable(0.25)
delta = K.variable(0.75)

span_detection_model.compile(..., loss_weights= "starts_0":alpha,"stops_0":beta,"starts_1":gamma,"stops_1":delta})

danielplatt · 2020-08-07T17:02:47Z

This does not work for me. print(model.__dict__['compiled_loss'].__dict__['_user_loss_weights']) shows that the loss weights remain unchanged. (Below you see the code from before plus one import statement and the two print statements) In another example I trained on X=[[1,0],[1,0],...] and y1=[[1],[1],...], y2=[[0],[0],...] with loss weights (0,1) in the beginning and loss weights (1,0) after a few epochs. There, the weights are almost constant after the loss weights change, even though the y1 is extremely poorly fitted.

import numpy as np
import tensorflow as tf
K = tf.keras.backend
import keras
from keras.layers import *
from keras.callbacks import Callback

input_1 = Input(shape=(1,))
hidden_0 = Dense(units=10, activation='tanh')(input_1)
dense_0 = Dense(1, activation='tanh')(hidden_0)
dense_1 = Dense(1, activation='tanh')(hidden_0)

model = keras.Model(inputs=input_1, outputs=[dense_0, dense_1])
model.summary()

alpha = K.variable(0)
beta = K.variable(1)

model.compile(optimizer='adam',
              loss=['mean_absolute_error', 'binary_crossentropy'],
              metrics=['accuracy'],
              loss_weights=[alpha, beta])

class CustomValidationLoss(Callback):
    def __init__(self, alpha, beta):
        self.alpha = alpha
        self.beta = beta

    def on_epoch_end(self, epoch, logs={}):
        print("loss_weights used:", K.get_value(self.alpha), K.get_value(self.beta))
        if epoch == 10:
            self.alpha = K.variable(1)
            self.beta = K.variable(0)

X = np.random.rand(100, 1)
Y1 = np.random.rand(100, 1)
Y2 = np.random.rand(100, 1)

print(model.__dict__['compiled_loss'].__dict__['_user_loss_weights'])

model.fit(X, [Y1, Y2],
          epochs=20,
          verbose=2,
          callbacks=[CustomValidationLoss(alpha, beta)])

print(model.__dict__['compiled_loss'].__dict__['_user_loss_weights'])

stale bot added the stale label May 23, 2017

stale bot closed this as completed Jun 22, 2017

rasto2211 mentioned this issue Apr 7, 2018

changing loss weight during training #6446

Closed

yushuinanrong mentioned this issue Jun 5, 2018

changeable loss weights for multiple output when using train_on_batch #10358

Closed

janzd mentioned this issue Jun 7, 2018

Serialisation doesn't allow variables as loss_weights #9444

Closed

krdav mentioned this issue Nov 21, 2018

Reconsider losses, including KL term matsengrp/vampire#29

Closed

matsen mentioned this issue Dec 15, 2018

adjust beta during training matsengrp/vampire#58

Closed

alexeyev mentioned this issue Mar 3, 2019

Introduce KL term annealing alexeyev/Keras-Generating-Sentences-from-a-Continuous-Space#1

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

changeable loss weights for multiple output #2595

changeable loss weights for multiple output #2595

xingdi-eric-yuan commented May 3, 2016

joelthchao commented May 4, 2016

xingdi-eric-yuan commented May 5, 2016

pGit1 commented Jan 7, 2017

joelthchao commented Jan 7, 2017

pGit1 commented Jan 7, 2017

joelthchao commented Jan 8, 2017

pGit1 commented Jan 8, 2017 via email

nabsabraham commented Oct 9, 2018

panwarnaveen9 commented Mar 16, 2019 •

edited

Loading

mycal-tucker commented May 6, 2019

panwarnaveen9 commented May 7, 2019

mycal-tucker commented May 7, 2019

tejal567 commented May 16, 2019

Barnonewdm commented May 16, 2019

Barnonewdm commented May 16, 2019

Falcatrua commented Aug 20, 2019 •

edited

Loading

lorinczszabolcs commented May 7, 2020 •

edited

Loading

dsadulla commented Jun 14, 2020 •

edited

Loading

danielplatt commented Aug 7, 2020

changeable loss weights for multiple output #2595

changeable loss weights for multiple output #2595

Comments

xingdi-eric-yuan commented May 3, 2016

joelthchao commented May 4, 2016

xingdi-eric-yuan commented May 5, 2016

pGit1 commented Jan 7, 2017

joelthchao commented Jan 7, 2017

pGit1 commented Jan 7, 2017

joelthchao commented Jan 8, 2017

pGit1 commented Jan 8, 2017 via email

nabsabraham commented Oct 9, 2018

panwarnaveen9 commented Mar 16, 2019 • edited Loading

mycal-tucker commented May 6, 2019

panwarnaveen9 commented May 7, 2019

mycal-tucker commented May 7, 2019

tejal567 commented May 16, 2019

Barnonewdm commented May 16, 2019

Barnonewdm commented May 16, 2019

Falcatrua commented Aug 20, 2019 • edited Loading

lorinczszabolcs commented May 7, 2020 • edited Loading

dsadulla commented Jun 14, 2020 • edited Loading

danielplatt commented Aug 7, 2020

panwarnaveen9 commented Mar 16, 2019 •

edited

Loading

Falcatrua commented Aug 20, 2019 •

edited

Loading

lorinczszabolcs commented May 7, 2020 •

edited

Loading

dsadulla commented Jun 14, 2020 •

edited

Loading