Loss not changing when training #2711

kevkid · 2016-05-12T22:47:31Z

I have a model that I am trying to train where the loss does not go down. I have a custom image set that I am using. These images are 106 x 106 px (black and white) and I have two (2) classes, Bargraph or Gels. These two classes are very different. I have run the Cifar10 dataset and it did reduce the loss, but I am very confused as to why my model will always predict only one class for everything.

Xtrain is a numpy array of images (which are numpy arrays), Ytrain is a numpy array of arrays ([0,1] or [1,0]) the shapes look like this:

np.shape(Xtrain)
Out[58]: (2000, 1, 106, 106)

np.shape(Ytrain)
Out[59]: (2000, 2)

Here is my model:

model = Sequential()
model.add(Convolution2D(32, 16, 16, border_mode='same',name='conv1', input_shape = (1, 106, 106)))
first_layer = model.layers[0]
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))

#2
model.add(Convolution2D(64, 15, 15, border_mode='same',name='conv2'))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))

#3
model.add(Convolution2D(128, 14, 14, border_mode='same',name='conv3'))
model.add(Activation("relu"))

#flatten
model.add(Flatten())
model.add(Dense(2))
model.add(Activation('softmax'))

rms = RMSprop()
sgd = SGD(lr=0.000001, decay=1e-6, momentum=1.9)
model.compile(loss='categorical_crossentropy', optimizer=sgd,metrics=["accuracy"])
model.fit(Xtrain[950:1050], Ytrain[950:1050], batch_size=32, nb_epoch=20,
          verbose=1, validation_split=0.2,
          callbacks=[EarlyStopping(monitor='val_loss', patience=0)])

Right now I am just doing very small training sets (I tried doing 1000 examples as well, with similar results).

I have also tried RMS and SDG with large and small learning rates.

What else can I try ?

Check that you are up-to-date with the master branch of Keras. You can update with:
pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps
If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

The text was updated successfully, but these errors were encountered:

dandxy89 · 2016-05-12T22:59:20Z

Try increasing the learning rate to a higher value, possibly to 0.1. That way you can ensure that noticeable changes to weights are made for each successive update.

kevkid · 2016-05-12T23:13:13Z

Tried it, it stopped after 2 epochs. Here are the results

sgd = SGD(lr=0.1, decay=1e-6, momentum=1.9)
model.compile(loss='categorical_crossentropy', optimizer=sgd,metrics=["accuracy"])
model.fit(Xtrain[950:1050], Ytrain[950:1050], batch_size=32, nb_epoch=20,
          verbose=1, validation_split=0.2,
          callbacks=[EarlyStopping(monitor='val_loss', patience=0)])

Train on 80 samples, validate on 20 samples
Epoch 1/20
80/80 [==============================] - 84s - loss: 5.9848 - acc: 0.5250 - val_loss: 1.1921e-07 - val_acc: 1.0000
Epoch 2/20
80/80 [==============================] - 85s - loss: 10.0738 - acc: 0.3750 - val_loss: 1.1921e-07 - val_acc: 1.0000
Out[60]: <keras.callbacks.History at 0x7f6b30891c90>

model.predict(Xtrain[995:1005])
Out[62]: 
array([[ 1.,  0.],
       [ 1.,  0.],
       [ 1.,  0.],
       [ 1.,  0.],
       [ 1.,  0.],
       [ 1.,  0.],
       [ 1.,  0.],
       [ 1.,  0.],
       [ 1.,  0.],
       [ 1.,  0.]])

The first class is from 0 to 999 and the second class is from 1000 to 1999

I tried to predict right at the border and got all [1,0]. Shuffling the training set should not matter should it?

kevkid · 2016-05-13T05:47:22Z

After reading some blogs, looks like the batch size is important, because if our data is not shuffled it will learn one class for a few batches and then another class for a few batches. Similarly My loss seems to stay the same, here is an interesting read on the loss function. I really am still unsure as to what I may be doing wrong.

Here are a few things I tried:

number of layers (reduction)
size of the filters (reduction)
SGD learning rate from 0.000000001 to 0.1
SGD decay to 1e-2
Batch size
Different images
Shuffling the images around

I am really unsure as to what I can do to get my loss to go down. Any other ideas?

Code:

model = Sequential()
model.add(Convolution2D(32, 10, 10, border_mode='same',name='conv1', input_shape = (1, 106, 106)))
#flatten
model.add(Flatten())
model.add(Dense(2))
model.add(Activation('softmax'))
rms = RMSprop()
sgd = SGD(lr=0.1, decay=1e-2, momentum=0.9)
model.compile(loss='categorical_crossentropy', optimizer=sgd,metrics=["accuracy"])
model.fit(Xtrain, Ytrain, batch_size=64, nb_epoch=10,
          verbose=1)

Here is the code for me to read in the images if it helps:

def loadCustomData():
    #read directories
    #files = []
    fileRoots = []
    for root, directories, filenames in os.walk('modified/convNetPanelsPaneled_sorted/'):
        for filename in filenames: 
            #files.append(os.path.join(root,filename))
            fileRoots.append(root)
    fileRoots = list(sorted(set(fileRoots)))
    bar = imgs.getFiles(fileRoots[19])
    bar = [fileRoots[19] + "/" + s for s in list(bar)]
    gel = imgs.getFiles(fileRoots[0])
    gel = [fileRoots[0] + "/" + s for s in list(gel)]
    Xtrain = []
    Ytrain = []
    Xtest = []
    Ytest = []
    #for i in range(int(len(bar)*.95)):
    for i in range(1000):
        imgArray = imgs.load_image(bar[i])
        if(len(np.shape(imgArray)) == 3):
            imgArray = imgArray[:,:,1]#set to 1 so we keep the greyscale
        Xtrain.append(imgArray)
        Ytrain.append([1])

    #for i in range(len(bar) - int(len(bar)*.95)):
    for i in range(1000,1100):
        imgArray = imgs.load_image(bar[i])
        if(len(np.shape(imgArray)) == 3):
            imgArray = imgArray[:,:,1]
        Xtest.append(imgArray)
        Ytest.append([1])

    for i in range(1000):
        imgArray = imgs.load_image(gel[i])
        if(len(np.shape(imgArray)) == 3):
            imgArray = imgArray[:,:,1]
        Xtrain.append(imgArray)
        Ytrain.append([0])

    #for i in range(len(gel) - int(len(gel)*.95)):
    for i in range(1000,1100):
        imgArray = imgs.load_image(gel[i])
        if(len(np.shape(imgArray)) == 3):
            imgArray = imgArray[:,:,1]
        Xtest.append(imgArray)
        Ytest.append([0])   

    Ytrain = np_utils.to_categorical(Ytrain, 2)
    Ytrain = np.array(Ytrain)
    Ytest = np_utils.to_categorical(Ytest, 2)
    Ytest = np.array(Ytest)
    c = np.c_[np.array(Xtrain).reshape(len(Xtrain), -1), np.array(Ytrain).reshape(len(Ytrain), -1)]
    np.random.shuffle(c)
    Xtrain = c[:, :np.array(Xtrain).size//len(Xtrain)].reshape(np.array(Xtrain).shape)
    Ytrain = c[:, np.array(Xtrain).size//len(Xtrain):].reshape(np.array(Ytrain).shape)
    #reshape to fit the network
    Xtrain = np.array(Xtrain).reshape(2000, 1, 106, 106)#np.array(Xtrain).reshape(4415, 1, 106, 106)
    Xtrain = Xtrain.astype('float32')
    Xtest = np.array(Xtest).reshape(200, 1, 106, 106)#np.array(Xtest).reshape(233, 1, 106, 106)
    Xtest = Xtest.astype('float32')

    Xtrain /= 255
    Xtest /= 255

Here is my output:

Epoch 1/10
2000/2000 [==============================] - 73s - loss: 8.0590 - acc: 0.5000     
Epoch 2/10
2000/2000 [==============================] - 75s - loss: 8.0590 - acc: 0.5000     
...SNIP...    
Epoch 7/10
2000/2000 [==============================] - 81s - loss: 8.0590 - acc: 0.5000

Links:
What is batch size in neural network?

http://cs231n.github.io/neural-networks-3/#loss

joelthchao · 2016-05-13T11:20:57Z

Miss activation (e.g. relu) after Convolution2D. I use your network on cifar10 data, loss does not decrease but increase. With activation, it can learn something basic.
Network is too shallow. It's hard to learn with only a convolutional layer and a fully connected layer. Try Alexnet or VGG style to build your network or read examples (cifar10, mnist) in Keras.
I recommend you to take some online courses about deep learning, it would be helpful.

kevkid · 2016-05-13T16:07:24Z

Hi @joelthchao,

I am unsure as to what you mean in 1 for miss activation. Are you saying if you remove the activation the loss increases and when you use activation it learns?

For 2: I actually tried with a deeper network, but I figured since it was giving me no improvement, it may be best to simplify the model and troubleshoot with that.

I am wondering if this could be an issue with my data.

I can try to increasing the depth. Anything else I should look at?

I will post my results from the cifar10.

EDIT: I found this example:

https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py

I will try it and adapt it to my needs. Can anyone explain why we do double convolution like this?

joelthchao · 2016-05-13T16:24:15Z

@kevkid Try this, does loss still not decrease?

model.add(Convolution2D(32, 10, 10, border_mode='same',name='conv1', input_shape = (1, 106, 106)))
model.add(Activation('relu'))
# ...

kevkid · 2016-05-13T16:29:46Z

@joelthchao

Just tried it:

Train on 1600 samples, validate on 400 samples
Epoch 1/20
1600/1600 [==============================] - 60s - loss: 8.8503 - acc: 0.3969 - val_loss: 1.1921e-07 - val_acc: 1.0000
Epoch 2/20
1600/1600 [==============================] - 60s - loss: nan - acc: 0.3750 - val_loss: nan - val_acc: 1.0000
Out[17]: <keras.callbacks.History at 0x7f0cc152f7d0>

loss goes to nan. Could the weights be blowing up? I will try the example from keras for cifar 10.

kevkid · 2016-05-13T21:11:52Z

Could this be my architecture? Are there any resources for designing the neural network? Here is how the model currently looks:

model = Sequential()

model.add(Convolution2D(32, 5, 5, border_mode='same',name='conv1_1', input_shape = (1, 106, 106)))
#first_layer = model.layers[0]
# this is a placeholder tensor that will contain our generated images
#input_img = first_layer.input
#dream = input_img
model.add(Activation("relu"))
model.add(Convolution2D(32, 5, 5, border_mode='same',name='conv1_2'))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

#2
model.add(Convolution2D(64, 5, 5, border_mode='same',name='conv2_1'))
model.add(Activation("relu"))
model.add(Convolution2D(64, 5, 5, border_mode='same',name='conv2_2'))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

#flatten
model.add(Flatten())
model.add(Dense(512))
model.add(Activation("relu"))
model.add(Dropout(0.5))

model.add(Dense(2))
model.add(Activation('softmax'))

rms = RMSprop()
sgd = SGD(lr=0.01, decay=1e-6, momentum=1, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd,metrics=["accuracy"])
model.fit(Xtrain, Ytrain, batch_size=32, nb_epoch=20,
          verbose=1, validation_split=0.2,
          callbacks=[EarlyStopping(monitor='val_loss', patience=2)])

Based on cifar10 example. I am currently running the model.

kevkid · 2016-05-16T17:58:05Z

I was able to make a decent model that gave me excellent results. I am unsure why rmsprop seems to make the loss go up, but here is my model:

model = Sequential()

model.add(Convolution2D(32, 5, 5, border_mode='same',name='conv1_1', input_shape = (1, 106, 106)))
#first_layer = model.layers[0]
# this is a placeholder tensor that will contain our generated images
#input_img = first_layer.input
#dream = input_img
model.add(Activation("relu"))
model.add(Convolution2D(32, 5, 5, border_mode='same',name='conv1_2'))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

#2
model.add(Convolution2D(64, 5, 5, border_mode='same',name='conv2_1'))
model.add(Activation("relu"))
model.add(Convolution2D(64, 5, 5, border_mode='same',name='conv2_2'))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

#flatten
model.add(Flatten())
#model.add(Dense(512))
#model.add(Activation("relu"))
model.add(Dropout(0.5))

model.add(Dense(2))
model.add(Activation('softmax'))

rms = RMSprop()
sgd = SGD(lr=0.001, decay=1e-6, momentum=0.5, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd,metrics=["accuracy"])
model.fit(Xtrain, Ytrain, batch_size=32, nb_epoch=100,
          verbose=1)

--SNIP--
Epoch 99/100
2000/2000 [==============================] - 11s - loss: 0.0570 - acc: 0.9795     
Epoch 100/100
2000/2000 [==============================] - 11s - loss: 0.0576 - acc: 0.9815     
Out[14]: <keras.callbacks.History at 0x7f204ba9f0d0>
--SNIP--
print('Classifcation rate %02.3f' % model.evaluate(Xtest, Ytest)[1])
200/200 [==============================] - 0s     
Classifcation rate 0.930

111hypo · 2016-05-17T13:19:58Z

I have got the same problem as you , but I guess there must be something worng with my function of load_data, and I am not sure if I get the right Xtrain and Ytrain, so if you please share your code of load_data , thanks a lot.

kevkid · 2016-05-17T13:27:26Z

@111hypo I have posted my loadCustomData function a few posts above this one. The portion :

    c = np.c_[np.array(Xtrain).reshape(len(Xtrain), -1), np.array(Ytrain).reshape(len(Ytrain), -1)]
    np.random.shuffle(c)
    Xtrain = c[:, :np.array(Xtrain).size//len(Xtrain)].reshape(np.array(Xtrain).shape)
    Ytrain = c[:, np.array(Xtrain).size//len(Xtrain):].reshape(np.array(Ytrain).shape)

is unnecessary because we do not need to shuffle the input (This was just a test to try and figure out why My network would not converge).

I still have problems with RMSprop. It quickly gains loss, and the accuracy goes to 0 (which to me is funky). I tried a few different SGDs and the one in my latest post seemed to work the best for me.

alyato · 2016-08-10T08:04:14Z

@kevkid I also meet your problem. I collect 1505 numbers pics as my dataset and use a simple model.
The valid_acc doesn't change. Do you give me some advices. Thanks.

alyato · 2016-08-10T08:32:00Z

@111hypo ,how do you solve your problem.

albertyou2 · 2016-09-14T09:11:57Z

@kevkid
Have you found the solution now?I have met the same problem to yours.

Have you tried to change the ' momentum=1.9 '.I found that this problem may connected to the argument named 'momentum' in SGD optimizter.

I did't find the solution yet but when I changed the momentum to 0.5, the loss changed.But after several epoch , the loss did not change again......

hope this can help you !

DanlanChen · 2016-10-28T02:33:28Z

You may want to reduce your drop out rate and shuffle your data

fatemaaa · 2016-12-26T04:53:51Z

Hello, I used vgg19 architecture to classify my data set into 2 classes but the problem is the value of accuracy doesn't change after 30 iteration and I don't what is the problem and this my code:
data,Label = shuffle(immatrix,label, random_state=2)
train_data = [data,Label]
print (train_data[0].shape) # the train data
print (train_data[1].shape) # the labels of these data
#batch_size to train
batch_size = 16

number of output classes

nb_classes = 3

number of epochs to train

nb_epoch = 150 #each epochs contains around 70000/128=468 batches with 128 images
img_rows, img_cols = 200, 200
(X, y) = (train_data[0],train_data[1])

STEP 1: split X and y into training and testing sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=4)
X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
#Normalization
X_train /= 255
X_test /= 255

compute the total time of prearing and importing the dataset

t_generateArray = time.time()
print('Generating Array Time:{}'.format(t_generateArray - t_start))

print('X_train shape:', X_train.shape)
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

convert class vectors to binary class matrices

Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)

Step 1:Network structure

model = Sequential()
model.add(ZeroPadding2D((1, 1), input_shape=(1, img_rows, img_cols)))
model.add(Convolution2D(64, 3, 3, activation='relu',init='glorot_uniform'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(64, 3, 3, activation='relu',init='glorot_uniform'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(128, 3, 3, activation='relu',init='glorot_uniform'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(128, 3, 3, activation='relu',init='glorot_uniform'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(256, 3, 3, activation='relu',init='glorot_uniform'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(256, 3, 3, activation='relu',init='glorot_uniform'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(256, 3, 3, activation='relu',init='glorot_uniform'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(256, 3, 3, activation='relu',init='glorot_uniform'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu',init='glorot_uniform'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu',init='glorot_uniform'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu',init='glorot_uniform'))
model.add(ZeroPadding2D((1, 1)))
model.add(Convolution2D(512, 3, 3, activation='relu',init='glorot_uniform'))
model.add(MaxPooling2D((2, 2), strides=(2, 2)))

model.add(Flatten())
model.add(Dense(1024, activation='relu',init='glorot_uniform'))
model.add(Dropout(0.5))
model.add(Dense(1024, activation='relu',init='glorot_uniform'))
model.add(Dropout(0.5))
model.add(Dense(nb_classes, activation='softmax')) # neuron number

#step 2: Learning target(computiong the loss using the entropy function
sgd = SGD(lr=0.0001, decay=1e-6, momentum=0.9, nesterov=True)

adagrad = Adagrad(lr=0.01, epsilon=1e-08)

model.compile(loss='categorical_crossentropy',optimizer= sgd,metrics=['accuracy'])

#checkpointer=ModelCheckpoint(filepath='exp_161123_best_lr0.0001_weights.h5',monitor='acc', verbose=1, save_best_only=True, mode='max')

#training the model
print ('Training Start....')
hist = model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
verbose=1, validation_data=(X_test, Y_test))
#,callbacks=[checkpointer]

model.save_weights('final_last5_weights.h5')

#model.save_weights('exp_161123_final_lr0.0001_weights.h5')

#evaluate the model
score = model.evaluate(X_test, Y_test,verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

redouanelg · 2017-03-13T11:49:02Z

If you have unbalanced classes, maybe you should consider weighting classes, check class_weight and sample_weight in Keras docs

Here's a similar question asked on stackoverflow
http://stackoverflow.com/questions/41881220/keras-predict-always-output-same-value-in-multi-classification

alyato · 2017-03-14T03:15:53Z

@redouanelg Do you mean by adding sample_weight in fit()?
Or do you give an example how to do ? thanks.

redouanelg · 2017-03-18T15:38:48Z

@alyato Sorry for the late reply.

My modest experience tells me that if you have only two classes use a dict in class_weight.

If you have more you'll get the error class_weight not supported for +3 dim.

A way to overcome this consists in adding sample_weight in fit() using a 2D weight array (one weight per timestep per sample), and adding sample_weight_mode="temporal" in compile()

It's not an elegant solution but it works. I'll be glad if someone has another answer.

dityas · 2017-03-24T21:16:26Z

Hey, i am having a similar problem i am trying to train a network to learn word embeddings using skip grams. i have a vocabulary of 256 and a sequence of about 166000 words. But when i train, the accuracy stays the same at around 0.1327 no matter what i do, i tried changing learning rates and batch_size. But no luck. This has happened every time i used keras. But it usually starts learning after tweaking the batch size a bit. But this one just doesn't work.

Here is the model:
`

  def make_model(self,vocab_size=256,vec_dim=100):
	model=Sequential()
	model.add(Dense(vec_dim,activation="sigmoid",input_dim=vocab_size))
	model.add(Dense(vocab_size,activation="sigmoid"))
	sgd=SGD(lr=1.0)
	model.compile(loss="categorical_crossentropy",optimizer=sgd,metrics=["accuracy"])
	return model

def _get_callbacks(self):
	earlystop=EarlyStopping(monitor="val_loss",min_delta=0.0001,patience=10,verbose=2)
	checkpoint=ModelCheckpoint("checkpt.hdf5",period=10,verbose=2)
	reducelr=ReduceLROnPlateau(monitor="val_loss",factor=0.1,patience=5,verbose=2)
	return [earlystop,checkpoint,reducelr]

def train(self,model,X,y):
	model.fit(X,y,nb_epoch=1000,callbacks=self._get_callbacks(),validation_split=0.1,verbose=2,batch_size=300)
	model.save("model.hdf5")

`
X is a one hot vector of len 256 for every word
y in a one hot vector of len 256 representing the skip word in the context of X.
so for instance is the sequence is [2,6,5,7,9]
X will be
[5,5,5,5,7,7,7...]

y will be
[2,6,7,9,6,5,9...]
and so on for every word in the sequence.

This is what happens when i try to train:

9s - loss: 4.5012 - acc: 0.1794 - val_loss: 4.5873 - val_acc: 0.1327
Epoch 2/1000
9s - loss: 4.2679 - acc: 0.1801 - val_loss: 4.8339 - val_acc: 0.1327
Epoch 3/1000
9s - loss: 4.2363 - acc: 0.1801 - val_loss: 4.7040 - val_acc: 0.1327
Epoch 4/1000
9s - loss: 4.2102 - acc: 0.1801 - val_loss: 4.6947 - val_acc: 0.1327
Epoch 5/1000
9s - loss: 4.1882 - acc: 0.1801 - val_loss: 4.6625 - val_acc: 0.1327
Epoch 6/1000
9s - loss: 4.1777 - acc: 0.1801 - val_loss: 4.6303 - val_acc: 0.1327

I ve waited for a about 50 epochs and the acc still does not change.

Any idea what i am doing wrong? Ive faced this problem everytime ive used keras even when training other models like language modelling using RNNs text generation using LSTMs.

arielbenitah · 2017-04-05T19:27:58Z

Hi @adityashinde1506, You may have already found a solution but if not, try to decrease your learning rate.

vijaydwivedi75 · 2017-06-18T12:19:53Z

Hi guys, I am having a similar problem. I am training an LSTM model for text classification and my loss does not improve on subsequent epochs. I tried many optimizers with different learning rates. But same problem.

max_length = 275
X_train = sequence.pad_sequences(train_data_new, maxlen=max_length, padding='post')
X_test = sequence.pad_sequences(test_data_new, maxlen=max_length, padding='post')

y_train = []
y_test = []

# preparing y_test and y_train
for label in train_label:
	if label == 'first':
		y_train.append([1,0])
	else:
		y_train.append([0,1])

y_train = np.array(y_train)

for label in test_label:
	if label == 'second':
		y_test.append([1,0])
	else:
		y_test.append([0,1])
	
y_test = np.array(y_test)

# Create the model
rmsprop = RMSprop(lr=0.1)
sgd = SGD(lr=0.1)
model = Sequential()
model.add(Embedding(len(word_index) + 1, EMBEDDING_DIM, 
	weights=[embedding_matrix], 
	input_length=max_length,
	trainable=True))
# model.add(Dropout(0.2))
model.add(LSTM(128, return_sequences=True))
# model.add(Dropout(0.2))
model.add(LSTM(64))
model.add(Dense(2, activation='sigmoid'))
model.compile(loss='categorical_crossentropy', optimizer=rmsprop, metrics=['accuracy'])
print(model.summary())

print('Training model...')
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=64)

# Final evaluation of the model
scores = model.evaluate(X_test, y_test, verbose=0)
print("Accuracy: %.2f%%" % (scores[1]*100))

The OUTPUT I get is this:

Total train samples:  931
Total test samples:  390
Processing text dataset
Found 6132 unique tokens.
Indexing word vectors.
Found 400000 word vectors.
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_1 (Embedding)      (None, 275, 200)          1226600   
_________________________________________________________________
lstm_1 (LSTM)                (None, 275, 128)          168448    
_________________________________________________________________
lstm_2 (LSTM)                (None, 64)                49408     
_________________________________________________________________
dense_1 (Dense)              (None, 2)                 130       
=================================================================
Total params: 1,444,586
Trainable params: 1,444,586
Non-trainable params: 0
_________________________________________________________________
None
Training model...
Train on 931 samples, validate on 390 samples
Epoch 1/10
931/931 [==============================] - 16s - loss: 0.8362 - acc: 0.5081 - val_loss: 0.6939 - val_acc: 0.4949
Epoch 2/10
931/931 [==============================] - 14s - loss: 0.6935 - acc: 0.5016 - val_loss: 0.6936 - val_acc: 0.4949
Epoch 3/10
931/931 [==============================] - 14s - loss: 0.6933 - acc: 0.5016 - val_loss: 0.6933 - val_acc: 0.4949
Epoch 4/10
931/931 [==============================] - 14s - loss: 0.6933 - acc: 0.5016 - val_loss: 0.6933 - val_acc: 0.4949
Epoch 5/10
931/931 [==============================] - 14s - loss: 0.6933 - acc: 0.5016 - val_loss: 0.6932 - val_acc: 0.4949
Epoch 6/10
931/931 [==============================] - 14s - loss: 0.6932 - acc: 0.5016 - val_loss: 0.6932 - val_acc: 0.4949
Epoch 7/10
931/931 [==============================] - 14s - loss: 0.6932 - acc: 0.5016 - val_loss: 0.6932 - val_acc: 0.4949
Epoch 8/10
931/931 [==============================] - 13s - loss: 0.6932 - acc: 0.5016 - val_loss: 0.6931 - val_acc: 0.4949
Epoch 9/10
931/931 [==============================] - 13s - loss: 0.6931 - acc: 0.5016 - val_loss: 0.6931 - val_acc: 0.4949
Epoch 10/10
931/931 [==============================] - 14s - loss: 0.6931 - acc: 0.5016 - val_loss: 0.6931 - val_acc: 0.4949
Accuracy: 49.49%

I am unable to figure out what the problem is.

td2014 · 2017-06-18T14:32:42Z

Out of curiosity, why are you passing in a "weights" matrix to the Embedding layer? Thanks.

vijaydwivedi75 · 2017-06-18T14:42:23Z

Hi @td2014 , that weights in Embedding layer is just because i want to give my own embeddings (GloVe in this case) for the word inputs. Even when i removed the weights and ran the file, my loss didnt change.

td2014 · 2017-06-18T15:02:07Z

Okay. And you set the learning rate to 0.1 for your optimizer(s). Just curious, but was the default not working? Thanks.

vijaydwivedi75 · 2017-06-18T18:05:56Z

Initially, it was default. Then I read on a similar issue page on stackoverflow where it told to alter learning rates. So I was trying to see the change on different values. Using lr=0.1 the loss starts from 0.83 and becomes constant at 0.69. When I was using default value, loss was stuck same at 0.69

td2014 · 2017-06-18T20:12:35Z

Okay. I created a simplified version of what you have implemented, and it does seem to work (loss decreases). Here is the code you can cut and paste. Note that the first section is setting up the environment for reproducible results (which I provide at the end in my case). In your case, you may want to check a few things:

Is your input data making sense? It could be that the preprocessing steps (the padding) are creating input sequences that cannot be separated (perhaps you are getting a lot of zeros or something of that sort).
You might want to simplify your architecture to include just a single LSTM layer (like I did) just until you convince yourself that the model is actually learning something.

I hope this helps. Thanks.

Start: Set up environment for reproduction of results

import numpy as np
import tensorflow as tf
import random as rn
import os
os.environ['PYTHONHASHSEED'] = '0'
np.random.seed(42)
rn.seed(12345)
#single thread
session_conf = tf.ConfigProto(
intra_op_parallelism_threads=1,
inter_op_parallelism_threads=1)

from keras import backend as K
tf.set_random_seed(1234)
sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)
K.set_session(sess)

End: Set up environment for reproduction of results

from keras.layers import LSTM, Dense, Embedding
from keras.models import Sequential
from keras.preprocessing import sequence

Create input sequences

word_index=21
train_data_new = []
train_data_new.append([1, 1, 1, 1, 1, 1, 1 ])
train_data_new.append([2, 2, 2, 2, 2, 2, 2 ])
train_data_new.append([2, 2, 2, 2, 2, 2, 2 ])
train_data_new.append([1, 1, 1, 1, 1, 1, 1 ])
train_data_new.append([2, 2, 2, 2, 2, 2, 2 ])
train_data_new.append([2, 2, 2, 2, 2, 2, 2 ])
train_data_new.append([1, 1, 1, 1, 1, 1, 1 ])
train_data_new.append([2, 2, 2, 2, 2, 2, 2 ])
train_data_new.append([2, 2, 2, 2, 2, 2, 2 ])

Preprocess

max_length = 5
X_train = sequence.pad_sequences(train_data_new, maxlen=max_length, padding='post')

preparing y_train

y_train = []
y_train.append([1,0])
y_train.append([0,1])
y_train.append([0,1])
y_train.append([1,0])
y_train.append([0,1])
y_train.append([0,1])
y_train.append([1,0])
y_train.append([0,1])
y_train.append([0,1])

y_train = np.array(y_train)

Create model

EMBEDDING_DIM=16

model = Sequential()
model.add(Embedding(word_index + 1, EMBEDDING_DIM, input_length=max_length))
model.add(LSTM(5))
model.add(Dense(2, activation='sigmoid'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model.summary())

Train

print('Training model...')
model.fit(X_train, y_train, epochs=10, shuffle=False)

output predictions

predictions = model.predict(X_train)

====OUTPUT BELOW====

Layer (type) Output Shape Param #

embedding_29 (Embedding) (None, 5, 16) 352

lstm_29 (LSTM) (None, 5) 440

dense_29 (Dense) (None, 2) 12

Total params: 804.0
Trainable params: 804
Non-trainable params: 0.0

None
Training model...
Epoch 1/10
9/9 [==============================] - 2s - loss: 0.7013 - acc: 0.3333
Epoch 2/10
9/9 [==============================] - 0s - loss: 0.6953 - acc: 0.3333
Epoch 3/10
9/9 [==============================] - 0s - loss: 0.6911 - acc: 0.3333
Epoch 4/10
9/9 [==============================] - 0s - loss: 0.6875 - acc: 1.0000
Epoch 5/10
9/9 [==============================] - 0s - loss: 0.6842 - acc: 1.0000
Epoch 6/10
9/9 [==============================] - 0s - loss: 0.6812 - acc: 1.0000
Epoch 7/10
9/9 [==============================] - 0s - loss: 0.6783 - acc: 1.0000
Epoch 8/10
9/9 [==============================] - 0s - loss: 0.6754 - acc: 1.0000
Epoch 9/10
9/9 [==============================] - 0s - loss: 0.6726 - acc: 1.0000
Epoch 10/10
9/9 [==============================] - 0s - loss: 0.6698 - acc: 1.0000

unnir · 2017-06-28T14:18:03Z

@td2014

Dude, you architecture just does not work. Try something different.

vijaydwivedi75 · 2017-06-28T14:27:12Z

It works, I had to clean the data. Then the loss started to converge

jianning-li · 2017-08-09T11:48:38Z

try 'sigmoid' activation for the last layer since it's a binary classification problem

whistler · 2017-08-30T23:20:22Z

Here is a good list of issues to check for that I have found useful: https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607

tebzito · 2017-09-14T05:40:08Z

Another reason could be class imbalance. So try upsampling or downsampling using SMOTE/OneSidedSelection from imblearn package, then reshape your data back to 4 dimensions for your model.

taninaim · 2018-01-22T07:29:27Z

I encountered the problem while I was trying to finetune a pretrained VGGFace model, using keras_vggface.utils.preprocess_input as my custom preprocessing function.

def preprocess_input(x, data_format=None, version=1):
    if data_format is None:
        data_format = K.image_data_format()
    assert data_format in {'channels_last', 'channels_first'}

    if version == 1:
        if data_format == 'channels_first':
            x = x[:, ::-1, ...]
            x[:, 0, :, :] -= 93.5940
            x[:, 1, :, :] -= 104.7624
            x[:, 2, :, :] -= 129.1863
        else:
            x = x[..., ::-1]
            x[..., 0] -= 93.5940
            x[..., 1] -= 104.7624
            x[..., 2] -= 129.1863

    elif version == 2:
        if data_format == 'channels_first':
            x = x[:, ::-1, ...]
            x[:, 0, :, :] -= 91.4953
            x[:, 1, :, :] -= 103.8827
            x[:, 2, :, :] -= 131.0912
        else:
            x = x[..., ::-1]
            x[..., 0] -= 91.4953
            x[..., 1] -= 103.8827
            x[..., 2] -= 131.0912
    else:
        raise NotImplementedError

    return x

The problem seems to come from the scaling. I used preprocessing_function=keras_vggface.utils.preprocess_input and got into that problem. However, when I rescale it with 1/255. the problem is fixed. I think it may be that the pretrained model was trained additionally with a scaling factor to normalize it to [0,1], but the preprocessing function only gives us the mean so we know which means to subtract to center the data. I'd recommend you check if your scaling makes sense; a bad scaling of inputs into a Neural Network may cause your updates to either move very slowly (i.e. the derivative of the sigmoid function beyond -3 and +3 are near 0 and so your gradients are almost 0), or if you're using something like the ReLU function, the updates may be big (the derivative is 1) and a wrong update makes you jump pass the local minima very easily.

ALSO, if you're rescaling in python 2, make sure you have that dot in 1/255., or else all your inputs will be multiplied by 0 and you aren't making any updates!!!

lixiang-ucas · 2018-02-27T00:53:00Z

sigmoid_cross_entropy_with_logits may encounters the gradients explosion problem, try using clip_gradients.

yosunpeng · 2018-08-18T13:29:43Z

In my case, It is the normalization problem:
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')

x_train /= 255
x_test /= 255 # normalization [0, 255]--->[0, 1]

sujanme25 · 2018-10-01T12:49:31Z

I am trying to Create DNN but it is not converging, any idea
model = Sequential()
model.add(Dense(5000, input_dim=5, activation='relu', kernel_regularizer=regularizers.l2(0.1)))
model.add(Dropout(0.1))
model.add(Dense(2000, kernel_regularizer=regularizers.l2(0.1),activation='relu'))
model.add(Dropout(0.1))
model.add(Dense(400, kernel_regularizer=regularizers.l2(0.1),activation='relu'))
model.add(Dropout(0.1))
model.add(Dense(1,activation='relu'))

rmsprop = optimizers.RMSprop(lr=0.01, rho=0.7, epsilon=1e-8, decay=0.0)
sgd = optimizers.SGD(lr=0.001, decay=1e-6, momentum=0.4, nesterov=True)
model.compile(loss='mse', optimizer='adam', metrics=["mae"])
callbacks = [EarlyStopping(monitor='val_loss', patience=5),
ModelCheckpoint(filepath='DNN_Adam.h5', monitor='val_loss', save_best_only=True)]
np.random.seed(3)
history=model.fit(X_train, y_train, epochs=500 , batch_size=5000, validation_split=0.1, verbose=2,callbacks=callbacks)
Train on 16109 samples, validate on 1790 samples
Epoch 1/500

20s - loss: 322.9844 - mean_squared_error: 7.8310e-04 - val_loss: 243.3298 - val_mean_squared_error: 2.3419e-08
Epoch 2/500
15s - loss: 216.4914 - mean_squared_error: 6.4440e-04 - val_loss: 156.6757 - val_mean_squared_error: 2.3419e-08
Epoch 3/500
15s - loss: 137.8335 - mean_squared_error: 6.8178e-04 - val_loss: 96.6401 - val_mean_squared_error: 2.3419e-08
Epoch 4/500
15s - loss: 84.1424 - mean_squared_error: 7.0834e-04 - val_loss: 57.3809 - val_mean_squared_error: 2.3419e-08
Epoch 5/500
15s - loss: 49.5767 - mean_squared_error: 7.1517e-04 - val_loss: 33.2029 - val_mean_squared_error: 2.3419e-08
Epoch 6/500
15s - loss: 28.6330 - mean_squared_error: 7.1524e-04 - val_loss: 19.2849 - val_mean_squared_error: 2.3419e-08
Epoch 7/500
15s - loss: 16.7852 - mean_squared_error: 7.1524e-04 - val_loss: 11.7314 - val_mean_squared_error: 2.3419e-08
Epoch 8/500
15s - loss: 10.4144 - mean_squared_error: 7.1524e-04 - val_loss: 7.7523 - val_mean_squared_error: 2.3419e-08
Epoch 9/500
15s - loss: 7.0391 - mean_squared_error: 7.1524e-04 - val_loss: 5.5379 - val_mean_squared_error: 2.3419e-08
Epoch 10/500
15s - loss: 5.0998 - mean_squared_error: 7.1524e-04 - val_loss: 4.1133 - val_mean_squared_error: 2.3419e-08
Epoch 11/500
15s - loss: 3.7908 - mean_squared_error: 7.1524e-04 - val_loss: 3.0279 - val_mean_squared_error: 2.3419e-08
Epoch 12/500
16s - loss: 2.7628 - mean_squared_error: 7.1524e-04 - val_loss: 2.1295 - val_mean_squared_error: 2.3419e-08
Epoch 13/500
16s - loss: 1.9126 - mean_squared_error: 7.1524e-04 - val_loss: 1.4014 - val_mean_squared_error: 2.3419e-08
Epoch 14/500
18s - loss: 1.2362 - mean_squared_error: 7.1524e-04 - val_loss: 0.8581 - val_mean_squared_error: 2.3419e-08
Epoch 15/500
18s - loss: 0.7441 - mean_squared_error: 7.1524e-04 - val_loss: 0.4902 - val_mean_squared_error: 2.3419e-08
Epoch 16/500
16s - loss: 0.4204 - mean_squared_error: 7.1524e-04 - val_loss: 0.2675 - val_mean_squared_error: 2.3419e-08
Epoch 17/500
16s - loss: 0.2305 - mean_squared_error: 7.1524e-04 - val_loss: 0.1482 - val_mean_squared_error: 2.3419e-08
Epoch 18/500
16s - loss: 0.1316 - mean_squared_error: 7.1524e-04 - val_loss: 0.0910 - val_mean_squared_error: 2.3419e-08
Epoch 19/500
16s - loss: 0.0850 - mean_squared_error: 7.1524e-04 - val_loss: 0.0645 - val_mean_squared_error: 2.3419e-08
Epoch 20/500
15s - loss: 0.0629 - mean_squared_error: 7.1524e-04 - val_loss: 0.0500 - val_mean_squared_error: 2.3419e-08
Epoch 21/500
16s - loss: 0.0496 - mean_squared_error: 7.1524e-04 - val_loss: 0.0388 - val_mean_squared_error: 2.3419e-08
Epoch 22/500
17s - loss: 0.0388 - mean_squared_error: 7.1524e-04 - val_loss: 0.0285 - val_mean_squared_error: 2.3419e-08
Epoch 23/500
16s - loss: 0.0289 - mean_squared_error: 7.1524e-04 - val_loss: 0.0196 - val_mean_squared_error: 2.3419e-08
Epoch 24/500
15s - loss: 0.0204 - mean_squared_error: 7.1524e-04 - val_loss: 0.0127 - val_mean_squared_error: 2.3419e-08
Epoch 25/500

SujanaC · 2018-10-12T22:48:58Z

I had a similar issue today when training on Google cloud GPU. I tried changing network architecture, weights, etc. The solution was to reset the TF graph: tf_reset_default_graph()
Somehow the GPU seemed to have a "memory" across different runs and was stuck at a local minima.

psy-mas · 2018-10-30T05:21:22Z

I had a similar issue today when training on Google cloud GPU. I tried changing network architecture, weights, etc. The solution was to reset the TF graph: tf_reset_default_graph()
Somehow the GPU seemed to have a "memory" across different runs and was stuck at a local minima.

Thx，I will have a try，hope it would work！

Fellfalla · 2018-12-12T16:34:49Z

For me, theese 3 things did the trick:

Lower the learning rate (0.1 converges too fast and already after the first epoch, there is no change anymore). Just for test purposes try a very low value like lr=0.00001.
Check the input for proper value range and normalize it
Add BatchNormalization (model.add(BatchNormalization())) after each layer

Alfatlawi · 2019-03-21T15:48:35Z

For me, this works:
Add BatchNormalization (model.add(BatchNormalization())) after each layer
Thanks @Fellfalla

DiyuanLu · 2019-03-25T13:49:58Z

I had the same problem. But for me it is just the learning_rate is too small. Try with a bigger learning_rate. It might help

qianzzzz · 2019-11-13T07:24:46Z

Based on my own experience as a starter, one possible reason or bug in your model is that you probably used a wrong activation function, i.e. the way you activated your result at the last output layer, for example, if you are trying to solve a multi class proplem, usually we use softmat rather sigmoid, while sigmoid is meant to activate the output for binary task. And in this case, it's a binary application, therefore just change your activation function as sigmoid, you should not find such exception.

Dulum · 2020-02-13T11:39:14Z

Try removing the Activation('softmax') layer.

QoT · 2020-03-12T06:31:19Z

I had a model that did not train at all. It just stucks at random chance of particular result with no loss improvement during training. Loss was constant 4.000 and accuracy 0.142 on 7 target values dataset.

It become true that I was doing regression with ReLU last activation layer, which is obviously wrong.

Before I was knowing that this is wrong, I did add Batch Normalisation layer after every learnable layer, and that helps. However, training become somehow erratic so accuracy during training could easily drop from 40% down to 9% on validation set. Accuracy on training dataset was always okay.

Then I realized that it is enough to put Batch Normalisation before that last ReLU activation layer only, to keep improving loss/accuracy during training. That probably did fix wrong activation method.

However, when I did replace ReLU with Linear activation (for regression), no Batch Normalisation was needed any more and model started to train significantly better.

1337-Pete · 2020-04-23T13:05:36Z

I'd highly recommend messing around with learning rates... Testing with extreme variability, like lr = 0.1 & lr = 1e-4, should do the trick in most instances.

amapic · 2020-05-07T09:05:24Z

I think that the problem comes from the learning rate, Mine was actually equal to 7 ahah. I wrote 10-3 instead of 1e-3.

stock-ds · 2020-09-17T11:48:26Z

My data had 3 classes but last layer was Dense(1, activation='sigmoid') changing it to Dense(3, activation='sigmoid') made the loss change. With the 1 output neuron it didn't return any errors, just had a constant loss.

Also if you are training binary classifier, you can just use Dense(1, activation='sigmoid') as output with binary_crossentropy, instead of Dense(2, activation='sigmoid') with categorical_crossentropy

wuditianyou · 2021-01-14T05:15:22Z

Happened many times, my solution is change or remove the activation function in the last layer.

soans1994 · 2021-09-01T17:15:10Z

Try removing the Activation('softmax') layer.

finally

kevkid closed this as completed May 16, 2016

liaoandi mentioned this issue May 12, 2018

train_loss, train_acc, valid_loss, valid_acc not changing when training UC-MACSS/persp-research_Spr18#34

Open

kumariko mentioned this issue Sep 20, 2021

precision and recall values kept unchanged for some training steps. tensorflow/tensorflow#19138

Closed

shibbirtanvin mentioned this issue Feb 22, 2022

RFC keras-team/governance#34

Closed

Loss not changing when training #2711

Loss not changing when training #2711

Comments

kevkid commented May 12, 2016

dandxy89 commented May 12, 2016

kevkid commented May 12, 2016 • edited Loading

kevkid commented May 13, 2016

Here are a few things I tried:

Code:

Here is the code for me to read in the images if it helps:

Here is my output:

joelthchao commented May 13, 2016

kevkid commented May 13, 2016 • edited Loading

joelthchao commented May 13, 2016

kevkid commented May 13, 2016

kevkid commented May 13, 2016 • edited Loading

kevkid commented May 16, 2016 • edited Loading

111hypo commented May 17, 2016

kevkid commented May 17, 2016

alyato commented Aug 10, 2016

alyato commented Aug 10, 2016

albertyou2 commented Sep 14, 2016

DanlanChen commented Oct 28, 2016

fatemaaa commented Dec 26, 2016

number of output classes

number of epochs to train

STEP 1: split X and y into training and testing sets

compute the total time of prearing and importing the dataset

convert class vectors to binary class matrices

Step 1:Network structure

adagrad = Adagrad(lr=0.01, epsilon=1e-08)

model.save_weights('final_last5_weights.h5')

redouanelg commented Mar 13, 2017

alyato commented Mar 14, 2017

redouanelg commented Mar 18, 2017

dityas commented Mar 24, 2017

arielbenitah commented Apr 5, 2017

vijaydwivedi75 commented Jun 18, 2017

td2014 commented Jun 18, 2017

vijaydwivedi75 commented Jun 18, 2017

td2014 commented Jun 18, 2017

vijaydwivedi75 commented Jun 18, 2017

td2014 commented Jun 18, 2017

Start: Set up environment for reproduction of results

End: Set up environment for reproduction of results

Create input sequences

Preprocess

preparing y_train

Create model

Train

output predictions

Layer (type) Output Shape Param #

dense_29 (Dense) (None, 2) 12

unnir commented Jun 28, 2017

vijaydwivedi75 commented Jun 28, 2017

jianning-li commented Aug 9, 2017

whistler commented Aug 30, 2017

tebzito commented Sep 14, 2017

taninaim commented Jan 22, 2018 • edited Loading

lixiang-ucas commented Feb 27, 2018

yosunpeng commented Aug 18, 2018

sujanme25 commented Oct 1, 2018 • edited Loading

SujanaC commented Oct 12, 2018

psy-mas commented Oct 30, 2018

Fellfalla commented Dec 12, 2018 • edited Loading

Alfatlawi commented Mar 21, 2019

DiyuanLu commented Mar 25, 2019

qianzzzz commented Nov 13, 2019

Dulum commented Feb 13, 2020

QoT commented Mar 12, 2020

1337-Pete commented Apr 23, 2020

amapic commented May 7, 2020 • edited Loading

stock-ds commented Sep 17, 2020 • edited Loading

wuditianyou commented Jan 14, 2021

soans1994 commented Sep 1, 2021

kevkid commented May 12, 2016 •

edited

Loading

kevkid commented May 13, 2016 •

edited

Loading

kevkid commented May 13, 2016 •

edited

Loading

kevkid commented May 16, 2016 •

edited

Loading

taninaim commented Jan 22, 2018 •

edited

Loading

sujanme25 commented Oct 1, 2018 •

edited

Loading

Fellfalla commented Dec 12, 2018 •

edited

Loading

amapic commented May 7, 2020 •

edited

Loading

stock-ds commented Sep 17, 2020 •

edited

Loading