You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I used the "2_model_SimCR.ipynb" to train a model. After two epochs, the loss function returns "nan" values and messes up the training.
I wonder if you have any solution for this?
Train for 53 steps, validate for 12 steps
Epoch 1/5
52/53 [============================>.] - ETA: 4s - loss: 582.9939
Epoch 00001: val_loss improved from inf to 497.00585, saving model to models/trashnet/SimCLR/SimCLR_05_05_11h_05.h5
53/53 [==============================] - 395s 7s/step - loss: 581.4498 - val_loss: 497.0059
Epoch 2/5
52/53 [============================>.] - ETA: 0s - loss: 421.8934
Epoch 00002: val_loss improved from 497.00585 to 342.78980, saving model to models/trashnet/SimCLR/SimCLR_05_05_11h_05.h5
53/53 [==============================] - 36s 675ms/step - loss: 420.4594 - val_loss: 342.7898
Epoch 3/5
52/53 [============================>.] - ETA: 0s - loss: 278.3572
Epoch 00003: val_loss improved from 342.78980 to 213.78286, saving model to models/trashnet/SimCLR/SimCLR_05_05_11h_05.h5
53/53 [==============================] - 37s 694ms/step - loss: 277.1834 - val_loss: 213.7829
Epoch 4/5
52/53 [============================>.] - ETA: 0s - loss: nan
Epoch 00004: val_loss did not improve from 213.78286
53/53 [==============================] - 34s 643ms/step - loss: nan - val_loss: nan
Epoch 5/5
52/53 [============================>.] - ETA: 0s - loss: nan
Epoch 00005: val_loss did not improve from 213.78286
53/53 [==============================] - 34s 639ms/step - loss: nan - val_loss: nan
trainable parameters: 11.86 M.
non-trainable parameters: 4.05 M.
Random guess accuracy: 0.0156
accuracy - test - before: 0.74
accuracy - test - after: nan
y_predict_test_before
0.73 | 0.77 | 0.66 | 0.92 | 0.95 | 0.22 | 0.51 | 0.92 | 0.71 | 0.9 | 0.11 | 0.84 | 0.84 | 0.8 | 0.69 |
The text was updated successfully, but these errors were encountered:
System used:
Ubuntu 18.04
Tensorflow-gpu 2.1
I used the "2_model_SimCR.ipynb" to train a model. After two epochs, the loss function returns "nan" values and messes up the training.
I wonder if you have any solution for this?
The text was updated successfully, but these errors were encountered: