Valid loss explode after several epochs

[Hugh0120]

here is the picture of my training procedure, after several epochs' of decreasing, the valid loss suddenly exploded, did it happened to you when you were training?

[AnaRhisT94]

@Hugh0120 Hi, how did you get the model to train in the first place? I'm still struggling.
About the valid loss that suddenly exploded, try to lower the LR by a factor of 10 when getting to 20 epochs.

[Hugh0120]

@Hugh0120 Hi, how did you get the model to train in the first place? I'm still struggling.
About the valid loss that suddenly exploded, try to lower the LR by a factor of 10 when getting to 20 epochs.

I just trained the model from scratch

[Hugh0120]

@Hugh0120 Hi, how did you get the model to train in the first place? I'm still struggling.
About the valid loss that suddenly exploded, try to lower the LR by a factor of 10 when getting to 20 epochs.

btw, I change the code from keras to tf estimator and the loss decreased properly

[dehanlazuardi]

@Hugh0120 for me the validation loss is exploding from the beginning. disable the ReduceLROnPlateau(verbose=1) in keras callback will do the job.