New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Early stopping does not working as expected when using ClassificationScoreCalculator #6237

Closed
ajan1019 opened this Issue Aug 21, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@ajan1019
Copy link

ajan1019 commented Aug 21, 2018

When I use early stopping with ClassificationScoreCalculator it terminates where it shouldn't break.

For example,

This is my LSTM architecture.

MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
//workspaces will be used for training (reduced memory and better performance)
.seed(123)
.weightInit(WeightInit.XAVIER)
.optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
.updater(new RmsProp(0.001))
.activation(Activation.ELU)
.l2(1e-5)
.gradientNormalization(GradientNormalization
.ClipElementWiseAbsoluteValue)
.gradientNormalizationThreshold(1.0)
.list()
.layer(0, new LSTM.Builder()
.nIn(300)
.nOut(200)
.activation(Activation.TANH)
.gateActivationFunction(Activation.SIGMOID)
.dropOut(0.5)
.build())
.layer(1, new RnnOutputLayer.Builder()
.nIn(200)
.nOut(outputs)
.activation(Activation.SOFTMAX)
.lossFunction(LossFunctions.LossFunction.MCXENT)
.build())
.pretrain(false).backprop(true).build();

Below is my early stopping configuration.
EarlyStoppingConfiguration<MultiLayerNetwork> esConf =
new EarlyStoppingConfiguration.Builder<MultiLayerNetwork>()
.epochTerminationConditions(
new MaxEpochsTerminationCondition(10),
new ScoreImprovementEpochTerminationCondition(1))
.iterationTerminationConditions(
new MaxTimeIterationTerminationCondition(10, TimeUnit.MINUTES))
.scoreCalculator(new ClassificationScoreCalculator(Evaluation.Metric.F1, testingDataSetIterator))
.modelSaver(saver)
.saveLastModel(true)
.build();

My output:

Completed training epoch 0
Score at epoch 0: 0.4175824175824176
Completed training epoch 1
New best model: score = 0.5755395683453237, epoch = 1 (previous: score = 0.4175824175824176, epoch = 0)
Hit epoch termination condition at epoch 1. Details: ScoreImprovementEpochTerminationCondition(maxEpochsWithNoImprovement=1, minImprovement=0.0)
Termination reason: EpochTerminationCondition
Termination details: ScoreImprovementEpochTerminationCondition(maxEpochsWithNoImprovement=1, minImprovement=0.0)
Total epochs: 2
Best epoch number: 1
Score at best epoch: 0.5755395683453237

Hope my example make it clear.
We had improvement in score(we used classification score calculator). But early stopping terminated with the reason of not improving score after one epoch.

@AlexDBlack AlexDBlack self-assigned this Aug 22, 2018

@AlexDBlack

This comment has been minimized.

Copy link
Member

AlexDBlack commented Aug 22, 2018

Thanks for reporting: fixed here - #6242
FYI the issue was that minimize vs. maximize of scores weren't being handled correctly for termination conditions.
The fix will be available on snapshots a few hours after the PR has been merged
https://deeplearning4j.org/docs/latest/deeplearning4j-config-snapshots

@ajan1019

This comment has been minimized.

Copy link

ajan1019 commented Aug 28, 2018

Thanks for the fix :)

@lock

This comment has been minimized.

Copy link

lock bot commented Sep 27, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Sep 27, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.