Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

java.lang.IllegalStateException: Cannot perform backprop: Dropout mask array is absent (already cleared?) #6756

Closed
wangfeng-skymind opened this issue Nov 23, 2018 · 3 comments

Comments

@wangfeng-skymind
Copy link

commented Nov 23, 2018

Description : this deeplearning4j version is 1.0.0-beta3
Run some code , and then show some exception, below code
is only a little part ==================================
ZooModel zooModel = ResNet50.builder().workspaceMode(WorkspaceMode.NONE).build();
ComputationGraph resnet50 = (ComputationGraph) zooModel.initPretrained();
System.out.println(resnet50.summary());

    FineTuneConfiguration fineTuneConf = new FineTuneConfiguration.Builder()
            .activation(Activation.LEAKYRELU)
            .weightInit(WeightInit.RELU)
            .updater(new Nesterovs(5e-5))
            .dropOut(0.5)
            .seed(seed)
            .build();

    graph = new TransferLearning.GraphBuilder(resnet50)
            .fineTuneConfiguration(fineTuneConf)
            .setInputTypes(InputType.convolutional(height, width, channels))
            .removeVertexKeepConnections("conv1")
            .addLayer("conv1", new ConvolutionLayer.Builder(new int[]{3, 3})
                    .nIn(channels).nOut(64).activation( Activation.RELU).build(), "input_1")
            .addLayer("fc2048",new DenseLayer.Builder().activation(Activation.TANH).nIn(1000).nOut(2048).build(),"fc1000")
            .removeVertexAndConnections("output")
            .addLayer("newOutput",new OutputLayer
                    .Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
                    .activation(Activation.SOFTMAX)
                    .nIn(2048)
                    .nOut(numLabels)
                    .build(),"fc2048")
            .setOutputs("newOutput")
            .build();
    System.out.println(graph.summary());

Exception message=============================================
Epoch=====================0
11:02:48.307 [AMDSI prefetch thread] DEBUG org.nd4j.linalg.memory.abstracts.Nd4jWorkspace - Steps: 4
Exception in thread "main" java.lang.IllegalStateException: Cannot perform backprop: Dropout mask array is absent (already cleared?)
at org.nd4j.base.Preconditions.throwStateEx(Preconditions.java:641)
at org.nd4j.base.Preconditions.checkState(Preconditions.java:268)
at org.deeplearning4j.nn.conf.dropout.Dropout.backprop(Dropout.java:154)
at org.deeplearning4j.nn.layers.AbstractLayer.backpropDropOutIfPresent(AbstractLayer.java:309)
at org.deeplearning4j.nn.layers.convolution.subsampling.SubsamplingLayer.backpropGradient(SubsamplingLayer.java:278)
at org.deeplearning4j.nn.graph.vertex.impl.LayerVertex.doBackward(LayerVertex.java:148)
at org.deeplearning4j.nn.graph.ComputationGraph.calcBackpropGradients(ComputationGraph.java:2621)
at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1369)
at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1329)
at org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(BaseOptimizer.java:160)
at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:63)
at org.deeplearning4j.optimize.Solver.optimize(Solver.java:52)
at org.deeplearning4j.nn.graph.ComputationGraph.fitHelper(ComputationGraph.java:1149)
at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1098)
at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1065)
at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:999)

@AlexDBlack

This comment has been minimized.

Copy link
Contributor

commented Nov 24, 2018

Thanks for reporting. Fixed here: #6758
Will be merged once tests are complete. A few hours after the merge, fix should be accessible on snapshots: https://deeplearning4j.org/docs/latest/deeplearning4j-config-snapshots

Alternatively, here's a workaround for 1.0.0-beta3:

        for(Layer l : graph.getLayers()){
            org.deeplearning4j.nn.conf.layers.Layer conf = (org.deeplearning4j.nn.conf.layers.Layer)l.getConfig();
            if(!(l.getConfig() instanceof org.deeplearning4j.nn.conf.layers.BaseLayer)){
                conf.setIDropout(null);
            } else {
                IDropout d = conf.getIDropout();
                conf.setIDropout(d == null ? null : d.clone());
            }
        }

https://gist.github.com/AlexDBlack/339991f8da0a5553223ccaaa690c4507

@wangfeng-skymind

This comment has been minimized.

Copy link
Author

commented Nov 24, 2018

okay,thanks

@lock

This comment has been minimized.

Copy link

commented Dec 24, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Dec 24, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
2 participants
You can’t perform that action at this time.