Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ComputationGraphConfiguration with AutoEncoder layer fails on Array workspace validation #5662

Closed
lacic opened this issue Jun 21, 2018 · 3 comments

Comments

@lacic
Copy link

commented Jun 21, 2018

Running a ComputationGraphConfiguration with AutoEncoder layers (as described bellow) causes
a ND4JWorkspaceException:

Feed forward (inference): array (ACTIVATIONS) workspace validation failed (vertex encoder - class: AutoEncoder) - array is defined in incorrect workspace

Array workspace validation failed: Array of type ACTIVATIONS should be in workspace "WS_ALL_LAYERS_ACT" but is in workspace "WS_LAYER_WORKING_MEM"

Code:

int mergeSize = 500;
int encdecSize = 250;
double learningRate = 1e-2;
int in1Size = 2000; // depends on the used dataset
int in2Size = 1500; // depends on the used dataset

ComputationGraphConfiguration conf = new NeuralNetConfiguration.Builder()
    .seed(SEED)
    .updater(new RmsProp(learningRate))
    .weightInit(WeightInit.XAVIER)
    .l2(1e-4)
    .graphBuilder()
    .addInputs("in1", "in2")
    .addLayer("L1", new DenseLayer.Builder().nIn(in1Size).nOut(mergeSize).build(), "in1")
    .addLayer("L2", new DenseLayer.Builder().nIn(in2Size).nOut(mergeSize).build(), "in2")
    .addVertex("merge", new MergeVertex(), "L1", "L2")
    .addLayer("encoder", 
		new AutoEncoder.Builder()
		   .nIn(2 * mergeSize)
		   .nOut(encdecSize)
		   .activation(Activation.TANH)
		   .weightInit(WeightInit.XAVIER)
		   .lossFunction(LossFunction.MCXENT)
		   .corruptionLevel(0.2)
		   .build(),  
		"merge")
    .addLayer("hidden", 
		new AutoEncoder.Builder()
		   .nIn(encdecSize)
		   .nOut(hiddenSize)
		   .activation(Activation.TANH)
		   .weightInit(WeightInit.XAVIER)
		   .lossFunction(LossFunction.MCXENT)
		   .build(), 
		"encoder")
    .addLayer("decoder", 
	      	new AutoEncoder.Builder()
		   .nIn(hiddenSize)
		   .nOut(encdecSize)
		   .activation(Activation.TANH)
		   .weightInit(WeightInit.XAVIER)
		   .lossFunction(LossFunction.MCXENT)
		   .corruptionLevel(0.2)
		   .build(), 
		"hidden")
    .addLayer("L4", new DenseLayer.Builder().nIn(encdecSize).nOut(mergeSize).build(), "decoder")
    .addLayer("out1", 
		new OutputLayer.Builder()
		   .lossFunction(LossFunctions.LossFunction.MSE)
		   .nIn(mergeSize).nOut(in1Size).build(), 
		"L4")
    .addLayer("out2", 
		new OutputLayer.Builder()
		   .lossFunction(LossFunctions.LossFunction.MSE)
		   .nIn(mergeSize).nOut(in2Size).build(), 
		"L4")
    .setOutputs("out1","out2")
    .pretrain(true).backprop(true)
    .build();

ComputationGraph net = new ComputationGraph(conf);
net.init();
net.setListeners(new ScoreIterationListener(1000));

LocalFileModelSaver modelSaver = new LocalFileModelSaver(new File("/my/path/").getCanonicalPath()); // setup model saver

MultiDataSetIterator trainIterator = null; // some MultiDataSetIterator implementation
MultiDataSetIterator validIterator = null; // some MultiDataSetIterator implementation

EarlyStoppingConfiguration esConf = new EarlyStoppingConfiguration.Builder<MultiLayerNetwork>()
                .epochTerminationConditions(new MaxEpochsTerminationCondition(100))
                .epochTerminationConditions(new ScoreImprovementEpochTerminationCondition(5))
                .evaluateEveryNEpochs(1)
                .modelSaver(modelSaver)
            	.scoreCalculator(new DataSetLossCalculator(validIterator, true));
		.build();

EarlyStoppingGraphTrainer trainer = new EarlyStoppingGraphTrainer(esConf, net, trainIterator, null);
EarlyStoppingResult result = trainer.fit();

If I for example replace the AutEncoder class with a DenseLayer class the training of the network proceeds normally and does not cause an exception.

@AlexDBlack

This comment has been minimized.

Copy link
Contributor

commented Jun 21, 2018

Thanks for reporting. Fixed here: #5663

Once merged, you can use snapshots (will take a few hours for snapshots to propagate after merge). https://deeplearning4j.org/snapshots

@lacic

This comment has been minimized.

Copy link
Author

commented Jun 21, 2018

Great! Thank you. 😄

@lock

This comment has been minimized.

Copy link

commented Sep 21, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Sep 21, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
2 participants
You can’t perform that action at this time.