"Expected no workspace active" exception during evaluation

[turambar]

Training a simple MLP on MNIST with an EvaluativeListener, I get the following workspace-related exception: Exception in thread "main" org.nd4j.linalg.workspace.ND4JWorkspaceException: Expected no workspace active in outputOfLayerDetached - Open/active workspaces: [WS_ALL_LAYERS_ACT]

It happens on the first call to doEvaluation(...) inside of the listener, so it likely has to do with the inference workspace mode...?

Here's a full stack trace for the exception:

Exception in thread "main" org.nd4j.linalg.workspace.ND4JWorkspaceException: Expected no workspace active in outputOfLayerDetached - Open/active workspaces: [WS_ALL_LAYERS_ACT]
	at org.nd4j.linalg.workspace.WorkspaceUtils.assertNoWorkspacesOpen(WorkspaceUtils.java:38)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.outputOfLayerDetached(MultiLayerNetwork.java:1030)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.doEvaluation(MultiLayerNetwork.java:3052)
	at org.deeplearning4j.optimize.listeners.EvaluativeListener.invokeListener(EvaluativeListener.java:211)
	at org.deeplearning4j.optimize.listeners.EvaluativeListener.onBackwardPass(EvaluativeListener.java:188)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore(MultiLayerNetwork.java:2460)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore(MultiLayerNetwork.java:2395)
	at org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(BaseOptimizer.java:178)
	at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:60)
	at org.deeplearning4j.optimize.Solver.optimize(Solver.java:54)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(MultiLayerNetwork.java:1459)
	at org.deeplearning4j.distributed.BasicTrainingExperimentDriver.entryPoint(BasicTrainingExperimentDriver.java:127)
	at org.deeplearning4j.distributed.BasicTrainingExperimentDriver.main(BasicTrainingExperimentDriver.java:58)

[turambar]

Scratch that, it must be the training workspace mode -- setting training workspace mode to NONE eliminated the exception.

[raver119]

Proposed fix above is just to scope internal evaluation call out of workspaces.

@AlexDBlack what would you say? Current DL4j mechanic will be ok with that?
If yes - i'll add some tests and make it PR.

[AlexDBlack]

It happens on the first call to doEvaluation(...) inside of the listener,

Proposed fix above is just to scope internal evaluation call out of workspaces.

That won't work: currently (to detect bugs) we expect no workspace to be active external to that forward pass method. Even if we scope out eval, the outer WS will still be open and we'll still fail validation.

So we need to mess with the validation conditions, or mess with the listener.
I'm inclined to do the latter - probably switch from onBackwardPas to iterationDone
(which makes more sense for memory use anyway)

deeplearning4j/deeplearning4j/deeplearning4j-nn/src/main/java/org/deeplearning4j/optimize/listeners/EvaluativeListener.java

Lines 186 to 189 in 74bd701

	public void onBackwardPass(Model model) {
	if (invocationType == InvocationType.ITERATION_END)
	invokeListener(model);
	}

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.