New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Expected no workspace active" exception during evaluation #5351

Closed
turambar opened this Issue May 27, 2018 · 4 comments

Comments

Projects
None yet
3 participants
@turambar
Copy link
Contributor

turambar commented May 27, 2018

Training a simple MLP on MNIST with an EvaluativeListener, I get the following workspace-related exception: Exception in thread "main" org.nd4j.linalg.workspace.ND4JWorkspaceException: Expected no workspace active in outputOfLayerDetached - Open/active workspaces: [WS_ALL_LAYERS_ACT]

It happens on the first call to doEvaluation(...) inside of the listener, so it likely has to do with the inference workspace mode...?

Here's a full stack trace for the exception:

Exception in thread "main" org.nd4j.linalg.workspace.ND4JWorkspaceException: Expected no workspace active in outputOfLayerDetached - Open/active workspaces: [WS_ALL_LAYERS_ACT]
	at org.nd4j.linalg.workspace.WorkspaceUtils.assertNoWorkspacesOpen(WorkspaceUtils.java:38)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.outputOfLayerDetached(MultiLayerNetwork.java:1030)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.doEvaluation(MultiLayerNetwork.java:3052)
	at org.deeplearning4j.optimize.listeners.EvaluativeListener.invokeListener(EvaluativeListener.java:211)
	at org.deeplearning4j.optimize.listeners.EvaluativeListener.onBackwardPass(EvaluativeListener.java:188)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore(MultiLayerNetwork.java:2460)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.computeGradientAndScore(MultiLayerNetwork.java:2395)
	at org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(BaseOptimizer.java:178)
	at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:60)
	at org.deeplearning4j.optimize.Solver.optimize(Solver.java:54)
	at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(MultiLayerNetwork.java:1459)
	at org.deeplearning4j.distributed.BasicTrainingExperimentDriver.entryPoint(BasicTrainingExperimentDriver.java:127)
	at org.deeplearning4j.distributed.BasicTrainingExperimentDriver.main(BasicTrainingExperimentDriver.java:58)

@turambar turambar added the Bug label May 27, 2018

@turambar turambar changed the title workspace issue "Expected no workspace active" exception during evaluation May 27, 2018

@turambar

This comment has been minimized.

Copy link
Contributor

turambar commented May 27, 2018

Scratch that, it must be the training workspace mode -- setting training workspace mode to NONE eliminated the exception.

raver119 added a commit that referenced this issue May 27, 2018

@raver119

This comment has been minimized.

Copy link
Contributor

raver119 commented May 27, 2018

Proposed fix above is just to scope internal evaluation call out of workspaces.

@AlexDBlack what would you say? Current DL4j mechanic will be ok with that?
If yes - i'll add some tests and make it PR.

@AlexDBlack

This comment has been minimized.

Copy link
Member

AlexDBlack commented May 28, 2018

It happens on the first call to doEvaluation(...) inside of the listener,

Proposed fix above is just to scope internal evaluation call out of workspaces.

That won't work: currently (to detect bugs) we expect no workspace to be active external to that forward pass method. Even if we scope out eval, the outer WS will still be open and we'll still fail validation.

So we need to mess with the validation conditions, or mess with the listener.
I'm inclined to do the latter - probably switch from onBackwardPas to iterationDone
(which makes more sense for memory use anyway)

public void onBackwardPass(Model model) {
if (invocationType == InvocationType.ITERATION_END)
invokeListener(model);
}

AlexDBlack added a commit that referenced this issue May 28, 2018

AlexDBlack added a commit that referenced this issue May 28, 2018

Merge pull request #5352 from deeplearning4j/ab_5351_eval
#5351 Fix EvaluativeListener workspace issue
@lock

This comment has been minimized.

Copy link

lock bot commented Sep 21, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Sep 21, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.