Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DifferentialFunctionClassHolder.getInstance().getInstance("abs") fails due to guava #7170

Closed
dd00f opened this issue Feb 15, 2019 · 3 comments

Comments

Projects
None yet
2 participants
@dd00f
Copy link

commented Feb 15, 2019

Environment :
Windows 10,
DL4J version 1.0.0-beta3.
CUDA 9.2
Java version : OpenJDK 11.0.1

When I first tried to run a ComputationGraph fit function, I got the following exception:

Exception in thread "main" java.lang.RuntimeException: java.lang.NullPointerException
	at org.nd4j.linalg.api.ops.factory.DefaultOpFactory.createTransform(DefaultOpFactory.java:299)
	at org.nd4j.linalg.api.ops.factory.DefaultOpFactory.createTransform(DefaultOpFactory.java:246)
	at org.nd4j.linalg.lossfunctions.impl.LossL1.scoreArray(LossL1.java:88)
	at org.nd4j.linalg.lossfunctions.impl.LossL1.computeScore(LossL1.java:108)
	at org.nd4j.linalg.lossfunctions.impl.LossMAE.computeScore(LossMAE.java:62)
	at org.deeplearning4j.nn.layers.BaseOutputLayer.computeScore(BaseOutputLayer.java:90)
	at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1395)
	at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1329)
	at org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(BaseOptimizer.java:160)
	at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:63)
	at org.deeplearning4j.optimize.Solver.optimize(Solver.java:52)
	at org.deeplearning4j.nn.graph.ComputationGraph.fitHelper(ComputationGraph.java:1149)
	at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1098)
	at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:1065)
	at kemet.ai.KemetNeuralNetwork.trainEpoch(KemetNeuralNetwork.java:72)
	at kemet.ai.KemetNeuralNetwork.train(KemetNeuralNetwork.java:55)
	at kemet.util.Coach.runTrainingIteration(Coach.java:273)
	at kemet.util.Coach.learn(Coach.java:212)
	at kemet.data.KemetLearn.main(KemetLearn.java:64)
Caused by: java.lang.NullPointerException
	at org.nd4j.linalg.api.ops.factory.DefaultOpFactory.createTransform(DefaultOpFactory.java:295)
	... 18 more

After a bit of investigation, I found that this method kept returning a null value :
DifferentialFunctionClassHolder.getInstance().getInstance("abs")

Turns out the introspection done in DifferentialFunctionClassHolder using com.google.common.reflect.ClassPath kept on returning an empty list of classes. I traced this to a similar bug in Guava :

google/guava#2152

I upgraded guava to version 27.0.1-jre and the problem went away.

The guava version seems to be specified here.

<guava.version>20.0</guava.version>

Hope this helps.

@AlexDBlack

This comment has been minimized.

Copy link
Member

commented Feb 15, 2019

Thanks for the detailed issue, though this is in part no longer relevant.
DefaultOpFactory was removed entirely here: #6980
https://github.com/deeplearning4j/deeplearning4j/search?q=DefaultOpFactory&unscoped_q=DefaultOpFactory

LossL1 has been updated to use Transforms.abs instead: https://github.com/deeplearning4j/deeplearning4j/blob/master/nd4j/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/linalg/lossfunctions/impl/LossL1.java#L88

Other than using the workaround of downloading guava, you can also consider using snapshots (though note there are a number of breaking changes there around ND4J datatypes, so you might need to make some modifications to your code to use snapshots) - https://deeplearning4j.org/docs/latest/deeplearning4j-config-snapshots

Now, I will leave this issue open for now because we use DifferentialFunctionClassHolder.getInstance().getInstance(String) in a few other places - though I note we haven't run into this anywhere yet. That doesn't mean we won't, however.

https://github.com/deeplearning4j/deeplearning4j/search?q=DifferentialFunctionClassHolder.getInstance%28%29.getInstance&unscoped_q=DifferentialFunctionClassHolder.getInstance%28%29.getInstance

@dd00f Was there anything special about your environment, other than java 11?
i.e., it's just a standard "run in IDE" or "build uberjar with maven shade" type workflow?
Just thinking how to test other similar uses of DifferentialFunctionClassHolder that might reproduce this...

@dd00f

This comment has been minimized.

Copy link
Author

commented Feb 15, 2019

@AlexDBlack I'm currently running this in a relatively recent version of the Eclipse IDE.

@lock

This comment has been minimized.

Copy link

commented May 18, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators May 18, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.