Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-Task/Outputs Classification Evaluation of Computation Graph #6497

Closed
mxiao08 opened this issue Sep 29, 2018 · 4 comments

Comments

@mxiao08
Copy link

commented Sep 29, 2018

I am using ComputationGraph to train a Multi-Task learning system (one input sequence + two output sequence). However, when I add evaluations, it does not work for the second output sequence. Here is the error:
(run-main-0) java.lang.IllegalArgumentException: Unable to evaluate. Predictions and labels arrays are not same shape. Predictions shape: [2989, 23], Labels shape: [2989, 45]
[error] java.lang.IllegalArgumentException: Unable to evaluate. Predictions and labels arrays are not same shape. Predictions shape: [2989, 23], Labels shape: [2989, 45]
[error] at org.deeplearning4j.eval.Evaluation.eval(Evaluation.java:382)
[error] at org.deeplearning4j.eval.Evaluation.eval(Evaluation.java:351)
[error] at org.deeplearning4j.eval.BaseEvaluation.evalTimeSeries(BaseEvaluation.java:91)
[error] at org.deeplearning4j.eval.BaseEvaluation.eval(BaseEvaluation.java:111)
[error] at org.deeplearning4j.nn.graph.ComputationGraph.doEvaluationHelper(ComputationGraph.java:4021)
[error] at org.deeplearning4j.nn.graph.ComputationGraph.evaluate(ComputationGraph.java:3960)
[error] at ai.minxiao.tutorial.dl4j.evaluation.Evaluator$.run(Evaluator.scala:55)
[error] at ai.minxiao.tutorial.dl4j.evaluation.Evaluator$.run(Evaluator.scala:72)
[error] at ai.minxiao.tutorial.dl4j.learning.Learner$$anonfun$runWithTestCGMDEval$1.apply$mcVI$sp(Learner.scala:313)
[error] at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
[error] at ai.minxiao.tutorial.dl4j.learning.Learner$.runWithTestCGMDEval(Learner.scala:308)
[error] at ai.minxiao.tutorial.dl4j.example.cgnn.WSJMultiTaskExample$.main(WSJMultiTaskExample.scala:185)
[error] at ai.minxiao.tutorial.dl4j.example.cgnn.WSJMultiTaskExample.main(WSJMultiTaskExample.scala)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.lang.reflect.Method.invoke(Method.java:498)
[error] at sbt.Run.invokeMain(Run.scala:89)
[error] at sbt.Run.run0(Run.scala:83)
[error] at sbt.Run.execute$1(Run.scala:61)
[error] at sbt.Run.$anonfun$run$4(Run.scala:73)
[error] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[error] at sbt.util.InterfaceUtil$$anon$1.get(InterfaceUtil.scala:10)
[error] at sbt.TrapExit$App.run(TrapExit.scala:252)
[error] at java.lang.Thread.run(Thread.java:745)

The first OutputLayer has 45 classes and the second OutputLayer has 23 classes.
My guess is that it still uses the first OutputLayer's label with the second OutputLayer's prediction for evolution.
If I only evaluate on the first OutputLayer, it works properly.

For references: the NN architecture is like:
Word ID Sequence --> EmbeddingLayer --> GLSTM --> GLSTM-A --> RnnOutputLayer-A
--> GLSTM-B --> RnnOutputLayer-B
The evaluation is like:
cg.evaluate(MultiDatasetIterator, Map(0 -> Array(new Evaluation), 1 -> Array(new Evaluation)).

@AlexDBlack

This comment has been minimized.

Copy link
Contributor

commented Sep 30, 2018

Can you share the actual code (network configuration + exactly how you are using evaluation) used to produce this error?
Given we have a number of tests for multi-task evaluation already, it might be something specific to the network or your code or network.

@AlexDBlack

This comment has been minimized.

Copy link
Contributor

commented Oct 1, 2018

Actually, nevermind - I was able to reproduce this.
It's definitely a bug. Thanks for reporting.

@AlexDBlack AlexDBlack self-assigned this Oct 1, 2018

AlexDBlack added a commit that referenced this issue Oct 1, 2018
@AlexDBlack AlexDBlack referenced this issue Oct 1, 2018
@mxiao08

This comment has been minimized.

Copy link
Author

commented Oct 1, 2018

AlexDBlack added a commit that referenced this issue Oct 3, 2018
Misc fixes (#6504)
* #6501 Fix multiple issues with Xception configuration

* #6500 Fix SqueezeNet (non-pretrained) config + add tests

* #6497 Fix multi-output net evaluation for ComputationGraph

* #6502 fix formatting for ComputationGraph summary

* #6502 fix formatting for MultiLayerNetwork summary

* #6489 fix JSON mapping for keras import preprocessors

* Switch default output layer activation to softmax

* Fix transposei issues with autoencoder

* Minor test fixes

* ComputationGraph: Fix possible double application of preprocessors and setting input for output layers when doing backprop

* Workspace fix for output layer dropout in compgraph
@lock

This comment has been minimized.

Copy link

commented Nov 2, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Nov 2, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
2 participants
You can’t perform that action at this time.