New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Evaluation Statistics Confusion Matrix Output #5096

Closed
steven-lang opened this Issue May 8, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@steven-lang
Copy link
Contributor

steven-lang commented May 8, 2018

Issue Description

The current confusion matrix output looks like this:

Predictions labeled as 0 classified by model as 0: 39 times
Predictions labeled as 1 classified by model as 1: 43 times
Predictions labeled as 2 classified by model as 2: 33 times
Predictions labeled as 3 classified by model as 3: 39 times
....

which prints numClasses^2 lines in the worst case and kind of obfuscates the actual information by showing too many unneccessary repeated words.

Suggestion

A more preferable way to display the confusion matrix would be e.g. how Weka prints it:

=== Confusion Matrix ===

  a  b  c  d  e  f  g  h  i  j   <-- classified as
 62  0  0  0  1  0  2  0  1  1 |  a = 0
  0 88  0  0  0  0  0  0  2  0 |  b = 1
  1  1 52  0  2  0  1  1  2  0 |  c = 2
  1  0  4 61  0  3  1  1  3  0 |  d = 3
  1  0  0  0 60  0  0  0  0  8 |  e = 4
  1  0  0  3  0 59  5  0  1  0 |  f = 5
  0  0  1  0  1  0 65  0  0  0 |  g = 6
  0  0  2  0  0  1  0 62  0  6 |  h = 7
  1  2  1  3  0  0  0  0 70  2 |  i = 8
  1  1  0  1  2  0  1  1  1 60 |  j = 9

Furthermore, it should be optional to include the confusion matrix when calling Evaluation.stats() as the ouput can become quite large for datasets with a high number of classes.

Version Information

  • Deeplearning4j version: 1.0.0-alpha
@AlexDBlack

This comment has been minimized.

Copy link
Member

AlexDBlack commented May 8, 2018

OK, so here's what I'm thinking:

  • We add stats(boolean printConfusion) overload, or similar
  • stats() basically calls stats(numClasses < 20) (i.e., default print for small classes, something like "confusion matrix omitted due to number of classes, use X" otherwise for no-arg method)
  • Maybe: Add the old "Predictions labelled as..." style under another (new) method

@AlexDBlack AlexDBlack self-assigned this May 9, 2018

AlexDBlack added a commit that referenced this issue May 9, 2018

AlexDBlack added a commit that referenced this issue May 9, 2018

@lock

This comment has been minimized.

Copy link

lock bot commented Sep 22, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Sep 22, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.