Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show accuracy and confusion matrix in classify_many #608

Merged
merged 2 commits into from
Mar 10, 2016

Conversation

gheinrich
Copy link
Contributor

Show some stats when ground truth is provided in text file:

  • top-1 accuracy,
  • top-5 accuracy,
  • per-class accuracy,
  • confusion matrix.

This may produce output like:
alexnet-classify-many-stats

@lukeyeager
Copy link
Member

Very nice!

If we merge, more people will probably be interested in using infer_many in DIGITS. That raises the importance of solving #611 before the next release so that Nginx users don't try running big jobs which shut down their server when it takes too long to respond.

@lukeyeager
Copy link
Member

Tested with 1000-class models and the UI fails pretty gracefully. The table just shoots off the screen to the right, which seems like the only reasonable option.

@gheinrich
Copy link
Contributor Author

The current code discards empty rows. I was wondering whether it would make sense to discard empty columns as well. When most rows and columns have non-zero numbers in them I didn't really want to implement things like "retain only the x most common classes" because that would be another configuration knob to add.

Perhaps we can limit the maximum number of labels we show on the result page, and add an option to download the full matrix, à la Travis log. That would be slightly more difficult to implement as I think that would require storing inference results on disk.

If user really feels offended by the wide result page, they can just not add the ground truth to their file and none of those stats will show.

@lukeyeager
Copy link
Member

I don't think any of those things need to be addressed right now. This looks good to me - merging!

lukeyeager added a commit that referenced this pull request Mar 10, 2016
 	Show accuracy and confusion matrix in classify_many
@lukeyeager lukeyeager merged commit 41996c8 into NVIDIA:master Mar 10, 2016
@gheinrich gheinrich deleted the dev/classify-stats branch May 24, 2016 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants