Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add results browser #195

Merged
merged 1 commit into from
Aug 28, 2015
Merged

Add results browser #195

merged 1 commit into from
Aug 28, 2015

Conversation

semisight
Copy link
Contributor

Idea

This pull request adds a central page to view results from different ModelJobs. From there, jobs can be compared and filtered in a paginated table. I'm using DataTable, a jquery plugin that seems fairly widely used. The goal is to make finding the best network easier.

Feedback

I'm looking for feedback on how to make this feature more useful. What columns should be added? How do users prefer to compare? Should there be a side-by-side comparison page?

Also, how should this be best integrated into digits? Currently, it is a green button on the front page that links to this. Is this ideal? Where/how should we link to this? Through the navbar?

@lukeyeager
Copy link
Member

I can't see the results yet (see the email I sent you), but from looking at the source this looks promising.

1 quick request: upload only minimized .js and .css files.

@lukeyeager
Copy link
Member

Suggestions

  • Move the sort icon next to the th
  • Sort as numbers, not strings
  • Show val_outputs
  • If no value, sort to bottom of page (asc or desc)
  • Sort by status
  • Sort by time started
  • Sort by runtime

Sorry that's a lot! Some should be easy, some may be hard.

@semisight semisight force-pushed the master branch 3 times, most recently from 2d260da to b343d1b Compare August 7, 2015 16:59
@lukeyeager
Copy link
Member

Thanks for the fixes! Here are some more suggestions:

  • Can we display the times as a readable string instead of a number of seconds?
  • Can we add an explanation somewhere that tells whether you're showing the maximum or minimum or last value for each column? It's not clear to me.
  • Can we show an arbitrarily large number of entries rather than choosing from 10,25,50,100?

And it looks like you haven't addressed these suggestions. Do you have reasons or just haven't gotten around to them yet?

  • If no value, sort to bottom of page (either asc or desc)
  • Sort by time started

@semisight semisight force-pushed the master branch 2 times, most recently from f995b41 to 3de17a8 Compare August 7, 2015 22:22
@semisight
Copy link
Contributor Author

Point by point:

  • What's your preferred time format? I've been working with seconds but I can make it a "H:M:S" format if that is preferable?
  • I think the intent of the little glyph on the side is that if it is getting "smaller" as it goes down then it is descending, or if it gets "larger" going down it is ascending. I agree it could be clearer. Maybe make it bigger?
  • I think that should be possible. I can turn off pagination, it's just on by default.

And as for the other two: I'm not quite finished yet. I'm still undecided on how best to override the default sorting per column. I believe sorting by ID should automatically sort by time started, but again, that's not exactly obvious. Maybe make another column for "start time?"

@lukeyeager
Copy link
Member

Preferred time format
I like this: https://github.com/NVIDIA/DIGITS/blob/v2.0.0-rc3/digits/utils/time_filters.py#L5-L16

Maximum/Minimum/Last value
Actually what I meant was that when I see "Accuracy" as a column, I don't know if the value in each row is the maximum accuracy for that job or just the last value.

@semisight
Copy link
Contributor Author

Preferred time format--okay. The runtime is a timedelta so I'd probably use the print_time_diff for that one.

Max/min/last--I can add a tooltip or comment above. All the values presented are from the last run. The assumption I'm making is that the final numbers are from the net that is saved by Caffe (unless it was aborted). I feel like it would be disingenuous to present an "all time minimum" because the current state of the net is not represented by that number.

@lukeyeager
Copy link
Member

I think many people will be interested in knowing which model had the best accuracy. See discussion at #71.

@semisight
Copy link
Contributor Author

Interesting. How about both?

@lukeyeager
Copy link
Member

Is there any way to view max/min/last for each output without creating 3x the number of columns? Some sort of radio button / select field for each column?

@semisight
Copy link
Contributor Author

I think for a first go, having a form to narrow down results would get us 90% of the way there with much less work. If this is a pain point we can always revisit it later.

@lukeyeager
Copy link
Member

@semisight, you said there's a way to view max/min/last now? I pulled the latest from semisight/master and I see the multi-select, but I don't see a way to sort by max accuracy.

Also, we should still display something by default when the page loads. Maybe the default columns and "loss" and "accuracy"?

@semisight
Copy link
Contributor Author

Yes, there is. It's pretty clunky right now. The multi select presents each column title 3 times: once with a "max" suffix, once with a "min" suffix, and once with no suffix. The max corresponds to the maximum value of that attribute, the min to the minimum value, and the "no suffix" to the latest value (last epoch).

I can add a default table in. Are loss and accuracy standard names? Is it okay to "bless" them, considering models aren't required to emit data under those names?

@lukeyeager
Copy link
Member

Since 9ff246e, all the standard networks output loss and accuracy. I think it's fine to show those by default to encourage standard naming on people's custom networks.

@lukeyeager lukeyeager mentioned this pull request Aug 11, 2015
@semisight semisight force-pushed the master branch 6 times, most recently from 905de5f to 3e7c269 Compare August 20, 2015 17:57
@semisight semisight force-pushed the master branch 3 times, most recently from 3337660 to e4cd028 Compare August 26, 2015 20:13
@lukeyeager
Copy link
Member

Can we add a simple page load test in these places:

https://github.com/NVIDIA/DIGITS/blob/v2.1.0/digits/test_views.py#L124
https://github.com/NVIDIA/DIGITS/blob/v2.1.0/digits/model/images/classification/test_views.py#L302
https://github.com/NVIDIA/DIGITS/blob/v2.1.0/digits/model/images/generic/test_views.py#L304

This looks pretty cool. Let's squash down to one commit and merge this thing, unless you have any other last-minute features you're trying to get done.

Show/hide columns:

Columns are grouped by 'type'. You can view the metadata like the name,
or view numeric values by latest value, min value, or max value.

Column filters:

* Regex search on non-numeric columns
* Range search on numeric columns (use ':' to separate min and max
  values; omission of min or max substitutes -inf or +inf)

Toggle filters:

Press 'Show/Hide Filters' to hide filters.

Sorting and Multi-sorting:

All columns are sortable. Hold shift to sort by multiple columns.
@semisight
Copy link
Contributor Author

Squashed everything. Here are some release notes, especially for features that aren't documented (yet).

Show/hide columns:

Columns are grouped by 'type'. You can view the metadata like the name,
or view numeric values by latest value, min value, or max value.

Column filters:

  • Regex search on non-numeric columns
  • Range search on numeric columns (use ':' to separate min and max
    values; omission of min or max substitutes -inf or +inf)

Toggle filters:

Press 'Show/Hide Filters' to hide filters.

Sorting and Multi-sorting:

All columns are sortable. Hold shift to sort by multiple columns.

@lukeyeager
Copy link
Member

Looks good and thanks for the release notes!

Once Travis passes, I'll merge ...

lukeyeager added a commit that referenced this pull request Aug 28, 2015
@lukeyeager lukeyeager merged commit fa6095f into NVIDIA:master Aug 28, 2015
lukeyeager added a commit to lukeyeager/DIGITS that referenced this pull request Sep 1, 2015
Cleanup link on homepage
Move template location
Remove "results browser" language
Add some simple tests
lukeyeager added a commit to lukeyeager/DIGITS that referenced this pull request Sep 1, 2015
Cleanup link on homepage
Move template location
Remove "results browser" language
Add some simple tests
lukeyeager added a commit to lukeyeager/DIGITS that referenced this pull request Sep 1, 2015
Cleanup link on homepage
Move template location
Remove "results browser" language
Add some simple tests
lukeyeager added a commit to lukeyeager/DIGITS that referenced this pull request Sep 1, 2015
Cleanup link on homepage
Move template location
Remove "results browser" language
Add some simple tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants