New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] Support individual scalar reporting without plots #400
Comments
Hi @idantene , Did you try adding this single scalar to the leaderboard and filter with it? can this do the trick? |
Interesting, @idantene are saying that maybe in the UI, when all the series on a graph have a single report, we should present it as table, is that it ? |
@bmartinn I think that would make a lot of sense, yes. At the same time, I believe the user should have absolute control on how these are displayed. This could be some toggleable button in the UI or some specific set of instructions in the SDK, that separates between a graph and a table. For example, one can imagine a table also for multiple reports, with e.g.
etc (where iteration is just a pseudonym for "description" in this case) |
@idantene Seems like this last example you provide steers further from the "default" scalars use case of value series. (This while not discounting @bmartinn's suggestion for a more suited display in certain scenarios) |
@ainoam Well, that depends. I would not expect these scalars to be reported as a single table. This is more of an extension of the original request, where an iteration does not really exist -- these are metrics that encapsulate the "end result". The above example is just an illustration of a use case we have - there's the test set, but we also split it by various characteristics. It's then useful to calculate e.g. the overall MAE on the test set, but also the MAE on various subsets that represent these characteristics. So the "iteration" concept does not exactly exist, but there are several metrics with the same name but with different descriptions. I see two ways to achieve this as of now - What about something along the lines of
What do you think? |
Hi @idantene
This could be a nice way to present an entire graph (multiple series) in a table as a summary view.
Creating a nice interface for it in the SDK can definitely help users understand how to use this feature. I'm thinking we will find a way to signal to the UI that this scalar graph should start as a table (always an option to switch back to graph view), then add an interface to "report single value" to the SDK. |
Hey @bmartinn, Sure, that sounds like a plan 👍🏻 Can't wait! Just to clarify though - the |
Hi, maybe this is the right topic for the issue we're currently facing -- right now we are using the workaround of "plotting" a table, with one scalar per row. However the table view doesn't support text selection / copying, so if we want to paste the results into a report, we need to still log the table to the console. Could you please enable text selection or a true html / TeX table copy/paste button? The JSON export is not very useful right now for this job. |
Hi @swigicat , I'll ask around and see what can be done 🙂 |
I think the easiest would be to download as CSV, no? |
Any updates on this UI feature request? |
Hi @idantene , thanks for the bump! Recapping here:
Question:
instead of plotting a "scalar graph" like we have here: We should get a table:
So basically, if we actually report scalars (instead of |
@bmartinn Yes that looks about right 👍🏻 Perhaps the |
Just jumping in as this would be a really useful feature - it would also be useful to think about how this appears when comparing experiments. E.g if it was just a table you could have
With perhaps some colouring to show which values are larger. It would be even more useful to plot this data, e.g. a bar chart grouped by value name or even a radar chart with the values normalised against the base experiment. |
Maybe an option to choose if a reported value is optimized to be maximal or minimal first (i.e. highlight larger F1 scores, but lower MSE scores...) |
i.e. sort the table by value?
|
Sorting is nice if one is interested in a single metric, coloring is more convenient if one is looking at multiple metrics in a single table. |
Any solution to this request? |
Hi @cpatrickalves @idantene @mmiller-max , while there isn't a solution yet, it is in the making! The plan is to have a logger interface in which you can do something like Anything I missed? |
@erezalg you're saying "In the scalars tab, you'll have a table with these reported metrics.". I would do one little modification to this design: instead of having "scalars" tab, I would do two tabs: "scalars (one value)" and "scalars (time series)". Reason: I am just afraid that the table with the values may get lost in the charts of the "scalars" tab (if there will be a lot of them). |
@ibobak , the plan is to make it appear first in the scalars tab. Makes sense? |
Closing this issue. Please reopen if it's still relevant. |
When using
logger.report_scalar
a plot is automatically generated to accommodate for a time-series axis.There is no current way for the user to report a single scalar with a name (that is not a time-series) and have it both aesthetically pleasing and visible enough in the WebUI. Using
iteration=0
just wastes space by creating a scatter plot with a single datum.It would be great to have e.g.
logger.report_scalar(name="MAE", value=mae)
, and have it visible as a table (or similar) with e.g.:Even better, once these are in place, they can automatically be aligned between multiple experiments for comparison.
The text was updated successfully, but these errors were encountered: