Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graph of measured values (Metrics API) #6

Closed
chovanecm opened this issue Jan 3, 2017 · 3 comments
Closed

Graph of measured values (Metrics API) #6

chovanecm opened this issue Jan 3, 2017 · 3 comments

Comments

@chovanecm
Copy link
Owner

As a user, I want to see a simple graph of training progress when in detail view

@chovanecm
Copy link
Owner Author

chovanecm commented Apr 24, 2017

Data can be collected using the Metrics API:
https://github.com/chovanecm/sacred/blob/feature/data-logger/docs/collected_information.rst#metrics-api

Sacredboard shall display a chart for each associated metric that gets measured during the execution of the experiment run.
Metrics are mostly qualitative variables of the adaptive model, such as the value of the loss function, accuracy of correctly classified items, etcetera
Instead of running the training process for a fixed number of iterations, researchers may opt to engage techniques that measure the metrics after each (or e.g. after each 20th) training iteration to decide whether the training process should continue or if it should be terminated because of a low expectation of future progress.

A visual chart of a metric shall be a 2D graph with the training step or elapsed time on the x-axis and the measured scalar value on the y-axis.
Sacredboard shall only display one chart per metric per run, there should be only one curve in the graph. A run can have one to a few metrics.

@chovanecm
Copy link
Owner Author

@chovanecm chovanecm self-assigned this May 8, 2017
@chovanecm chovanecm added the Epic label Jun 30, 2017
@chovanecm chovanecm changed the title Graph of measured values Graph of measured values (Metrics API) Jun 30, 2017
@chovanecm chovanecm added this to the scipy2017 milestone Jun 30, 2017
@chovanecm
Copy link
Owner Author

Scenario:

  1. The user indicates he wants to see the progress of a specific metric or metrics.
  2. The system responds with a list of available metrics
  3. The user selects the metrics he wants to plot
  4. The system plots the selected metrics with x-axis as the step and y-axis as value.

Alternate scenarios:

  1. In step 3., the user chooses to set the x-axis as time instead of step number.
    1.1 The system draws the selected metrics with date-based x axis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant