Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to get the best run in experiment based on a given metric #15

Closed
aslotte opened this issue Jun 2, 2020 · 1 comment · Fixed by #75
Closed

Add ability to get the best run in experiment based on a given metric #15

aslotte opened this issue Jun 2, 2020 · 1 comment · Fixed by #75
Assignees

Comments

@aslotte
Copy link
Owner

aslotte commented Jun 2, 2020

We need the ability to get the best run in an experiement based on a specific metric.
For example, let's say we value the F1 score for a given machine learning model we are training (contained in an experiment). Each time we train the model (run), we want to know if this model is better than a previously recorded model or not.

The reason we want this is so we don't need to upload every model that we don't care about.

I'm thinking something like this

public void IsBestRun(Guid runId, string metricName) 
{
    // 1. Fetch the given metric value for that run
    // 2. Given the experiment that run is apart of, fetch the best run for that metric
    // 3. Compare if this is the best run or not
}
@aslotte aslotte changed the title Add ability to get best run in experiment based on metric Add ability to get the best run in experiment based on a given metric Jun 3, 2020
@AnoojNair
Copy link
Collaborator

AnoojNair commented Jun 5, 2020

@aslotte , i feel we should save the model irrespective with each run and let the user decide which model to use. What if the best score was a result of a bug in the code or underlying data changed.
Or
We can pass in a boolean flag so that the user decides to save it not?

Can grab this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants