Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to record uncertainties along with each prediction #42

Closed
ardunn opened this issue Aug 11, 2021 · 3 comments
Closed

Add ability to record uncertainties along with each prediction #42

ardunn opened this issue Aug 11, 2021 · 3 comments
Assignees
Labels
code Anything having to do with matbench python package code

Comments

@ardunn
Copy link
Collaborator

ardunn commented Aug 11, 2021

No description provided.

@ardunn ardunn added the code Anything having to do with matbench python package code label Aug 11, 2021
@ardunn ardunn self-assigned this Aug 11, 2021
@ml-evs
Copy link
Contributor

ml-evs commented Aug 11, 2021

Just to carry on the discussion here:

Do you think it should be a 95% conf interval with assumed normal distribution? So like for each sample you would get [prediction, lower bound, higher bound] or something

Originally posted by @ardunn in #39 (comment)

This works for us!

It probably shouldn't be included in the metadata for now just because that will be a lot of data to be shown on the MODNet algorithm page for the website.

Originally posted by @ardunn in #39 (comment)

Maybe a more general way of providing per-prediction data/metadata (that doesn't clog up the web page) could be useful too, though beyond the scope of this issue.

@ardunn
Copy link
Collaborator Author

ardunn commented Aug 11, 2021

@ml-evs That actually sounds like a better idea, just having a metadata field for each prediction rather than having an array or something. So for each prediction it would look like:

{"mb-XXXX": 
    {
        "prediction": 8.94
        "metadata": {
            { <whatever metadata goes here, e.g. uncertainties>}
    }
}

@ardunn
Copy link
Collaborator Author

ardunn commented Oct 10, 2021

no longer needed, made the prediction field unnecessarily complex. arbitrary metadata can be kept per fold in params and as of #58 it will no longer spam the webpage if very large.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code Anything having to do with matbench python package code
Projects
None yet
Development

No branches or pull requests

2 participants