New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Label and Score #158

Closed
cheesemacfly opened this Issue May 15, 2018 · 7 comments

Comments

Projects
5 participants
@cheesemacfly

cheesemacfly commented May 15, 2018

Not sure if it's the best place for this question but how to get the labels back when getting the score? I remember from the build 2018 video that during the demo they used PredictedLabelColumnOriginalValueConverter to get the label back instead of the index for the PredictedLabel but is there a way to apply this converter to all the values in the Score? If yes, how so?

I also remember they were using the ClassificationMetrics.ConfusionMatrix.ClassNames to get the labels back from the score but this isn't really doable if you load the output in an app and don't need/want to test your data.

I found the file Terms.txt in the output zip archive and I can definitely parse it to find the labels but I'm sure there's a better way to do it.

@shauheen shauheen added the question label May 15, 2018

@GalOshri

This comment has been minimized.

Show comment
Hide comment
@GalOshri

GalOshri May 16, 2018

Contributor

Getting the class names for the other scores (not just the top score) is currently available in ML.NET by going through IDV (IDataView), but not in the high-level APIs (e.g. LearningPipeline). This is something we definitely need to expose!

@codemzs, @glebuk, @TomFinley, any thoughts on how we can enable getting this type of data in the high-level APIs?

Contributor

GalOshri commented May 16, 2018

Getting the class names for the other scores (not just the top score) is currently available in ML.NET by going through IDV (IDataView), but not in the high-level APIs (e.g. LearningPipeline). This is something we definitely need to expose!

@codemzs, @glebuk, @TomFinley, any thoughts on how we can enable getting this type of data in the high-level APIs?

@cheesemacfly

This comment has been minimized.

Show comment
Hide comment
@cheesemacfly

cheesemacfly May 16, 2018

Is there any available documentation/example on how to use IDataView vs LearningPipeline? Also, is it safe to assume that the float[] from Score will have the same order as the labels in the Terms.txt file? Is there a public API to access the files once they have been loaded with PredictionModel.ReadAsync()?

cheesemacfly commented May 16, 2018

Is there any available documentation/example on how to use IDataView vs LearningPipeline? Also, is it safe to assume that the float[] from Score will have the same order as the labels in the Terms.txt file? Is there a public API to access the files once they have been loaded with PredictionModel.ReadAsync()?

@shauheen shauheen added this to Backlog in v0.2 via automation May 16, 2018

@shauheen shauheen added this to the 0518 milestone May 16, 2018

@shauheen shauheen moved this from Backlog to Todo in v0.2 May 22, 2018

@Ivanidzo4ka Ivanidzo4ka moved this from Todo to In Progress in v0.2 May 25, 2018

@codemzs codemzs closed this in #239 May 25, 2018

v0.2 automation moved this from In Progress to Done May 25, 2018

@GalOshri

This comment has been minimized.

Show comment
Hide comment
@GalOshri

GalOshri May 30, 2018

Contributor

@cheesemacfly, just wanted to quickly follow up to see if #239 addresses your question. Take a look at TryGetScoreLabelNames here, which gives you the class names (in the same order as the scores).

Contributor

GalOshri commented May 30, 2018

@cheesemacfly, just wanted to quickly follow up to see if #239 addresses your question. Take a look at TryGetScoreLabelNames here, which gives you the class names (in the same order as the scores).

@cheesemacfly

This comment has been minimized.

Show comment
Hide comment
@cheesemacfly

cheesemacfly May 30, 2018

@GalOshri it definitely looks like this would address my question. Is there a pre-release version of 0.2 of the nuget package available somewhere so I can test it?

cheesemacfly commented May 30, 2018

@GalOshri it definitely looks like this would address my question. Is there a pre-release version of 0.2 of the nuget package available somewhere so I can test it?

@GalOshri

This comment has been minimized.

Show comment
Hide comment
@GalOshri

GalOshri May 30, 2018

Contributor

Yes, the daily NuGet builds are available here.

Contributor

GalOshri commented May 30, 2018

Yes, the daily NuGet builds are available here.

@cheesemacfly

This comment has been minimized.

Show comment
Hide comment
@cheesemacfly

cheesemacfly May 30, 2018

@GalOshri it works as expected thanks.

Also I'm now wondering, is there a reason why we couldn't get the labels from the Predict() call result right away? So using something like [ColumnName("MappedScore")] (keeping [ColumnName("Score")] as it is) we would get back a Tuple<float, string>[] for example? It would be easier to read than a second method call on the PredictionModel object.

cheesemacfly commented May 30, 2018

@GalOshri it works as expected thanks.

Also I'm now wondering, is there a reason why we couldn't get the labels from the Predict() call result right away? So using something like [ColumnName("MappedScore")] (keeping [ColumnName("Score")] as it is) we would get back a Tuple<float, string>[] for example? It would be easier to read than a second method call on the PredictionModel object.

@onema

This comment has been minimized.

Show comment
Hide comment
@onema

onema Sep 12, 2018

Just to recap and for future reference, because I've been looking for this, the code that gets the label for the scores is this:

string[] scoreLabels;
model.TryGetScoreLabelNames(out scoreLabels);

The and the order matches the scores in the model!

onema commented Sep 12, 2018

Just to recap and for future reference, because I've been looking for this, the code that gets the label for the scores is this:

string[] scoreLabels;
model.TryGetScoreLabelNames(out scoreLabels);

The and the order matches the scores in the model!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment