Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Q: A calibrator is required for Bin Classification Evaluate #4630

Closed
lefig opened this issue Jan 5, 2020 · 2 comments
Closed

Q: A calibrator is required for Bin Classification Evaluate #4630

lefig opened this issue Jan 5, 2020 · 2 comments
Assignees

Comments

@lefig
Copy link

@lefig lefig commented Jan 5, 2020

Hi,

Concerning Binary Classification features analysis I find that for evaluation of scores a calibrator is required, such as

Logger.Info("===== Evaluating Model's accuracy with Test data =====");
var trainingPipeline = dataProcessPipeline.Append(algorithm)
.Append(mlContext.BinaryClassification.Calibrators.Platt());

I can understand this being standard behaviour for uncalibrated learners but even with calibrated learners such as the following this is still needed (for the Probability column to be found):

var algorithm = mlContext.BinaryClassification.Trainers.AveragedPerceptron(labelColumnName: "Label", featureColumnName: "Features");

Is this correct behaviour?

Many thanks

Fig

@yaeldekel

This comment has been minimized.

Copy link
Member

@yaeldekel yaeldekel commented Jan 6, 2020

Hi @lefig , thanks for your question.
The AveragedPerceptron algorithm does not return a calibrated model, it returns a model that only knows how to compute a Score. In order to make it calibrated you would need to add a calibrator like in your example. It is true that some of the binary classification trainers automatically return a calibrated model without the need to explicitly add one (such as FastTree), and in these cases you do not need to add a calibrator before calling the Evaluate API.

Another option for models that are not calibrated, is to use the EvaluateNonCalibrated API instead of Evaluate. This API does not require a Probability column, and returns only metrics that can be computed directly from the Score column.

@antoniovs1029 antoniovs1029 self-assigned this Jan 6, 2020
@antoniovs1029

This comment has been minimized.

Copy link
Member

@antoniovs1029 antoniovs1029 commented Jan 7, 2020

Hi @lefig , it seems your question has been answered so I will close this issue. If you have anything else to add, please feel free to reopen it. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.