Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strange error message #4648

Closed
CBrauer opened this issue Jan 13, 2020 · 3 comments
Closed

Strange error message #4648

CBrauer opened this issue Jan 13, 2020 · 3 comments
Assignees
Labels

Comments

@CBrauer
Copy link

@CBrauer CBrauer commented Jan 13, 2020

Hello,

I just upgraded my project to the new pre-release versions of ML.NET and I got the following error message when I ran my program:

bug

I have added a Zip file of my program and dataset. I hope you guys can help me find out why I'm getting this error,

Charles

bug.zip

@Lynx1820 Lynx1820 self-assigned this Jan 13, 2020
@Lynx1820

This comment has been minimized.

Copy link
Member

@Lynx1820 Lynx1820 commented Jan 15, 2020

@justinormont This appears to be an AutoML issue, could you take a look at it?

@justinormont

This comment has been minimized.

Copy link
Member

@justinormont justinormont commented Jan 15, 2020

This is an error in the AutoML API code.

It appears due to the handling of NaN values in the AutoML code, which can occur for the F1Score metric that you're optimizing towards. Along with F1, log-loss reduction can also return a NaN.

F1 can return a NaN value when both the precision and recall are zero, as F1 = 2 * precision * recall / (precision + recall), which turns into 0 / 0. Precision/recall being zero occurs for a trivial model which always guesses the majority class zero.

Two things to fix in ML.NET/AutoML:

  1. Check why AutoML isn't handling the NaN for the metric values
  2. Consider returning F1=0.0 instead of NaN -- this is done in sklearn (code), and others (note)

    Our F1 calculation code:
    public Double F1 { get { return 2 * PrecisionPos * RecallPos / (PrecisionPos + RecallPos); } }
@CBrauer

This comment has been minimized.

Copy link
Author

@CBrauer CBrauer commented Jan 16, 2020

Thanks Justin. I'll change your code as your suggest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.