Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LightGBMMulticlass doesn't normalize its Score column #4450

Closed
LittleLittleCloud opened this issue Nov 7, 2019 · 11 comments · Fixed by #4472
Assignees
Labels
bug

Comments

@LittleLittleCloud
Copy link
Contributor

@LittleLittleCloud LittleLittleCloud commented Nov 7, 2019

when I try AutoML v0.16.0 multi-classification, sometimes I get prediction score <0, and the summation of all scores doesn't equal to 1. which is unexpected. Maybe some trainer's output doesn't normalize.

image

I publish a mini-reproducible project in this repo for facilitating debug.

@justinormont justinormont added the bug label Nov 7, 2019
@justinormont

This comment has been minimized.

Copy link
Member

@justinormont justinormont commented Nov 7, 2019

With 100k rows, you are going to be in TrainTest mode instead of CV.

My guess for the non-normalized Score column:

Perhaps LightGBMMulticlass changed its behavior, or we modified our wrapper for it.

In TLC (internally) LightGBMMulticlass returns a normalized Score column. In current ML.NET, LightGBMMulticlass returns the raw values of each class. This is unexpected behavior to me.

Work around for Model Builder:

I'd recommend just normalizing the Score column temporarily.

Todo:

  • Investigate why it changed
    • Check if LightGBMMulticlass or our wrapper for it has changed its behavior
    • Check if the above was expected
  • Ensure we normalize the score column in our wrapper for LightGBMMulticlass

/cc @codemzs, @eerhardt

@justinormont justinormont changed the title [AutoML v0.16.0] Multiclass classification score sum doesn't alwasy equals 1 LightGBMMulticlass doesn't normalize its Score column Nov 8, 2019
@harishsk

This comment has been minimized.

Copy link
Member

@harishsk harishsk commented Nov 11, 2019

When I run the sample code in LightGBM.cs I do see that the scores are being normalized.

Let me take a look at your sample repro and get back to you.

@harishsk

This comment has been minimized.

Copy link
Member

@harishsk harishsk commented Nov 11, 2019

I downloaded your project file and tried to build it and ran into the following error:

AutoMLMultiClassificationBug-master\AutoMLMultiClassBug\obj\project.assets.json' not found. Run a NuGet package restore to generate this file

Running nuget restore isn't helping.
Are you able to build your project locally from the downloaded zip file?

@harishsk harishsk self-assigned this Nov 11, 2019
@harishsk

This comment has been minimized.

Copy link
Member

@harishsk harishsk commented Nov 12, 2019

After messing with the nuget package sources, I was able to build and run the project. But I can't reproduce the issue.

This is my output:

Running AutoML multiclass classification experiment for 5 seconds...
Total models produced: 1
Best model's trainer: AveragedPerceptronOva
Metrics of best model from validation data --
LogLoss: 0.6980664075650147
LogLossReduction: 0.006876844013740892
MacroAccuracy: 0.22116438506449212
MicroAccuracy: 0.5572916666666666
Predicted number for test data:
0.5885952
0.4082032
0.0027440954
0.00040919252
4.829548E-05
Press any key to continue...

I see that my run produces only 1 model and that the out normalizes to 1.

Do you get the same results if you download the code from github and run?

@justinormont

This comment has been minimized.

Copy link
Member

@justinormont justinormont commented Nov 12, 2019

@harishsk: You'll need additional runtime. As you note, only one model was tested. With longer runtime, LightGBM will have a chance to run and be selected as the winning model.

Try changing Program.cs line 53 to from 5s to 60s:
https://github.com/LittleLittleCloud/AutoMLMultiClassificationBug/blob/c034bf95568fee8f439633fb374c906ba71365f8/AutoMLMultiClassBug/Program.cs#L53

With the longer runtime, I see LightGBM chosen as the winning model; and the predicted Scores column is not normalized:
image

@harishsk

This comment has been minimized.

Copy link
Member

@harishsk harishsk commented Nov 12, 2019

When I change the run time to 60 seconds I get 5 models, with the best model as FastTreeOva and normalized output. But if I change the run time to 30 seconds, I get the output similar to yours.
Thanks for the tip. Let me take a look and report back.

@harishsk

This comment has been minimized.

Copy link
Member

@harishsk harishsk commented Nov 12, 2019

@LittleLittleCloud Thank you for raising this issue.
It turned out to be a bug in how OVA model parameters were saved. I have submitted a PR for this issue.

@justinormont

This comment has been minimized.

Copy link
Member

@justinormont justinormont commented Nov 12, 2019

@harishsk: Wow. That had to be difficult to locate the source of the issue. Thanks.

@rauhs

This comment has been minimized.

Copy link
Contributor

@rauhs rauhs commented Nov 13, 2019

Related: #3647 ?

@harishsk

This comment has been minimized.

Copy link
Member

@harishsk harishsk commented Nov 13, 2019

It is very likely that this PR fixes that issue as well. Let me verify tomorrow and report back.

@eerhardt

This comment has been minimized.

Copy link
Member

@eerhardt eerhardt commented Nov 13, 2019

#4051 could also be related here as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.