Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LightGBM Error, code is -1, error message is 'bad allocation' #3340

Closed
PeterPann23 opened this Issue Apr 15, 2019 · 5 comments

Comments

Projects
None yet
3 participants
@PeterPann23
Copy link

PeterPann23 commented Apr 15, 2019

System information

  • **Windows 10 **:10.0.18362 Build 18362
  • .NET: Version 4.7.3, Microsoft.ML 1.0 preview
    -** Microsoft.ML.LightGMB**: 0.11.27505.13@build By dlab -14DDVS...

Issue

  • What did you do?
    Started training using following code:
var pipeline = mlContext.Transforms.Conversion.MapValueToKey(outputColumnName: nameof(PredictedResult.LabelIndex) , inputColumnName: nameof(Data.Label), sort: ValueToKeyMappingEstimator.SortOrder.Value)
                       .Append(mlContext.Transforms.Normalize(mode: NormalizingEstimator.NormalizerMode.MinMax, outputColumnName: nameof(Data.Features)))
                       .Append(mlContext.MulticlassClassification.Trainers.LightGbm(labelColumnName:nameof(Data.LabelIndex),  featureColumnName:nameof(Data.Features)))
                       .Append(mlContext.Transforms.Conversion.MapValueToKey(nameof(PredictedResult.PredictedLabelIndex), nameof(PredictedResult.PredictedLabel)))
                       .Append(mlContext.Transforms.CopyColumns(outputColumnName: nameof(PredictedResult.Scores), inputColumnName: DefaultColumnNames.Score));
            // Train the model.
            var model = pipeline.Fit(dataset.TrainSet);
  • What happened?
    Received error on log line:
Exception(s): LightGBM Error, code is -1, error message is 'bad allocation'.
Log: 18:20:52:[Source=LightGBMMulticlass; Loading data for LightGBM, Kind=Trace] Channel disposed. Elapsed 02:24:12.0168791.6010992.81611.0110556.
  • What did you expect?
    A error message that makes sense that I can understand what the issue is.

Source code / logs

No additional logs recorded,

@Ivanidzo4ka

This comment has been minimized.

Copy link
Member

Ivanidzo4ka commented Apr 15, 2019

This is something coming from LightGBM side.
Somewhat related:
Microsoft/LightGBM#1598

If you can scramble small repro we can pass it to LightGBM folks

Also not sure why you using:
.Append(mlContext.Transforms.Conversion.MapValueToKey(nameof(PredictedResult.PredictedLabelIndex), nameof(PredictedResult.PredictedLabel)))
Shouldn't it be KeyToValue instead?

@sfilipi sfilipi added the need info label Apr 15, 2019

@PeterPann23

This comment has been minimized.

Copy link
Author

PeterPann23 commented Apr 15, 2019

Hi,

The code comes from
Samples.Dynamic.Trainers.MulticlassClassification class LightGbm

I added only normalisation and used NameOf instead of strings, see sample code on #3290

         var pipeline = mlContext.Transforms.Conversion.MapValueToKey("LabelIndex", "Label")
                        .Append(mlContext.MulticlassClassification.Trainers.LightGbm(labelColumnName: "LabelIndex"))
                        .Append(mlContext.Transforms.Conversion.MapValueToKey("PredictedLabelIndex", "PredictedLabel"))
                        .Append(mlContext.Transforms.CopyColumns("Scores", "Score"));
@PeterPann23

This comment has been minimized.

Copy link
Author

PeterPann23 commented Apr 16, 2019

Hi,

text label to MapValueToKey results in to error:

Schema mismatch for input column 'Label': expected KeyType, got Text
Parameter name: inputSchema
@Ivanidzo4ka

This comment has been minimized.

Copy link
Member

Ivanidzo4ka commented Apr 16, 2019

That sample is definitely confusing. Good thing we rewriting it.

So you have following pipeline:

You have original data where labels are strings and features are vector of floats.
You apply first MapValueToKey on top of Label column which would produce LabelIndex column with respectful keys. (I dunno who come up with Index part of name)
That part is great and actually necessary since we multiclass works only on Key columns.
So my comment wasn't about first MapValueToKey transform.

By running trainer we will pop up two more columns PredictedLabel which is type Key and Score column which is vector of floats.

Applying MapValueToKey on top of key type column is doable, but looks weird I don't understand what's the point of doing it.

If you want to get original values back (from Key type for PredictedLabel) you either can use annotations

engine.OutputSchema[nameof(IrisPrediction.PredictedLabel)].GetKeyValues(ref keys);

or use MapKeyToValue
.Append(mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabelInOriginalType", "PredictedLabel"))

@PeterPann23

This comment has been minimized.

Copy link
Author

PeterPann23 commented Apr 16, 2019

cool, thanks for the update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.