Getting OutOfMemory exception while training model on large datasets in file

[prathyusha12345]

System information

• OS version/distro: Windows

Issue

I am trying to create a sample dotnet/machinelearning-samples#520 to train a model on large datasets that are stored in a file. I am using BinaryClassification trainer. While training the model I am getting the OutOfMemory exception at the Fit() method as shown below.

var model = trainingPipeLine.Fit(trainTestData.TrainSet);

complete details of error

System.FormatException
  HResult=0x80131537
  Message=Parsing failed with an exception: Stream reading encountered exception
  Source=Microsoft.ML.Data
  StackTrace:
   at Microsoft.ML.Data.TextLoader.Cursor.<ParseParallel>d__33.MoveNext()
   at Microsoft.ML.Data.TextLoader.Cursor.MoveNextCore()
   at Microsoft.ML.Data.RootCursorBase.MoveNext()
   at Microsoft.ML.Data.LinkedRowFilterCursorBase.MoveNextCore()
   at Microsoft.ML.Data.RootCursorBase.MoveNext()
   at Microsoft.ML.Transforms.ValueToKeyMappingTransformer.Train(IHostEnvironment env, IChannel ch, ColInfo[] infos, IDataView keyData, ColumnOptionsBase[] columns, IDataView trainingData, Boolean autoConvert)
   at Microsoft.ML.Transforms.ValueToKeyMappingTransformer..ctor(IHostEnvironment env, IDataView input, ColumnOptionsBase[] columns, IDataView keyData, Boolean autoConvert)
   at Microsoft.ML.Transforms.ValueToKeyMappingEstimator.Fit(IDataView input)
   at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
   at Microsoft.ML.Transforms.OneHotEncodingTransformer..ctor(ValueToKeyMappingEstimator term, IEstimator`1 toVector, IDataView input)
   at Microsoft.ML.Transforms.OneHotEncodingEstimator.Fit(IDataView input)
   at Microsoft.ML.Data.EstimatorChain`1.Fit(IDataView input)
   at LargeDatasetsInSqlServer.Program.Main() in C:\GitRepos\Fork\ML-samples\ML-samples-LargeDataInFile\samples\csharp\getting-started\LargeDatasetsInFile\LargeDatasetsInFile\Program.cs:line 107

Inner Exception 1:
FormatException: Stream reading encountered exception

Inner Exception 2:
OutOfMemoryException: Insufficient memory to continue the execution of the program.

The data set is copied from shared folder \ct01\data\Criteo\Spark\day_0_withHeader.tsv.

Source code / logs

Please find the entire source code from the https://github.com/prathyusha12345/machinelearning-samples/tree/LargeDatasetsInFile/samples/csharp/getting-started/LargeDatasetsInFile

[harishsk]

I enlisted in your branch and I ran the LargeDatasets project. I don't see any exception.

What should I be doing differently?

Can you please help with the steps to reproduce the problem you are seeing?

[prathyusha12345]

@harishsk There are no steps in reproducing... I just ran the sample and got the exception. we tested in 2 computers one is of 8gb Ram and 256gb Hard disk and other one is 16gb RAM and 1TB hard disk.

By the way I am curious about how long did it take for train and evaluation as the file size is 42 gb?

[prathyusha12345]

@harishsk By the way the sample is not published yet. Have you tried the sample that i have mentioned in the PR above https://github.com/prathyusha12345/machinelearning-samples/tree/LargeDatasetsInFile/samples/csharp/getting-started/LargeDatasetsInFile ?

[harishsk]

Ah, Looks like i was running the wrong sample. Where do I find the file you reference above?

[harishsk]

Never mind. I found it. (There is a small typo in your text above. It should read \\ct01\data\Criteo\Spark\day_0_withHeader.tsv.)

[prathyusha12345]

ohh thats great..... or the dataset can be downloaded from citeo website here http://labs.criteo.com/2013/12/download-terabyte-click-logs/ .

[PeterPann23]

I have found that my datasets may be maximum 1/10th so 64GB = 640GB file of my ram, having more will cause and "allocation error" on ECC and Non ECC memory. Setting cashing will make the problem get worse.

[prathyusha12345]

@codemzs Could you please check and let me know if you have any idea on this

[najeeb-kazmi]

@prathyusha12345 this is not a bug. There is some incorrect usage, and some non-optimal usage, that is causing the memory usage to blow up.

Incorrect: You are loading the 13 numerical features as string, and then applying FeaturizeText on each of them. This is completely unnecessary, and it also means that for each feature, a dictionary is trained and then the feature is vectorized when it does not need to. You can just load it in as as a float. Moreover, you can load all 13 into a vector of floats.

Non-optimal: The categorical features in the Criteo dataset are very high cardinality. Because of this, OneHotEncoding can be costly. A further simplification is to load all the categorical features into one vector as well, and use OneHotHashEncoding instead. This will significantly cut down memory usage, as large dictionaries for each categorical feature will not have to be learnt and stored.

You can also load the label as bool, then you won't need to convert its type (this doesn't impact memory usage, its just a simplification).

Btw, there is some bad � character in the last line of the day_0_withHeader.tsv from the \ct01 location. My code ran till the first pass of training LightGbm and then threw an error complaining about this. I'd suggest pre-processing to remove this line, e.g. sed -i '$d' day_0_withHeader.tsv in the Linux subsystem.

Here are the changes I'd suggest:

    public class UrlClicks
    {
        [LoadColumn(0)]
        public bool Label { get; set; }

        [LoadColumn(1, 13)]
        [VectorType(13)]
        public float[] NumericalFeatures { get; set; }

        [LoadColumn(14, 39)]
        [VectorType(26)]
        public string[] CategoricalFeatures { get; set; }
    }

var categoriesTransformer = mlContext.Transforms.Categorical.OneHotHashEncoding("CatEncoded", "CategoricalFeatures");
            
var finalTransformerPipeLine = categoriesTransformer.Append(mlContext.Transforms.Concatenate("Features", "NumericalFeatures", "CatEncoded"));

[najeeb-kazmi]

Closing this issue. Please feel free to reopen if there are more questions.