Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of Memory Exception In Model Builder #4499

Closed
mayoatte opened this issue Nov 22, 2019 · 4 comments

Comments

@mayoatte
Copy link

@mayoatte mayoatte commented Nov 22, 2019

System information

  • OS version/distro: Windows 10
  • .NET Version (eg., dotnet --info): .Net Core 3.0

Issue

  • What did you do?

Tried to use a Model Builder (AutoML) with a 1 GB csv file.

  • What happened?

Out of Memory Exception

image

  • What did you expect?

The file would get loaded. It looks like the Model Builder is trying to use a StringBuilder here. That's naturally going to fall down after a certain size of csv. Stream?

@justinormont

This comment has been minimized.

Copy link
Member

@justinormont justinormont commented Nov 22, 2019

For the model builder team:
Recommend using Microsoft.ML.AutoML api for inspecting the file, and creating the loader function. Then reading the file w/ the ML.NET text loader by feeding it the created loader parameters.

This avoids incompatibilities between the two (and it streams nicely).

@mayoatte

This comment has been minimized.

Copy link
Author

@mayoatte mayoatte commented Nov 22, 2019

Sounds good. I believe some of the csv columns in my data are also mucking up the larger csv (spaces!). Will close this for now and switch to using the AutoML API.

Thanks again for responding so fast.

@mayoatte mayoatte closed this Nov 22, 2019
@justinormont

This comment has been minimized.

Copy link
Member

@justinormont justinormont commented Nov 22, 2019

@mayoatte : My apologies, my comment was more for the Model Builder team as a way to fix your reported bug. I've updated my comment to that effect.

Though, you can also use the AutoML API directly for the same reason. You can also try the ML.NET CLI: https://docs.microsoft.com/en-us/dotnet/machine-learning/how-to-guides/install-ml-net-cli

@mayoatte

This comment has been minimized.

Copy link
Author

@mayoatte mayoatte commented Nov 22, 2019

@justinormont No worries at all.

After looking at it again, i wasn't sure if my csv was in the right formt (I also have csv columns). And in case, the API is a better fit for me as i was able to reuse all the data loading and prep code I had already without going back to csv.

Please free to reopen the issue if you'd still like the Model Builder team to have a look.

Thanks again for all your help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.