You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Using master, it appears there's a memory leak when reading in the initial dataset. Specifying the --disk flag, it appears to happen when checking for the max feature size. It runs until OOM ends up killing it, which is after consuming around 105gbs of memory on this particular machine.
Our dataset of choice is 68Gbs in size, about 160 million samples, with 1 million features total (sparse). Happy to provide more information as needed.
Any thoughts?
The text was updated successfully, but these errors were encountered:
Hi, @Refefer Thanks for reporting this important bug. I have already fixed it and you can try it now.
Please let me know if there has any other problem!
Hi there,
Using master, it appears there's a memory leak when reading in the initial dataset. Specifying the --disk flag, it appears to happen when checking for the max feature size. It runs until OOM ends up killing it, which is after consuming around 105gbs of memory on this particular machine.
Our dataset of choice is 68Gbs in size, about 160 million samples, with 1 million features total (sparse). Happy to provide more information as needed.
Any thoughts?
The text was updated successfully, but these errors were encountered: