Memory leak during disk reading on large dataset

[Refefer]

Hi there,

Using master, it appears there's a memory leak when reading in the initial dataset. Specifying the --disk flag, it appears to happen when checking for the max feature size. It runs until OOM ends up killing it, which is after consuming around 105gbs of memory on this particular machine.

Our dataset of choice is 68Gbs in size, about 160 million samples, with 1 million features total (sparse). Happy to provide more information as needed.

Any thoughts?

[Refefer]

It appears to be a leak in general, most likely in the OndiskReader since it continues to leak when using a smaller dataset during train.

[aksnzhy]

Thanks for your report. We will check and fix this problem as soon as possible.

[aksnzhy]

Hi, @Refefer Thanks for reporting this important bug. I have already fixed it and you can try it now.
Please let me know if there has any other problem!

[Refefer]

@aksnzhy ran some tests and it appears your patch fixed the problem. Closing and thanks!