[R] LightGBM uses a lot of RAM when #feature and #leaves are both large

[Laurae2]

I ran LightGBM on a very big dataset. It is twice faster than xgboost. But for comparing both, I am building from all the dataset and using all the features, in contrary to LightGBM default behavior.

Dataset properties:

• 2,250,000 observations
• 23,636 features

(Committed) RAM usage:

• LightGBM: 164GB (173GB when building from 100000 observations using bin_construct_sample_cnt only)
• xgboost fast histogram: 63GB
• xgboost exact: 25GB (not sure, but it didn't use a lot)

Time per iteration (seems a big dataset fixes issue #542, but this one is really big...):

• LightGBM: 8-12 seconds
• xgboost fast histogram: 16-20 seconds
• xgboost exact: 90 seconds (gave up)

I think with MinGW, the dataset must be really large for LightGBM to scale well. Otherwise, we see the overhead issue with multithreading (and LightGBM ends up with the same speed of xgboost or is worse).

Now the issue is LightGBM using so much RAM... it uses 54GB RAM, swaps, and commits a total of approx 170,000,000 bytes. In the case of xgboost, as I capped to 54GB RAM it swaps but it's unnoticeable at all thanks to RAID 0 2Gbps NVMe drives (adding more RAM just made the histogram faster, the training was not faster).

I can provide you the building scripts for the dataset if you need to reproduce it. It takes about 30 minutes to prepare them.

I am using:

• 255 bins (intentional)
• depth of 10
• number of leaves of 1023
• 8 threads

[guolinke]

@Laurae2 try the parameter histogram_pool_size

[guolinke]

@Laurae2
Can you save the dataset to binary format, and give me its size?

I think the memory consuming here is used for the histogram cache. it needs about num_leaves * 20Bytes * num_features * num_bins.

You also can try to reduce the num_leaves, and see how much it costs.

[Laurae2]

@guolinke

binary dataset: 1,499,944 KB

with histogram_pool_size = 16384 it manages to keep under 54GB RAM.

When num_leaves decreases, the RAM required decreases exponentially. I think I'll drop some of my tests on this big dataset.

With num_leaves = 255, drops to 65GB RAM required.

[guolinke]

@Laurae2
So the dataset only cost 1GB memory.
And the histogram cache is 16GB when set histogram_pool_size=16384.
I am not sure what cost these other 40GB+ memories...

Did you run it by R-package? can you try it with the CLI version(you can use the binary dataset) ?

[guolinke]

@Laurae2 can you give me the reproduce dataset ?

[Laurae2]

@guolinke Do you have an email address where I can send you a link to the file?

[Laurae2]

@guolinke I think 10GB is caused by the data in memory, but the rest I am not sure what it is.

Do you know if it is possible to do this in R?:

• Remove the original dataset while being able to train (like training from a binary file, but directly in memory)
• Predict from lgb.Dataset instead of using the original dataset

xgboost also has the issue of exponential RAM increase when the maximum number of leaves increases, but it's getting bigger slower only.

[guolinke]

@Laurae2
can you give the memory usage before calling lgb.train ?
BTW, we cannot predict from lgb.Dataset .

[guolinke]

@Laurae2
I debugged on your dataset, some information:

1. Cost of Dataset: about 6GB.
2. Cost of other data structure: about 1GB
3. Cost of histogram cache: set by histogram_pool_size.

I run it on visual studio and mingw, parameter: data=train.bin app=binary histogram_pool_size=1024 .
they both cost about 8GB memory.

As a result, I think it is not reason to cost 54GB memories...

Another thing is: The binary dataset you build seems is by V2.0 version, not the master version.
Can you delete all local code, and re-build for the master branch version?

[Laurae2]

I'll recompile with master branch and come back in 2 days when server is free (no idea why it keeps using old LightGBM, I think the dll didn't get freed as I encounter this issue a lot - going to wipe the folder instead of trusting the reinstall for it from now).

can you give the memory usage before calling lgb.train ?

Approximately 8GB if I remember (+2GB of Windows).

[guolinke]

@Laurae2
can you share your R script for the reproducing?

[Laurae2]

@guolinke Check here: https://github.com/Laurae2/gbt_benchmarks/tree/master/zzz_new

It is the "reput" dataset. You can run the Creator scripts (there are 3 of them bundled together) manually if you want to recreate the dataset, then you can manually run the "code" script to run benchmark. The number inside the sh scripts define number of threads to use.

I can try to provide you an easier script when I finish work for training model if needed.

Run "debug" script for reput R script only if your objective is to test RAM usage.

(Will be back in 11h)