Opinion on LightGBM scalability

[Laurae2]

@guolinke What's your opinion on the current LightGBM scalability? Visual Studio has a strange behavior without CPU affinity, which makes it faster than if it had CPU affinity setup.

https://medium.com/@Laurae2/getting-the-most-of-xgboost-and-lightgbm-speed-compiler-cpu-pinning-374c38d82b86

I did some tests using R, and Visual Studio is still best for LightGBM as we saw before.

Note that when increasing the max depth/leaves to a large number, it takes forever to learn using MinGW.

Does Visual Studio has optimizations on Windows to be better without CPU pinning/affinity? It seems strange a roaming CPU is faster than with a pinned CPU.

[guolinke]

@Laurae2 It seems you are using a very sparse dataset.
Do you have the results on dense dataset ?

[Laurae2]

@guolinke Do you have any large dense dataset with enough features? (something like 1M x 1K, or smaller)

Higgs is too small in number of features.

[guolinke]

@Laurae2 I think you can generate some new features in higgs dataset.
e.g. sum/sub/mul/div of two features.

[Laurae2]

I started one run today using mul of all features. 33.3 GB matrix. Will come back later for results.

[Laurae2]

@guolinke The difference is huge with CPU pinning on Higgs 33.3GB.

[guolinke]

@Laurae2 so for dense dataset, the pinned solution is better.
I guess the reason is the effect of memory bandwidth (latency).
For dense dataset, LightGBM prefers the high memory bandwidth, so pin solution is better.
While for sparse dataset, the memory bandwidth is not the bottleneck.