feat: add singleton dataset mode for faster performance and use old sparse dataset create method to reduce memory usage #1066
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Codecov Report
@@ Coverage Diff @@
## master #1066 +/- ##
==========================================
- Coverage 85.74% 85.54% -0.21%
==========================================
Files 252 254 +2
Lines 11605 11801 +196
Branches 599 619 +20
==========================================
+ Hits 9951 10095 +144
- Misses 1654 1706 +52
Continue to review full report at Codecov.
|
imatiach-msft
force-pushed
the
ilmat/improve-csr-matrix
branch
from
ccd45fa
to
8511d54
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
imatiach-msft
force-pushed
the
ilmat/improve-csr-matrix
branch
from
736e93a
to
90e5f0a
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
imatiach-msft
force-pushed
the
ilmat/improve-csr-matrix
branch
from
06c35d7
to
15fa45d
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
imatiach-msft
force-pushed
the
ilmat/improve-csr-matrix
branch
from
15fa45d
to
f5f9829
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
imatiach-msft
force-pushed
the
ilmat/improve-csr-matrix
branch
from
f5f9829
to
4237fe7
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
imatiach-msft
force-pushed
the
ilmat/improve-csr-matrix
branch
from
4237fe7
to
5e003ae
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
imatiach-msft
force-pushed
the
ilmat/improve-csr-matrix
branch
from
5e003ae
to
174a5b6
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
imatiach-msft
force-pushed
the
ilmat/improve-csr-matrix
branch
from
bd54562
to
fbfe4f1
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Hello, how can I get the latest snapshot jar with this feature please? Thank you. |
This was referenced Dec 27, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding single (or "singleton") dataset mode to lightgbm learners.
User can enable this new mode by setting the parameter useSingleDatasetMode=True (it is false by default).
In this mode, each executor creates a single LightGBMDataset. By default, currently each task within an executor creates a dataset:
In this PR, a new mode is added to only create one dataset per executor:
This means that there is lower network communication overhead since fewer nodes are initialized and more parallelization is done within the machine in the native code with default number of threads. This also seems to reduce memory usage significantly for some datasets.
Note in most cluster configurations there is usually only one executor per machine anyway.
In performance tests, we've found this mode sometimes outperforms the default in certain scenarios, both in terms of memory and execution time.
On a sparse dataset with 9 GB of data and large parameter values (num_leaves=768, num_trees=1000, min_data_in_leaf=15000, max_bin=512) and 5 machines with 8 cores and 28 GB of RAM, runtime was 17.54 minutes with this new mode. When specifying tasks=5 it took 106 minutes and in default mode it failed with OOM.
However in other scenarios the default mode is much faster.
On dense Higgs dataset (4GB) with default parameters and 8 workers with 14 GB memory, 4 cores each the default run took 54 seconds but new single dataset mode took 1.1 minutes (used to be 2 minutes with recent optimization on dataset conversion code to native this was speeded up a lot), which was a bit slower.
For this reason we will keep this mode as non-default for now as we continue to do more benchmarking/experimentation.