[SYSTEMDS-2719] Lineage exploitation in buffer pool#1096
Closed
sergred wants to merge 2 commits intoapache:masterfrom
Closed
[SYSTEMDS-2719] Lineage exploitation in buffer pool#1096sergred wants to merge 2 commits intoapache:masterfrom
sergred wants to merge 2 commits intoapache:masterfrom
Conversation
… buffer pool.
When buffers are to be evicted, we check if there is a valid lineage item that can be used for further recompute.
In this case, we do not write the buffer to FS and keep the lineage item instead.
The test files are : src/test/java/org/apache/sysds/test/functions/lineage/LineageExploitationBufferPoolTest.java
src/test/scripts/functions/lineage/LineageExploitationBufferPool1.dml
The issue is that, so far, all to-be-evicted buffers still go to FS.
… RAND, we store their lineage alongside the data. Under memory pressure, these matrices do not go to the buffer pool. We recompute them from lineage instead of reading evicted buffers from FS. This potentially results in fewer evictions and IO.
Contributor
|
LGTM - thanks @sergred for this great feature. During the merge I just made a few tweaks: added missing license header, fixed wrong formatting (tabs over spaces), changed the way we present the new statistics, fixed the recomputation by lineage, removed an unnecessary |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This task introduces a change to object storage mechanisms. For matrices that are results of data generation instructions such as RAND, we store their lineage alongside the data. Under memory pressure, these matrices do not go to the buffer pool. We recompute them from lineage instead of reading evicted buffers from FS. This potentially results in fewer evictions and IO.