Closed
Conversation
Baunsgaard
force-pushed
the
DecompressionOptimization
branch
from
201cd8f to
c0f0d6a
Compare
Baunsgaard
force-pushed
the
DecompressionOptimization
branch
from
86db6f5 to
e117dff
Compare
This commit change the blocking of the decompression, to no longer align perfectly with 64k blocks, since if a column group contain many columns this is sub optimal. A future update is to introduce skip lists in SDC col Group since these suffer from iterating through their offset lists, of reach thread starting at a offset. [SYSTEMDS-3000] CLA MM Most Common Element Addition This commit adds an exploitation of the compressed representation that allows add the most common element when multiplying on the left side with a compressed transposed matrix. This is a common occurrence in MMChain and TSMM and allows sparsity exploitation of dense compressed column groups.
add sparse dictionary, and cocodeMatrixCost mm mult cost update remove scaling with sparsity
Compressed left multiplication have two phases, first preaggregation then a matrix multiplication. This commit make the matrix mult use the default systemds kernels. This allows for exploitation of the various dedicated mm kernels already in SystemDS.
Initial version of cost based mm cocoding
This commit change the dictionary of the column groups to support MatrixBlocks, this further enforce the previous design of using already implemented kernels, and allow for sparse dictionary exploitation in operations.
Add various tests compression - InsertionSorterTests - OffsetTests - MappingTests Minor bug fixes and better mapping test, Better compression ration on SDC with 3 distinct elements since the dictionary contains number of distinct elements -1 for SDC. Therefore if the SDC contain 3 distinct values, it only need 2 distinct identifiers in the dictionary.
Baunsgaard
force-pushed
the
DecompressionOptimization
branch
from
e117dff to
af16b5d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains multiple updates to the compression framework:
The PR is not ready for merge, but here for testing.