MCOL-987 LZ4 compression support. #1842
Merged
+1,302
−540
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
denis0x0D
force-pushed
the
MCOL-987_LZ
branch
6 times, most recently
from
c24f6a9
to
5d3d766
Compare
denis0x0D
changed the title
|
Note, I added LZ4 as default by separate commit on top on this pull request to trigger test suite under LZ4 compression, this patch should be removed, before merging. |
denis0x0D
force-pushed
the
MCOL-987_LZ
branch
3 times, most recently
from
c2b07c4
to
9c886a5
Compare
mariadb-AlexeyAntipovsky
suggested changes
Apr 9, 2021
|
@mariadb-AlexeyAntipovsky thanks for review! I'll address your comments. |
mariadb-AlexeyAntipovsky
approved these changes
Apr 9, 2021
denis0x0D
force-pushed
the
MCOL-987_LZ
branch
2 times, most recently
from
8b0a506
to
394a356
Compare
denis0x0D
force-pushed
the
MCOL-987_LZ
branch
2 times, most recently
from
98a6b22
to
863bd36
Compare
mariadb-AlexeyAntipovsky
suggested changes
Jul 5, 2021
| @@ -696,13 +696,25 @@ void loadBlock ( | |||
| i = fp->pread( &cmpHdrBuf[0], 0, 4096 * 3); | |||
There was a problem hiding this comment.
4096 here and below is some random number or HDR_BUF_LEN?
There was a problem hiding this comment.
Seems like it's HDR_BUF_LEN, I'm not sure, why author wrote 4096, instead of HDR_BUF_LEN.
drrtuy
reviewed
Jul 5, 2021
dbcon/mysql/columnstore_info.sql
Outdated
| @@ -95,7 +95,11 @@ DROP PROCEDURE IF EXISTS `compression_ratio` // | |||
|
|
|||
| CREATE PROCEDURE compression_ratio() SQL SECURITY INVOKER | |||
| BEGIN | |||
| SELECT CONCAT((SELECT SUM(data_size) FROM information_schema.columnstore_extents ce left join information_schema.columnstore_columns cc on ce.object_id = cc.object_id where compression_type='Snappy') / (SELECT SUM(compressed_data_size) FROM information_schema.columnstore_files WHERE compressed_data_size IS NOT NULL), ':1') COMPRESSION_RATIO; | |||
|
|
|||
| SELECT CONCAT((SELECT SUM(data_size) FROM information_schema.columnstore_extents ce left join information_schema.columnstore_columns cc on ce.object_id = cc.object_id where compression_type='Snappy') / (SELECT SUM(compressed_data_size) FROM information_schema.columnstore_files co left join information_schema.columnstore_columns cc on (co.object_id = cc.object_id) left join information_schema.columnstore_extents ce on (ce.object_id = co.object_id) where compression_type='Snappy' and compressed_data_size IS NOT NULL /* could be a situation when compressed_data_size != NULL but data_size == 0, in this case we will get wrong ratio */ and data_size > 0), ':1') COMPRESSION_RATIO_SNAPPY; | |||
There was a problem hiding this comment.
I think we need to put a literal column to tell which compression type is this and combine two SELECTs into a single UNION if UNION doesn't crash with queries from I_S :)
There was a problem hiding this comment.
I don't know an SQL good enough, can you please show an example, thanks.
|
@mariadb-AlexeyAntipovsky @drrtuy thanks for review, updated. For some comments I've added a questions. Thanks. |
* Adds CompressInterfaceLZ4 which uses LZ4 API for compress/uncompress. * Adds CMake machinery to search LZ4 on running host. * All methods which use static data and do not modify any internal data - become `static`, so we can use them without creation of the specific object. This is possible, because the header specification has not been modified. We still use 2 sections in header, first one with file meta data, the second one with pointers for compressed chunks. * Methods `compress`, `uncompress`, `maxCompressedSize`, `getUncompressedSize` - become pure virtual, so we can override them for the other compression algos. * Adds method `getChunkMagicNumber`, so we can verify chunk magic number for each compression algo. * Renames "s/IDBCompressInterface/CompressInterface/g" according to requirement.
mariadb-AlexeyAntipovsky
approved these changes
Jul 7, 2021
drrtuy
approved these changes
Jul 7, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
LZ4 compression support.