Compressed binary log #270
Conversation
We add new event types to support compress the binlog as follow:
QUERY_COMPRESSED_EVENT,
WRITE_ROWS_COMPRESSED_EVENT_V1,
UPDATE_ROWS_COMPRESSED_EVENT_V1,
DELETE_POWS_COMPRESSED_EVENT_V1,
WRITE_ROWS_COMPRESSED_EVENT,
UPDATE_ROWS_COMPRESSED_EVENT,
DELETE_POWS_COMPRESSED_EVENT
We introduce two option for this feature: "log_bin_compress " and
"log_bin_compress_min_len", the
former is a switch of whether the binlog should be compressed and the
latter is the minimum length of
sql statement(in statement mode) or record(in row mode) that can be
compressed.
In master, it can be described by the code:
if binlog_format == statement {
if log_bin_compress == true and query_len >= log_bin_compress_min_len
create a Query_compressed_log_event;
else
create a Query_log_event;
}
if binlog_format == row {
if log_bin_compress == true and record_len >= log_bin_compress_min_len
create a Write_rows_compressed_log_event(when INSERT)
else
create a Write_log_event(when INSERT);
}
And in slave, the compressed binlog events would be converted to the
uncompressed form in IO thread, such as QUERY_COMPRESSED_EVENT convert
to QUERY_EVENT.
So the SQL and worker threads can be stay unchanged. And the sql thread
will be executed as fast as executing uncompressed binlog.
Now, we use zlib as compressed algrithm.
And the compressed record is
Compressed Record(3 parts)
Record Header: 1 Byte
0 Bit: Always 1, mean compressed;
1-3 Bit: Reversed, compressed algorithm�0 means zlib, It's always 0
by now.
4-7 Bit: Bytes of "Record Original Length"
Record Original Length: 1-4 Bytes
Compressed Buf: the real compressed part.
The feture can reduce 42% ~ 70% binlog capacity in our production
environment.
|
Hi, thank you for submitting this pull request. In order to consider your code we need you to sign the Oracle Contribution Agreement (OCA). Please review the details and follow the instructions at http://www.oracle.com/technetwork/community/oca-486395.html |
|
Hi, thank you for your contribution. Please confirm this code is submitted under the terms of the OCA (Oracle's Contribution Agreement) you have previously signed by cutting and pasting the following text as a comment: |
|
I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it. |
|
Hi, thank you for your contribution. Your code has been assigned to an internal queue. Please follow |
We add new event types to support compress the binlog as follow:
QUERY_COMPRESSED_EVENT,
WRITE_ROWS_COMPRESSED_EVENT_V1,
UPDATE_ROWS_COMPRESSED_EVENT_V1,
DELETE_POWS_COMPRESSED_EVENT_V1,
WRITE_ROWS_COMPRESSED_EVENT,
UPDATE_ROWS_COMPRESSED_EVENT,
DELETE_POWS_COMPRESSED_EVENT
We introduce two option for this feature: "log_bin_compress " and "log_bin_compress_min_len", the
former is a switch of whether the binlog should be compressed and the latter is the minimum length of
sql statement(in statement mode) or record(in row mode) that can be compressed.
In master, it can be described by the code:
if binlog_format == statement {
if log_bin_compress == true and query_len >= log_bin_compress_min_len
create a Query_compressed_log_event;
else
create a Query_log_event;
}
if binlog_format == row {
if log_bin_compress == true and record_len >= log_bin_compress_min_len
create a Write_rows_compressed_log_event(when INSERT)
else
create a Write_log_event(when INSERT);
}
And in slave, the compressed binlog events would be converted to the uncompressed form in IO thread, such as QUERY_COMPRESSED_EVENT convert to QUERY_EVENT.
So the SQL and worker threads can be stay unchanged. And the sql thread will be executed as fast as executing uncompressed binlog.
Now, we use zlib as compressed algrithm.
And the compressed record is
Compressed Record(3 parts)
Record Header: 1 Byte
0 Bit: Always 1, mean compressed;
1-3 Bit: Reversed, compressed algorithm。0 means zlib, It's always 0 by now.
4-7 Bit: Bytes of "Record Original Length"
Record Original Length: 1-4 Bytes
Compressed Buf: the real compressed part.
The feture can reduce 42% ~ 70% binlog capacity in our production environment.