-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
MDEV-31404 Implement binlog_space_limit
binlog_space_limit is a variable in Percona server used to limit the total size of all binary logs. This implementation is based on code from Percona server 5.7. In MariaDB we decided to call the variable max-binlog-total-size to be similar to max-binlog-size. This makes it easier to find in the output from 'mariadbd --help --verbose'). MariaDB will also support binlog_space_limit for compatibility with Percona. Some internal notes to explain implementation notes: - When running MariaDB does not delete binary logs that are either used by slaves or have active xid that are not yet committed. Some implementation notes: - max-binlog-total-size is by default 0 (no limit). - max-binlog-total-size can be changed without server restart. - Binlog file sizes are checked on startup, or if max-binlog-total-size is set to a value > 0, not for every log write. The total size of all binary logs is cached and dynamically updated when updating the binary log on binary log rotation. - max-binlog-total-size is checked against existing log files during serverstart, binlog rotation, FLUSH LOGS, when writing to binary log or when max-binlog-total-size changes value. - Option --slave-connections-needed-for-purge with 1 as default added. This allows one to ensure that we do not delete binary logs if there is less than 'slave-connections-needed-for-purge' connected. Without this option max-binlog-total-size would potentially delete binlogs needed by slaves on server startup or when a slave disconnects as there are then no connected slaves to protect active binlogs. - PURGE BINARY LOGS TO ... will be executed as if slave-connectitons-needed-for-purge would be zero. In other words it will do the purge even if there is no slaves connected. If there are connected slaves working on the logs, these will be protected. - If binary log is on and max-binlog-total_size <> 0 then the status variable 'Binlog_disk_use' shows the current size of all old binary logs + the state of the current one. - Removed test of strcmp(log_file_name, log_info.log_file_name) in purge_logs_before_date() as this is tested in can_purge_logs() - To avoid expensive calls of log_in_use() we cache the result for the last log that is in use by a slave. Future calls to can_purge_logs() for this binary log will be quickly detected and false will be returned until a slave starts working on a new log. - Note that after a binary log rotation caused by max_binlog_size, the last log will not be purged directly as it is still in use internally. The next binary log write will purge binlogs if needed. Reviewer:Kristian Nielsen <knielsen@knielsen-hq.org>
- Loading branch information
Showing
23 changed files
with
906 additions
and
46 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
!include include/default_my.cnf | ||
|
||
[mariadbd] | ||
slave_connections_needed_for_purge=0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,123 @@ | ||
select @@global.max_binlog_total_size; | ||
@@global.max_binlog_total_size | ||
1500 | ||
select @@global.max_binlog_size; | ||
@@global.max_binlog_size | ||
4096 | ||
# | ||
# MDEV-31404 Implement binlog_space_limit | ||
# | ||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
show binary logs; | ||
Log_name File_size | ||
binary.000001 # | ||
binary.000002 # | ||
binary.000003 # | ||
binary.000004 # | ||
show status like "binlog_disk_use"; | ||
Variable_name Value | ||
Binlog_disk_use 1552 | ||
set @@global.slave_connections_needed_for_purge= 0; | ||
# binary.000001 should be deleted now | ||
show binary logs; | ||
Log_name File_size | ||
binary.000002 # | ||
binary.000003 # | ||
binary.000004 # | ||
show status like "binlog_disk_use"; | ||
Variable_name Value | ||
Binlog_disk_use 1183 | ||
CREATE TABLE `t1` ( | ||
`v1` int(11) DEFAULT NULL, | ||
`v2` varchar(8000) DEFAULT NULL, | ||
KEY `v1` (`v1`) | ||
) engine=myisam; | ||
INSERT INTO t1 VALUES (0,repeat("a",3000)); | ||
show status like "binlog_disk_use"; | ||
Variable_name Value | ||
Binlog_disk_use 3863 | ||
# First binary should be binary.000004 | ||
show binary logs; | ||
Log_name File_size | ||
binary.000004 # | ||
INSERT INTO t1 VALUES (2,repeat("b",10)); | ||
# First binary should be binary.000004 | ||
show binary logs; | ||
Log_name File_size | ||
binary.000004 # | ||
binary.000005 # | ||
FLUSH LOGS; | ||
# First binary should be binary.000005 | ||
show binary logs; | ||
Log_name File_size | ||
binary.000005 # | ||
binary.000006 # | ||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
show binary logs; | ||
Log_name File_size | ||
binary.000008 # | ||
binary.000009 # | ||
binary.000010 # | ||
show status like "binlog_disk_use"; | ||
Variable_name Value | ||
Binlog_disk_use 1225 | ||
PURGE BINARY LOGS TO 'binary.000009'; | ||
# First binary should be binary.000009 | ||
show binary logs; | ||
Log_name File_size | ||
binary.000009 # | ||
binary.000010 # | ||
INSERT INTO t1 VALUES (3,repeat("c",4000)); | ||
# First binary should be binary.000010 | ||
show binary logs; | ||
Log_name File_size | ||
binary.000010 # | ||
binary.000011 # | ||
INSERT INTO t1 VALUES (4,repeat("d",3000)); | ||
# First binary should be binary.000011 | ||
show binary logs; | ||
Log_name File_size | ||
binary.000011 # | ||
RESET MASTER; | ||
show binary logs; | ||
Log_name File_size | ||
binary.000001 # | ||
show status like "binlog_disk_use"; | ||
Variable_name Value | ||
Binlog_disk_use 325 | ||
INSERT INTO t1 VALUES (5,"e"); | ||
FLUSH LOGS; | ||
INSERT INTO t1 VALUES (6,repeat("f",3000)); | ||
show binary logs; | ||
Log_name File_size | ||
binary.000002 # | ||
show status like "binlog_disk_use"; | ||
Variable_name Value | ||
Binlog_disk_use 3647 | ||
INSERT INTO t1 VALUES (7,repeat("g",3000)); | ||
# binary.000001 should be deleted now | ||
show binary logs; | ||
Log_name File_size | ||
binary.000002 # | ||
binary.000003 # | ||
show status like "binlog_disk_use"; | ||
Variable_name Value | ||
Binlog_disk_use 7338 | ||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
# binary.000002 should be deleted now | ||
show binary logs; | ||
Log_name File_size | ||
binary.000003 423 | ||
binary.000004 423 | ||
binary.000005 379 | ||
show status like "binlog_disk_use"; | ||
Variable_name Value | ||
Binlog_disk_use 1225 | ||
DROP TABLE IF EXISTS t1; | ||
set @@global.slave_connections_needed_for_purge= default; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
--log-bin=binary | ||
--max-binlog-total-size=1500 | ||
--max-binlog-size=4096 | ||
--binlog-format=row | ||
--slave_connections_needed_for_purge=1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,94 @@ | ||
--source include/have_log_bin.inc | ||
select @@global.max_binlog_total_size; | ||
select @@global.max_binlog_size; | ||
|
||
# Note that this test is only using MyISAM tables. | ||
# The reason for this is that we do not want to have engine | ||
# checkpoints in the binary log as it would change the number of | ||
# bytes in the log, which could cause random rotations. | ||
|
||
--echo # | ||
--echo # MDEV-31404 Implement binlog_space_limit | ||
--echo # | ||
|
||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
source include/show_binary_logs.inc; | ||
show status like "binlog_disk_use"; | ||
set @@global.slave_connections_needed_for_purge= 0; | ||
--echo # binary.000001 should be deleted now | ||
source include/show_binary_logs.inc; | ||
show status like "binlog_disk_use"; | ||
|
||
CREATE TABLE `t1` ( | ||
`v1` int(11) DEFAULT NULL, | ||
`v2` varchar(8000) DEFAULT NULL, | ||
KEY `v1` (`v1`) | ||
) engine=myisam; | ||
INSERT INTO t1 VALUES (0,repeat("a",3000)); | ||
show status like "binlog_disk_use"; | ||
--echo # First binary should be binary.000004 | ||
source include/show_binary_logs.inc; | ||
|
||
INSERT INTO t1 VALUES (2,repeat("b",10)); | ||
--echo # First binary should be binary.000004 | ||
# The reson why we have logs 00004 and 00005 at this point is that we first | ||
# do a rotate and then try to purge. However as there are still existing | ||
# xid's pointing to the 00004, we cannot yet purge 000004. | ||
# After rotate a checkpoint record will be written to 00005 which | ||
# will release pointers to 00004. On next binary log write or flush | ||
# the purge will be retried and succeed. | ||
|
||
source include/show_binary_logs.inc; | ||
|
||
FLUSH LOGS; | ||
--echo # First binary should be binary.000005 | ||
source include/show_binary_logs.inc; | ||
|
||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
source include/show_binary_logs.inc; | ||
show status like "binlog_disk_use"; | ||
|
||
PURGE BINARY LOGS TO 'binary.000009'; | ||
--echo # First binary should be binary.000009 | ||
source include/show_binary_logs.inc; | ||
|
||
INSERT INTO t1 VALUES (3,repeat("c",4000)); | ||
--echo # First binary should be binary.000010 | ||
source include/show_binary_logs.inc; | ||
|
||
INSERT INTO t1 VALUES (4,repeat("d",3000)); | ||
--echo # First binary should be binary.000011 | ||
source include/show_binary_logs.inc; | ||
|
||
RESET MASTER; | ||
source include/show_binary_logs.inc; | ||
show status like "binlog_disk_use"; | ||
|
||
INSERT INTO t1 VALUES (5,"e"); | ||
FLUSH LOGS; | ||
INSERT INTO t1 VALUES (6,repeat("f",3000)); | ||
source include/show_binary_logs.inc; | ||
show status like "binlog_disk_use"; | ||
|
||
INSERT INTO t1 VALUES (7,repeat("g",3000)); | ||
--echo # binary.000001 should be deleted now | ||
source include/show_binary_logs.inc; | ||
show status like "binlog_disk_use"; | ||
|
||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
--echo # binary.000002 should be deleted now | ||
show binary logs; | ||
show status like "binlog_disk_use"; | ||
|
||
--disable_warnings | ||
DROP TABLE IF EXISTS t1; | ||
--enable_warnings | ||
|
||
set @@global.slave_connections_needed_for_purge= default; | ||
# End of 11.4 tests |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
--slave_connections_needed_for_purge=0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
include/master-slave.inc | ||
[connection master] | ||
# | ||
# MDEV-31404 Implement binlog_space_limit | ||
# | ||
# Test that master is not deleting binary logs before slave has a | ||
# chance to digest them | ||
select @@global.max_binlog_total_size; | ||
@@global.max_binlog_total_size | ||
1500 | ||
select @@global.max_binlog_size; | ||
@@global.max_binlog_size | ||
4096 | ||
connection slave; | ||
STOP SLAVE IO_THREAD; | ||
include/wait_for_slave_io_to_stop.inc | ||
connection master; | ||
kill DUMP_THREAD; | ||
CREATE TABLE `t1` ( | ||
`v1` int(11) DEFAULT NULL, | ||
`v2` varchar(8000) DEFAULT NULL, | ||
KEY `v1` (`v1`) | ||
); | ||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
FLUSH LOGS; | ||
show binary logs; | ||
Log_name File_size | ||
binary.000001 # | ||
binary.000002 # | ||
binary.000003 # | ||
binary.000004 # | ||
INSERT INTO t1 VALUES (0,repeat("a",3000)); | ||
show binary logs; | ||
Log_name File_size | ||
binary.000001 # | ||
binary.000002 # | ||
binary.000003 # | ||
binary.000004 # | ||
connection slave; | ||
START SLAVE IO_THREAD; | ||
connection master; | ||
connection slave; | ||
connection master; | ||
DROP TABLE t1; | ||
show binary logs; | ||
Log_name File_size | ||
binary.000004 # | ||
include/rpl_end.inc |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
--log-bin=binary | ||
--max-binlog-total-size=1500 | ||
--max-binlog-size=4096 | ||
--binlog-format=row | ||
--slave_connections_needed_for_purge=1 |
Oops, something went wrong.