This repository has been archived by the owner. It is now read-only.
Permalink
Browse files

Add basic suite of innodb stress tests

Summary:
Feature: Stress Tests

This is a port of all stress, bigstress, and hugestress tests to 5.6.

This is a complete port, except:

Removed all *_gc.test.  These tested a group commit mechanism that
is not in 5.6, and will never be ported to it.

This can stress test secondary indexes as well, but this is disabled.

Removed extra innodb_file_format_check.   It is no longer needed in 5.6.

Updated results files (partly with --record, partly manually).

Commit every 100 inserts while populating the tables.

Properly restart the master in innodb_stress.inc

Verify secondary index count after all stress test runs.

Removed use of vars we've not (yet) ported.  These included:
	innodb_zlib_wrap
	innodb_log_compressed_pages
	innodb_background_checkpoint
	innodb_prepare_commit_mutex
	innodb_prefix_index_cluster_optimization

Updated and verified changes to all results files.

Test Plan:
Ran all the non-huge tests, and one huge test, myself, so far so good.
Jenkins nightlies will start hammering these after this is pushed.

Reviewers: flamingcow, pivanof, jeremycole, andrew-ford, inaam-rana

Reviewed By: pivanof

CC: jtolmer, MarkCallaghan

Differential Revision: https://reviews.facebook.net/D15759
  • Loading branch information...
steaphangreene committed Apr 2, 2013
1 parent bbda2dd commit 8b6adf69913226cab5cf8aaf45914e66b812692d
Showing with 2,363 additions and 1 deletion.
  1. +27 −0 mysql-test/include/wait_for_slave_to_sync_with_master.inc
  2. +1 −1 mysql-test/mysql-test-run.pl
  3. +164 −0 mysql-test/suite/innodb_stress/include/innodb_stress.inc
  4. +26 −0 mysql-test/suite/innodb_stress/r/innodb_bigstress.result
  5. +26 −0 mysql-test/suite/innodb_stress/r/innodb_bigstress_blob.result
  6. +24 −0 mysql-test/suite/innodb_stress/r/innodb_bigstress_blob_nocompress.result
  7. +32 −0 mysql-test/suite/innodb_stress/r/innodb_bigstress_crash.result
  8. +32 −0 mysql-test/suite/innodb_stress/r/innodb_bigstress_crash_blob.result
  9. +30 −0 mysql-test/suite/innodb_stress/r/innodb_bigstress_crash_blob_nocompress.result
  10. +30 −0 mysql-test/suite/innodb_stress/r/innodb_bigstress_crash_nocompress.result
  11. +24 −0 mysql-test/suite/innodb_stress/r/innodb_bigstress_nocompress.result
  12. +26 −0 mysql-test/suite/innodb_stress/r/innodb_hugestress.result
  13. +26 −0 mysql-test/suite/innodb_stress/r/innodb_hugestress_blob.result
  14. +24 −0 mysql-test/suite/innodb_stress/r/innodb_hugestress_blob_nocompress.result
  15. +32 −0 mysql-test/suite/innodb_stress/r/innodb_hugestress_crash.result
  16. +32 −0 mysql-test/suite/innodb_stress/r/innodb_hugestress_crash_blob.result
  17. +30 −0 mysql-test/suite/innodb_stress/r/innodb_hugestress_crash_blob_nocompress.result
  18. +30 −0 mysql-test/suite/innodb_stress/r/innodb_hugestress_crash_nocompress.result
  19. +24 −0 mysql-test/suite/innodb_stress/r/innodb_hugestress_nocompress.result
  20. +26 −0 mysql-test/suite/innodb_stress/r/innodb_stress.result
  21. +26 −0 mysql-test/suite/innodb_stress/r/innodb_stress_blob.result
  22. +24 −0 mysql-test/suite/innodb_stress/r/innodb_stress_blob_nocompress.result
  23. +27 −0 mysql-test/suite/innodb_stress/r/innodb_stress_crash.result
  24. +27 −0 mysql-test/suite/innodb_stress/r/innodb_stress_crash_blob.result
  25. +25 −0 mysql-test/suite/innodb_stress/r/innodb_stress_crash_blob_nocompress.result
  26. +25 −0 mysql-test/suite/innodb_stress/r/innodb_stress_crash_nocompress.result
  27. +24 −0 mysql-test/suite/innodb_stress/r/innodb_stress_nocompress.result
  28. +7 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress-master.opt
  29. +4 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress-slave.opt
  30. +35 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress.test
  31. +7 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_blob-master.opt
  32. +4 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_blob-slave.opt
  33. +35 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_blob.test
  34. +7 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_blob_nocompress-master.opt
  35. +4 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_blob_nocompress-slave.opt
  36. +34 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_blob_nocompress.test
  37. +7 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_crash-master.opt
  38. +4 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_crash-slave.opt
  39. +35 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_crash.test
  40. +7 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_crash_blob-master.opt
  41. +4 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_crash_blob-slave.opt
  42. +35 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_crash_blob.test
  43. +7 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_crash_blob_nocompress-master.opt
  44. +4 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_crash_blob_nocompress-slave.opt
  45. +33 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_crash_blob_nocompress.test
  46. +7 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_crash_nocompress-master.opt
  47. +4 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_crash_nocompress-slave.opt
  48. +33 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_crash_nocompress.test
  49. +7 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_nocompress-master.opt
  50. +4 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_nocompress-slave.opt
  51. +33 −0 mysql-test/suite/innodb_stress/t/innodb_bigstress_nocompress.test
  52. +7 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress-master.opt
  53. +4 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress-slave.opt
  54. +38 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress.test
  55. +7 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_blob-master.opt
  56. +4 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_blob-slave.opt
  57. +38 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_blob.test
  58. +7 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_blob_nocompress-master.opt
  59. +4 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_blob_nocompress-slave.opt
  60. +37 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_blob_nocompress.test
  61. +7 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_crash-master.opt
  62. +4 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_crash-slave.opt
  63. +35 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_crash.test
  64. +7 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_crash_blob-master.opt
  65. +4 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_crash_blob-slave.opt
  66. +35 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_crash_blob.test
  67. +7 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_crash_blob_nocompress-master.opt
  68. +4 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_crash_blob_nocompress-slave.opt
  69. +33 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_crash_blob_nocompress.test
  70. +7 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_crash_nocompress-master.opt
  71. +4 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_crash_nocompress-slave.opt
  72. +33 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_crash_nocompress.test
  73. +7 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_nocompress-master.opt
  74. +4 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_nocompress-slave.opt
  75. +36 −0 mysql-test/suite/innodb_stress/t/innodb_hugestress_nocompress.test
  76. +6 −0 mysql-test/suite/innodb_stress/t/innodb_stress-master.opt
  77. +3 −0 mysql-test/suite/innodb_stress/t/innodb_stress-slave.opt
  78. +34 −0 mysql-test/suite/innodb_stress/t/innodb_stress.test
  79. +6 −0 mysql-test/suite/innodb_stress/t/innodb_stress_blob-master.opt
  80. +3 −0 mysql-test/suite/innodb_stress/t/innodb_stress_blob-slave.opt
  81. +34 −0 mysql-test/suite/innodb_stress/t/innodb_stress_blob.test
  82. +6 −0 mysql-test/suite/innodb_stress/t/innodb_stress_blob_nocompress-master.opt
  83. +3 −0 mysql-test/suite/innodb_stress/t/innodb_stress_blob_nocompress-slave.opt
  84. +33 −0 mysql-test/suite/innodb_stress/t/innodb_stress_blob_nocompress.test
  85. +6 −0 mysql-test/suite/innodb_stress/t/innodb_stress_crash-master.opt
  86. +3 −0 mysql-test/suite/innodb_stress/t/innodb_stress_crash-slave.opt
  87. +37 −0 mysql-test/suite/innodb_stress/t/innodb_stress_crash.test
  88. +6 −0 mysql-test/suite/innodb_stress/t/innodb_stress_crash_blob-master.opt
  89. +3 −0 mysql-test/suite/innodb_stress/t/innodb_stress_crash_blob-slave.opt
  90. +37 −0 mysql-test/suite/innodb_stress/t/innodb_stress_crash_blob.test
  91. +6 −0 mysql-test/suite/innodb_stress/t/innodb_stress_crash_blob_nocompress-master.opt
  92. +3 −0 mysql-test/suite/innodb_stress/t/innodb_stress_crash_blob_nocompress-slave.opt
  93. +35 −0 mysql-test/suite/innodb_stress/t/innodb_stress_crash_blob_nocompress.test
  94. +6 −0 mysql-test/suite/innodb_stress/t/innodb_stress_crash_nocompress-master.opt
  95. +3 −0 mysql-test/suite/innodb_stress/t/innodb_stress_crash_nocompress-slave.opt
  96. +35 −0 mysql-test/suite/innodb_stress/t/innodb_stress_crash_nocompress.test
  97. +6 −0 mysql-test/suite/innodb_stress/t/innodb_stress_nocompress-master.opt
  98. +3 −0 mysql-test/suite/innodb_stress/t/innodb_stress_nocompress-slave.opt
  99. +32 −0 mysql-test/suite/innodb_stress/t/innodb_stress_nocompress.test
  100. +436 −0 mysql-test/suite/innodb_stress/t/load_generator.py
@@ -0,0 +1,27 @@
#
# Similar to sync_slave_with_master but supports a configurable timeout
# let $slave_sync_timeout = 300;
# source include/wait_for_slave_to_sync_with_master;
#

connection slave;
stop slave;
start slave;

connection master;

let $file = query_get_value(SHOW MASTER STATUS, File, 1);
let $pos = query_get_value(SHOW MASTER STATUS, Position, 1);

connection slave;
--disable_result_log
--disable_query_log
eval select master_pos_wait("$file", $pos, $slave_sync_timeout);
--enable_result_log
--enable_query_log

connection master;
sync_slave_with_master;

connection master;

@@ -163,7 +163,7 @@ END

# If you add a new suite, please check TEST_DIRS in Makefile.am.
#
my $DEFAULT_SUITES= "main,sys_vars,binlog,federated,rpl,innodb,innodb_fts,innodb_zip,perfschema,funcs_1,opt_trace,parts,auth_sec";
my $DEFAULT_SUITES= "main,sys_vars,binlog,federated,rpl,innodb,innodb_fts,innodb_zip,perfschema,funcs_1,opt_trace,parts,auth_sec,innodb_stress";
my $opt_suites;

our $opt_verbose= 0; # Verbose output, enable with --verbose
@@ -0,0 +1,164 @@
# Populate a table with 1000 records. Allow the replica to sync with the master.
# Run concurrent threads that run OLTP transactions on master.
# Kill the master database server at random points.
# Check the table against the replica.
# Reinvoke the threads.

# create the directory for temporary log files.
--exec mkdir -p $MYSQL_TMP_DIR/load_generator

if ($fake_changes)
{
--exec mkdir -p $MYSQL_TMP_DIR/load_generator_slave
}

--connection master

# since this test generates lot of errors in log, suppress checking errors
call mtr.add_suppression(".*");

--sync_slave_with_master

--connection master
--let $pid_file = `SELECT @@pid_file`
--let $crash_num = 0
--let $master_host = 127.0.0.1
--let $table = test
--let $user = root
--let $checksum = 0
--let $secondary_index_checks = 0

if ($do_checksum)
{
# populate the table and store its checksum before any load.
let $exec =
python suite/innodb_stress/t/load_generator.py $pid_file $kill_db_after
$num_records 0 0 $user $master_host $MASTER_MYPORT
$table 0 $max_rows $MYSQL_TMP_DIR/load_generator 0 0 0;
exec $exec;
let $checksum=query_get_value(CHECKSUM TABLE t1, Checksum, 1);
# Master needs to be restarted to start with an empty buffer pool so
# that logical read ahead gets used.
let rpl_server_number = 1;
source include/rpl_restart_server.inc;
connection slave;
# Start slave to avoid I/O thread retry errors
disable_warnings;
source include/start_slave.inc;
enable_warnings;
let $num_records = 0;
}

while ($num_crashes)
{
connection master;
exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect;
if ($crash_num)
{
let $num_records = 0; # do not populate the table except for the first run.
}

if ($use_blob)
{
let $exec =
python suite/innodb_stress/t/load_generator.py $pid_file $kill_db_after
$num_records $num_workers $num_transactions $user $master_host $MASTER_MYPORT
$table 1 $max_rows $MYSQL_TMP_DIR/load_generator 0 $checksum $secondary_index_checks;
}
if (!$use_blob)
{
let $exec =
python suite/innodb_stress/t/load_generator.py $pid_file $kill_db_after
$num_records $num_workers $num_transactions $user $master_host $MASTER_MYPORT
$table 0 $max_rows $MYSQL_TMP_DIR/load_generator 0 $checksum $secondary_index_checks;
}

exec $exec;

if ($do_crash)
{
--echo Wait for reconnect
enable_reconnect;
# Call script that will poll the server waiting for it to be back online again
source include/wait_until_connected_again.inc;
connection slave;
source include/wait_until_connected_again.inc;
connection master;
}

--echo Checksum master
let $master_checksum = query_get_value(CHECKSUM TABLE t1, Checksum, 1);

# if sync_slave_with_master had a configurable timeout this would not be needed
let $slave_sync_timeout = 7200;
--source include/wait_for_slave_to_sync_with_master.inc

connection slave;
if ($fake_changes)
{
--echo applying fake updates to the slave
let $slave_pid_file = `SELECT @@pid_file`;
let $slave_exec =
python suite/innodb_stress/t/load_generator.py $slave_pid_file $kill_db_after
0 $num_workers $num_transactions $user $master_host $SLAVE_MYPORT
$table 0 $max_rows $MYSQL_TMP_DIR/load_generator_slave 1 $checksum $secondary_index_checks;
exec $slave_exec;
}
--echo Checksum slave
let $slave_checksum=query_get_value(CHECKSUM TABLE t1, Checksum, 1);
let $not_same = `SELECT $master_checksum-$slave_checksum`;
if ($not_same)
{
let $msg =
The checksums of table t1 for master and slave do not match for $crash_num th
crash. This may happen if there is a corrupt recovery log or a bug in crash
recovery. You can take a look at the logs in $MYSQL_TMP_DIR/load_generator to see the
queries issued before the crash.;
echo $msg;

connection master;
eval select * into outfile '$MYSQLTEST_VARDIR/tmp/master_all' from t1 order by id;
eval select id into outfile '$MYSQLTEST_VARDIR/tmp/master_id' from t1 order by id;
show master status;

connection slave;
eval select * into outfile '$MYSQLTEST_VARDIR/tmp/slave_all' from t1 order by id;
eval select id into outfile '$MYSQLTEST_VARDIR/tmp/slave_id' from t1 order by id;
show slave status;

die;
}
dec $num_crashes;
inc $crash_num;
}

# final cleanup
--connection master

let $primary=`select count(*) from t1 use index (primary)`;
let $secondary=`select count(*) from t1 use index (msg_i)`;
if ($primary != $secondary)
{
--echo Secondary index inconsistent! $primary != $secondary
--die
}

DROP TABLE t1;

# if sync_slave_with_master had a configurable timeout this would not be needed
let $slave_sync_timeout = 7200;
--source include/wait_for_slave_to_sync_with_master.inc

--connection slave
--source include/stop_slave.inc
# For stress tests sometimes the replication thread can not connect to master
# temporarily. This is either because the master crashed and it is recovering
# or the master is too busy and could not service the slave's requests.
# mtr's internal check requires that there be no errors in slave status.
# restarting replication clears the errors.
--source include/start_slave.inc
--source include/stop_slave.inc

connection master;

# --exec rm -rf $MYSQL_TMP_DIR/load_generator
@@ -0,0 +1,26 @@
include/master-slave.inc
Warnings:
Note #### Sending passwords in plain text without SSL/TLS is extremely insecure.
Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
[connection master]
DROP TABLE IF EXISTS t1;
CREATE TABLE t1(id INT AUTO_INCREMENT PRIMARY KEY,
msg_prefix VARCHAR(255),
msg VARCHAR(255),
msg_length int,
msg_checksum varchar(128),
KEY msg_i(msg_prefix))
ENGINE=INNODB
ROW_FORMAT=COMPRESSED
KEY_BLOCK_SIZE=1;
call mtr.add_suppression(".*");
Checksum master
stop slave;
start slave;
Checksum slave
DROP TABLE t1;
stop slave;
start slave;
include/stop_slave.inc
include/start_slave.inc
include/stop_slave.inc
@@ -0,0 +1,26 @@
include/master-slave.inc
Warnings:
Note #### Sending passwords in plain text without SSL/TLS is extremely insecure.
Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
[connection master]
DROP TABLE IF EXISTS t1;
CREATE TABLE t1(id INT AUTO_INCREMENT PRIMARY KEY,
msg_prefix VARCHAR(255),
msg longtext,
msg_length int,
msg_checksum varchar(128),
KEY msg_i(msg_prefix))
ENGINE=INNODB
ROW_FORMAT=COMPRESSED
KEY_BLOCK_SIZE=1;
call mtr.add_suppression(".*");
Checksum master
stop slave;
start slave;
Checksum slave
DROP TABLE t1;
stop slave;
start slave;
include/stop_slave.inc
include/start_slave.inc
include/stop_slave.inc
@@ -0,0 +1,24 @@
include/master-slave.inc
Warnings:
Note #### Sending passwords in plain text without SSL/TLS is extremely insecure.
Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
[connection master]
DROP TABLE IF EXISTS t1;
CREATE TABLE t1(id INT AUTO_INCREMENT PRIMARY KEY,
msg_prefix VARCHAR(255),
msg longtext,
msg_length int,
msg_checksum varchar(128),
KEY msg_i(msg_prefix))
ENGINE=INNODB;
call mtr.add_suppression(".*");
Checksum master
stop slave;
start slave;
Checksum slave
DROP TABLE t1;
stop slave;
start slave;
include/stop_slave.inc
include/start_slave.inc
include/stop_slave.inc
@@ -0,0 +1,32 @@
include/master-slave.inc
Warnings:
Note #### Sending passwords in plain text without SSL/TLS is extremely insecure.
Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
[connection master]
DROP TABLE IF EXISTS t1;
CREATE TABLE t1(id INT AUTO_INCREMENT PRIMARY KEY,
msg_prefix VARCHAR(255),
msg VARCHAR(255),
msg_length int,
msg_checksum varchar(128),
KEY msg_i(msg_prefix))
ENGINE=INNODB
ROW_FORMAT=COMPRESSED
KEY_BLOCK_SIZE=1;
call mtr.add_suppression(".*");
Wait for reconnect
Checksum master
stop slave;
start slave;
Checksum slave
Wait for reconnect
Checksum master
stop slave;
start slave;
Checksum slave
DROP TABLE t1;
stop slave;
start slave;
include/stop_slave.inc
include/start_slave.inc
include/stop_slave.inc
@@ -0,0 +1,32 @@
include/master-slave.inc
Warnings:
Note #### Sending passwords in plain text without SSL/TLS is extremely insecure.
Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
[connection master]
DROP TABLE IF EXISTS t1;
CREATE TABLE t1(id INT AUTO_INCREMENT PRIMARY KEY,
msg_prefix VARCHAR(255),
msg longtext,
msg_length int,
msg_checksum varchar(128),
KEY msg_i(msg_prefix))
ENGINE=INNODB
ROW_FORMAT=COMPRESSED
KEY_BLOCK_SIZE=1;
call mtr.add_suppression(".*");
Wait for reconnect
Checksum master
stop slave;
start slave;
Checksum slave
Wait for reconnect
Checksum master
stop slave;
start slave;
Checksum slave
DROP TABLE t1;
stop slave;
start slave;
include/stop_slave.inc
include/start_slave.inc
include/stop_slave.inc
@@ -0,0 +1,30 @@
include/master-slave.inc
Warnings:
Note #### Sending passwords in plain text without SSL/TLS is extremely insecure.
Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
[connection master]
DROP TABLE IF EXISTS t1;
CREATE TABLE t1(id INT AUTO_INCREMENT PRIMARY KEY,
msg_prefix VARCHAR(255),
msg longtext,
msg_length int,
msg_checksum varchar(128),
KEY msg_i(msg_prefix))
ENGINE=INNODB;
call mtr.add_suppression(".*");
Wait for reconnect
Checksum master
stop slave;
start slave;
Checksum slave
Wait for reconnect
Checksum master
stop slave;
start slave;
Checksum slave
DROP TABLE t1;
stop slave;
start slave;
include/stop_slave.inc
include/start_slave.inc
include/stop_slave.inc
Oops, something went wrong.

0 comments on commit 8b6adf6

Please sign in to comment.