Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
This section focus on Monitor v1.2.1, as it introduces multiple improvements compared to v1.2.0
Variables removed as unused or deprecated:
Variables currently not in use:
The Monitor Module is responsible for a series of check against the backends. It currently supports 4 types of checks:
connect : it connects to all the backends, and success/failure is logged in table
ping : it pings to all the backends, and success/failure is logged in table
mysql_server_ping_log. In case of
mysql-monitor_ping_max_failuresmissed heartbeat, sends a signal to MySQL_Hostgroups_Manager to kill all connections;
replication lag : it checks
Seconds_Behind_Masterto all backends configured with
max_replication_laggreater than 0, and check is logged in table
max_replication_lagthe server is shunned until
read only : it checks
read_onlyfor all hosts in the hostgroups in table
mysql_replication_hostgroups, and check is logged in table
read_only=1the host is copied/moved to the
reader_hostgroup, while if
read_only=0the host is copied/moved to the
Specifies the username that the Monitor module will use to connect to the backend. The user needs only
USAGEprivileges to connect, ping and check
read_only. The user needs also
REPLICATION CLIENTif it needs to monitor replication lag.
Password for user mysql-monitor_username
It enables or disables MySQL Monitor. Since MySQL Monitor can interfere with changed applied directly on the Admin interface, this variable allows to temporary disable it.
How frequently a connect check is performed, in milliseconds.
Connection timeout in milliseconds. The current implementation rounds this value to an integer number of seconds less or equal to the original interval, with 1 second as minimum. This lazy rounding is done because SSL connections are blocking calls.
How frequently a ping check is performed, in milliseconds.
Ping timeout in milliseconds.
If a host misses mysql-monitor_ping_max_failures pings in a row, MySQL_Monitor informs MySQL_Hostgroup_Manager that the node is unreachable and that should immediately kill all connections. It is important to note that in case a connection to the backend is not available, MySQL_Monitor will first try to connect in order to ping, therefore the time to detect a node down could be one of the two:
- mysql-monitor_ping_max_failures * mysql-monitor_connect_timeout
- mysql-monitor_ping_max_failures * mysql-monitor_ping_timeout
Read only variables:
How frequently a read only check is performed, in milliseconds.
Read only check timeout in milliseconds.
When a node change its
read_onlyvalue from 1 to 0, this variable determines if the node should be present in both hostgroups or not:
false : node will be moved in
writer_hostgroupand removed from
true : node will be copied in
writer_hostgroupand stay also in
- false : node will be moved in
Replication lag variables:
How frequently a replication lag check is performed, in milliseconds.
Replication lag check timeout in milliseconds.
To prevent that log tables grow without limit, Monitor Module will automatically purge records older than mysql-monitor_history milliseconds. Since ping checks relies on history table to determine if a node is missing heartbeats, the value of mysql-monitor_history is automatically adjusted to the follows if less than it:
- (mysql-monitor_ping_max_failures + 1 ) * mysql-monitor_ping_timeout
The Monitor Module has several internal threads. There are currently 5 main threads:
- Monitor: master thread, responsible to start and coordinate all the others
- monitor_connect_thread: main thread and scheduler for the connect checks
- monitor_ping_thread: main thread and scheduler for the ping checks
- monitor_read_only_thread: main thread and scheduler for the read only checks
- monitor_replication_lag_thread: main thread and scheduler for the replication lag checks Up to version v1.2.0 the above threads but Monitor were also responsible to perform the checks
The implementation in v1.2.0 has a limitation with SSL implementation: with SSL,
connect() is a blocking call, causing the threads to stall while performing the connect phase.
Version v1.2.1 tries to overcome this limitation with a new implementation. Now:
- Monitor initializes a Thread Pool of workers and creates a queue;
- monitor_connect_thread, monitor_ping_thread, monitor_read_only_thread and monitor_replication_lag_thread are producers that generate tasks and sent them to the workers using the queue;
- the workers process the tasks and perform the requires actions;
- if Monitor detects that the queue is growing too fast, it creates new temporary worker threads
Monitor implements its own connection pool. Connections that are alive for more than 3 *
mysql_thread___monitor_ping_interval milliseconds are automatically purged
To prevent that backends terminated connections, Monitor module automatically configures
mysql_thread___monitor_ping_interval * 10