Server shunned due to replication lag re-enabled due to replication check timing out

[JavierJF]

• A clear description of the issue

If a server is shunned due to replication lag, this is: SHUNNED_REPLICATION_LAG. It can get re-enabled even due to replication-lag checks timing out.

• ProxySQL version

• v2.1.1

• OS version

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling

• The steps to reproduce the issue

For reproducing the issue it's enough configuring MySQL servers to a slow max_replication_lag:

mysql> select hostgroup_id, hostname, port, gtid_port, status, max_replication_lag from mysql_servers;
+--------------+-----------+-------+-----------+--------+---------------------+
| hostgroup_id | hostname  | port  | gtid_port | status | max_replication_lag |
+--------------+-----------+-------+-----------+--------+---------------------+
| 0            | 127.0.0.1 | 13306 | 0         | ONLINE | 3                   |
| 1            | 127.0.0.1 | 13306 | 0         | ONLINE | 3                   |
| 1            | 127.0.0.1 | 13307 | 0         | ONLINE | 3                   |
| 1            | 127.0.0.1 | 13308 | 0         | ONLINE | 3                   |
+--------------+-----------+-------+-----------+--------+---------------------+

Later enforce the MySQL server to report a higher replication lag than the configure max_replication_lag configured for ProxySQL,
server status should change to:

+-----------+-----------+----------+-------------------------+---------+
| hostgroup | srv_host  | srv_port | status                  | Queries |
+-----------+-----------+----------+-------------------------+---------+
| 0         | 127.0.0.1 | 13306    | ONLINE                  | 0       |
| 1         | 127.0.0.1 | 13306    | ONLINE                  | 0       |
| 1         | 127.0.0.1 | 13307    | SHUNNED_REPLICATION_LAG | 0       |
| 1         | 127.0.0.1 | 13308    | ONLINE                  | 0       |
+-----------+-----------+----------+-------------------------+---------+

While this is status is taking place, enforce a delay in the shunned replica, that is higher than the configured: mysql-monitor_replication_lag_timeout. The following events can be appreciated in ProxySQL log:

2021-03-12 16:30:27 ProxySQL_Admin.cpp:2330:admin_handler_command_load_or_save(): [INFO] Received load mysql variables to runtime command
2021-03-12 16:30:28 MySQL_HostGroups_Manager.cpp:3331:replication_lag_action(): [WARNING] Shunning server 127.0.0.1:13307 from HG 1 with replication lag of 10 second, count number: '1'
2021-03-12 16:30:32 ProxySQL_Admin.cpp:1448:admin_handler_command_set(): [INFO] Received command set mysql-monitor_replication_lag_timeout=200
2021-03-12 16:30:33 ProxySQL_Admin.cpp:2330:admin_handler_command_load_or_save(): [INFO] Received load mysql variables to runtime command
2021-03-12 16:30:34 MySQL_HostGroups_Manager.cpp:3355:replication_lag_action(): [WARNING] Re-enabling server 127.0.0.1:13307 from HG 1 with replication lag of -2 second

The previous shunned server should now be back online due to the timeout of the mysql-monitor_replication_lag_timeout.

• The full ProxySQL error log (default location: /var/lib/proxysql/proxysql.log)

proxysql_replication_lag_timeout.log