Galera nodes permanently dropped from runtime_mysql_servers for writer_is_also_reader=2

[JavierJF]

• A clear description of the issue

If a node is placed just in the reader_hostgroup in a cluster configured with writer_is_also_reader=2, the node will be permanently dropped from the 'runtime_mysql_servers' table.

• ProxySQL version

v2.0.16
v2.1.0

• OS version

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling

• The steps to reproduce the issue

Create a Galera cluster with the following config:

mysql> select * from runtime_mysql_galera_hostgroups;
+------------------+-------------------------+------------------+-------------------+--------+-------------+-----------------------+-------------------------+---------+
| writer_hostgroup | backup_writer_hostgroup | reader_hostgroup | offline_hostgroup | active | max_writers | writer_is_also_reader | max_transactions_behind | comment |
+------------------+-------------------------+------------------+-------------------+--------+-------------+-----------------------+-------------------------+---------+
| 20               | 30                      | 10               | 40                | 1      | 1           | 2                     | 100                     | NULL    |
+------------------+-------------------------+------------------+-------------------+--------+-------------+-----------------------+-------------------------+---------+
1 row in set (0.00 sec)

Initial runtime_mysql_servers looks like this:

mysql> select * from runtime_mysql_servers;
+--------------+-------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname    | port | gtid_port | status  | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10           | 172.18.1.12 | 3306 | 0         | ONLINE  | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 30           | 172.18.1.12 | 3306 | 0         | ONLINE  | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 30           | 172.18.1.11 | 3306 | 0         | ONLINE  | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 20           | 172.18.1.13 | 3306 | 0         | ONLINE  | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 20           | 172.18.1.12 | 3306 | 0         | SHUNNED | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 20           | 172.18.1.11 | 3306 | 0         | SHUNNED | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 10           | 172.18.1.11 | 3306 | 0         | ONLINE  | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
+--------------+-------------+------+-----------+---------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
7 rows in set (0.00 sec)

Force a server to be placed only in the reader_hostgroup, with: SET GLOBAL read_only=ON;. The server has now completely disappear from the runtime_mysql_servers table, leaving it like this:

mysql: [Warning] Using a password on the command line interface can be insecure.
+-----------+-------------+----------+---------+---------+
| hostgroup | srv_host    | srv_port | status  | Queries |
+-----------+-------------+----------+---------+---------+
| 20        | 172.18.1.11 | 3306     | SHUNNED | 0       |
| 20        | 172.18.1.13 | 3306     | ONLINE  | 0       |
| 30        | 172.18.1.11 | 3306     | ONLINE  | 0       |
| 10        | 172.18.1.11 | 3306     | ONLINE  | 0       |
+-----------+-------------+----------+---------+---------+

Now, when trying to recover the sever, reverting the read_only config, the server never gets back, after setting: SET GLOBAL read_only=OFF;, the runtime_mysql_servers table keeps being:

mysql: [Warning] Using a password on the command line interface can be insecure.
+-----------+-------------+----------+---------+---------+
| hostgroup | srv_host    | srv_port | status  | Queries |
+-----------+-------------+----------+---------+---------+
| 20        | 172.18.1.11 | 3306     | SHUNNED | 0       |
| 20        | 172.18.1.13 | 3306     | ONLINE  | 0       |
| 30        | 172.18.1.11 | 3306     | ONLINE  | 0       |
| 10        | 172.18.1.11 | 3306     | ONLINE  | 0       |
+-----------+-------------+----------+---------+---------+

• The full ProxySQL error log (default location: /var/lib/proxysql/proxysql.log)

The attached tar contains:

• The full ProxySQL error log.
• The logs from the individual mysql containers during the testing.
• The cluster configuration.
• The final cluster status after resetting the read only flag.
• Sysbench output log running during the test.

reader_gone_logs.tar.gz