When the mysqlConf is updated and the master node becomes the slave node, the node fails. #692
Description
The two clusters present different problems. The new master node is OK.
# kubectl get mysql -nkme-mysql-10272
NAME READY REPLICAS AGE
lzx1 False 2 145m
lzx2 False 2 145m
# kubectl get pod -nkme-mysql-10272
NAME READY STATUS RESTARTS AGE
lzx1-mysql-0 4/4 Running 0 52m
lzx1-mysql-1 4/4 Running 0 83m
lzx2-mysql-0 4/4 Running 0 52m
lzx2-mysql-1 4/4 Running 0 83m
lzx1 lzx1-mysql-0 log:
2021-06-04T10:11:31.028529Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20-11) starting as process 1
2021-06-04T10:11:31.041698Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-06-04T10:11:31.771601Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-06-04T10:11:32.125889Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/lib/mysql/mysqlx.sock' bind-address: '::' port: 33060
2021-06-04T10:11:32.250403Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-06-04T10:11:32.354015Z 0 [Warning] [MY-010604] [Repl] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=lzx1-mysql-0-relay-bin' to avoid this problem.
2021-06-04T10:11:32.357097Z 0 [Warning] [MY-010539] [Repl] Recovery from master pos 175196 and file mysql-bin.000001 for channel ''. Previous relay log pos and relay log file had been set to 61754, ./lzx1-mysql-0-relay-bin.000002 respectively.
2021-06-04T10:11:32.392419Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.20-11' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona Server (GPL), Release 11, Revision 5b5a5d2.
2021-06-04T10:11:59.189718Z 33 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL '' executed'. Previous state master_host='lzx1-mysql-1.mysql.kme-mysql-10272', master_port= 3306, master_log_file='mysql-bin.000001', master_log_pos= 175196, master_bind=''. New state master_host='lzx1-mysql-1.mysql.kme-mysql-10272', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''.
2021-06-04T10:11:59.282748Z 35 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2021-06-04T10:11:59.286711Z 35 [System] [MY-010562] [Repl] Slave I/O thread for channel '': connected to master 'sys_replication@lzx1-mysql-1.mysql.kme-mysql-10272:3306',replication started in log 'FIRST' at position 4
2021-06-04T10:11:59.289104Z 35 [ERROR] [MY-010557] [Repl] Error reading packet from server for channel '': Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been (server_errno=1236)
2021-06-04T10:11:59.289138Z 35 [ERROR] [MY-013114] [Repl] Slave I/O for channel '': Got fatal error 1236 from master when reading data from binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been', Error_code: MY-013114
2021-06-04T10:12:50.966490Z 77 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2021-06-04T10:12:50.968930Z 77 [System] [MY-010562] [Repl] Slave I/O thread for channel '': connected to master 'sys_replication@lzx1-mysql-1.mysql.kme-mysql-10272:3306',replication started in log 'FIRST' at position 4
2021-06-04T10:12:50.971123Z 77 [ERROR] [MY-010557] [Repl] Error reading packet from server for channel '': Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been (server_errno=1236)
2021-06-04T10:12:50.971166Z 77 [ERROR] [MY-013114] [Repl] Slave I/O for channel '': Got fatal error 1236 from master when reading data from binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been', Error_code: MY-013114
lzx2 log:
2021-06-04T11:05:29.381914Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20-11) starting as process 1
2021-06-04T11:05:29.393882Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-06-04T11:05:30.008040Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-06-04T11:05:30.274956Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/lib/mysql/mysqlx.sock' bind-address: '::' port: 33060
2021-06-04T11:05:30.393645Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-06-04T11:05:30.474279Z 0 [Warning] [MY-010604] [Repl] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=lzx2-mysql-0-relay-bin' to avoid this problem.
2021-06-04T11:05:30.479093Z 0 [Warning] [MY-010539] [Repl] Recovery from master pos 1082920 and file mysql-bin.000001 for channel ''. Previous relay log pos and relay log file had been set to 1070, ./lzx2-mysql-0-relay-bin.000002 respectively.
2021-06-04T11:05:30.535846Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.20-11' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona Server (GPL), Release 11, Revision 5b5a5d2.
2021-06-04T11:05:36.914128Z 22 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL '' executed'. Previous state master_host='lzx2-mysql-1.mysql.kme-mysql-10272', master_port= 3306, master_log_file='mysql-bin.000001', master_log_pos= 1082920, master_bind=''. New state master_host='lzx2-mysql-1.mysql.kme-mysql-10272', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''.
2021-06-04T11:05:59.217593Z 44 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL '' executed'. Previous state master_host='lzx2-mysql-1.mysql.kme-mysql-10272', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='lzx2-mysql-1.mysql.kme-mysql-10272', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''.
2021-06-04T11:05:59.225106Z 46 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2021-06-04T11:05:59.228631Z 46 [System] [MY-010562] [Repl] Slave I/O thread for channel '': connected to master 'sys_replication@lzx2-mysql-1.mysql.kme-mysql-10272:3306',replication started in log 'FIRST' at position 4
2021-06-04T11:05:59.238725Z 47 [ERROR] [MY-010584] [Repl] Slave SQL for channel '': Could not execute Update_rows event on table sys_operator.heartbeat; Can't find record in 'heartbeat', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000001, end_log_pos 1168, Error_code: MY-001032
2021-06-04T11:05:59.238754Z 47 [Warning] [MY-010584] [Repl] Slave: Can't find record in 'heartbeat' Error_code: MY-001032
2021-06-04T11:05:59.238768Z 47 [ERROR] [MY-010586] [Repl] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.000001' position 808
2021-06-04T11:18:50.117442Z 694 [ERROR] [MY-010584] [Repl] Slave SQL for channel '': Could not execute Update_rows event on table sys_operator.heartbeat; Can't find record in 'heartbeat', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000001, end_log_pos 1168, Error_code: MY-001032
2021-06-04T11:18:50.117468Z 694 [Warning] [MY-010584] [Repl] Slave: Can't find record in 'heartbeat' Error_code: MY-001032
2021-06-04T11:18:50.117477Z 694 [ERROR] [MY-010586] [Repl] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.000001' position 808