自动主从切换部署不能选出master

[weihubeats]

• rocketmq版本: 5.1.0
• nameserve: 2个节点
• broker: 2个节点
  

重启broker、nameserve依旧报这个错

执行

sh bin/mqadmin getBrokerEpoch -n localhost:9876 -b broker-a

没反应

错误log

2023-03-15 16:59:04 INFO ReplicasManager_ScheduledService_1 - Update controller leader address to x.x.x.1:9878
2023-03-15 16:59:06 WARN ReplicasManager_ExecutorService_1 - Failed to start replicasManager, retry times:35, current state:FIRST_TIME_SYNC_CONTROLLER_METADATA_DONE, try it again
2023-03-15 16:59:06 ERROR ReplicasManager_ExecutorService_1 - Failed to register broker to controller
org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2012  DESC: The broker has not master, and this new registered broker can't not be elected as master
For more information, please visit the url, https://rocketmq.apache.org/docs/bestPractice/06FAQ
        at org.apache.rocketmq.broker.out.BrokerOuterAPI.registerBrokerToController(BrokerOuterAPI.java:1183)
        at org.apache.rocketmq.broker.controller.ReplicasManager.registerBrokerToController(ReplicasManager.java:302)
        at org.apache.rocketmq.broker.controller.ReplicasManager.startBasicService(ReplicasManager.java:151)
        at org.apache.rocketmq.broker.controller.ReplicasManager.lambda$start$0(ReplicasManager.java:133)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2023-03-15 16:59:11 WARN ReplicasManager_ExecutorService_1 - Failed to start replicasManager, retry times:36, current state:FIRST_TIME_SYNC_CONTROLLER_METADATA_DONE, try it again
2023-03-15 16:59:11 ERROR ReplicasManager_ExecutorService_1 - Failed to register broker to controller
org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2012  DESC: The broker has not master, and this new registered broker can't not be elected as master
For more information, please visit the url, https://rocketmq.apache.org/docs/bestPractice/06FAQ
        at org.apache.rocketmq.broker.out.BrokerOuterAPI.registerBrokerToController(BrokerOuterAPI.java:1183)
        at org.apache.rocketmq.broker.controller.ReplicasManager.registerBrokerToController(ReplicasManager.java:302)
        at org.apache.rocketmq.broker.controller.ReplicasManager.startBasicService(ReplicasManager.java:151)
        at org.apache.rocketmq.broker.controller.ReplicasManager.lambda$start$0(ReplicasManager.java:133)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2023-03-15 16:59:14 INFO ReplicasManager_ScheduledService_2 - Update controller leader address to x.x.x.1:9878

[weihubeats]

添加参数 enableElectUncleanMaster = true,可以选出新的master节点。但是不明白为什么就这两个节点什么都没干回出现两个节点选不出master的情况

[TheR1sing3un]

是否在重启的过程中发生ip的变化(用wsl或者容器部署会发生变化，可参考#5745) ，因为我看这个副本组中syncStateSet中只有一个broker

如果这个broker重启的时候ip变化了，重启前后这个broker就没办法对应上，所以在未开启enableElectUncleanMaster 的时候，就没办法在注册的时候从当前syncStateSet中选出master了, 这个目前develop分支已修复该问题，可以参考：#5989

[weihubeats]

没有使用wsl和容器部署，所以机器ip应该是不会变的

[weihubeats]

应该是我本地又起来一个broker，然后本地下线，服务器上的两个broker由于落后于本地broker，所以他们都当选不了主

[RongtongJin]

@weihubeats 从你图中可以看到，当前SyncStateSet中只有一个同步副本（InSyncReplica为同步副本，NotInSyncReplica为非同步副本），因此该情况下为了保证不丢消息，只有等InSyncReplica的副本上线，才会被选为Master（除非unclean选举）。就像@TheR1sing3un所说，是否由于ip变化导致上线的两个broker均非InSyncReplica，导致无法选主，如果是的话，可以参考@TheR1sing3un解决方法（已经合入develop分支），该方案已经从Controller识别brokerIP作为broker身份改为持久化的broker编号。

此外，这种情况下，可以如下操作恢复：
（1）下线主备Broker
（2）利用cleanBrokerData命令清除Controller中该组Broker元数据
（3）删除主备Broker下的文件～/store/epochFileCheckpoint和epochFileCheckpoint.bak
（4）Broker重新上线（尽量保证旧的主备关系，先主后备上线）

[weihubeats]

好的，非常感觉，我好像明白了，因为为了查看原因又在本地启了一个broker，导致本地的broker的数据文件是最新的，所以服务器上的两个broker都当不了master

[zuoHx580]

Comment options
cleanBrokerData 这个命令是哪个工具的，可以发个完整的吗

[RongtongJin]

Comment options
cleanBrokerData 这个命令是哪个工具的，可以发个完整的吗

mqadmin工具的，最新develop分支命名为cleanBrokerMetadata

[zuoHx580]

Comment options
cleanBrokerData 这个命令是哪个工具的，可以发个完整的吗

mqadmin工具的，最新develop分支命名为cleanBrokerMetadata

好的，非常感谢