In HA mode, the master is down and cannot be consumed from the slave

[MrLYG]

ISSUE REPORT

1. Please describe the issue you observed:

• What did you do (The steps to reproduce)?
  step1. K8S使用operator部署HA模式集群1master&1slave
  step1. K8S uses operator to deploy HA mode cluster 1master&1slave
  
  step2. 往集群名为slave-test的topic写入一些数据
  step2. Write some data to the topic of the cluster named slave-test
  
  
  step3.模拟master宕机，关闭master节点
  step3. Simulate the master downtime and shut down the master
  
  step4. 启动消费者服务，消费数据该topic数据
  step4. Start the consumer service and consume data for the topic data
  

• What is expected to see?
  在master宕机的情况下，slave应当承担数据消费职责，不会影响rocketmq集群内topic的消息消费任务。
  In the case of master downtime, the slave should assume the responsibility of data consumption and will not affect the message consumption task of topics in the rocketmq cluster.

• What did you see instead?
  消费者消费消息失败；
  Consumer failed to consume message

nameserverF =
topicF = SELF_TEST_TOPIC
consumerGroupF = slave-test
producerGroupF = slave-test
time="2022-08-18T15:37:54+08:00" level=info msg="the consumer start beginning" consumerGroup=slave-test messageModel=Clustering unitMode=false
time="2022-08-18T15:37:54+08:00" level=info msg="the topic route info changed" changeTo="{\"OrderTopicConf\":\"\",\"queueDatas\":[{\"brokerName\":\"rocketmq-broker-0\",\"readQueueNums\":4,\"writeQueueNums\":4,\"perm\":6,\"topicSy
nFlag\":0}],\"brokerDatas\":[{\"cluster\":\"rocketmq-broker\",\"brokerName\":\"rocketmq-broker-0\",\"brokerAddrs\":{\"1\":\"10.53.70.232:23333\"}}]}" changedFrom="<nil>" topic=slave-test
time="2022-08-18T15:37:54+08:00" level=warning msg="query topic route from server error" underlayError="topic not exist"
time="2022-08-18T15:37:54+08:00" level=warning msg="queryTopicRouteInfoFromServer return nil" topic="%RETRY%slave-test" 
time="2022-08-18T15:37:54+08:00" level=warning msg="query topic route from server error" underlayError="topic not exist"
time="2022-08-18T15:37:54+08:00" level=warning msg="queryTopicRouteInfoFromServer return nil" topic="%RETRY%slave-test"
time="2022-08-18T15:37:55+08:00" level=info msg="receive broker's notification to consumer group" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="do balance in group failed, the topic does not exist" consumerGroup=slave-test topic="%RETRY%slave-test"
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="do balance in group failed, the topic does not exist" consumerGroup=slave-test topic="%RETRY%slave-test"
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=0]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=1]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=2]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=warning msg="delete mq from offset table" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" consumerGroup=slave-test
time="2022-08-18T15:37:55+08:00" level=error msg="fecth offset of mq from broker error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" consumerGroup=slave-test underlayError="broker: rocke
tmq-broker-0 address not found"
time="2022-08-18T15:37:55+08:00" level=warning msg="query max offset error" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" underlayError="the broker [rocketmq-broker-0] does not exist"
time="2022-08-18T15:37:55+08:00" level=warning msg="do defaultConsumer, add a new mq failed" MessageQueue="MessageQueue [topic=slave-test, brokerName=rocketmq-broker-0, queueId=3]" consumerGroup=slave-test

[RongtongJin]

Hi, @MrLYG two conditions need to be met for consumption from slave broker: 1. The configuration slavReadEnable must be true 2. It cannot be the consumption of order messages.

[MrLYG]

Hi, @MrLYG two conditions need to be met for consumption from slave broker: 1. The configuration slavReadEnable must be true 2. It cannot be the consumption of order messages.

@RongtongJin Thanks a ton for your fastly reply.
I had met the two conditions that you gave. But the consumer still not consume from slave when master done.

1. The configuration slavReadEnable is true
   
2. I didn't open the consumption of order messages.
   

[RongtongJin]

Hi, @MrLYG two conditions need to be met for consumption from slave broker: 1. The configuration slavReadEnable must be true 2. It cannot be the consumption of order messages.

@RongtongJin Thanks a ton for your fastly reply. I had met the two conditions that you gave. But the consumer still not consume from slave when master done.

1. The configuration slavReadEnable is true
   
2. I didn't open the consumption of order messages.
   

@MrLYG Could you try whether the Java client has the same issue?

[MrLYG]

Hi, @MrLYG two conditions need to be met for consumption from slave broker: 1. The configuration slavReadEnable must be true 2. It cannot be the consumption of order messages.

@RongtongJin Thanks a ton for your fastly reply. I had met the two conditions that you gave. But the consumer still not consume from slave when master done.

1. The configuration slavReadEnable is true
   
2. I didn't open the consumption of order messages.
   

@MrLYG Could you try whether the Java client has the same issue?

The Java client can consume messages when the master has done.
That's this means the go client has some bug or problem?
If I need to use the go client to connect the rocektmq service, what should I do to avoid that problem? Could you give me some suggestions?

[RongtongJin]

That's this means the go client has some bug or problem?
If I need to use the go client to connect the rocektmq service, what should I do to avoid that problem? Could you give me some suggestions?

IIRC, the latest version of rocketmq-client-go has fixed this issue. @guyinyou @ShannonDing Could you help to take a look？

[MrLYG]

That's this means the go client has some bug or problem?
If I need to use the go client to connect the rocektmq service, what should I do to avoid that problem? Could you give me some suggestions?

IIRC, the latest version of rocketmq-client-go has fixed this issue. @guyinyou @ShannonDing Could you help to take a look？

I used the latest version of rocketmq-client-go to consume messages from slave but still fail. The configuration slave ReadEnable is true, and I didn't open order messages.

There is my go.mod file in which I require the latest version of rocketmq-client-go 2.1.1

module go-rocket

go 1.17

require github.com/apache/rocketmq-client-go/v2 v2.1.1

require (
	github.com/emirpasic/gods v1.12.0 // indirect
	github.com/golang/mock v1.3.1 // indirect
	github.com/json-iterator/go v1.1.12 // indirect
	github.com/konsorten/go-windows-terminal-sequences v1.0.1 // indirect
	github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421 // indirect
	github.com/modern-go/reflect2 v1.0.2 // indirect
	github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
	github.com/pkg/errors v0.8.1 // indirect
	github.com/satori/go.uuid v1.2.0 // indirect
	github.com/sirupsen/logrus v1.4.0 // indirect
	github.com/stathat/consistent v1.0.0 // indirect
	github.com/tidwall/gjson v1.13.0 // indirect
	github.com/tidwall/match v1.1.1 // indirect
	github.com/tidwall/pretty v1.2.0 // indirect
	go.uber.org/atomic v1.5.1 // indirect
	golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2 // indirect
	golang.org/x/lint v0.0.0-20190930215403-16217165b5de // indirect
	golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f // indirect
	golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c // indirect
	gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
)

[github-actions]

This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs.

[github-actions]

This issue was closed because it has been inactive for 3 days since being marked as stale.