Why are messages lost?

[liwashington]

I use nanomq as a message queue service with default configuration, I publish 100000 message after sleep(10) with qos=0, I found "msg lost due to reach the limit of lmq" in logs, the message are lost! So I try to publish a message with qos=1, but found " ACK failed! qos msg XXX not found!" in logs, and at last , "msg lost due to reach the limit of lmq" in logs . Can I use nanomq without lost message?

[JaylinYu]

Nano drops msg if the in-flight queue is full. In your case, Most likely is the subscribe client not able to handle same msg volume/velocity as the Publishing rate. It has nothing to do with QoS, and ack will failed if Other side(your sUB client) is not able to send ACK in time, So that previous packet id is overwrite by new coming msg

In this case, suggest you extend “max_mqueue_len”, by default it is 2048, you can have 65535 if you don't care memory usage.

[liwashington]

Yes, thanks. I produce messages much faster than I consume, I tried to change max_mqueue_len from 2048 to 204800，it works. But it cannot fundamentally solve the problem, memory is limited. if the in-flight queue is full,Can I publish messages through blocking? Now, client works normally, deliveryComplete method is callback, message is dropped at server! the client is unaware that the message has been dropped on the server side. If the server is about to lose messages, I can have the client wait when publishing messages.

[JaylinYu]

According to MQTT design. Pub client has no way knowing which client subscribe the topic to receive msg in what rate. All things going asynchronously. but you can limit the the coming rate of Pub client from Broker side. This is a feature of MQTT5.

[JaylinYu]

And Even if you set the rate limit. The msg coming from to is still being dropped when exceeding The limitation. There is no way to inform client an MQTT event from TCP/Ip perspective. I guess QoS 1/2 is an option though. Client could slow itself down when Ack is not received.

[liwashington]

Thanks a lot! We cannot allow any data loss, we have to take a look at other message queues.

[JaylinYu]

In your case, AMQP based message queue is a better solution, but it requires robust network