"Quota exceeded" errors when no limits are configured

[NiJoao]

We have run into an issue where many clients were being disconnected with
Bad socket read/write on client ...... : Quota exceeded after a mosquitto broker upgrade to version 2.0.21.

Big underused server, with just 300 clients and ~1k messages per second.
Configured with "unlimited" settings almost everywhere:

max_queued_messages 0
max_queued_bytes 0
queue_qos0_messages true
memory_limit 0

We found the problem to be a bug in the code dealing with the config max_queued_messages 0, not handling the 0 case as per documentation Set to 0 for no maximum (not recommended).
Tests for this case are also missing.

Solved temporarily by setting max_queued_messages 65000

Buggy code:

mosquitto/src/handle_publish.c

Lines 386 to 388 in a61bfe1

	if(context->out_packet_count >= db.config->max_queued_messages){
	rc = MQTT_RC_QUOTA_EXCEEDED;
	}

Should be:

if(db.config->max_queued_messages > 0 && context->out_packet_count >= db.config->max_queued_messages){
rc = MQTT_RC_QUOTA_EXCEEDED;
}

[ralight]

Thank you, I've committed the fix for 2.0.22