Bad socket read/write on client __ Unknown Error

[Izerous]

I have two mosquitto brokers both are running on Ubuntu 20.04. A test environment running Mosquitto 2.0.10 and a live environment running 2.0.13. And for clarification test has a fraction of the traffic of the live server even the the live server floats at like 7% CPU usage most of the time give or take a little bit.

The current issue is only occurring on the live server not the test server.

I have had a few issues that I have resolved 1 by 1 as they came up. Both are using go auth plugin to talk to separate auth servers (also test & live versions) and some of the calls were far from optimized occasionally causing timeouts. Those calls are now super fast and no longer appear to be relevant to the now current issue.

The live install sees a lot more traffic but considering the low CPU usage i'm not sure if it is relevant since it is no where even near 10% let alone 50% on a dual core machine.

About once every 7-14 days the server seems to reject all traffic and go into a state of full unresponsiveness for about a minute or two causing a ripple through everything connected to it. This affects all client types that are connected C#, JS, Java & Swift based clients.

The Logs look like the following and will last nearly a minute
(Ip addresses and stuff removed)

1656424491: Bad socket read/write on client [snip]: Unknown error.
1656424491: New connection from [snip] on port [snip].
1656424491: Bad socket read/write on client [snip]: Unknown error.
1656424491: New connection from [snip] on port [snip].
1656424491: Bad socket read/write on client [snip]: Unknown error.

1656424491: New connection from [snip] on port [snip].
1656424491: Bad socket read/write on client [snip]: Unknown error.
1656424491: New connection from [snip] on port [snip].
1656424491: Bad socket read/write on client [snip]: Unknown error.
1656424491: New connection from [snip] on port [snip].
1656424491: Bad socket read/write on client [snip]: Unknown error.
1656424491: New connection from [snip] on port [snip].
1656424491: Bad socket read/write on client [snip]: Unknown error.
1656424491: New connection from [snip] on port [snip].
1656424491: Bad socket read/write on client [snip]: Unknown error.
1656424492: New connection from [snip] on port [snip].
1656424492: Bad socket read/write on client [snip]: Unknown error.
1656424492: New client connected from [snip] as [snip]

1656424492: New connection from [snip] on port [snip].
1656424492: Bad socket read/write on client [snip]: Unknown error.
1656424492: New connection from [snip] on port [snip].
1656424492: Bad socket read/write on client [snip]: Unknown error.
1656424492: New connection from [snip] on port [snip].
1656424492: New connection from [snip] on port [snip].
1656424492: Client [snip] disconnected, not authorised.
1656424492: Bad socket read/write on client [snip]: Unknown error.
1656424492: New connection from [snip] on port [snip].
1656424492: Bad socket read/write on client [snip]: Unknown error.
1656424492: New client connected from [snip] as [snip]
1656424492: New connection from [snip] on port [snip].
1656424492: Bad socket read/write on client [snip]: Unknown error.
1656424492: New connection from [snip] on port [snip].
1656424492: Bad socket read/write on client [snip]: Unknown error.
1656424492: New connection from [snip] on port [snip].
1656424492: Bad socket read/write on client [snip]: Unknown error.
1656424492: New connection from [snip] on port [snip].
1656424492: Bad socket read/write on client [snip]: Unknown error.
1656424492: New connection from [snip] on port [snip].
1656424492: Bad socket read/write on client [snip]: Unknown error.
1656424492: New connection from [snip] on port [snip].
1656424492: Bad socket read/write on client [snip]: Unknown error.
1656424492: New connection from [snip] on port [snip].
1656424492: Bad socket read/write on client [snip]: Unknown error.
1656424493: New connection from [snip] on port [snip].
1656424493: Bad socket read/write on client [snip]: Unknown error.
1656424493: New connection from [snip] on port [snip].
1656424493: Bad socket read/write on client [snip]: Unknown error.
1656424493: New connection from [snip] on port [snip].
1656424493: Bad socket read/write on client [snip]: Unknown error.
1656424493: New connection from [snip] on port [snip].

Because these are all thrown as Unknown errors i'm not even sure where to proceed investigating right now.

$SYS/broker/uptime is reporting an uptime over a month (bounced it to change logging) so it isn't like mosquitto is crashing and restarting or anything like that.

[NorbertHeusser]

This "Unknown error" messages will be written to the log file, if a new client connections was created to the broker, but the broken was not able to receive or send the first network packet to this client. In the MQTT protocol this may happen in with the CONN or CONN_ACK. So the connection to the client will be closed as the communication could not be established.

As you report that different clients are effected at the same point in time I would assume this is not simply a misbehaving client, but instead a more general problem with the network stack. Perhaps you find some information in the /var/log/syslog about the network being unresponsive or down for a minute (e.g. renewing IP addresses, restarting due to some updates).

[AryanRezHer]

Did you solve it? Now I have this problem and I don´t know what I need to do for solve it

[inqode-lars]

i have the same error message since updating from 2.0.15 to 2.0.18.

700045316: New client connected from 127.0.0.1:35848 as auto-A98586C7-F88A-B33D-01A8-7D8F9477AE1D (p5, c1, k60, u'client-1234').                                                                          
1700045316: Bad socket read/write on client auto-A98586C7-F88A-B33D-01A8-7D8F9477AE1D: Unknown error. 

It can be reproduced by publishing to a topic that is restricted by ACL.

# for anonymous
pattern readwrite client/%u/#

user admin
topic readwrite #

Producing the error:

mosquitto_pub -h localhost -p 1883 -u client-1234 -t test/test/x -m "Hello"

Working:

mosquitto_pub -h localhost -p 1883 -u client-1234 -t client/client-1234/x -m "Hello"

[shurkanTwo]

We are facing the same issue.
Has there been any progress regarding a fix?

[corpix]

I think this is an issue with mosquitto logging being too general. I often debug this using tcpdump to dump communication and then check my configuration files.
Last time I do this is when setting up bridge, I noticed client published message to $SYS/broker/connection/%c/state topic, but it has not corresponding ACL defined, so broker just sent TCP RST.

[ftapajos]

related: #3088