mosquitto dies after SIGHUP on Raspberry Pi

[wollud1969]

I've a quite strange behavior on my RPi running Raspian Stretch since the last update a few days ago: mosquitto dies after SIGHUP (happens every morning due to the logrotate script).

I just cloned the repo and built from source and found the same behavior. My configuration is

pid_file /var/run/mosquitto.pid
log_dest file /var/log/mosquitto/mosquitto.log

persistence true
persistence_location /var/lib/mosquitto/

per_listener_settings true


# listener 1883 0.0.0.0
allow_anonymous true


listener 8883 0.0.0.0
password_file /etc/mosquitto/pwfile
allow_anonymous false
cafile /etc/mosquitto/ssl/ca/server-ca.crt
certfile /etc/mosquitto/ssl/internal-mqtt-broker.crt
keyfile /etc/mosquitto/ssl/internal-mqtt-broker.pem
tls_version tlsv1


# connection to-external
# address 172.16.1.10:8883
# local_password xxx
# local_username argus
# remote_password xxx
# remote_username homegear
# bridge_insecure true
# bridge_tls_version tlsv1.2
# bridge_cafile /etc/mosquitto/ssl/ca/server-ca.crt
# topic IoT/espThermometer2/measurement in

This configuration opens two listeners, the default listener at port 1883 without tls and without passwords and a second listener at port 8883 with tls and with passwords.

I started debugging on the RPi and inserted some printf's and found that mosquitto died in security_default.c, mosquitto_security_apply_default, line 864 (line 870 in my fork at github.com:wollud1969/mosquitto) since context->listener is null.

printf("msad4 %p\n", context);
printf("msad4.1 %p\n", context->listener);
printf("msad4.2 %p\n", &(context->listener->security_options));
printf("msad4.3 %d\n", context->listener->security_options.allow_anonymous);
                        allow_anonymous = context->listener->security_options.allow_anonymous;
printf("msad4b\n");

msad4 0x2aa4e8
msad4.1 (nil)
msad4.2 0x50
Segmentation fault

The listener is not set.

For easier debugging I switched to my laptop (Debian Stretch), prepared an environment with exactly the same configuration (well, nearly the same: paths for persistence, pwfile, certs and keys are different, content is the same).

But on the laptop I got the following output:

msad4 0x55a293f5ee40
msad4.1 0x55a293f4b290
msad4.2 0x55a293f4b318
msad4.3 0
msad4b

Here, the listener is set.

I found a difference in the evaluation of the configuration between RPi and laptop:
When I try to set a listener statement on the RPi to port 1883

listener 1883 0.0.0.0

I got in the logfile:

1528375568: mosquitto version 1.5 starting
1528375568: Config loaded from /etc/mosquitto/mosquitto.conf.
1528375568: Opening ipv4 listen socket on port 1883.
1528375568: Opening ipv4 listen socket on port 8883.
1528375568: Opening ipv4 listen socket on port 1883.
1528375568: Error: Address already in use

So, appearently it trys to open the default listener on port 1883 and an additional listener on port 1883, which fails.

On the laptop this works, only one listener is opened on port 1883. Actually, when I change the first listener statement on the laptop to port 1884 no listener on port 1883 (default listener) is opened at all.

I'm trying to track this down even more but currently I got lost in all the context and HASH stuff. But maybe someone else has an idea why mosquitto dies on the RPi on a SIGHUP. I'm trying to continue in understanding the internals of mosquitto while tracking down this even closer but so far I want to share this intermediate results with you.
Help would be real appreciated since I'm about to use mosquitto on RPIs at work in an industrial IoT setup and also here I need the coexistance of port 1883 and port 8883 listener, same as in my smarthome environment, were it is currently failing.

[wollud1969]

Well, it appears that the context with the missing listener was loaded from the database. It wasn't a current context. Maybe it remained from before the update and the context structure has been changed in between or so. Has it?
However, after removing the database it problem disappeared.

Is there a tool to dump the database and inspect it?

[ralight]

There is a crude tool if you look in the source tree at src/db_dump, it's not normally distributed though.

[ralight]

The problem with it trying to open 1883 listeners is caused by char being unsigned on the Pi, I'm just about to push a fix for that.

[ralight]

Your other problem I'm not sure about yet. A persistent client that is disconnected will not have a listener set because it isn't connected to the broker.

[ralight]

I think this should be fixed in the latest fixes branch, could you give it a try and close this if you're happy?

[BozoTClown]

To start with I am a total noob to Mosquitto. I have been using it for approximately 3 weeks and up until today everything has been working great. I am running pure Mosquitto one broker and three clients. I reboot my raspberry pi's every might. Last night the brokers database got corrupted on the reboot so today when I finally figured out what the issue was I removed the database. Now whenever I try to start the broker I get the following in the log and no other messages.

1544283446: mosquitto version 1.4.10 (build date Wed, 17 Oct 2018 19:03:03 +0200) starting
1544283446: Config loaded from /etc/mosquitto/mosquitto.conf.

I have reboot my broker pi with no changes. I am to the point of removing and re-installing unless someone has a solution.

Here is a copy of my conf file

user mosquitto
max_queued_messages 200
message_size_limit 0
allow_zero_length_clientid true
allow_duplicate_messages false
listener 1883
autosave_interval 60
autosave_on_changes false
persistence true
persistence_file mosquitto.db
allow_anonymous true
#password_file /etc/mosquitto/passwd