New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mosquitto dies after SIGHUP on Raspberry Pi #849

Closed
wollud1969 opened this Issue Jun 7, 2018 · 5 comments

Comments

Projects
None yet
2 participants
@wollud1969
Contributor

wollud1969 commented Jun 7, 2018

I've a quite strange behavior on my RPi running Raspian Stretch since the last update a few days ago: mosquitto dies after SIGHUP (happens every morning due to the logrotate script).

I just cloned the repo and built from source and found the same behavior. My configuration is

pid_file /var/run/mosquitto.pid
log_dest file /var/log/mosquitto/mosquitto.log

persistence true
persistence_location /var/lib/mosquitto/

per_listener_settings true


# listener 1883 0.0.0.0
allow_anonymous true


listener 8883 0.0.0.0
password_file /etc/mosquitto/pwfile
allow_anonymous false
cafile /etc/mosquitto/ssl/ca/server-ca.crt
certfile /etc/mosquitto/ssl/internal-mqtt-broker.crt
keyfile /etc/mosquitto/ssl/internal-mqtt-broker.pem
tls_version tlsv1


# connection to-external
# address 172.16.1.10:8883
# local_password xxx
# local_username argus
# remote_password xxx
# remote_username homegear
# bridge_insecure true
# bridge_tls_version tlsv1.2
# bridge_cafile /etc/mosquitto/ssl/ca/server-ca.crt
# topic IoT/espThermometer2/measurement in

This configuration opens two listeners, the default listener at port 1883 without tls and without passwords and a second listener at port 8883 with tls and with passwords.

I started debugging on the RPi and inserted some printf's and found that mosquitto died in security_default.c, mosquitto_security_apply_default, line 864 (line 870 in my fork at github.com:wollud1969/mosquitto) since context->listener is null.

printf("msad4 %p\n", context);
printf("msad4.1 %p\n", context->listener);
printf("msad4.2 %p\n", &(context->listener->security_options));
printf("msad4.3 %d\n", context->listener->security_options.allow_anonymous);
                        allow_anonymous = context->listener->security_options.allow_anonymous;
printf("msad4b\n");
msad4 0x2aa4e8
msad4.1 (nil)
msad4.2 0x50
Segmentation fault

The listener is not set.

For easier debugging I switched to my laptop (Debian Stretch), prepared an environment with exactly the same configuration (well, nearly the same: paths for persistence, pwfile, certs and keys are different, content is the same).

But on the laptop I got the following output:

msad4 0x55a293f5ee40
msad4.1 0x55a293f4b290
msad4.2 0x55a293f4b318
msad4.3 0
msad4b

Here, the listener is set.

I found a difference in the evaluation of the configuration between RPi and laptop:
When I try to set a listener statement on the RPi to port 1883

listener 1883 0.0.0.0

I got in the logfile:

1528375568: mosquitto version 1.5 starting
1528375568: Config loaded from /etc/mosquitto/mosquitto.conf.
1528375568: Opening ipv4 listen socket on port 1883.
1528375568: Opening ipv4 listen socket on port 8883.
1528375568: Opening ipv4 listen socket on port 1883.
1528375568: Error: Address already in use

So, appearently it trys to open the default listener on port 1883 and an additional listener on port 1883, which fails.

On the laptop this works, only one listener is opened on port 1883. Actually, when I change the first listener statement on the laptop to port 1884 no listener on port 1883 (default listener) is opened at all.

I'm trying to track this down even more but currently I got lost in all the context and HASH stuff. But maybe someone else has an idea why mosquitto dies on the RPi on a SIGHUP. I'm trying to continue in understanding the internals of mosquitto while tracking down this even closer but so far I want to share this intermediate results with you.
Help would be real appreciated since I'm about to use mosquitto on RPIs at work in an industrial IoT setup and also here I need the coexistance of port 1883 and port 8883 listener, same as in my smarthome environment, were it is currently failing.

@wollud1969

This comment has been minimized.

Show comment
Hide comment
@wollud1969

wollud1969 Jun 7, 2018

Contributor

Well, it appears that the context with the missing listener was loaded from the database. It wasn't a current context. Maybe it remained from before the update and the context structure has been changed in between or so. Has it?
However, after removing the database it problem disappeared.

Is there a tool to dump the database and inspect it?

Contributor

wollud1969 commented Jun 7, 2018

Well, it appears that the context with the missing listener was loaded from the database. It wasn't a current context. Maybe it remained from before the update and the context structure has been changed in between or so. Has it?
However, after removing the database it problem disappeared.

Is there a tool to dump the database and inspect it?

@ralight

This comment has been minimized.

Show comment
Hide comment
@ralight

ralight Aug 2, 2018

Contributor

There is a crude tool if you look in the source tree at src/db_dump, it's not normally distributed though.

Contributor

ralight commented Aug 2, 2018

There is a crude tool if you look in the source tree at src/db_dump, it's not normally distributed though.

@ralight

This comment has been minimized.

Show comment
Hide comment
@ralight

ralight Aug 2, 2018

Contributor

The problem with it trying to open 1883 listeners is caused by char being unsigned on the Pi, I'm just about to push a fix for that.

Contributor

ralight commented Aug 2, 2018

The problem with it trying to open 1883 listeners is caused by char being unsigned on the Pi, I'm just about to push a fix for that.

ralight added a commit that referenced this issue Aug 2, 2018

Fix problem on Pi caused by unsigned char being default.
Found via #849.

Signed-off-by: Roger A. Light <roger@atchoo.org>
@ralight

This comment has been minimized.

Show comment
Hide comment
@ralight

ralight Aug 2, 2018

Contributor

Your other problem I'm not sure about yet. A persistent client that is disconnected will not have a listener set because it isn't connected to the broker.

Contributor

ralight commented Aug 2, 2018

Your other problem I'm not sure about yet. A persistent client that is disconnected will not have a listener set because it isn't connected to the broker.

@ralight

This comment has been minimized.

Show comment
Hide comment
@ralight

ralight Sep 19, 2018

Contributor

I think this should be fixed in the latest fixes branch, could you give it a try and close this if you're happy?

Contributor

ralight commented Sep 19, 2018

I think this should be fixed in the latest fixes branch, could you give it a try and close this if you're happy?

@ralight ralight closed this in 55ca61f Sep 19, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment