Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not able to reload mosquitto on Ubuntu 16.04 #1424

Closed
JinPingChng opened this issue Sep 25, 2019 · 16 comments

Comments

@JinPingChng
Copy link

commented Sep 25, 2019

Hi,

This case happened on yesterday morning (GMT + 08). and it doesn't restart/start the mosquitto service. I get this issue from my syslog.

// This when I do /etc/init.d/mosquitto reload

Sep 25 10:55:47 nodus-rtls-ytl-prod systemd[1]: Reloading LSB: mosquitto MQTT v3.1 message broker.
Sep 25 10:55:47 nodus-rtls-ytl-prod mosquitto[7370]: * Reloading network daemon configuration: mosquitto
Sep 25 10:55:47 nodus-rtls-ytl-prod mosquitto[7370]: ...done.
Sep 25 10:55:47 nodus-rtls-ytl-prod systemd[1]: Reloaded LSB: mosquitto MQTT v3.1 message broker.
Sep 25 10:55:47 nodus-rtls-ytl-prod mosquitto[7348]: Reloading config.
Sep 25 10:55:47 nodus-rtls-ytl-prod mosquitto[7348]: Loading config file /etc/mosquitto/conf.d/default.conf
Sep 25 10:55:47 nodus-rtls-ytl-prod kernel: [12620.743395] mosquitto[7348]: segfault at 88 ip 0000000000423f5c sp 00007ffd9d7bb2a0 error 4 in mosquitto[400000+3d000]

As I understand that

  1. Logrotate will do a killall -HUP on all mosquitto, so it will hang up the mosquitto and reload back. Apparently, the mosquitto reload is no longer work (before Yesterday, the mosquitto still work like a charm). So, I decided to temporary remove the killall -HUP mosquitto first
  2. Every time I tried to do /etc/init.d/mosquitto reload, I will receive the segfault at 88 ip. And I need to restart manually the mosquitto by restarting
  3. I have tried reboot the server, but it's seem like didn't do the trick.
  4. Tried to look for solutions and google around.

Please help and I wouldn't hesitate to give out what information you need...

@ralight

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2019

Is this a Raspberry Pi? Others have been having problems with Pi 1 and Pi 2 devices.

@JinPingChng

This comment has been minimized.

Copy link
Author

commented Sep 25, 2019

No, this is a server that runs on Ubuntu 16.04 Server.

@ralight

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2019

What version of mosquitto, and where did you install from?

@JinPingChng

This comment has been minimized.

Copy link
Author

commented Sep 25, 2019

Mosquitto Version 1.6.6
I install through sudo apt-get.

This is an example of how do I install mosquitto MQTT broker.
https://www.vultr.com/docs/how-to-install-mosquitto-mqtt-broker-server-on-ubuntu-16-04

@JinPingChng

This comment has been minimized.

Copy link
Author

commented Sep 25, 2019

Hi railight

Thank for respond. Currently, the main issue is that mosquitto service is not able to reload. However, it still can run as usual. Even when I restart the services. Just that it couldn't reload...

@ralight

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2019

I understand. Are all files in /etc/mosquitto accessible by the mosquitto user? It shouldn't crash, but that is a possible cause for problems.

@ralight

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2019

Are you able to debug on this machine? If so, could you apt install mosquitto-dbgsym, then stop mosquitto and run it manually: sudo valgrind --log-file=vglog /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf. Once it is running, try to reload in the same way. If it doesn't crash, then stop the valgrind process (it will be called memcheck). Please then send me the vglog file to roger@atchoo.org

If you aren't able to debug on this machine, then could you please share more of your configuration? I haven't been able to reproduce this so far on a 16.04 machine.

@JinPingChng

This comment has been minimized.

Copy link
Author

commented Sep 25, 2019

Hi ralight,

I will get back on you once I tested on what you have suggested. Thanks!

@JinPingChng

This comment has been minimized.

Copy link
Author

commented Sep 25, 2019

I understand. Are all files in /etc/mosquitto accessible by the mosquitto user? It shouldn't crash, but that is a possible cause for problems.

the server only run in root user, temporary no other super user

@ralight

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2019

Mosquitto will try to change to the mosquitto user if run as root, unless you explicitly tell it to stay as root. Which is not recommended.

@JinPingChng

This comment has been minimized.

Copy link
Author

commented Sep 25, 2019

Hi railight,

I failed to debug by using the valgrind, It will stop the services, I will send you the vglog. And I wasn't able to install the mosquitto-dbgsym.

This will be my configuration

/etc/mosquitto/conf.d/default.conf

General Configuration
log_type all
log_dest syslog
log_dest stdout
connection_messages

Default Listener
port 8883
cafile /root/<root_directory>/mqtt_ca.crt
certfile /root/<root_directory>/mqtt_server.crt
keyfile /root/<root_directory>/mqtt_server.key

Listener 8884 (Third Party Customer)
listener 8884
cafile /root/<root_directory>/mqtt_server_ca.crt
certfile /root/<root_directory>/mqtt_client_server.crt
keyfile /root/<root_directory>/mqtt_client_server.key
require_certificate

WebSocket
listener 9001
protocol websockets

Insecure Connection 1883
listener 1883

/etc/mosquitto/mosquitto.conf

pid_file /var/run/mosquitto.pid

persistence true
persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d'

@ralight ralight closed this in 9584463 Sep 25, 2019
@ralight

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2019

Could you please try updating, this should be fixed.

@JinPingChng

This comment has been minimized.

Copy link
Author

commented Sep 26, 2019

Hi Ralight

Thank You for your support and help.

I have updated on the servers and yes I am able to manual reload the mosquitto.

For another question is that, will also affect other versions of Mosquitto? As I have some cloud servers that used another version (I believe is 1.4.8).

Once again, thank for your help.

@ralight

This comment has been minimized.

Copy link
Contributor

commented Sep 26, 2019

The other versions should be unaffected.

@JinPingChng

This comment has been minimized.

Copy link
Author

commented Oct 1, 2019

Hi ralight, one question as our end customer come to us and ask me on it.

I would like to know about the issue, could you specify what kind of issue we were faced before?

Once again, thank for your help

With Regards
Jpchng

@ralight

This comment has been minimized.

Copy link
Contributor

commented Oct 1, 2019

It was something like if you had a TLS listener and non-TLS listener configured, and had a persistent client that was disconnected at the time the reload occurred, then an incomplete check could cause a null pointer to be derefenced.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.