Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mosquitto seg. fault during client connection (websockets / TLS) #406

Closed
gmatt76 opened this issue Mar 17, 2017 · 8 comments
Closed

mosquitto seg. fault during client connection (websockets / TLS) #406

gmatt76 opened this issue Mar 17, 2017 · 8 comments

Comments

@gmatt76
Copy link

gmatt76 commented Mar 17, 2017

Hi All,

I got the following seg. fault when a client is trying to connect to the broker using the TLS over websockets:

root@peamicms-vm:/tmp# gdb mosquitto
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/local/sbin/mosquitto...(no debugging symbols found)...done.
(gdb) r -c /etc/mosquitto/mosquitto.conf
Starting program: /usr/local/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0xb7d6508f in lws_ssl_server_name_cb () from /usr/local/lib/libwebsockets.so.10
(gdb) bt
#0 0xb7d6508f in lws_ssl_server_name_cb () from /usr/local/lib/libwebsockets.so.10
#1 0xb7f67b97 in ?? () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
#2 0xb7f4f744 in ?? () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
#3 0xb7f545b1 in ?? () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
#4 0xb7f7b5ea in SSL_accept () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
#5 0xb7f6399e in ?? () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
#6 0xb7f642e2 in ?? () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
#7 0xb7f7b5ea in SSL_accept () from /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0
#8 0xb7d64bfe in lws_server_socket_service_ssl () from /usr/local/lib/libwebsockets.so.10
#9 0xb7d692f1 in lws_adopt_descriptor_vhost () from /usr/local/lib/libwebsockets.so.10
#10 0xb7d693cb in lws_adopt_socket_vhost () from /usr/local/lib/libwebsockets.so.10
#11 0xb7d6966a in lws_server_socket_service () from /usr/local/lib/libwebsockets.so.10
#12 0xb7d5a0c0 in lws_service_fd_tsi () from /usr/local/lib/libwebsockets.so.10
#13 0xb7d67386 in _lws_plat_service_tsi () from /usr/local/lib/libwebsockets.so.10
#14 0xb7d675cb in lws_plat_service () from /usr/local/lib/libwebsockets.so.10
#15 0xb7d5a653 in lws_service () from /usr/local/lib/libwebsockets.so.10
#16 0x0805553b in ?? ()
#17 0x0804bd9c in ?? ()
#18 0xb7bfee46 in __libc_start_main () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#19 0x0804bfd9 in ?? ()

Mosquitto compiled with the following libraries:

root@peamicms-vm:/tmp# ldd /usr/local/sbin/mosquitto
linux-gate.so.1 => (0xb76fd000)
libdl.so.2 => /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xb76ed000)
libm.so.6 => /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xb76c7000)
librt.so.1 => /lib/i386-linux-gnu/i686/cmov/librt.so.1 (0xb76bd000)
libanl.so.1 => /lib/i386-linux-gnu/i686/cmov/libanl.so.1 (0xb76b9000)
libssl.so.1.0.0 => /usr/lib/i386-linux-gnu/i686/cmov/libssl.so.1.0.0 (0xb765e000)
libcrypto.so.1.0.0 => /usr/lib/i386-linux-gnu/i686/cmov/libcrypto.so.1.0.0 (0xb749c000)
libuuid.so.1 => /lib/i386-linux-gnu/libuuid.so.1 (0xb7496000)
libwebsockets.so.10 => /usr/local/lib/libwebsockets.so.10 (0xb746c000)
libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb7304000)
/lib/ld-linux.so.2 (0xb76fe000)
libpthread.so.0 => /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xb72eb000)
libz.so.1 => /lib/i386-linux-gnu/libz.so.1 (0xb72d2000)

on a debian 7.11

Thanks in advance,
Gianluca

@ralight
Copy link
Contributor

ralight commented Mar 17, 2017

Could you confirm which exact versions of mosquitto, openssl and libwebsockets you are using please?

@gmatt76
Copy link
Author

gmatt76 commented Mar 20, 2017

mosquitto 1.4.11

libwebsocket 2.2.0

compiled
with:

Compiling with SSL support
Found OpenSSL: /usr/lib/i386-linux-gnu/libssl.so;/usr/lib/i386-linux-gnu/libcrypto.so (found version 1.0.1t)
OpenSSL include dir: /usr/include
OpenSSL libraries: /usr/lib/i386-linux-gnu/libssl.so;/usr/lib/i386-linux-gnu/libcrypto.so
Looking for openssl/ecdh.h
Looking for openssl/ecdh.h - found
Looking for SSL_CTX_set1_param
Looking for SSL_CTX_set1_param - found
Looking for X509_VERIFY_PARAM_set1_host
Looking for X509_VERIFY_PARAM_set1_host - not found
Searching for OpenSSL executable and dlls
OpenSSL executable: /usr/bin/openssl
GENCERTS = 1
Generating SSL Certificates for the test-server...
SUCCESSFULLY generated SSL certificate

Looking for RPMTools... - rpmbuild NOT FOUND

Settings: (For more help do cmake -LH )

LWS_WITH_STATIC = ON

LWS_WITH_SHARED = ON
LWS_WITH_SSL = ON (SSL Support)
LWS_SSL_CLIENT_USE_OS_CA_CERTS = 1
LWS_USE_WOLFSSL = OFF (wolfSSL/CyaSSL replacement for OpenSSL)
LWS_WITHOUT_BUILTIN_SHA1 = OFF
LWS_WITHOUT_BUILTIN_GETIFADDRS = OFF
LWS_WITHOUT_CLIENT = OFF
LWS_WITHOUT_SERVER = OFF
LWS_LINK_TESTAPPS_DYNAMIC = OFF
LWS_WITHOUT_TESTAPPS = OFF
LWS_WITHOUT_TEST_SERVER = OFF
LWS_WITHOUT_TEST_SERVER_EXTPOLL = OFF
LWS_WITHOUT_TEST_PING = OFF
LWS_WITHOUT_TEST_ECHO = OFF
LWS_WITHOUT_TEST_CLIENT = OFF
LWS_WITHOUT_TEST_FRAGGLE = OFF
LWS_WITHOUT_EXTENSIONS = OFF
LWS_WITH_LATENCY = OFF
LWS_WITHOUT_DAEMONIZE = ON
LWS_USE_LIBEV =
LWS_USE_LIBUV =
LWS_IPV6 = OFF
LWS_UNIX_SOCK = OFF
LWS_WITH_HTTP2 = OFF
LWS_SSL_SERVER_WITH_ECDH_CERT = OFF
LWS_MAX_SMP = 32
LWS_WITH_CGI = OFF
LWS_HAVE_OPENSSL_ECDH_H = 1
LWS_HAVE_SSL_CTX_set1_param = 1
LWS_WITH_HTTP_PROXY = OFF
LIBHUBBUB_LIBRARIES =
PLUGINS =
LWS_WITH_ACCESS_LOG = OFF
LWS_WITH_SERVER_STATUS = OFF
LWS_WITH_LEJP = OFF
LWS_WITH_LEJP_CONF = OFF
LWS_WITH_SMTP = OFF
LWS_WITH_GENERIC_SESSIONS = OFF
LWS_STATIC_PIC = OFF
LWS_WITH_RANGES = ON
LWS_PLAT_OPTEE = OFF
LWS_WITH_ESP32 = OFF
LWS_WITH_ZIP_FOPS = ON

openssl version
OpenSSL 1.0.1e 11 Feb 2013 (Library: OpenSSL 1.0.1t 3 May 2016)

@mkeyno
Copy link

mkeyno commented Apr 8, 2017

I have same problem , I've installed last version of mosquitto on my raspberry pi Jessi , I have no problem to pub/sub either through terminal in raspberry pi or MQTT.fx 1.3.1 on my windows but when I use following paho mqtt client js it throw disconnected error , I might add I disable mod_websocket_mosquitto module on Apache server to prevent other websocket port conflict, however problem not resolved
error

1491637310: New connection from 192.168.1.101 on port 1883.
1491637310: Socket error on client <unknown>, disconnecting.

html page

<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <script type="text/javascript" src="mqttws31.js"></script>

  <script type="text/javascript">
  
  var client;
  var form = document.getElementById("tutorial");
  var  host =location.hostname;

  function doConnect() {
      client = new Paho.MQTT.Client(host, 1883, "ClientId");
      client.onConnect = onConnect;
      client.onMessageArrived = onMessageArrived;
      client.onConnectionLost = onConnectionLost;
      client.connect({onSuccess:onConnect});
  }
  
  function doSubscribe() {
      client.subscribe("/World");
  }
  
  function doSend() {
      message = new Paho.MQTT.Message("Hello");
      message.destinationName = "/World";
      client.send(message);
  }

  function doDisconnect() {
      client.disconnect();
  }
  
  // Web Messaging API callbacks
  
  function onConnect() {
      var form = document.getElementById("example");
      form.connected.checked= true;
  }
  
  function onConnectionLost(responseObject) {
      var form = document.getElementById("example");
      form.connected.checked= false;
      if (responseObject.errorCode !== 0)
          alert(client.clientId+"\n"+responseObject.errorCode);
  }
  
  function onMessageArrived(message) {
      var form = document.getElementById("example");
      form.receiveMsg.value = message.payloadString;
  }
  
  </script>
</head> 

<body>
  <h1>Example Web Messaging web page.</h1>
  <form id="example">
  <fieldset>
  <legend id="Connect" > Connect </legend>
    Make a connection to the server, and set up a call back used if a 
    message arrives for this client.
    <br>
    <input type="button" value="Connect" onClick="doConnect(this.form)" name="Connect"/>
    <input type="checkbox" name="connected" disabled="disabled"/>
  </fieldset>
  
  <fieldset>
  <legend id="Subscribe" > Subscribe </legend>
  Make a subscription to topic "/World".
  <br> <input type="button" value="Subscribe" onClick="doSubscribe(this.form)"/>
  </fieldset>
  
  <fieldset>
  <legend id="Send" > Send </legend>
    Create a Message object containing the word "Hello" and then publish it at
    the server.
    <br>
    <input type="button" value="Send" onClick="doSend(this.form)"/>
  </fieldset>
  
  <fieldset>
  <legend id="Receive" > Receive </legend>
    A copy of the published Message is received in the callback we created earlier.
  <textarea name="receiveMsg" rows="1" cols="40" disabled="disabled"></textarea>
  </fieldset>
  
  <fieldset>
  <legend id="Disconnect" > Disconnect </legend>
    Now disconnect this client from the server.
  <br> <input type="button" value="Disconnect" onClick="doDisconnect()"/>
  </fieldset>
  </form>
<!-- /body>
</html>

@karlp
Copy link
Contributor

karlp commented Apr 10, 2017 via email

@mkeyno
Copy link

mkeyno commented Apr 10, 2017

I configure this /etc/mosquitto/conf.d/mosquitto.conf file to enable websocket on 9001 as well , and just follow couple of example across of internet , but I'm not sure where other place should configure websockets listener? are you saying I should open websocket port in my page ?

@AlReem
Copy link

AlReem commented Jun 11, 2017

I am also having the same issue with the following installations.

Server: Ubuntu 16.04.2 LTS (Xenial)
Mosquitto: v1.4.8
libwebsockets: v2.2.0 (Compiled from source)

Extract from journalctl:
Jun 11 05:57:28 myserver kernel: mosquitto[23293]: segfault at 8 ip 00007a55e0779b29 sp 00007ffd5a966a80 error 4 in libwebsockets.so.7[7a65f0771000+1f000]

Please help

@jeffsf
Copy link

jeffsf commented Jan 1, 2018

mosquitto ungracefully exits SIGSEGV on websockets TLS connection, seems to work OK with websockets without TLS. Crash appears to be during TLS handshake. I have never been able to successfully connect and run with TLS websockets though using the same certificates and Python (paho) client code, with only the port/transport change from the TCP MQTT connection. I can't say that it crashes mosquitto every time, but the inability to connect seems to be consistent.

FreeBSD 11.1-RELEASE-p6
All ports built in a clean environment with poudriere from ports tree and sources obtained on 2017-12-31

autoconf-wrapper-20131203.txz
automake-1.15.1.txz
automake-wrapper-20131203.txz
c-ares-1.12.0_2.txz
ca_root_nss-3.34.1.txz
cmake-3.10.1_1.txz
curl-7.57.0.txz
expat-2.2.5.txz
gettext-runtime-0.19.8.1_1.txz
gettext-tools-0.19.8.1.txz
gmake-4.2.1_1.txz
help2man-1.47.5.txz
indexinfo-0.3.1.txz
jsoncpp-1.8.1_2.txz
libarchive-3.3.2,1.txz
libffi-3.2.1_2.txz
libgcrypt-1.8.2.txz
libgpg-error-1.27.txz
liblz4-1.8.0,1.txz
libnghttp2-1.29.0.txz
libtool-2.4.6.txz
libuv-1.18.0.txz
libwebsockets-2.2.2.txz
libxml2-2.9.7.txz
libxslt-1.1.29_1.txz
lzo2-2.10_1.txz
m4-1.4.18,1.txz
mosquitto-1.4.14.txz
ninja-1.8.2,2.txz
p5-Locale-gettext-1.07.txz
perl5-5.24.3.txz
pkg-1.10.3_1.txz
pkgconf-1.3.10,1.txz
py27-Babel-2.5.1.txz
py27-Jinja2-2.10.txz
py27-MarkupSafe-1.0.txz
py27-alabaster-0.7.6.txz
py27-docutils-0.14_1.txz
py27-imagesize-0.7.1.txz
py27-pygments-2.2.0.txz
py27-pystemmer-1.3.0_1.txz
py27-pytz-2017.3,1.txz
py27-setuptools-36.5.0.txz
py27-six-1.11.0.txz
py27-snowballstemmer-1.2.0_1.txz
py27-sphinx-1.4.8_2,1.txz
py27-sphinx_rtd_theme-0.2.4.txz
python27-2.7.14_1.txz
python36-3.6.4.txz
readline-7.0.3_1.txz
rhash-1.3.5.txz
scons-3.0.1.txz
texinfo-6.5,1.txz
# This file is auto-generated by 'make config'.
# Options for mosquitto-1.4.14
_OPTIONS_READ=mosquitto-1.4.14
_FILE_COMPLETE_OPTIONS_LIST=WS
OPTIONS_FILE_SET+=WS
# This file is auto-generated by 'make config'.
# Options for libwebsockets-2.2.2
_OPTIONS_READ=libwebsockets-2.2.2
_FILE_COMPLETE_OPTIONS_LIST=HTTP2 HTTP_PROXY IPV6 LIBEV LIBUV PLUGINS UNIX_SOCK WEBSERVER
OPTIONS_FILE_SET+=HTTP2
OPTIONS_FILE_UNSET+=HTTP_PROXY
OPTIONS_FILE_SET+=IPV6
OPTIONS_FILE_UNSET+=LIBEV
OPTIONS_FILE_UNSET+=LIBUV
OPTIONS_FILE_UNSET+=PLUGINS
OPTIONS_FILE_UNSET+=UNIX_SOCK
OPTIONS_FILE_UNSET+=WEBSERVER

(all other ports have their options unchanged from FreeBSD defaults)

DNS does not supply an IPv6 address for the mosquitto host and wireshark is showing an IPv4 connection.

Edit:

Disabling HTTP2 in libwebsockets and rebuilding both libwebsockets and mosquitto does not change the behavior. On connect attempt with TLS over websockets, mosquitto exists with signal 11. The Python client shows

  File "/Users/jeff/venv/incubator-pid/lib/python3.6/site-packages/paho/mqtt/client.py", line 768, in connect
    return self.reconnect()
  File "/Users/jeff/venv/incubator-pid/lib/python3.6/site-packages/paho/mqtt/client.py", line 927, in reconnect
    sock.do_handshake()
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:777)

@ralight
Copy link
Contributor

ralight commented Dec 2, 2020

I'm fairly certain that the issue here is something to do with libwebsockets. Version 2.4.0 is the first version I'm happy to recommend, and this is before that and now well out of date, so I'm going to close this issue. If it's still a problem with more recent versions of lws, please reopen.

@ralight ralight closed this as completed Dec 2, 2020
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 11, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants