Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

apache plugin crashes on Ubuntu Lucid when 2 https instances are used #720

Closed
gilesw opened this issue Aug 29, 2014 · 5 comments
Closed

apache plugin crashes on Ubuntu Lucid when 2 https instances are used #720

gilesw opened this issue Aug 29, 2014 · 5 comments
Labels
Pending contributor action

Comments

@gilesw
Copy link

gilesw commented Aug 29, 2014

ii collectd 4.8.2-1ubuntu0.1 statistics collection and monitoring daemon
ii collectd-core 4.8.2-1ubuntu0.1 statistics collection and monitoring daemon
ii collectd-dbg 4.8.2-1ubuntu0.1 statistics collection and monitoring daemon
ii collectd-utils 4.8.2-1ubuntu0.1 statistics collection and monitoring daemon

specifically with 2 https instances

<Plugin "apache">

   <Instance "blar">
       URL "https://xxxx.net/server-status?auto"
   </Instance>
</Plugin>

<Plugin "apache">

   <Instance "localhost">
       URL "https://localhost/server-status?auto"
   </Instance>
</Plugin>

Only found on Ubuntu Lucid. Same conditions on Precise run fine. Also 2 http instances runs fine.

Bug seen in collectd-core 5.3.0-ppa8~lucid1
and own backport of 5.4.1-git master.

sudo gdb /usr/sbin/collectd /var/lib/collectd/core
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/collectd...Reading symbols from /usr/lib/debug/usr/sbin/collectd...done.
done.
[New Thread 28910]
[New Thread 28908]
[New Thread 28906]
[New Thread 28909]
[New Thread 28912]
[New Thread 28911]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /usr/lib/libltdl.so.7...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libltdl.so.7
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/collectd/syslog.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/syslog.so...done.
done.
Loaded symbols for /usr/lib/collectd/syslog.so
Reading symbols from /usr/lib/collectd/apache.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/apache.so...done.
done.
Loaded symbols for /usr/lib/collectd/apache.so
Reading symbols from /usr/lib/libcurl-gnutls.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libcurl-gnutls.so.4
Reading symbols from /usr/lib/libidn.so.11...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libidn.so.11
Reading symbols from /usr/lib/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /usr/lib/libgnutls.so.26...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /lib/libgcrypt.so.11...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcrypt.so.11
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /usr/lib/libtasn1.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /lib/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libgpg-error.so.0
Reading symbols from /usr/lib/collectd/cpu.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/cpu.so...done.
done.
Loaded symbols for /usr/lib/collectd/cpu.so
Reading symbols from /usr/lib/collectd/df.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/df.so...done.
done.
Loaded symbols for /usr/lib/collectd/df.so
Reading symbols from /usr/lib/collectd/vmem.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/vmem.so...done.
done.
Loaded symbols for /usr/lib/collectd/vmem.so
Reading symbols from /usr/lib/collectd/disk.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/disk.so...done.
done.
Loaded symbols for /usr/lib/collectd/disk.so
Reading symbols from /usr/lib/collectd/entropy.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/entropy.so...done.
done.
Loaded symbols for /usr/lib/collectd/entropy.so
Reading symbols from /usr/lib/collectd/interface.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/interface.so...done.
done.
Loaded symbols for /usr/lib/collectd/interface.so
Reading symbols from /usr/lib/collectd/irq.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/irq.so...done.
done.
Loaded symbols for /usr/lib/collectd/irq.so
Reading symbols from /usr/lib/collectd/load.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/load.so...done.
done.
Loaded symbols for /usr/lib/collectd/load.so
Reading symbols from /usr/lib/collectd/memory.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/memory.so...done.
done.
Loaded symbols for /usr/lib/collectd/memory.so
Reading symbols from /usr/lib/collectd/processes.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/processes.so...done.
done.
Loaded symbols for /usr/lib/collectd/processes.so
Reading symbols from /usr/lib/collectd/protocols.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/protocols.so...done.
done.
Loaded symbols for /usr/lib/collectd/protocols.so
Reading symbols from /usr/lib/collectd/swap.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/swap.so...done.
done.
Loaded symbols for /usr/lib/collectd/swap.so
Reading symbols from /usr/lib/collectd/unixsock.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/unixsock.so...done.
done.
Loaded symbols for /usr/lib/collectd/unixsock.so
Reading symbols from /usr/lib/collectd/uptime.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/uptime.so...done.
done.
Loaded symbols for /usr/lib/collectd/uptime.so
Reading symbols from /usr/lib/collectd/users.so...Reading symbols from /usr/lib/debug/usr/lib/collectd/users.so...done.
done.
Loaded symbols for /usr/lib/collectd/users.so
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Core was generated by `collectd -C /etc/collectd/collectd.conf -f'.
Program terminated with signal 6, Aborted.
#0  0x00007f6c4c701b65 in raise () from /lib/libc.so.6
(gdb) bt full
#0  0x00007f6c4c701b65 in raise () from /lib/libc.so.6
No symbol table info available.
#1  0x00007f6c4c7056b0 in abort () from /lib/libc.so.6
No symbol table info available.
#2  0x00007f6c4c6faa31 in __assert_fail () from /lib/libc.so.6
No symbol table info available.
#3  0x00007f6c4ae9b37a in ?? () from /lib/libgcrypt.so.11
No symbol table info available.
#4  0x00007f6c4aed1620 in ?? () from /lib/libgcrypt.so.11
No symbol table info available.
#5  0x00007f6c4aed176e in ?? () from /lib/libgcrypt.so.11
No symbol table info available.
#6  0x00007f6c4aed281e in ?? () from /lib/libgcrypt.so.11
No symbol table info available.
#7  0x00007f6c4b13e7e5 in ?? () from /usr/lib/libgnutls.so.26
No symbol table info available.
#8  0x00007f6c4b129379 in ?? () from /usr/lib/libgnutls.so.26
No symbol table info available.
#9  0x00007f6c4b123305 in ?? () from /usr/lib/libgnutls.so.26
No symbol table info available.
#10 0x00007f6c4b1205e5 in ?? () from /usr/lib/libgnutls.so.26
No symbol table info available.
#11 0x00007f6c4b120e10 in gnutls_handshake () from /usr/lib/libgnutls.so.26
No symbol table info available.
#12 0x00007f6c4c0b3cf8 in ?? () from /usr/lib/libcurl-gnutls.so.4
No symbol table info available.
#13 0x00007f6c4c0b42bc in ?? () from /usr/lib/libcurl-gnutls.so.4
No symbol table info available.
#14 0x00007f6c4c0b52c7 in ?? () from /usr/lib/libcurl-gnutls.so.4
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#15 0x00007f6c4c093efb in ?? () from /usr/lib/libcurl-gnutls.so.4
No symbol table info available.
#16 0x00007f6c4c09a552 in ?? () from /usr/lib/libcurl-gnutls.so.4
No symbol table info available.
#17 0x00007f6c4c0a09e4 in ?? () from /usr/lib/libcurl-gnutls.so.4
No symbol table info available.
#18 0x00007f6c4c0a8960 in ?? () from /usr/lib/libcurl-gnutls.so.4
No symbol table info available.
#19 0x00007f6c4c2c8554 in apache_read_host (user_data=<value optimized out>) at apache.c:647
        i = <value optimized out>
        ptr = <value optimized out>
        saveptr = <value optimized out>
        lines = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6204a0 "", 0x0, 0x7f6c4ce6daf6 "H\201Ā", 0x0, 0x0,
          0x0,
          0x40f96a "\205\300~:H\213E0N\215<8J\213\f(I\213\027I\211\017J\211\024(\213D$\f\205\300\017\224\300H\205\333t\016\204\300tzH\321\353\353\205\017\037D"}
        lines_num = <value optimized out>
        fields = {0x0, 0x100000000 <Address 0x100000000 out of bounds>,
          0x7f6c4ce6ec83 "H\205\300I\017D\306H\203\304\060A_A^A]A\\\303\350\224M", 0x0}
        fields_num = <value optimized out>
        st = 0xc6c9a0
        __PRETTY_FUNCTION__ = "apache_read_host"
#20 0x000000000040b6bc in plugin_read_thread (args=<value optimized out>) at plugin.c:392
        rf = 0xc6cb10
        now = {tv_sec = 1409337161, tv_usec = 112750}
        status = <value optimized out>
        rf_type = 1
        __PRETTY_FUNCTION__ = "plugin_read_thread"
#21 0x00007f6c4ce699ca in start_thread () from /lib/libpthread.so.0
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#22 0x00007f6c4c7b81cd in clone () from /lib/libc.so.6
No symbol table info available.
#23 0x0000000000000000 in ?? ()
No symbol table info available.
@mfournier
Copy link

The symptom remind me of #513. Try checking if 5f2f969 is present in the various versions of collectd you use (it definetely won't be in 4.8, but should apply easily).

A possible workaround could be to run collectd with ReadThreads 1, but this will negatively impact performance.

@gilesw
Copy link
Author

gilesw commented Aug 29, 2014

Hi Mournier yes that's a thread I've been looking at for a while. I thought the curl init change would resolve it but it turns out that was in the codebase I was testing too.

@octo
Copy link
Member

octo commented Sep 2, 2014

From http://curl.haxx.se/libcurl/c/libcurl-tutorial.html:

libcurl is completely thread safe, except for two issues: signals and SSL/TLS handlers. Signals are used for timing out name resolves (during DNS lookup) - when built without c-ares support and not on Windows.

If you are accessing HTTPS or FTPS URLs in a multi-threaded manner, you are then of course using the underlying SSL library multi-threaded and those libs might have their own requirements on this issue. Basically, you need to provide one or two functions to allow it to function properly. For all details, see this:

When using multiple threads you should set the CURLOPT_NOSIGNAL option to 1 for all handles. Everything will or might work fine except that timeouts are not honored during the DNS lookup […]

We do set the CURLOPT_NOSIGNAL option, but calling gcry_control, as mentioned in the GnuTLS manual, is only done in the network plugin. We should probably move this initialization to a more central place. Which crypto library does your version of libCURL use?

@octo
Copy link
Member

octo commented Sep 2, 2014

Just saw /usr/lib/libgnutls.so.26, so it looks like GnuTLS 2.6. It appears that the initialization code has been changed with GnuTLS 3.

@octo
Copy link
Member

octo commented Sep 12, 2016

Version 5.6 fixes many gcrypt initialization issues; please re-open if this is still an issue.

@octo octo closed this as completed Sep 12, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Pending contributor action
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mfournier @octo @gilesw