Failed to start thread manager

[fschaefer]

# uname -a
Linux raspberrypi 3.10.24+ #614 PREEMPT Thu Dec 19 20:38:42 GMT 2013 armv6l GNU/Linux

# git branch
* master

# git rev-parse HEAD
e1d8a2a96501418605dd3df686df708162b24b1d

# /home/pi/sslsplit/sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443 tcp 0.0.0.0 8080
Generated RSA key for leaf certs.
SSLsplit 0.4.7-42-ge1d8a2a (built 2014-01-14)
Copyright (c) 2009-2014, Daniel Roethlisberger <daniel@roe.ch>
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter:  IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using SSL_MODE_RELEASE_BUFFERS
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected
proxyspecs:
- [0.0.0.0]:8080 tcp plain netfilter
- [0.0.0.0]:8443 ssl plain netfilter
Loaded CA: '/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd'
Using libevent backend 'epoll'
Event base supports: edge yes, O(1) yes, anyfd no
Inserted events:
  0x5d240 [fd 7] Read Persist
  0x5e1cc [fd 8] Read Persist
  0x5f98c [fd 9] Read Persist
  0x5d130 [fd 6] Read Persist
  0x5df98 [fd 3] Signal Persist
  0x5fb48 [fd 1] Signal Persist
  0x5fc28 [fd 2] Signal Persist
  0x5fd08 [fd 13] Signal Persist
Failed to start thread manager

# /home/pi/sslsplit/sslsplit -V
SSLsplit 0.4.7-42-ge1d8a2a (built 2014-01-14)
Copyright (c) 2009-2014, Daniel Roethlisberger <daniel@roe.ch>
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter:  IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using SSL_MODE_RELEASE_BUFFERS
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected

# gdb --args /home/pi/sslsplit/sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443 tcp 0.0.0.0 8080
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/pi/sslsplit/sslsplit...done.
(gdb) r
Starting program: /home/pi/sslsplit/sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443 tcp 0.0.0.0 8080
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
0xb6e5a5e0 in ?? () from /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.0.0
(gdb) c
Continuing.
Cannot access memory at address 0x0

Program received signal SIGILL, Illegal instruction.
0xb6e5a5e8 in ?? () from /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.0.0
(gdb) c
Continuing.
Cannot access memory at address 0x0
Warning: not seeding OpenSSL RAND due to PURITY!
Generated RSA key for leaf certs.
SSLsplit 0.4.7-42-ge1d8a2a-dirty (built 2014-01-14)
Copyright (c) 2009-2014, Daniel Roethlisberger <daniel@roe.ch>
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DPURIFY -DDEBUG_PROXY -DDEBUG_CERTIFICATE -DDEBUG_SESSION_CACHE -DDEBUG_SNI_PARSER -DDEBUG_THREAD -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter:  IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using SSL_MODE_RELEASE_BUFFERS
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected
proxyspecs:
- [0.0.0.0]:8080 tcp plain netfilter
- [0.0.0.0]:8443 ssl plain netfilter
Loaded CA: '/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd'
Certificate:
<snip>
-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
Using libevent backend 'epoll'
Event base supports: edge yes, O(1) yes, anyfd no
[New Thread 0xb6bd7470 (LWP 30094)]
[New Thread 0xb63d7470 (LWP 30095)]
[New Thread 0xb5bd7470 (LWP 30096)]
Failed to start thread manager
[Thread 0xb6bd7470 (LWP 30094) exited]
[Thread 0xb63d7470 (LWP 30095) exited]
[Thread 0xb5bd7470 (LWP 30096) exited]
[Inferior 1 (process 30091) exited normally]

[droe]

Does make test pass on the system? Can openssl s_client connect to SSL servers successfully?

[fschaefer]

# /home/pi/sslsplit/sslsplit.test 
Running suite(s): 
 main
 opts
Cannot resolve address '::1' port '10443': Name or service not known
Cannot resolve address '::1' port '10443': Name or service not known
Cannot resolve address '::1' port '10443': Name or service not known
 dynbuf
 cert
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
 cachemgr
 cachefkcrt
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
 cachetgcrt
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
 cachedsess
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
 cachessess
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
 ssl
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
Warning: not seeding OpenSSL RAND due to PURITY!
 sys
 base64
 url
 util
97%: Checks: 113, Failures: 0, Errors: 3
opts.t.c:96:E:proxyspec_parse:proxyspec_parse_02:0: (after this point) Early exit with return value 1
opts.t.c:298:E:proxyspec_parse:proxyspec_parse_13:0: (after this point) Early exit with return value 1
opts.t.c:331:E:proxyspec_parse:proxyspec_parse_14:0: (after this point) Early exit with return value 1

# openssl s_client -host www.google.com -port 443
CONNECTED(00000003)
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
   i:/C=US/O=Google Inc/CN=Google Internet Authority G2
 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
   i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
Server certificate
<snip>

[droe]

a80cbf7 will tell us what exactly fails. This will complain that creating one of the evbases or the dnsbases failed. Since it dies before printing the number of threads it initialized, the only other possible source of the problem is memory exhaustion, which seems very unlikely. In other words, this is very likely an issue in libevent2, not SSLsplit.

[droe]

Btw, the fact that evutil_getaddrinfo() cannot handle IPv6 addresses on that box (the three failed tests) suggests that there is generally something wrong with that system's setup. That may or may not be related to the failure of starting the thread manager (or libevent creating event or evdns bases).

[fschaefer]

@droe, thanks for investigating! evutil_getaddrinfo() can't resolve IPv6 addresses on that box, because IPv6 is disabled. If I enable it there are no failing tests, but sslsplit still bails out. Failed to create dnsbase 0 is what fails.

[droe]

Personally, I would argue that disabled IPv6 in 2014 definitely qualifies for "something wrong" :) But seriously, is your resolv.conf ok? That's one of the reasons libevent can fail to create a dnsbase.

I added some code in 9338200 which should detect resolv.conf loading problems both in make test and at runtime.

[fschaefer]

Well, this is a private box on a private network. There is really no need to enable IPv6. Security-wise it would be a bad idea to enable it just because it's there. :P

I pulled 9338200 and it just starts up normally without trowing Failed to start thread manager. O_o
I changed back to a80cbf7 and it bails out again. Then I changed to 9338200 once more and it works again.

[droe]

Oh wow, that's unexpected. I will leave that code in there then, and close the ticket. Thanks for the bug report and feedback.