Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation fault starting thread manager #10

Closed
linuxton opened this issue Aug 23, 2013 · 11 comments
Closed

Segmentation fault starting thread manager #10

linuxton opened this issue Aug 23, 2013 · 11 comments
Assignees
@droe
Labels
bug released
Milestone
0.4.8

Comments

@linuxton
Copy link

Hello,

when I try to run sslsplit (having libssl-dev and libevent-dev installed,generating self-signed certificates and using port forwarding before this) I get the following error:

Inserted events:
  0x9a01148 [fd 7] Read Persist
  0x9a0291c [fd 8] Read Persist
  0x9a01038 [fd 6] Read Persist
  0x9a02968 [fd 3] Signal Persist
  0x9a02ad0 [fd 1] Signal Persist
  0x9a02bb0 [fd 2] Signal Persist
  0x9a02c90 [fd 13] Signal Persist
Failed to start thread manager
Segmentation fault

I'm using:

./sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 9000

on the command line.

Could this be a resurfacing of previously reported bug (issue #9)?

Thank you
@ghost ghost assigned droe Aug 23, 2013
@droe
Copy link
Owner

droe commented Aug 23, 2013

No, this is definitely not issue #9. Could you post the output of sslsplit -V, a more complete debug log, plus ideally also a stack trace?

@linuxton
Copy link
Author

Sure, here they are:

./sslsplit -V
SSLsplit 0.4.7 (built 2013-08-23)
Copyright (c) 2009-2013, Daniel Roethlisberger daniel@roe.ch
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected

(gdb) run -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
Starting program: /root/sslsplit-0.4.7/sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Generated RSA key for leaf certs.
SSLsplit 0.4.7 (built 2013-08-23)
Copyright (c) 2009-2013, Daniel Roethlisberger daniel@roe.ch
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected
proxyspecs:

  • [0.0.0.0]:8443 ssl plain netfilter
    Loaded CA: '/C=US/ST=test/L=test/O=test/OU=test/CN=N/emailAddress=test@test.com'
    Using libevent backend 'epoll'
    Event base supports: edge yes, O(1) yes, anyfd no
    [New Thread 0xb7b87b70 (LWP 16616)]
    [New Thread 0xb7386b70 (LWP 16617)]
    [New Thread 0xb6b85b70 (LWP 16618)]
    Inserted events:
    0x8075160 [fd 11] Read Persist
    0x8077f1c [fd 12] Read Persist
    0x8075050 [fd 10] Read Persist
    0x80751b0 [fd 3] Signal Persist
    0x8078080 [fd 1] Signal Persist
    0x8078160 [fd 2] Signal Persist
    0x8078240 [fd 13] Signal Persist
    Failed to start thread manager

Program received signal SIGSEGV, Segmentation fault.
0xb7c527e1 in free () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thanks for your help

droe added a commit that referenced this issue Aug 23, 2013
@droe
Fix segfault after thread manager start failure
080604e 
This should fix the segmentation fault in issue #10 but not the
underlying reason why the thread manager fails to start in the first
place.

Issue:          #10
Reported by:    linuxton
@linuxton
Copy link
Author

Could I just replace proxy.c and pxythrmgr.c in sslsplit directory with the ones updated by you or should I tar the original archive again and use make with the replaced files?

I tried the former but it still gives segmentation fault issue. Is there anything else I'd log to show you?

Thanks for your time

@droe
Copy link
Owner

droe commented Aug 24, 2013

I suggest you do a git clone of master in order to test the latest code, that is safer than updating only those two files. Can you post the output of bt full in gdb after each segmentation fault you encounter?

droe added a commit that referenced this issue Aug 24, 2013
@droe
Fix off by one error in thrmgr error cleanup code
fc29806 
This should remove another potential source of segmentation faults when
the thread manager fails to start.

Issue:          #10
Reported by:    linuxton
@linuxton
Copy link
Author

Ok this is a -V of latest sslsplit git clone:

./sslsplit -V
SSLsplit fc (built 2013-08-25)
Copyright (c) 2009-2013, Daniel Roethlisberger daniel@roe.ch
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected

This is a try to run it:
./sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
Generated RSA key for leaf certs.
SSLsplit fc (built 2013-08-25)
Copyright (c) 2009-2013, Daniel Roethlisberger daniel@roe.ch
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected
proxyspecs:

  • [0.0.0.0]:8443 ssl plain netfilter
    Loaded CA: '/C=aa/ST=aa/L=aaa/O=aaaa/OU=aaa/CN=aaa/emailAddress=aaa@aaaa.aaa'
    Using libevent backend 'epoll'
    Event base supports: edge yes, O(1) yes, anyfd no
    Inserted events:
    0x8555198 [fd 7] Read Persist
    0x8556ccc [fd 8] Read Persist
    0x8555088 [fd 6] Read Persist
    0x8556d18 [fd 3] Signal Persist
    0x8556e80 [fd 1] Signal Persist
    0x8556f60 [fd 2] Signal Persist
    0x8557040 [fd 13] Signal Persist
    Failed to start thread manager

And this is a gdb run of it:
(gdb) run -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
Starting program: /root/sslsplit-fc/sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Generated RSA key for leaf certs.
SSLsplit fc (built 2013-08-25)
Copyright (c) 2009-2013, Daniel Roethlisberger daniel@roe.ch
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected
proxyspecs:

  • [0.0.0.0]:8443 ssl plain netfilter
    Loaded CA: '/C=aa/ST=aa/L=aaa/O=aaaa/OU=aaa/CN=aaa/emailAddress=aaa@aaaa.aaa'
    Using libevent backend 'epoll'
    Event base supports: edge yes, O(1) yes, anyfd no
    [New Thread 0xb7b87b70 (LWP 3970)]
    [New Thread 0xb7386b70 (LWP 3971)]
    [New Thread 0xb6b85b70 (LWP 3972)]
    Inserted events:
    0x8075160 [fd 11] Read Persist
    0x8077d14 [fd 12] Read Persist
    0x8075050 [fd 10] Read Persist
    0x80751b0 [fd 3] Signal Persist
    0x8077e78 [fd 1] Signal Persist
    0x8077f58 [fd 2] Signal Persist
    0x8078038 [fd 13] Signal Persist
    Failed to start thread manager
    [Thread 0xb7b87b70 (LWP 3970) exited]
    [Thread 0xb7386b70 (LWP 3971) exited]
    [Thread 0xb6b85b70 (LWP 3972) exited]
    [Inferior 1 (process 3967) exited normally]

Using a 'bt full' at this point gives a 'No stack.' message. Should I configure a ulimit number to successfully execute a bt full in gdb after running the program or is it normal since no segmentation fault error was in output?

I ran the previous "standard" 0.4.7 version of sslsplit and did a bt full, thought it may have been useful to you since I forgot to add it last time:

(gdb) run -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
Starting program: /root/sslsplit-0.4.7/sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Generated RSA key for leaf certs.
SSLsplit 0.4.7 (built 2013-08-26)
Copyright (c) 2009-2013, Daniel Roethlisberger daniel@roe.ch
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected
proxyspecs:

  • [0.0.0.0]:8443 ssl plain netfilter
    Loaded CA: '/C=US/ST=Alabama/L=Some/O=Internet Windows Ltd/OU=On/CN=N/emailAddress=internetwindows@internetwindowsltd.com'
    Using libevent backend 'epoll'
    Event base supports: edge yes, O(1) yes, anyfd no
    [New Thread 0xb7b87b70 (LWP 6251)]
    [New Thread 0xb7386b70 (LWP 6252)]
    [New Thread 0xb6b85b70 (LWP 6253)]
    Inserted events:
    0x8075160 [fd 11] Read Persist
    0x8077efc [fd 12] Read Persist
    0x8075050 [fd 10] Read Persist
    0x80751b0 [fd 3] Signal Persist
    0x8078060 [fd 1] Signal Persist
    0x8078140 [fd 2] Signal Persist
    0x8078220 [fd 13] Signal Persist
    Failed to start thread manager

Program received signal SIGSEGV, Segmentation fault.
0xb7c527e1 in free () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
(gdb) bt full
#0 0xb7c527e1 in free () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
No symbol table info available.
#1 0x080590c4 in pxy_thrmgr_free ()
No symbol table info available.
#2 0x08054e44 in proxy_free ()
No symbol table info available.
#3 0x080536b6 in main ()
No symbol table info available.

quit
A debugging session is active.

Inferior 1 [process 6293] will be killed.

Quit anyway? (y or n)

Is it normal that executing two different 'run' in gdb gives different LWP numbers (see above 'run' of latest github clone, LWP 3970/3971/3972 while this log gives LWP 4071/4072/4073)?

(gdb) run -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
Starting program: /root/sslsplit-fc/sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Generated RSA key for leaf certs.
SSLsplit fc (built 2013-08-25)
Copyright (c) 2009-2013, Daniel Roethlisberger daniel@roe.ch
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected
proxyspecs:

  • [0.0.0.0]:8443 ssl plain netfilter
    Loaded CA: '/C=aa/ST=aa/L=aaa/O=aaaa/OU=aaa/CN=aaa/emailAddress=aaa@aaaa.aaa'
    Using libevent backend 'epoll'
    Event base supports: edge yes, O(1) yes, anyfd no
    [New Thread 0xb7b87b70 (LWP 4071)]
    [New Thread 0xb7386b70 (LWP 4072)]
    [New Thread 0xb6b85b70 (LWP 4073)]
    Inserted events:
    0x8075160 [fd 11] Read Persist
    0x8077ccc [fd 12] Read Persist
    0x8075050 [fd 10] Read Persist
    0x8077d18 [fd 3] Signal Persist
    0x8077e80 [fd 1] Signal Persist
    0x8077f60 [fd 2] Signal Persist
    0x8078040 [fd 13] Signal Persist
    Failed to start thread manager
    [Thread 0xb7b87b70 (LWP 4071) exited]
    [Thread 0xb7386b70 (LWP 4072) exited]
    [Thread 0xb6b85b70 (LWP 4073) exited]
    [Inferior 1 (process 4068) exited normally]

Is there anything else I'd log?

Thanks for your time

@edannenberg
Copy link

Same error here, seems to be related to the -j option. If i try without -j it starts just fine.

sslsplit -V
SSLsplit 0.4.7-6-gfc29806 (built 2013-11-21)
Copyright (c) 2009-2013, Daniel Roethlisberger daniel@roe.ch
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.21-stable
rtlinked against libevent 2.0.21-stable
2 CPU cores detected

@exvance
Copy link

exvance commented Dec 2, 2013

[Comment moved to new separate issue #11 since it is possibly unrelated.]

@droe droe mentioned this issue Dec 2, 2013
Closed
@antonywu
Copy link

If I get Failed to start thread manager without Segmentation fault does that mean it is a different issue?

FYI, This is on a RaspberryPi

$ uname -a
Linux chaitea 3.6.11+ #538 PREEMPT Fri Aug 30 20:42:08 BST 2013 armv6l GNU/Linux
$ sudo ./sslsplit -V
SSLsplit 0.4.7-6-gfc29806 (built 2013-12-12)
Copyright (c) 2009-2013, Daniel Roethlisberger <daniel@roe.ch>
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter:  IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected

@droe
Copy link
Owner

droe commented Dec 17, 2013

There are actually two issues; one issue is that the thread manager fails to start (which I cannot reproduce so far), and the other is that sslsniff crashes after the thread manager fails to start (which I can reproduce by simulating a thread manager startup failure). The two commits referenced in this issue fixed many (all?) of potential sources of segmentation faults after thread manager startup failure, so you should not get any segmentation faults anymore if you use latest master. There is no fix for the failing thread manager yet.

@droe
Copy link
Owner

droe commented Dec 23, 2013

If you are using latest the latest git master version and still encounter a segmentation fault following Failed to start thread manager - please attach a stack trace to this issue.

If you are able to reproduce the Failed to start thread manager in a stable way, please attach your operating system details to this issue: uname -a and specific information on the distribution and version. Also, details on the compiler and libc used to build SSLsplit would be great.

@droe
Copy link
Owner

droe commented Jan 13, 2014

I'm closing the issue since I believe the segmentation faults to be fixed. If you encounter any Failed to start thread manager errors in latest master, please open a new ticket.

@droe droe closed this as completed Jan 13, 2014
@droe droe added the released label Apr 23, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@droe droe

Labels
bug released
Projects
None yet
Milestone
0.4.8
Development

No branches or pull requests
5 participants
@antonywu @droe @edannenberg @linuxton @exvance