seg faults on 'general protection' using AMI linux

[kylezeeuwen]

Hello,

I am using toobusy v 0.2.4 in an express server running on AMI linux and getting repeatable segmentation faults. The only non standard thing I am doing is calling toobusy.lag() in a periodic loop (once every 5 seconds currently) to print out some instantaneous metrics.

I observe crashes every ~ 90 seconds under no load, I stop the server, I remove all references to toobusy from the app, I start the server , I observe no crashes, I stop the server, I add back references, then observe more crashes. Pretty sure it's toobusy.

I completely remove express and created a script to isolate the problem:

[ec2-user@ip-10-250-202-250 html]$ cat api/server.minimal.js 
// Generated by CoffeeScript 1.8.0
(function() {
  'use strict';
  var toobusy;

  toobusy = require('toobusy');

  setInterval(function() {
    var lag;
    lag = toobusy.lag();
    return console.log("lag is " + lag);
  }, 5000);

}).call(this);

Here is the output:

[ec2-user@ip-10-250-202-250 html]$ node api/server.minimal.js 
lag is 0
lag is 0
lag is 0
lag is 0
lag is 0
lag is 0
lag is 0
lag is 0
lag is 0
lag is 0
lag is 0
lag is 0
lag is 0
lag is 0
Segmentation fault

I deleted node_modules/toobusy and reinstalled via npm install toobusy on the box producing the seg faults to make sure this wasnt an issue of compiling on different hardware.

dmesg shows the following output when the crash occurs (one per crash):

[102094.365224] traps: node[31832] general protection ip:7f6d55bf123b sp:7fff5d4de8d0 error:0 in libuv.so.0.10[7f6d55be0000+1a000]
[102179.755771] traps: node[31850] general protection ip:7f8411e6423b sp:7fffaa71dba0 error:0 in libuv.so.0.10[7f8411e53000+1a000]
[102514.037806] traps: node[31929] general protection ip:7fc7fe54823b sp:7ffff1c10280 error:0 in libuv.so.0.10[7fc7fe537000+1a000]
[102680.646695] traps: node[31990] general protection ip:7fbbe813223b sp:7ffffd1fb840 error:0 in libuv.so.0.10[7fbbe8121000+1a000]

this is the uname -a printout:

[ec2-user@ip-10-250-202-250 html]$ uname -a
Linux ip-10-250-202-250 3.14.26-24.46.amzn1.x86_64 #1 SMP Wed Dec 10 10:02:43 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

node version:

[root@ip-10-250-202-250 html]# node --version
v0.10.33

Here is an ldd of the build binary:

[ec2-user@ip-10-250-202-250 html]$ ldd node_modules/toobusy/build/Release/toobusy.node 
    linux-vdso.so.1 =>  (0x00007fff30977000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fa359f20000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fa359c22000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fa359a0b000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa3597ef000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fa35944a000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa35a42d000)

Some node-gyp / g++ / gcc info (sorry i dont know how bindings + node-gyp works):

[ec2-user@ip-10-250-202-250 html]$ node-gyp --version
v0.10.6
[ec2-user@ip-10-250-202-250 html]$ g++ --version
g++ (GCC) 4.8.2 20140120 (Red Hat 4.8.2-16)
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[ec2-user@ip-10-250-202-250 html]$ gcc --version
gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-16)
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Here's the end of the strace for when it crashes (nothing good here):

epoll_wait(5, {}, 1024, 500)            = 0
clock_gettime(CLOCK_MONOTONIC, {103259, 202722380}) = 0
clock_gettime(CLOCK_MONOTONIC, {103259, 202750468}) = 0
clock_gettime(CLOCK_MONOTONIC, {103259, 202806928}) = 0
epoll_wait(5, {}, 1024, 500)            = 0
clock_gettime(CLOCK_MONOTONIC, {103259, 703592252}) = 0
clock_gettime(CLOCK_MONOTONIC, {103259, 703619899}) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=0} ---
+++ killed by SIGSEGV +++
Segmentation fault

Some attempt at GDB:

[ec2-user@ip-10-250-202-250 html]$ gdb node
GNU gdb (GDB) Amazon Linux (7.6.1-51.24.amzn1)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-amazon-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/node...(no debugging symbols found)...done.
Missing separate debuginfos, use: debuginfo-install nodejs-0.10.33-1.el6.x86_64
(gdb) set args api/server.minimal.js
(gdb) r
Starting program: /usr/bin/node api/server.minimal.js
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff7fe4700 (LWP 32326)]
lag is 0
lag is 0
lag is 0

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff75a923b in ?? () from /usr/lib64/libuv.so.0.10
(gdb) bt
#0  0x00007ffff75a923b in ?? () from /usr/lib64/libuv.so.0.10
#1  0x00007ffff75a00ab in uv_run () from /usr/lib64/libuv.so.0.10
#2  0x0000000000422f22 in node::Start(int, char**) ()
#3  0x00007ffff59b97d5 in __libc_start_main () from /lib64/libc.so.6
#4  0x0000000000419489 in _start ()

[asilvas]

https://github.com/STRML/node-toobusy

[kylezeeuwen]

@asilvas thanks for the suggestion I am trialing it out and looks good