Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Segmentation fault with Ruby Enterprise Edition 2011.12 #81

Open
stonegao opened this Issue · 26 comments
@stonegao

God works perfectly with previous ree versions, we have been running it for almost 2 years without problem.

After upgrading ree to ree-1.8.7-2011.12, got the following error, and god process itself disappears after starting monitored processes.

os is ubuntu Ubuntu 10.04.3 LTS

/usr/local/rvm/rubies/ree-1.8.7-2011.12/lib/ruby/1.8/monitor.rb:173: [BUG] Segmentation fault
ruby 1.8.7 (2011-12-28 MBARI 8/0x6770 on patchlevel 357) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2011.12
@jasonroelofs

Specifically for me, we upgraded to 2011.12 just fine 32-bit and it was working without any problems. I recently moved to a 64-bit box, 2011.12, and now I'm getting this segfault.

@stcatz

I've got this problem too, have you found some workaround of this issue ?

@dudupeters

I'm with the same problem. Any idea?

I [2012-02-24 17:17:16]  INFO: Gateway staging-10000 [ok] process is running (ProcessRunning)
I [2012-02-24 17:17:17]  INFO: Gateway staging-10000 [ok] memory within bounds [38992kb, 38992kb, 38992kb, 38992kb, 38992kb] (MemoryUsage)
I [2012-02-24 17:17:17]  INFO: Gateway staging-10000 [ok] cpu within bounds [1.75055938927828%%, 1.75660823724432%%, 1.74224387723063%%, 1.72631347950894%%, 1.73060544008095%%] (CpuUsage)
I [2012-02-24 17:17:18]  INFO: Gateway staging-10000 [ok] (RestartFileTouched)
I [2012-02-24 17:17:21]  INFO: Gateway staging-10000 [ok] process is running (ProcessRunning)
I [2012-02-24 17:17:22]  INFO: Gateway staging-10000 [ok] memory within bounds [38992kb, 38992kb, 38992kb, 38992kb, 38992kb] (MemoryUsage)
/home/xxxx/.rvm/rubies/ree-1.8.7-2012.01/lib/ruby/1.8/monitor.rb:173: [BUG] Segmentation fault
ruby 1.8.7 (2011-12-28 MBARI 8/0x8770 on patchlevel 357) [i686-linux], MBARI 0x8770, Ruby Enterprise Edition 2012.01
@jasonroelofs

REE 2012.02 was just released which claims to have a much more stable set of MBARI patches. I've yet to try it but it may fix this segfault issue.

@xavier

REE 2012.02 doesn't fix the issue on Debian running 2.6.38.2 (x86_64)

@jaredonline

Same issue with ruby 1.8.7 (2012-02-08 MBARI 8/0x6770 on patchlevel 358) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2012.02

@gbc-pfischer

same issue here:

${HOME}/.rvm/rubies/ree-1.8.7-2011.12/lib/ruby/1.8/monitor.rb:173: [BUG] Segmentation fault

ruby version (64bit):

ruby 1.8.7 (2011-12-28 MBARI 8/0x6770 on patchlevel 357) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2011.12

on Scientific Linux release 6.0 (Carbon) with 64bit (SELINUX disabled)

@dovadi

Same issue here:

/usr/local/lib/ruby/1.8/monitor.rb:173: [BUG] Segmentation fault
ruby 1.8.7 (2012-02-08 MBARI 8/0x6770 on patchlevel 358) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2012.02

on Ubuntu 10.04.4 LTS

@jasonroelofs

So as it stands, I'm no longer using god at all but have instead switched to Upstart.

@ahwatts

I'm seeing this issue, too. I have a cron job which runs periodically to check to see if god is running, and it seems to have to restart god every few hours.

@ahwatts

I'm also on REE 2012.02, on Fedora 8 and 11:

Sometimes I see this message:

/usr/lib/ruby/1.8/monitor.rb:173: [BUG] gc_sweep(): unknown data type 0x0(0x12efe18)
ruby 1.8.7 (2012-02-08 MBARI 8/0x6770 on patchlevel 358) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2012.02

and ometimes I see this message:

/usr/lib/ruby/1.8/monitor.rb:173: [BUG] Segmentation fault
ruby 1.8.7 (2012-02-08 MBARI 8/0x6770 on patchlevel 358) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2012.02
@drywheat

Experiencing same issue on Centos 5.4 ( 2.6.21.7-2.fc8xen x86_64 ) using REE 2012.02

@andresbravog

is anyone debugging the error? so we can try to solve the issue? ...

one solution that may work is to use rvm or another tool to define which ruby you use to execute god, so you can use ruby 1.9.2 for god and ree for the other things.

update: it worked for me to use 1.8 debian system ruby for god, in our linode debian server. ;)

@jasonroelofs

The only thing I've been able to find is that other libraries are having this same issue:

https://github.com/fastestforward/instrumental_agent/wiki/Using-with-Ruby-Enterprise-Edition

They claim to have fixed it, maybe someone can tease out the fix from their commit log?

@cromulus

having the same problem. gonna try using a different ruby

@donovanbray

I'm also having the same issue on 3 boxes, running ubuntu 10.04

/custom/ree/lib/ruby/1.8/monitor.rb:173: [BUG] Segmentation fault
ruby 1.8.7 (2012-02-08 MBARI 8/0x6770 on patchlevel 358) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2012.02

@donovanbray

we've also seen the gc_sweep error

/custom/ree/lib/ruby/1.8/monitor.rb:173: [BUG] gc_sweep(): unknown data type 0x0(0x34a6458)
ruby 1.8.7 (2012-02-08 MBARI 8/0x6770 on patchlevel 358) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2012.02

@hui
hui commented

having the same issue:

hui@ubuntu:/data/god/god_config$ tail -f /var/log/god.log
I [2012-07-07 09:00:11] INFO: Syslog enabled.
I [2012-07-07 09:00:11] INFO: Using pid file directory: /var/run/god
/home/hui/.rvm/rubies/ree-1.8.7-2012.02/lib/ruby/1.8/monitor.rb:173: [BUG] Segmentation fault
ruby 1.8.7 (2012-02-08 MBARI 8/0x8770 on patchlevel 358) [i686-linux], MBARI 0x8770, Ruby Enterprise Edition 2012.02

@tonybyrne

Oh the irony! I've just installed god to keep an eye on a service that periodically segfaults on me and now I find I'm bitten by the REE bug.

@donovanbray

I haven't been able to fix the issue; but I made a temporary bandaid using upstart. It will at least restart god when it segfaults.

put this (with your correct paths) in /etc/init/god.conf (assuming your on a dist with upstart or have otherwise installed it)

description "Ruby God Monitor"
author "Donovan Bray donnoman@donovanbray.com"


# automatically start
start on (hostname and syslog)

stop on runlevel [016]

# Run before process
pre-start script
  mkdir -p `dirname /var/www/application/shared/log/god.log`
  mkdir -p `dirname /var/www/application/shared/pids/god.pid`
end script

# Essentially lets upstart know the process will detach itself to the background
expect fork
kill timeout 10
respawn

# command to run
exec /opt/ree/bin/god -c /var/www/application/current/config/daemons.god -P /var/www/application/shared/pids/god.pid --log-level info --log /var/www/application/shared/log/god.log

# Run after process (this allows any user to issue god commands; remove it if you don't want it)
post-start script
    sleep 3 && sh -c "chmod 0777 /tmp/god.*.sock;true"
end script
@eric
Collaborator

I've ran into this as well. I'll see what I can dig up...

@zekefast

Have the error like this, but I don't use god.
Error:
$ gem install bundler
/home/zekefast/.rvm/rubies/ree-head/lib/ruby/1.8/timeout.rb:60: [BUG] Segmentation fault
ruby 1.8.7 (2012-02-08 patchlevel 358) [x86_64-linux]

Same for 2012.02. I use RVM.
$ gcc --version
gcc-4.7.real (Debian 4.7.1-2) 4.7.1
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Able to run ree without this error when install it like this: CC=/usr/bin/gcc-4.4 rvm install ree

Possibly, it could help someone else ...

@vakuum

Same error on SUSE Linux Enterprise Server 11 SP2 and Ruby Enterprise Edition 2012.02 with god 0.12.1 on x86_64:

...
/opt/erec/ruby-enterprise-1.8.7-2012.02/lib/ruby/1.8/monitor.rb:173: [BUG] Bus Error
ruby 1.8.7 (2012-02-08 MBARI 8/0x6770 on patchlevel 358) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2012.02
...
/opt/erec/ruby-enterprise-1.8.7-2012.02/lib/ruby/1.8/monitor.rb:173: [BUG] Segmentation fault
ruby 1.8.7 (2012-02-08 MBARI 8/0x6770 on patchlevel 358) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2012.02
...

Maybe this has something to do with an already fixed problem:

Ruby Enterprise Edition 1.8.7-2010.01 released:

Fix a crash bug in the zero-copy context switching patch set

This crash can be reproduced by running "god", which will eventually cause a crash. Aman Gupta has fixed this problem.

Please note that the zero-copy context switching patch set is disabled by default, and must be explicitly enabled by passing -fast-threading to the installer. It is currently still marked as experimental because there are some known issues with the Kernel::fork method. Issue #9.

Ruby Enterprise Edition 1.8.7-2012.02 released:

Experimental zero-copy context switch patch removed

This experimental patch set was never production-ready, so as of this release it has been removed.

@bonyiii

This worked for me (on opensuse 12.2): http://deadc.org/blog/2012/10/19/rvm-install-ruby-1-dot-8-7-with-gcc-4-dot-7/

rvm remove ree
export CFLAGS="-O2 -fno-tree-dce -fno-optimize-sibling-calls"
rvm install ree
@mikhailov

any ideas howto fix that bug? should we use just a standard Ruby with God monitoring?

@mikhailov

fixed by installing non-enterprise Ruby edition

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.