New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

native_support auto-build race condition #1570

Closed
bensie opened this Issue Aug 3, 2015 · 10 comments

Comments

Projects
None yet
5 participants
@bensie
Contributor

bensie commented Aug 3, 2015

Since updating to 5.0.15 (ubuntu trusty apt package), getting the following error:

App 1654 stderr: Passenger RubyApp: /mnt/apps/myapp/current/public (production): symbol lookup error: /home/deploy/.passenger/native_support/5.0.15/ruby-2.2.2-x86_64-linux/passenger_native_support.so: undefined symbol: rb_thread_blocking_region

Let me know if you need additional info!

@FooBarWidget

This comment has been minimized.

Show comment
Hide comment
@FooBarWidget

FooBarWidget Aug 3, 2015

Member

It looks like your native_support wasn't being compiled properly. Try recompiling it:

rm -f /home/deploy/.passenger/native_support/5.0.15/ruby-2.2.2-x86_64-linux/passenger_native_support.so
passenger-config build-native-support
Member

FooBarWidget commented Aug 3, 2015

It looks like your native_support wasn't being compiled properly. Try recompiling it:

rm -f /home/deploy/.passenger/native_support/5.0.15/ruby-2.2.2-x86_64-linux/passenger_native_support.so
passenger-config build-native-support
@bensie

This comment has been minimized.

Show comment
Hide comment
@bensie

bensie Aug 3, 2015

Contributor

That did the trick. At what point does that compilation happen? I upgraded to 5.0.15 with apt the same way as usual...

Contributor

bensie commented Aug 3, 2015

That did the trick. At what point does that compilation happen? I upgraded to 5.0.15 with apt the same way as usual...

@FooBarWidget

This comment has been minimized.

Show comment
Hide comment
@FooBarWidget

FooBarWidget Aug 3, 2015

Member

It happens when your app is first executed, and Passenger detects that there is no native_support compiled for the current combination of Passenger version and Ruby version.

Are you using Apache or Nginx? What does your PassengerRuby/passenger_ruby option look like?

Member

FooBarWidget commented Aug 3, 2015

It happens when your app is first executed, and Passenger detects that there is no native_support compiled for the current combination of Passenger version and Ruby version.

Are you using Apache or Nginx? What does your PassengerRuby/passenger_ruby option look like?

@bensie

This comment has been minimized.

Show comment
Hide comment
@bensie

bensie Aug 3, 2015

Contributor

Using Nginx, with multiple apps running on each node. When adding the node back to the load balancer after the update, it can get pretty slammed with initial requests. Maybe a race condition where 2 different apps didn't see native_support compiled and they both attempted it and 💥?

passenger_ruby /opt/rubies/2.2.2/bin/ruby

Contributor

bensie commented Aug 3, 2015

Using Nginx, with multiple apps running on each node. When adding the node back to the load balancer after the update, it can get pretty slammed with initial requests. Maybe a race condition where 2 different apps didn't see native_support compiled and they both attempted it and 💥?

passenger_ruby /opt/rubies/2.2.2/bin/ruby

@FooBarWidget

This comment has been minimized.

Show comment
Hide comment
@FooBarWidget

FooBarWidget Aug 24, 2015

Member

I can't see how a race condition would trigger this error, but I admit I haven't quite tested that case, so it's worth investigating.

Member

FooBarWidget commented Aug 24, 2015

I can't see how a race condition would trigger this error, but I admit I haven't quite tested that case, so it's worth investigating.

@bensie

This comment has been minimized.

Show comment
Hide comment
@bensie

bensie Sep 6, 2015

Contributor

FYI - The same thing happened on the upgrade from 5.0.15 to 5.0.16. Had to manually compile native_support on all web servers and restart Nginx.

Contributor

bensie commented Sep 6, 2015

FYI - The same thing happened on the upgrade from 5.0.15 to 5.0.16. Had to manually compile native_support on all web servers and restart Nginx.

@felixbuenemann

This comment has been minimized.

Show comment
Hide comment
@felixbuenemann

felixbuenemann Sep 7, 2015

Maybe the environment the app server runs in picks up a different ruby version?

If the problem occurs on the next upgrade, capture the output of

ldd /home/deploy/.passenger/native_support/5.0.17/ruby-2.2.2-x86_64-linux/passenger_native_support.so

before doing the manual recompile and post it here.

It should show you which version of libruby it was compiled against. The missing symbol points to something pre ruby 2.2.

felixbuenemann commented Sep 7, 2015

Maybe the environment the app server runs in picks up a different ruby version?

If the problem occurs on the next upgrade, capture the output of

ldd /home/deploy/.passenger/native_support/5.0.17/ruby-2.2.2-x86_64-linux/passenger_native_support.so

before doing the manual recompile and post it here.

It should show you which version of libruby it was compiled against. The missing symbol points to something pre ruby 2.2.

@NeoPhi

This comment has been minimized.

Show comment
Hide comment
@NeoPhi

NeoPhi Sep 25, 2015

I had this same issue happen on an upgrade to 5.0.19. Definitely a race condition as you can see overlapping compiles in the nginx/error.log:

App 71010 stdout:
App 71010 stderr:  --> Compiling passenger_native_support.so for the current Ruby interpreter...
App 71010 stderr:      (set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable
)
App 71118 stdout:
App 71118 stderr:  --> Compiling passenger_native_support.so for the current Ruby interpreter...
App 71118 stderr:
App 71118 stderr:      (set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable)
App 71118 stderr:      Compilation succesful. The logs are here:
App 71118 stderr:      /tmp/passenger_native_support-1729nsa.log
App 71118 stderr: ruby: symbol lookup error: /home/web/.passenger/native_support/5.0.19/ruby-2.0.0-x86_64-linux/passenger_native_support.so: undefined symbol: rb_defi
App 71010 stderr:      Compilation succesful. The logs are here:
App 71010 stderr:      /tmp/passenger_native_support-1m4cu8g.log
App 71010 stderr: ruby: symbol lookup error: /home/web/.passenger/native_support/5.0.19/ruby-2.0.0-x86_64-linux/passenger_native_support.so: undefined symbol: rb_defi

NeoPhi commented Sep 25, 2015

I had this same issue happen on an upgrade to 5.0.19. Definitely a race condition as you can see overlapping compiles in the nginx/error.log:

App 71010 stdout:
App 71010 stderr:  --> Compiling passenger_native_support.so for the current Ruby interpreter...
App 71010 stderr:      (set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable
)
App 71118 stdout:
App 71118 stderr:  --> Compiling passenger_native_support.so for the current Ruby interpreter...
App 71118 stderr:
App 71118 stderr:      (set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable)
App 71118 stderr:      Compilation succesful. The logs are here:
App 71118 stderr:      /tmp/passenger_native_support-1729nsa.log
App 71118 stderr: ruby: symbol lookup error: /home/web/.passenger/native_support/5.0.19/ruby-2.0.0-x86_64-linux/passenger_native_support.so: undefined symbol: rb_defi
App 71010 stderr:      Compilation succesful. The logs are here:
App 71010 stderr:      /tmp/passenger_native_support-1m4cu8g.log
App 71010 stderr: ruby: symbol lookup error: /home/web/.passenger/native_support/5.0.19/ruby-2.0.0-x86_64-linux/passenger_native_support.so: undefined symbol: rb_defi

@FooBarWidget FooBarWidget changed the title from Passenger 5.0.15 error to native_support auto-build race condition Sep 25, 2015

@FooBarWidget FooBarWidget added this to the 5.0.22 milestone Sep 25, 2015

@FooBarWidget

This comment has been minimized.

Show comment
Hide comment
@FooBarWidget

FooBarWidget Nov 19, 2015

Member

The problem has now been fixed.

Member

FooBarWidget commented Nov 19, 2015

The problem has now been fixed.

@FooBarWidget FooBarWidget self-assigned this Nov 19, 2015

@bensie

This comment has been minimized.

Show comment
Hide comment
@bensie

bensie Nov 19, 2015

Contributor

Thanks @FooBarWidget!

Contributor

bensie commented Nov 19, 2015

Thanks @FooBarWidget!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment