String conversion while handling native Ruby support lib LoadErrors can cause errors

[dannyfallon]

Issue report

Question 1: What is the problem?

• What is the expected behavior? Failures to load the native support library for Passenger should be rescued and Passenger should attempt to operate normally without the support lib.

• What is the actual behavior? Certain messages for LoadError cannot be converted to US-ASCII and throw a Encoding::UndefinedConversionError error, crashing the application server.

• How can we reproduce it?

Your answer:

When loading the Ruby native support library. Passenger appears to want to rescue LoadError, print out any dlopen errors and ignore the rest of them before returning false and proceeding to run the app without the native support library:

The string conversation to US-ASCII in this error handling block can throw Encoding::UndefinedConversionError which will result in the Passenger not continuing to run. For example:

App 25379 output: /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/native_support.rb:415:in `encode': U+0352 from UTF-8 to US-ASCII (Encoding::UndefinedConversionError)
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/native_support.rb:415:in `rescue in load_native_extension'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/native_support.rb:410:in `load_native_extension'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/native_support.rb:87:in `load_from_buildout_dir'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/native_support.rb:23:in `try_load'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/native_support.rb:34:in `start'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/native_support.rb:429:in `<top (required)>'
App 25379 output: 	from <internal:/apps/muster-internal/releases/muster-internal-55b3ef0aee8d80024ebe778f671054d735192422-2023-12-05-235837/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
App 25379 output: 	from <internal:/apps/muster-internal/releases/muster-internal-55b3ef0aee8d80024ebe778f671054d735192422-2023-12-05-235837/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger.rb:226:in `require_passenger_lib'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/preloader_shared_helpers.rb:13:in `<top (required)>'
App 25379 output: 	from <internal:/apps/muster-internal/releases/muster-internal-55b3ef0aee8d80024ebe778f671054d735192422-2023-12-05-235837/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
App 25379 output: 	from <internal:/apps/muster-internal/releases/muster-internal-55b3ef0aee8d80024ebe778f671054d735192422-2023-12-05-235837/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger.rb:226:in `require_passenger_lib'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/helper-scripts/rack-preloader.rb:41:in `init_passenger'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/helper-scripts/rack-preloader.rb:167:in `<module:App>'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/helper-scripts/rack-preloader.rb:14:in `<module:PhusionPassenger>'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/helper-scripts/rack-preloader.rb:13:in `<main>'
App 25379 output: <internal:/apps/muster-internal/releases/muster-internal-55b3ef0aee8d80024ebe778f671054d735192422-2023-12-05-235837/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require': linked to incompatible h�͒�U - /usr/local/lib/passenger-enterprise-server/buildout/ruby/ruby-3.2.2-x86_64-linux/passenger_native_support.so (LoadError)
App 25379 output: 	from <internal:/apps/muster-internal/releases/muster-internal-55b3ef0aee8d80024ebe778f671054d735192422-2023-12-05-235837/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/native_support.rb:411:in `load_native_extension'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/native_support.rb:87:in `load_from_buildout_dir'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/native_support.rb:23:in `try_load'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/native_support.rb:34:in `start'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/native_support.rb:429:in `<top (required)>'
App 25379 output: 	from <internal:/apps/muster-internal/releases/muster-internal-55b3ef0aee8d80024ebe778f671054d735192422-2023-12-05-235837/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
App 25379 output: 	from <internal:/apps/muster-internal/releases/muster-internal-55b3ef0aee8d80024ebe778f671054d735192422-2023-12-05-235837/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger.rb:226:in `require_passenger_lib'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger/preloader_shared_helpers.rb:13:in `<top (required)>'
App 25379 output: 	from <internal:/apps/muster-internal/releases/muster-internal-55b3ef0aee8d80024ebe778f671054d735192422-2023-12-05-235837/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
App 25379 output: 	from <internal:/apps/muster-internal/releases/muster-internal-55b3ef0aee8d80024ebe778f671054d735192422-2023-12-05-235837/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/ruby_supportlib/phusion_passenger.rb:226:in `require_passenger_lib'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/helper-scripts/rack-preloader.rb:41:in `init_passenger'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/helper-scripts/rack-preloader.rb:167:in `<module:App>'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/helper-scripts/rack-preloader.rb:14:in `<module:PhusionPassenger>'
App 25379 output: 	from /usr/local/lib/passenger-enterprise-server/src/helper-scripts/rack-preloader.rb:13:in `<main>'
[ E 2023-12-05 23:59:33.7051 2777/T6c age/Cor/App/Implementation.cpp:208 ]: Could not spawn process for application /apps/muster-internal/current: The application process exited prematurely.

Our application bundles its own Ruby (3.2.2) which is specified via the nginx option: passenger_ruby /apps/muster-internal/current/bin/ruby_wrapper.sh This script runs bin/slugexec ruby "$@".

When testing the native support library with this version of Ruby:

bin/slugexec ruby -e "require '/usr/local/lib/passenger-enterprise-server/buildout/ruby/ruby-3.2.2-x86_64-linux/passenger_native_support.so'"
I, [2023-12-06T10:27:11.176899 #216771]  INFO -- : Fetching configuration variables for muster-internal at version 20 from S3
<internal:/apps/muster-internal/releases/muster-internal-d0288ae453604566952cbffda2f3cbf52a2c7b7c-2023-12-06-010428/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require': linked to incompatible %��LlU - /usr/local/lib/passenger-enterprise-server/buildout/ruby/ruby-3.2.2-x86_64-linux/passenger_native_support.so (LoadError)
	from <internal:/apps/muster-internal/releases/muster-internal-d0288ae453604566952cbffda2f3cbf52a2c7b7c-2023-12-06-010428/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from -e:1:in `<main>'

Investigation revealed that there is a bug in Ruby 3.2 and lower, fixed in ruby/ruby#7552, which is responsible for producing the mangled character codes that throw this error. The fix is on 3.3 and is not currently backported.

On reproducibility: this failure is not triggered every time I start Passenger with the same version of my application code which makes it hard to reproduce. There are no configuration changes or Ruby version changes between deploys so why this sometimes blows up and sometimes doesn't is a mystery.

Question 2: Passenger version and integration mode:

Your answer: Passenger Enterprise 6.0.18 / nginx 1.24.0

Question 3: OS or Linux distro, platform (including version):

Your answer: Amazon Linux 2023

Question 4: Passenger installation method:

Your answer:
[ ] RubyGems + Gemfile
[x] RubyGems, no Gemfile
[ ] Phusion APT repo
[ ] Phusion YUM repo
[ ] OS X Homebrew
[ ] source tarball
[ ] Other, please specify:

Question 5: Your app's programming language (including any version managers) and framework (including versions):

Your answer:

System Ruby is 3.2.2.
Application is 3.2.2, but using a bundled version of Ruby (via Heroku Ruby Buildpack).

Question 6: Are you using a PaaS and/or containerization? If so which one?

Your answer: Amazon EC2 / t3.medium

Question 7: Anything else about your setup that we should know?

When using system Ruby (coincidentally also 3.2.2), no LoadError is thrown:

[ec2-user@muster-internal-production-internal-web current]$ ruby -v
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]

[ec2-user@muster-internal-production-internal-web current]$ which ruby
/usr/bin/ruby

[ec2-user@muster-internal-production-internal-web current]$ ruby -e "require '/usr/local/lib/passenger-enterprise-server/buildout/ruby/ruby-3.2.2-x86_64-linux/passenger_native_support.so'"

I can surmise that this version of the native support library was built out using system ruby at some point rather than my application's Ruby.

[dannyfallon]

This is a follow up to #2479

Eventually the exception doesn't have any invalid byte sequences in it and passenger starts the application normally and carries on normally until the next web server (Apache) restart. This is random though.

This explains why it works eventually/is intermittent for me.