Segmentation fault with rails

[aliismayilov]

Describe the bug
When using puma as rails server, I get segmentation fault. My app works fine with other web servers (webrick and thin tested).

Puma config:

# config/puma.rb
workers Integer(ENV['WEB_CONCURRENCY'] || 4)
threads_count = Integer(ENV['RAILS_MAX_THREADS'] || 5)
threads threads_count, threads_count

port        ENV['PORT']     || 3000
environment ENV['RACK_ENV'] || 'development'

preload_app!

on_worker_boot do
  # Worker specific setup for Rails 4.1+
  # See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
  ActiveRecord::Base.establish_connection
end

To Reproduce

Sadly, I can't share much of the codebase. The app is started with the following command:

$ rails s

It starts fine, but upon receiving any request crashes. Full crash report can be found here: https://gist.github.com/aliismayilov/bbfd82826202dc7e8c7b9913b5973799

Expected behavior
Serve requests without crashing.

Desktop (please complete the following information):

• OS: macOS Ventura 13.4 (22F66)
• Puma version: 6.3.0
• Ruby version: 3.2.2

Please let me know if I can provide more insight.

[dentarg]

This is not a Puma issue. The segfault is happening in the pg gem.

[aliismayilov]

@dentarg you are right, it does have something to do with pg gem.

Here's a small reproducible crash:

# config.ru
require "pg"

run Proc.new { |env|
  PG.connect(host: "localhost", user: "postgres", dbname: "testdb")

  ["200", {"Content-Type" => "text/plain"}, ["Hello World"]]
}

# Gemfile.lock
GEM
  remote: https://rubygems.org/
  specs:
    nio4r (2.5.9)
    pg (1.5.3)
    puma (6.3.0)
      nio4r (~> 2.0)

PLATFORMS
  arm64-darwin-22

DEPENDENCIES
  pg (~> 1.5)
  puma (~> 6.3)

BUNDLED WITH
   2.4.13

When I start puma with bundle exec puma -w 0, then it works fine. When I start with non-zero worker count it crashes as soon as it receives request:

$ bundle exec puma -w 2
[69889] Puma starting in cluster mode...
[69889] * Puma version: 6.3.0 (ruby 3.2.2-p53) ("Mugi No Toki Itaru")
[69889] *  Min threads: 0
[69889] *  Max threads: 5
[69889] *  Environment: development
[69889] *   Master PID: 69889
[69889] *      Workers: 2
[69889] *     Restarts: (✔) hot (✔) phased
[69889] * Listening on http://0.0.0.0:9292
[69889] Use Ctrl-C to stop
[69889] - Worker 0 (PID: 70021) booted in 0.01s, phase: 0
[69889] - Worker 1 (PID: 70022) booted in 0.01s, phase: 0
objc[70021]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[70021]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[70022]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[70022]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
[69889] - Worker 0 (PID: 70025) booted in 0.03s, phase: 0
[69889] - Worker 1 (PID: 70026) booted in 0.03s, phase: 0
objc[70025]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[70025]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[70026]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[70026]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
[69889] - Worker 0 (PID: 70027) booted in 0.02s, phase: 0
[69889] - Worker 1 (PID: 70028) booted in 0.02s, phase: 0
objc[70027]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[70027]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[70028]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[70028]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
[69889] - Worker 0 (PID: 70029) booted in 0.02s, phase: 0
[69889] - Worker 1 (PID: 70030) booted in 0.02s, phase: 0
objc[70029]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[70029]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[70030]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[70030]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
[69889] - Worker 0 (PID: 70031) booted in 0.02s, phase: 0
[69889] - Worker 1 (PID: 70032) booted in 0.01s, phase: 0
objc[70031]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[70031]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[70032]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[70032]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
[69889] - Worker 0 (PID: 70033) booted in 0.02s, phase: 0
[69889] - Worker 1 (PID: 70034) booted in 0.02s, phase: 0

[dentarg]

I think you need to disconnect from the database before Puma forks (starts workers) You can do that in before_fork.

Should be lots of docs on this in the various projects (active record, pg, rails ...)

[MSP-Greg]

See #3122. Your logs indicate that Puma is calling fork before all the threads setting up your pg database and/or connections are finished.

I haven't checked for a while, but I believe both webrick and thin do not make use of fork to increase server capacity/speed.

[aliismayilov]

@dentarg @MSP-Greg thanks for taking time to look into this! I'm still not sure where to put my finger at 😕 I just created a new rails app with postgres database and I get the above crash when starting the server with non-zero worker count.

On the other hand, if I put this code into a Docker container I get no crash:

# config.ru
require "pg"

run Proc.new { |env|
  # connect to host postgres instance on the host machine
  PG.connect(host: "host.docker.internal", user: "postgres", dbname: "testdb")

  ["200", {"Content-Type" => "text/plain"}, ["Hello World"]]
}

# Dockerfile
# docker build -t puma_crash .
# docker run -v $(pwd):/app -p 9292:9292 -it puma_crash puma -w 2

FROM ruby:3.2.2-alpine

RUN apk --update --no-cache add build-base postgresql-dev

WORKDIR /app

COPY Gemfile Gemfile.lock .
RUN bundle install

[dentarg]

I experienced this problem myself later on, how I worked around it in the tests (that actually boots Puma) in one of my apps: Starkast/wikimum@e256d61 – see the links in that commit message for more info

This is something that happens on macOS and has surfaced again because the krb5 formula in Homebrew was bumped to 1.21 with krb5/krb5#1221 (what ged/ruby-pg#311 (comment) says)

[dentarg]

See ged/ruby-pg#311 (comment) for more info and possible workarounds