Reactor suddenly throws NoMethodError but can't find reason

[taeyeob]

Describe the bug
A clear and concise description of what the bug is.

I use capistrano as a deploying method, and this is my deploy script:
bundle exec puma -C config/puma.rb -e production > logs/puma.log 2>&1 &
I started to use this method when I've upgraded my app ruby version from 2.5.3 to 3.2.2, and before then, I used '-d' option to daemonize.
After ruby version upgrade and use the build script, it works well in many times but sometimes I get this kind of errors and the server stops running.

Error in reactor loop escaped: undefined method `<' for nil:NilClass (NoMethodError)
/home/ubuntu/www/jj-server/shared/bundle/ruby/3.2.0/gems/puma-6.4.0/lib/puma/request.rb:607:in `str_headers'
/home/ubuntu/www/jj-server/shared/bundle/ruby/3.2.0/gems/puma-6.4.0/lib/puma/request.rb:172:in `prepare_response'
/home/ubuntu/www/jj-server/shared/bundle/ruby/3.2.0/gems/puma-6.4.0/lib/puma/server.rb:550:in `response_to_error'
/home/ubuntu/www/jj-server/shared/bundle/ruby/3.2.0/gems/puma-6.4.0/lib/puma/server.rb:516:in `client_error'
/home/ubuntu/www/jj-server/shared/bundle/ruby/3.2.0/gems/puma-6.4.0/lib/puma/server.rb:303:in `rescue in reactor_wakeup'
/home/ubuntu/www/jj-server/shared/bundle/ruby/3.2.0/gems/puma-6.4.0/lib/puma/server.rb:292:in `reactor_wakeup'
/home/ubuntu/www/jj-server/shared/bundle/ruby/3.2.0/gems/puma-6.4.0/lib/puma/server.rb:244:in `block in run'
/home/ubuntu/www/jj-server/shared/bundle/ruby/3.2.0/gems/puma-6.4.0/lib/puma/reactor.rb:119:in `wakeup!'
/home/ubuntu/www/jj-server/shared/bundle/ruby/3.2.0/gems/puma-6.4.0/lib/puma/reactor.rb:76:in `block in select_loop'
/home/ubuntu/www/jj-server/shared/bundle/ruby/3.2.0/gems/puma-6.4.0/lib/puma/reactor.rb:76:in `select'
/home/ubuntu/www/jj-server/shared/bundle/ruby/3.2.0/gems/puma-6.4.0/lib/puma/reactor.rb:76:in `select_loop'
/home/ubuntu/www/jj-server/shared/bundle/ruby/3.2.0/gems/puma-6.4.0/lib/puma/reactor.rb:39:in `block in run'

And then, I restart the server by this command:
bundle exec pumactl -S tmp/pids/puma.state phased-restart
Then it works for few hours, and the same thing repeats again.

Somebody please save me from this never ending disaster.

Puma config:

Please copy-paste your Puma config AND your command line options here.

threads_min_count = ENV.fetch('RAILS_MIN_THREADS') { 0 }
threads_max_count = ENV.fetch('RAILS_MAX_THREADS') { 32 }
env = ENV.fetch('RAILS_ENV') { 'development' }

threads threads_min_count, threads_max_count

port ENV.fetch('PORT') { 3030 }

# development / staging / production
environment env

workers ENV.fetch('WEB_CONCURRENCY') { 1 } unless RUBY_PLATFORM =~ /mingw/

pidfile 'tmp/pids/puma.pid'
state_path 'tmp/pids/puma.state'

lowlevel_error_handler do |ex, env|
  Sentry.capture_exception(
    ex,
    :message => ex.message,
    :extra => { :puma => env },
    :transaction => "Puma"
  )
end

bind 'unix:///tmp/puma.sock' unless RUBY_PLATFORM =~ /mingw/

prune_bundler

# on_worker_boot do
#   $hackle_client = Hackle.client(sdk_key: ENV['HACKLE_SDK_KEY'])
# end

To Reproduce
This is my output(and it worked well):

bundle exec puma -C config/puma.rb hello.ru
[90317] * Pruning Bundler environment
[90317] Puma starting in cluster mode...
[90317] * Puma version: 6.4.0 (ruby 3.2.2-p53) ("The Eagle of Durango")
[90317] *  Min threads: 0
[90317] *  Max threads: 32
[90317] *  Environment: development
[90317] *   Master PID: 90317
[90317] *      Workers: 1
[90317] *     Restarts: (✔) hot (✔) phased
[90317] * Listening on http://0.0.0.0:3030
[90317] * Listening on unix:///tmp/puma.sock
[90317] Use Ctrl-C to stop
[90317] ! WARNING: Detected running cluster mode with 1 worker.
[90317] ! Running Puma in cluster mode with a single worker is often a misconfiguration.
[90317] ! Consider running Puma in single-mode (workers = 0) in order to reduce memory overhead.
[90317] ! Set the `silence_single_worker_warning` option to silence this warning message.
[90334] + Gemfile in context: /Users/vb/Desktop/VB/jj-server/jj-server/Gemfile
[90317] - Worker 0 (PID: 90334) booted in 0.39s, phase: 0

Expected behavior
A clear and concise description of what you expected to happen.

Desktop (please complete the following information):

• OS: [e.g. Mac, Linux] Mac
• Puma Version [e.g. 4.1.1] 6.4.0
• Ruby Version: 3.2.2
• Rails Version: 7.1.1

[MSP-Greg]

The error is in the line shown below, which indicates that the status being returned by your app is nil. The rack spec states that
"It must be an Integer greater than or equal to 100."

It seems the error is in the lowlevel_error_handler you've shown above. It must return a valid response.

puma/lib/puma/request.rb

Line 607 in cdb2341

resp_info[:no_body] ||= status < 200 || STATUS_WITH_NO_ENTITY_BODY[status]

[dentarg]

Yes, since #3094 (Puma v6.4.0) the result of the lowlevel_error_handler is sent as a response to the client.

By the way, if you use a recent enough version of sentry-ruby, it will automatically monkey-patch Puma to capture the low-level errors.

[dentarg]

By the way, if you use a recent enough version of sentry-ruby, it will automatically monkey-patch Puma to capture the low-level errors.

So you don't need to capture the exception yourself in that case.

[taeyeob]

By the way, if you use a recent enough version of sentry-ruby, it will automatically monkey-patch Puma to capture the low-level errors.

So you don't need to capture the exception yourself in that case.

The error is in the line shown below, which indicates that the status being returned by your app is nil. The rack spec states that "It must be an Integer greater than or equal to 100."

It seems the error is in the lowlevel_error_handler you've shown above. It must return a valid response.

puma/lib/puma/request.rb

Line 607 in cdb2341

resp_info[:no_body] ||= status < 200 || STATUS_WITH_NO_ENTITY_BODY[status]

Thanks to all :) It was my only experience to upgrade ruby all the way down from 2.5 to 3.2, so I didn't get a notice on what should I do to prepare for the change.
With your advise, I have commented out the lowlevel_error_handler part and this is my final output:

threads_min_count = ENV.fetch('RAILS_MIN_THREADS') { 0 }
threads_max_count = ENV.fetch('RAILS_MAX_THREADS') { 32 }
env = ENV.fetch('RAILS_ENV') { 'development' }

threads threads_min_count, threads_max_count

port ENV.fetch('PORT') { 3030 }

# development / staging / production
environment env

workers ENV.fetch('WEB_CONCURRENCY') { 1 } unless RUBY_PLATFORM =~ /mingw/

pidfile 'tmp/pids/puma.pid'
state_path 'tmp/pids/puma.state'

# lowlevel_error_handler do |ex, env, status|
#   Sentry.capture_exception(
#     ex,
#     :message => ex.message,
#     :extra => { :puma => env },
#     :transaction => "Puma"
#   )
# end

bind 'unix:///tmp/puma.sock' unless RUBY_PLATFORM =~ /mingw/

prune_bundler

# on_worker_boot do
#   $hackle_client = Hackle.client(sdk_key: ENV['HACKLE_SDK_KEY'])
# end

Just to check for any misunderstandings that might have arisen from my poor English, could you please verify whether itis correct or if I've did the wrong way?

[dentarg]

Yes, exactly, that is the suggestion.

You can also still use lowlevel_error_handler but the last line in it must be a proper response. The README has an example of that: https://github.com/puma/puma#error-handling

[taeyeob]

Okay, I think I got the point.
I will try another if there is any further issues.
Thank you so much 😊😊