Error reached top of thread-pool: EOFError (EOFError) #1502

vincentwoo · 2018-01-10T00:54:03Z

Frequently for us in dev, puma hits an error in its thread pool, grinding the web server to a halt. This happens on page refreshes on our Rails app. I'm making an issue out of desperation, but I have not a lot to go on.

It appears this block catches some kind of internal error, but no backtrace is attached for us:

https://github.com/puma/puma/blob/master/lib/puma/thread_pool.rb#L121

          begin
            block.call(work, *extra)
          rescue Exception => e
            STDERR.puts "Error reached top of thread-pool: #{e.message} (#{e.class})"
          end

After a few of those in the logs, we also sometimes see:

16:51:09 web.1 | Error in reactor loop escaped: undefined method join' for nil:NilClass (NoMethodError)`

Does anyone have a hunch about where this might be coming from?

System configuration

Ruby version: 2.5.0
Rails version: 5.1.4
Puma version: 3.11.0

The text was updated successfully, but these errors were encountered:

vincentwoo · 2018-01-10T00:59:17Z

This is probably related to #1483, since we also run SSL in dev.

vincentwoo · 2018-01-10T01:03:20Z

If relevant, this is our puma config:

workers Integer(ENV['WEB_CONCURRENCY'] || 1) # 1 worker 5 threads in dev
threads_count = Integer(ENV['MAX_THREADS'] || 5)
threads threads_count, threads_count

preload_app!

_port = ENV['PORT'] || 8080
_environment = ENV['RACK_ENV'] || 'development'

if _environment == 'development' && File.exists?('/etc/ssl/localhost.ssl.key')
  ssl_bind '0.0.0.0', _port, {
    key: '/etc/ssl/localhost.ssl.key',
    cert: '/etc/ssl/localhost.ssl.crt',
    verify_mode: 'none'
  }
else
  port _port
end

rackup DefaultRackup
environment _environment

on_worker_boot do
  # Worker specific setup for Rails 4.1+
  # See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
  ActiveRecord::Base.establish_connection
end

plugin :tmp_restart

vincentwoo · 2018-01-10T02:06:20Z

After a lot of print debugging, I believe the exception-throwing line is: https://github.com/puma/puma/blob/master/lib/puma/minissl.rb#L66

@engine.inject(data)

when data is nil. This seems to happen pretty reliably once I added a bunch of print debugging, suggesting the problem is timing-related. The exception happens after the server has hung for many seconds, suggesting that the root cause is deeper.

vincentwoo · 2018-01-10T02:15:26Z

This is about as far as I am liable to get on my own, I fear. Help is desperately appreciated!

vincentwoo · 2018-01-10T03:31:08Z

Oh, in case this is related: I'm running OSX. It's possible that the issues outlined in https://blog.phusion.nl/2017/10/13/why-ruby-app-servers-break-on-macos-high-sierra-and-what-can-be-done-about-it/ matter.

edit: the manual fix in the article didn't seem to do it

jtzero · 2018-01-11T17:56:40Z

another data point,
I have the same issue but on
ruby 2.4.3p205 (2017-12-14 revision 61247) [x86_64-darwin16]
and Sierra (not high sierra)

Temporary work around for “no implicit conversion of nil into String” See puma/puma#1483 and puma/puma#1502

rrevi · 2018-02-01T14:54:40Z

So I updated to 3.11.2 to fix this issue. While the issue log errors mentioned here no longer show up, I am now getting the following errors in the puma.log file with most HTTP requests (over SSL of course):

=== puma startup: 2018-02-01 08:00:55 -0500 ===
=== puma startup: 2018-02-01 08:00:55 -0500 ===
Error reached top of thread-pool: EOFError (EOFError)
Error reached top of thread-pool: EOFError (EOFError)
Error reached top of thread-pool: EOFError (EOFError)
Error reached top of thread-pool: EOFError (EOFError)
Error reached top of thread-pool: EOFError (EOFError)
Error reached top of thread-pool: EOFError (EOFError)Error reached top of thread-pool: EOFError (EOFError)
Error reached top of thread-pool: EOFError (EOFError)

anyone have any idea what could be going on? I haven't noticed any bad user experience yet (e.g. puma crash or http request hang)

Thanks,
Rafael

chytreg · 2018-02-01T16:17:02Z

I have exactly the same issue and cannot track down the cause either.

rssluca · 2018-02-01T21:00:37Z

Same issue here although non disruptive.

MarkMurphy · 2018-02-04T19:27:54Z

Also having the same problem

gcoderre · 2018-02-22T16:20:41Z

I'm seeing the same issue after updating to 3.11.2 while using SSL:

Puma: 3.11.2
Ruby: 2.4.3p205
Rails: 5.1.5
OS: macOS 10.13.1

kylekyle · 2018-03-01T02:53:56Z

Also getting this with the exact same setup and versions as @gcoderre but on Rack/Roda not Rails.

kylekyle · 2018-03-02T16:20:54Z

Can we reopen this? I'm seeing it on both OS X and RHEL7 ....

dbpqdb · 2018-03-04T23:02:51Z

+1

Seeing same issue as @rrevi on Puma 3.11.2, Ubuntu 16.04, Ruby 2.4.3p205, Rails 5.1.4

tuwukee · 2018-03-06T11:51:22Z

hey @evanphx,
Error reached top of thread-pool: EOFError (EOFError) is fixed with your commit b9ff900

but I'm still getting EOFErrors with puma server

2018-03-06 14:24:05 +0300: Read error: #<EOFError: EOFError>

/Users/july/.rbenv/versions/2.4.3/lib/ruby/gems/2.4.0/gems/puma-3.11.3/lib/puma/client.rb:292:in `try_to_finish'
/Users/july/.rbenv/versions/2.4.3/lib/ruby/gems/2.4.0/gems/puma-3.11.3/lib/puma/client.rb:106:in `reset'
/Users/july/.rbenv/versions/2.4.3/lib/ruby/gems/2.4.0/gems/puma-3.11.3/lib/puma/server.rb:450:in `process_client'
/Users/july/.rbenv/versions/2.4.3/lib/ruby/gems/2.4.0/gems/puma-3.11.3/lib/puma/server.rb:302:in `block in run'
/Users/july/.rbenv/versions/2.4.3/lib/ruby/gems/2.4.0/gems/puma-3.11.3/lib/puma/thread_pool.rb:120:in `block in spawn_thread'

Your change adds exception handling for client.eagerly_finish method call https://github.com/puma/puma/blob/master/lib/puma/server.rb#L280

The EOFError is also can be thrown by client.reset method https://github.com/puma/puma/blob/master/lib/puma/server.rb#L450

Can you fix this one as well please?
Thanks!

nateberkopec · 2018-03-20T17:52:45Z

Reopening until @tuwukee's comment is addressed.

bruno-campos · 2018-06-04T15:03:15Z

I'm also seeing the same error reported by tuwukee on 3.11.4

2018-06-01 12:55:57 +0000: Read error: #<EOFError: EOFError>
/usr/local/bundle/gems/puma-3.11.3/lib/puma/client.rb:292:in `try_to_finish'
/usr/local/bundle/gems/puma-3.11.3/lib/puma/client.rb:106:in `reset'
/usr/local/bundle/gems/puma-3.11.3/lib/puma/server.rb:450:in `process_client'
/usr/local/bundle/gems/puma-3.11.3/lib/puma/server.rb:302:in `block in run'

Ruby 2.5.0
Rails 5.2.0

andrykonchin · 2020-03-09T09:53:44Z

Looks like the issue with EOFError is already fixed.

The code version where the EOFError error was reported to happen - source code. Currently any IO error is rescued and swallowed here source code.

This change was introduce in #1728 and released in v4.0.0 (Changelog)

vincentwoo mentioned this issue Jan 10, 2018

Puma SSL mode produces “no implicit conversion of nil into String” #1483

Closed

evanphx closed this as completed in acb709b Jan 19, 2018

lgebhardt added a commit to lgebhardt/publishers that referenced this issue Jan 22, 2018

Regress puma to 3.10

69e474c

Temporary work around for “no implicit conversion of nil into String” See puma/puma#1483 and puma/puma#1502

lgebhardt mentioned this issue Jan 22, 2018

Regress Puma to 3.10 to fix “no implicit conversion of nil into String” brave-intl/publishers#504

Merged

7 tasks

chytreg mentioned this issue Feb 5, 2018

Problems shutting down gracefully after EOFErrors in 3.11.2 #1507

Closed

tuwukee mentioned this issue Mar 5, 2018

Raise EOFError if the socket has been closed #1524

Closed

This was referenced Mar 15, 2018

Bump puma from 3.9.1 to 3.11.3 fsek/web#744

Merged

Bump puma from 3.9.1 to 3.11.3 bustikiller/raffle-bee#27

Merged

nateberkopec added the bug label Mar 20, 2018

nateberkopec assigned evanphx Mar 20, 2018

dependabot-preview bot mentioned this issue Mar 22, 2018

Bump puma from 3.10.0 to 3.11.3 sumyapp/basecamp3_slack_notify#29

Merged

nateberkopec reopened this Sep 2, 2019

nateberkopec mentioned this issue Sep 2, 2019

Unix socket binding on WSL > 3.11.0 #1521

Closed

nateberkopec added the contrib-wanted label Oct 20, 2019

nateberkopec changed the title ~~Error reached top of thread-pool: no implicit conversion of nil into String (TypeError)~~ Error reached top of thread-pool: EOFError (EOFError) Oct 20, 2019

nateberkopec unassigned evanphx Nov 4, 2019

nateberkopec closed this as completed Mar 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error reached top of thread-pool: EOFError (EOFError) #1502

Error reached top of thread-pool: EOFError (EOFError) #1502

vincentwoo commented Jan 10, 2018 •

edited

vincentwoo commented Jan 10, 2018

vincentwoo commented Jan 10, 2018

vincentwoo commented Jan 10, 2018 •

edited

vincentwoo commented Jan 10, 2018

vincentwoo commented Jan 10, 2018 •

edited

jtzero commented Jan 11, 2018

rrevi commented Feb 1, 2018

chytreg commented Feb 1, 2018

rssluca commented Feb 1, 2018

MarkMurphy commented Feb 4, 2018

gcoderre commented Feb 22, 2018

kylekyle commented Mar 1, 2018

kylekyle commented Mar 2, 2018

dbpqdb commented Mar 4, 2018 •

edited

tuwukee commented Mar 6, 2018

nateberkopec commented Mar 20, 2018

bruno-campos commented Jun 4, 2018

andrykonchin commented Mar 9, 2020 •

edited

Error reached top of thread-pool: EOFError (EOFError) #1502

Error reached top of thread-pool: EOFError (EOFError) #1502

Comments

vincentwoo commented Jan 10, 2018 • edited

System configuration

vincentwoo commented Jan 10, 2018

vincentwoo commented Jan 10, 2018

vincentwoo commented Jan 10, 2018 • edited

vincentwoo commented Jan 10, 2018

vincentwoo commented Jan 10, 2018 • edited

jtzero commented Jan 11, 2018

rrevi commented Feb 1, 2018

chytreg commented Feb 1, 2018

rssluca commented Feb 1, 2018

MarkMurphy commented Feb 4, 2018

gcoderre commented Feb 22, 2018

kylekyle commented Mar 1, 2018

kylekyle commented Mar 2, 2018

dbpqdb commented Mar 4, 2018 • edited

tuwukee commented Mar 6, 2018

nateberkopec commented Mar 20, 2018

bruno-campos commented Jun 4, 2018

andrykonchin commented Mar 9, 2020 • edited

vincentwoo commented Jan 10, 2018 •

edited

vincentwoo commented Jan 10, 2018 •

edited

vincentwoo commented Jan 10, 2018 •

edited

dbpqdb commented Mar 4, 2018 •

edited

andrykonchin commented Mar 9, 2020 •

edited