Error in reactor loop escaped: mode not supported for this object: r (ArgumentError)

[jboler]

I did see a similar error reported here but the error message is slightly different.

Describe the bug
Rails 5.2.3
jRuby 9.2.7.0 (OpenJDK 8)
puma (4.2.1-java)
nio4r (2.5.2-java)

Ubuntu 16.04

Seeing this error:

Error in reactor loop escaped: mode not supported for this object: r (ArgumentError)
org/nio4r/Selector.java:124:in `register'
/var/www/my-api/shared/bundle/jruby/2.5.0/gems/puma-4.2.1-java/lib/puma/reactor.rb:172:in `block in run_internal'
org/jruby/RubyArray.java:1792:in `each'
/var/www/my-api/shared/bundle/jruby/2.5.0/gems/puma-4.2.1-java/lib/puma/reactor.rb:162:in `block in run_internal'
org/jruby/ext/thread/Mutex.java:165:in `synchronize'
/var/www/my-api/shared/bundle/jruby/2.5.0/gems/puma-4.2.1-java/lib/puma/reactor.rb:159:in `block in run_internal'
org/jruby/RubyArray.java:1792:in `each'
/var/www/my-api/shared/bundle/jruby/2.5.0/gems/puma-4.2.1-java/lib/puma/reactor.rb:157:in `run_internal'
/var/www/my-api/shared/bundle/jruby/2.5.0/gems/puma-4.2.1-java/lib/puma/reactor.rb:312:in `block in run_in_thread'

Puma config:

environment 'staging'
threads 8, 8
directory "/var/www/my-api/current"
state_path "/var/www/my-api/current/tmp/puma/state"
pidfile "/var/www/my-api/current/tmp/puma/pid"
bind "unix:///var/www/my-api/shared/puma.sock"
activate_control_app

bundle exec puma

[nateberkopec]

I'm guessing you get this right when it starts up, right?

[nateberkopec]

Or maybe after serving the first request?

[jboler]

It doesn't happen straight away or on the very first request. It only seems to happen after holding down refresh in the browser to trigger a few concurrent requests. Once the error appears in the puma log the page that I was refreshing stops responding

[nateberkopec]

@ioquatix Have you seen this before? (Also are you ok with me mentioning you on anything possibly nio4r related?)

[ioquatix]

There are various bugs with JRuby, yes I've seen this one but I don't know why it's happening. I've been discussing it with @headius but honestly it's a bit outside what I'm currently focusing on. I welcome any PRs to fix this issue in nio4r, and investigation/discussion as to why it's happening.

[jboler]

It didn't happen with puma 3.12.1 so we're reverting back to that.

I'll check to see if this was introduced on a specific version of 4.x and I'll try to create a minimal example app to reproduce the error.

[ioquatix]

If this is a commercial project, you are welcome to sponsor my open source efforts to prioritise this issue. If you can create a minimal repro, that is most useful, but it shouldn't need to involve puma, it's a socket level issue with JRuby and nio4r.

[nateberkopec]

Thanks for confirming this is probably a nio4r thing, I’ll close here.

[jboler]

This is most likely due to socketry/nio4r#67 as noted in jruby/jruby#2750

[jboler]

An attempted fix for this issue was merged in #2477 . I ran a load test on puma v5.1.1 using unix:// sockets with a jRuby Rails app and puma was able to sustain many parallel connections for some minutes but eventually it did stop responding completely with no error in the logs. I ran the same load test using tcp:// sockets with the same jRuby Rails app and couldn't get it to lock up so it looks like tcp:// sockets are the way to go for jRuby apps.

[headius]

@jboler That hang might indicate a bug in our UNIXSocket implementation. Can you show me how to reproduce it and file a JRuby issue?

[jboler]

@headius I will try to reproduce it with a minimal jRuby Rails repo because I can't publish the Rails app I used. I think it unlikely that the bug is in the jRuby UNIXSocket implementation because we've been using unix:// sockets with our jRuby Rails app + puma 3.x for years without issue. I believe the problem is with nio4r which was introduced in puma 4.x.

[MSP-Greg]

@jboler

puma was able to sustain many parallel connections for some minutes

Do you recall approx how many connections before the failure? I may be able to try to repo...

[jboler]

I was running puma with min_threads & max_threads set to 8 on jRuby 9.2.13.0. For requests I opened ~15 terminals and ran:

while true; do curl http://host; done

It took a few minutes to lock up. I could imagine that increasing the load might trigger it more quickly.

The puma app is being proxied by nginx with this config:

/etc/nginx/nginx.conf

user www-data;
worker_processes auto;
env PATH;

events {
  use epoll;
  worker_connections 1024;
  multi_accept on;
}

http {
  client_max_body_size 1024m;
  client_body_timeout 3600s;

  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log crit;

  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;
  server_tokens off;
  keepalive_timeout 70;

  include mime.types;
  default_type application/octet-stream;

  gzip on;
  gzip_http_version 1.1;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_min_length 1100;
  gzip_buffers 64 8k;
  gzip_comp_level 3;
  gzip_proxied any;
  gzip_types
    application/atom+xml
    application/javascript
    application/json
    application/xml+rss
    application/vnd.ms-fontobject
    application/x-font-ttf
    application/x-web-app-manifest+json
    application/x-javascript
    application/vnd.api+json
    application/xhtml+xml
    application/xml
    font/opentype
    image/svg+xml
    image/x-icon
    text/css
    text/xml
    text/plain
    text/javascript
    text/x-component;

  map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
  }

  include /etc/nginx/sites-enabled/*;
}

/etc/nginx/sites-enabled/host.conf

upstream app {
  server unix:///var/www/site/shared/puma.sock;
}

server {
  listen 80 default_server;

  location / {
    root /var/www/site/current/dist;
    try_files /down.html @app;
  }

  location @app {
    add_header X-Frame-Options DENY;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto http;
    proxy_http_version 1.1;
    proxy_redirect off;
    proxy_buffering off;
    proxy_read_timeout 7200s;
    proxy_pass http://app;
  }
}