Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IOError: stream closed in another thread #2499

Open
bantonj opened this issue Sep 14, 2023 · 4 comments
Open

IOError: stream closed in another thread #2499

bantonj opened this issue Sep 14, 2023 · 4 comments
Milestone

Comments

@bantonj
Copy link

bantonj commented Sep 14, 2023

Issue report

Question 1: What is the problem?
After upgrading to version 6.0.18 from 6.0.15 we started to see this exception in our logs IOError: stream closed in another thread. The backtrace is:

/gems/passenger-6.0.18/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:133 in accept
/gems/passenger-6.0.18/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:133 in accept_and_process_next_request
/gems/passenger-6.0.18/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:110 in main_loop
/gems/passenger-6.0.18/src/ruby_supportlib/phusion_passenger/request_handler.rb:434 in block (2 levels) in start_threads
/gems/passenger-6.0.18/src/ruby_supportlib/phusion_passenger/utils.rb:113 in block in create_thread_and_abort_on_exception

Question 2: Passenger version and integration mode:
open source 6.0.18 apache

Question 3: OS or Linux distro, platform (including version):
Ubuntu 20.04 on arm

Question 4: Passenger installation method:

Your answer:
[x] RubyGems + Gemfile
[ ] RubyGems, no Gemfile
[ ] Phusion APT repo
[ ] Phusion YUM repo
[ ] OS X Homebrew
[ ] source tarball
[ ] Other, please specify:

Plus passenger-install-apache2-module --apxs2-path='/usr/bin/apxs' and passenger-config build-native-support

Question 5: Your app's programming language (including any version managers) and framework (including versions):
Ruby 2.7.7

Question 6: Are you using a PaaS and/or containerization? If so which one?
Docker based on ubuntu:20.04

@CamJN
Copy link
Member

CamJN commented Oct 19, 2023

I wonder if this is related: 213c9a0

Would you mind editing src/ruby_supportlib/phusion_passenger/utils.rb and commenting out PhusionPassenger.call_event(:unhandled_exception_before_exit, e) and see if the error goes away?

@CamJN CamJN added this to the 6.0.20 milestone Nov 20, 2023
@toncid
Copy link

toncid commented Mar 14, 2024

@bantonj Any new findings that you might share?

@CamJN We are also getting the same error in cases when the application is under heavier load and then being restarted following a restart.txt touch.

As if there is a race condition between application unloading and web server accepting the next request.

Worker processes remain in this status indefinitely:

  * PID: 550179   Sessions: 1       Processed: 312     Uptime: 37m 42s
    CPU: 21%     Memory  : 206M    Last used: 32m 44s ag
    Shutting down...

Setup: Passenger Standalone v6.0.20, Amazon Linux 2023, Ruby 3.2.2, x86_64.

I have tried commenting the line in utils.rb (line 118), but it is after line 113 highlighted in the stack trace:

App 517567 output: [ 2024-03-14 10:13:08.5795 517567/0x00007ff93bc369a0(HTTP helper worker) utils.rb ]: *** Exception IOError (stream closed in another thread) (process 517567, thread 0x00007ff93bc369a0(HTTP helper worker)):
.../passenger-6.0.20/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:133:in `accept'
.../passenger-6.0.20/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:133:in `accept_and_process_next_request'
.../passenger-6.0.20/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
.../passenger-6.0.20/src/ruby_supportlib/phusion_passenger/request_handler.rb:434:in `block (2 levels) in start_threads'
.../passenger-6.0.20/src/ruby_supportlib/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'  <<<<<<<<<
.../newrelic_rpm-9.7.1/lib/new_relic/agent/tracer.rb:434:in `block (2 levels) in thread_block_with_current_transaction'
.../newrelic_rpm-9.7.1/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
.../newrelic_rpm-9.7.1/lib/new_relic/agent/tracer.rb:433:in `block in thread_block_with_current_transaction'

@CamJN
Copy link
Member

CamJN commented May 5, 2024

I can reliably reproduce this with ruby 3.2.4 on macOS just by running rake test:ruby. @FooBarWidget any idea what this is?

@FooBarWidget
Copy link
Member

When the app process starts, it creates a server socket and multiple threads. Each thread accepts connections from that single server socket. The main thread then waits until the Passenger Core tells us to shut down.

Upon receiving the shutdown signal, the main thread first terminate all threads by calling Thread#raise followed Thread#join. After that, it closes the server socket.

So I don't really know why one would get a "stream closed in another thread" error from one of the threads. By the time we close the server sockets, the threads should already have terminated.

I can't reproduce the issue. Maybe someone who can, can help by increasing log level to 7.

@CamJN CamJN modified the milestones: 6.0.20, 6.0.22 May 17, 2024
@CamJN CamJN modified the milestones: 6.0.22, 6.0.24 Aug 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants