Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errno::ETIMEDOUT #1232

Closed
wants to merge 1 commit into from
Closed

Errno::ETIMEDOUT #1232

wants to merge 1 commit into from

Conversation

dpep
Copy link

@dpep dpep commented Nov 2, 2023

I'm seeing Errno::ETIMEDOUT occasionally being raised in production for Redis.evalsha calls (apparently due to openssl timing out). The Redis client catches a bunch of system errors and converts them into Redis::ConnectionError, but does not have similar logic in place for timeouts (it only handles TimeoutError)

stack trace

Errno::ETIMEDOUT: Connection timed out
/usr/local/lib/ruby/3.0.0/openssl/buffering.rb:406:in `syswrite_nonblock': Connection timed out (Errno::ETIMEDOUT)
	
from /usr/local/lib/ruby/3.0.0/openssl/buffering.rb:406:in `write_nonblock'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/connection/ruby.rb:78:in `block in write'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/connection/ruby.rb:77:in `loop'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/connection/ruby.rb:77:in `write'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/connection/ruby.rb:378:in `write'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/client.rb:320:in `block in write'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/client.rb:299:in `io'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/client.rb:318:in `write'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/client.rb:276:in `block (3 levels) in process'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/client.rb:270:in `each'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/client.rb:270:in `block (2 levels) in process'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/client.rb:411:in `ensure_connected'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/client.rb:269:in `block in process'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/client.rb:356:in `logging'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/client.rb:268:in `process'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/client.rb:161:in `call'
	
from /usr/local/bundle/ruby/3.0.0/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/instrumentation.rb:28:in `block in call'
	
from /usr/local/bundle/ruby/3.0.0/gems/ddtrace-0.54.2/lib/ddtrace/tracer.rb:283:in `trace'
	
from /usr/local/bundle/ruby/3.0.0/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/instrumentation.rb:22:in `call'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis.rb:270:in `block in send_command'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis.rb:269:in `synchronize'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis.rb:269:in `send_command'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/commands/scripting.rb:110:in `_eval'
	
from /usr/local/bundle/ruby/3.0.0/gems/redis-4.8.1/lib/redis/commands/scripting.rb:97:in `evalsha'

@dpep dpep changed the title ErrnoETIMEDOUT Errno::ETIMEDOUT Nov 2, 2023
@byroot
Copy link
Collaborator

byroot commented Nov 2, 2023

Thanks but redis-4.x is is no longer maintained.

@byroot byroot closed this Nov 2, 2023
@dpep
Copy link
Author

dpep commented Nov 2, 2023

bummer...we are trying to upgrade, but currently blocked by a Sidekiq requirement 😞

@sbiswas-chime
Copy link

@byroot would it be possible for you to open up a patch for redis-4.x with this fix ? As @dpep mentioned, our redis upgrade is blocked by a Sidekiq requirement.

@byroot
Copy link
Collaborator

byroot commented Nov 2, 2023

Why not just point your Gemfile at your branch?

@dpep
Copy link
Author

dpep commented Nov 2, 2023

security doesn't like branches in Gemfiles...we need to be more legit 🤣

@byroot
Copy link
Collaborator

byroot commented Nov 2, 2023

Shouldn't be hard to monkey patch.

@dpep dpep deleted the ErrnoETIMEDOUT branch November 21, 2023 16:51
dpep added a commit to dpep/network_resiliency_rb that referenced this pull request Nov 21, 2023
redis 4.x does not properly catch all system calls.  see redis/redis-rb#1232 for details.

this can be reverted once redis 5 is required
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants