Sporadic RedisClient::ConnectionError: Broken pipe

[davidgm0]

I have not been able to reproduce this, but I'm seeing this error daily when a job gets enqueued from a model callback:

RedisClient::ConnectionError: Broken pipe
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client/ruby_connection.rb:76:in `rescue in write'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client/ruby_connection.rb:73:in `write'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client/connection_mixin.rb:30:in `call'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:256:in `block (2 levels) in call'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client/middlewares.rb:16:in `call'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:255:in `block in call'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:675:in `ensure_connected'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:254:in `call'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:13:in `block (2 levels) in delete_lock'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:200:in `block (2 levels) in synchronize'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:219:in `with'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:200:in `block in synchronize'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:200:in `synchronize'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:200:in `synchronize'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:35:in `synced_redis_connection'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:12:in `block in delete_lock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:11:in `each'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:11:in `delete_lock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/strategies/base.rb:30:in `unlock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/strategies/until_executed.rb:12:in `after_perform'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/active_job_patch.rb:52:in `block (2 levels) in <module:ActiveJobPatch>'

Another similar error I see is

Redlock::LockAcquisitionError: failed to acquire lock on 'Too many Redis errors prevented lock acquisition:
RedisClient::ConnectionError: Broken pipe'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:323:in `lock_instances'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:275:in `block in try_lock_instances'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:271:in `times'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:271:in `try_lock_instances'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:79:in `lock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/strategies/base.rb:24:in `lock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/strategies/base.rb:59:in `before_enqueue'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/active_job_patch.rb:50:in `block (2 levels) in <module:ActiveJobPatch>'
/usr/local/bundle/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:467:in `instance_exec'

This does not happen if I downgrade to 0.2.5 and redlock 1.x. I'm using rails 7.0.8

[micahtessler]

I have the same error if I use activejob-uniqueness v0.3.1 and redlock 2.0.6
The error goes away if I downgrade redlock with
gem "redlock", "=2.0.4"

It's intermittant, not every lock check throws the error.

[micahtessler]

I have the same error if I use activejob-uniqueness v0.3.1 and redlock 2.0.6 The error goes away if I downgrade redlock with gem "redlock", "=2.0.4"

It's intermittant, not every lock check throws the error.

Now it's happening with redlock 2.04 as well. Downgrading redlock did not fix the issue for me...

[micahtessler]

I have the same error if I use activejob-uniqueness v0.3.1 and redlock 2.0.6 The error goes away if I downgrade redlock with gem "redlock", "=2.0.4"
It's intermittant, not every lock check throws the error.

Now it's happening with redlock 2.04 as well. Downgrading redlock did not fix the issue for me...

To stop the error, I downgraded to

gem "activejob-uniqueness", "=0.2.5"
gem "redlock", "=1.3.2"

[jorge-d]

Hey there, I did encounter the same issue using rails 7.0.8.

After investigation I stumbled upon this fix that was shipped recently in Redlock 2.0.6.

Updating to Rails 7.1 seems to fix the issue. I suggest that you add an activejob >= 7.1 dependency for the next gem versions.

Best,

EDIT: Ok it turns out it reduced the number of errors but we still have intermittent Broken Pipes, I believe it is related to the discussion in this issue: redis-rb/redis-client#119

EDIT 2: It's been a few hours now that I think I've fixed it by setting reconnect_attempts: 1 in my redlock server initialization, as suggested in this answer
Here's the interesting part in my config/initializers/active_job_uniqueness.rb:

  config.redlock_servers = [{
    url: REDIS_URL,
    db: REDIS_WORKER_DB,
    reconnect_attempts: 1,
    ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }
  }]

[jeffbax]

Hey there, I did encounter the same issue using rails 7.0.8.

After investigation I stumbled upon this fix that was shipped recently in Redlock 2.0.6.

Updating to Rails 7.1 seems to fix the issue. I suggest that you add an activejob >= 7.1 dependency for the next gem versions.

Best,

EDIT: Ok it turns out it reduced the number of errors but we still have intermittent Broken Pipes, I believe it is related to the discussion in this issue: redis-rb/redis-client#119

EDIT 2: It's been a few hours now that I think I've fixed it by setting reconnect_attempts: 1 in my redlock server initialization, as suggested in this answer Here's the interesting part in my config/initializers/active_job_uniqueness.rb:

  config.redlock_servers = [{
    url: REDIS_URL,
    db: REDIS_WORKER_DB,
    reconnect_attempts: 1,
    ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }
  }]

I just noticed a similar error on Heroku running Rails 6.1, redis-rb 4 (due to ActionCable) on Redis 7 itself in a review app testing the upgrades, will try this but it has not been super smooth sailing to update so far 🥲

[jeffbax]

FWIW I have not run into this since doing the full upgrade in our app (although do have reconnect_attempts: 1 specified)