Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restart Action Cable when Redis reconnect_attempt limit is reached #51687

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion actioncable/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
* Restart Action Cable when Redis reconnect_attempt limit is reached.

Please check [7-2-stable](https://github.com/rails/rails/blob/7-2-stable/actioncable/CHANGELOG.md) for previous changes.
*Heinrich Lee Yu*, *Elijah Buck*
4 changes: 4 additions & 0 deletions actioncable/lib/action_cable/subscription_adapter/redis.rb
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,10 @@ def ensure_listener_running
if retry_connecting?
when_connected { resubscribe }
retry
else
@adapter.server.logger.info("Redis reconnect_attempts limit exceeded; restarting.")
@thread = nil
@adapter.server.restart
end
end
end
Expand Down
41 changes: 41 additions & 0 deletions actioncable/test/subscription_adapter/redis_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -150,3 +150,44 @@ def connection_id
end
end
end

class RedisAdapterTest::ConnectionError < RedisAdapterTest
require "redis"
class FailingRedis < ::Redis
cattr_accessor :should_raise
def subscribe(*channels, &block)
if FailingRedis.should_raise
FailingRedis.should_raise = false
raise RedisClient::ConnectionError.new
end
super
end
end

def test_connection_restart_on_connection_error
ActionCable::SubscriptionAdapter::Redis.redis_connector = ->(config) do
FailingRedis.new(config.except(:adapter, :channel_prefix))
end
server = ActionCable::Server::Base.new
@adapter = server.config.pubsub_adapter.new(server)
subscribe_as_queue("channel", @adapter) do |queue|
thread_id = @adapter.send(:listener).instance_variable_get("@thread").object_id

# the first reconnect will fail, triggering the restart
FailingRedis.should_raise = true
drop_pubsub_connections

# wait for server to restart
10.times do
new_thread_id = @adapter.send(:listener).instance_variable_get("@thread")&.object_id
break if new_thread_id && new_thread_id != thread_id
sleep 0.1
end

subscribe_as_queue("newchannel", @adapter) do |queue|
@tx_adapter.broadcast("newchannel", "hello world")
assert_equal "hello world", queue.pop
end
end
end
end