-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Would this work to avoid Redis::InheritedError? #364
Comments
Hi, As you say, there's not really a way around using a separate connection per process. There is no simple way to coordinate usage of a single connection across processes, and in general you don't want to go there ;-). The approach that you mention would work fine, though it is almost exactly what the Redis client already does. The main difference is that it raises the I hope this helps. Cheers, |
I am a bit confused here. Looking at https://github.com/redis/redis-rb/wiki/redis-rb-on-Phusion-Passenger looks that from v 3 there is no need to reconnect after a fork. PhusionPassenger.on_event(:starting_worker_process) do |forked|
if forked
# If the actual cache respond to reconnect go on.
Rails.cache.reconnect if Rails.cache.respond_to? :reconnect
# Reconnect Resque Redis instance.
Resque.redis.client.reconnect
end
end Is this still needed? Are you here suggests that a better option may be something like: ....
if forked
# If the actual cache respond to reconnect go on.
Rails.cache.redis = Redis.new
# Reconnect Resque Redis instance.
Resque.redis = Redis.new
end
.... What do you think? |
Agreed. I completely understand the need for separate processes to use separate connections.
That's a huge difference, though; the solution I proposed above (and implemented in our project via a proxy object) transparently handles reconnecting when needed rather than raising
I think that would be a ton of effort: we'd have to wrap every place we use redis in rescue-reconnect-retry logic. You're probably thinking of putting the rescue-reconnect-retry at some higher-up place that can handle it in all our jobs (or whatever) automatically, but the problem is that we have over 100 redis connections in this application, and it would be non-trivial for such a bit of code to guess which redis connection to reconnect to. Plus, if we did that, it could involve wasting a lot of work: imagine if some heavy computation was done, then the error was raised: it would retry the heavy computation as well.
This isn't really feasible for us. We have lots of objects floating around that, when constructed, have been given a redis connection to talk to, and these objects may be used in the child process. We would basically have to re-architect how our application uses redis, which would be non-trivial. Given that you said that the approach I suggested above would work fine and is almost exactly what the redis client already does (even though the results are really quite different)...is there any reason you wouldn't accept a PR implementing the approach above? FWIW, we've been using it in production for a couple weeks and we have yet to see a problem. |
I understand your argument and agree that some handling should be added. Instead of adding a proxy object, however, what do you think of just handling this in the code path where the What do you think? |
+1 for adding this is a part of the gem itself. What I had in mind is something in the form of: if Process.pid != @pid
reconnect and return if @autoreconnect
raise InheritedError,
"Tried to use a connection from a child process without reconnecting. " +
"You need to reconnect to Redis after forking."
end _@autoreconnect_ should than be an optional parameter, it will simply remove the need of controlling the after_fork sequence from the user side and will maintain backwards compatibility, if it works fine for a while, it can probably than be set to be the default behavior with the next version change. |
I'm leaning towards including this. It will be nice to run a Google Trends search for "Redis::InheritedError" a few months after releasing :) |
Will a PR help to make this happen? Or maybe an animated gif? |
I just added a PR, these are the changes https://github.com/redis/redis-rb/pull/389/files |
@djanowski What do you think about removing the InheritedError altogether and always reconnect? |
Fixed by #414. |
Here at Moz we have a large distributed system that uses lots of forked processes (via our background job queue, Qless, which forks a process to give each job a sandbox like resque does) and also uses a lot of redis connections. We have a total of 112 of them, split both across different roles, and sharding on user id within those roles. We've been dealing with
Redis::InheritedError
by having each job reconnect to the redis connections that it might use. This works, but has also been causing us pain:We'd like to come up with a better way to deal with this, where reconnects will happen lazily happen automatically, and wanted to get your feedback on it. Here's what we're currently thinking:
delegate
stdlib) in place of theRedis
instances.Process.pid
when the instance is created.Redis
instance, checkProcess.pid
against the storedProcess.pid
, and, if they are different, reconnect to redis and update the pid instance variable.multi
blocks: if theProcess.pid
changed since the block opened, allow the normalRedis::Inherited
error to be raised (rather than reconnecting), since it would change the semantics of subsequent commands if we reconnected in the middle of a multi block.A few questions about this:
multi
, are there any otherRedis
methods that need special handling?Thanks!
/cc @proby @benkirzhner
The text was updated successfully, but these errors were encountered: