Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Per lock timeout values do not appear to work #163

Closed
jdejong opened this issue Dec 27, 2014 · 13 comments
Closed

Per lock timeout values do not appear to work #163

jdejong opened this issue Dec 27, 2014 · 13 comments

Comments

@jdejong
Copy link

jdejong commented Dec 27, 2014

I have the following lock defined in a class

lock :service, :expiration => 15.minutes, :timeout => 20.seconds 

I have also tried several other variations

lock :service, :expiration => 15.minutes, :timeout => 20 
lock :service, { :expiration => 15.minutes, :timeout => 20.seconds }

for setting the lock up. However when I use the lock and I cause a collision that should block the second thread for 10-15 seconds I end up with the following error.

Timeout on lock service:1:service_lock exceeded 5 sec

It seems like it is not taking the options I am passing into the lock and just using the default.

Not that I think it matters but this is how I am grabbing the lock

service.service_lock.lock do

I am on Ruby 1.9.3p448, redis 3.0.7 and redis-objects 0.9.1.

Thanks in advance for any suggestions.

@nateware
Copy link
Owner

nateware commented Apr 3, 2015

I'm not sure what to do here, as there are tests for this in the gem (I think). Any chance you have timeout code somewhere else that could be causing this?

@nateware nateware added the bug label Apr 3, 2015
@jdejong
Copy link
Author

jdejong commented Apr 3, 2015

Not that I know of. The only timeout besides the lock is on the connection pool for redis which is 5 seconds (very similar setup to your example in the readme). I also played with different values and it seemed to stick with 5 seconds. I have code that is running in Rails Controllers, Resque Workers and Ruby Daemons and I have seen this error from the Workers and the Daemons but I don't recall seeing it in the controllers if that helps any. But I don't see how that would change anything.

@nateware
Copy link
Owner

nateware commented May 1, 2015

Can you please try redis-objects 1.2.0 which includes a new internal implementation of expiration? References #173 and #174

@jdejong
Copy link
Author

jdejong commented Jul 9, 2015

After a good amount of testing in production and staging we are still seeing

Timeout on lock service:2296:service_lock exceeded 5 sec 

where service lock is defined as

lock :service, :expiration => 15.minutes, :timeout => 20.seconds 

So it would appear this issue is still occurring.

@jdejong
Copy link
Author

jdejong commented Jul 11, 2015

On top of that we added a new lock to our code and often the lock is getting left behind in Redis. We are stuck on a slightly older version of Redis gem due to some dependencies (3.0.7) and we use RedisLabs MultiAZ cluster. Any thoughts?

@nateware
Copy link
Owner

I'm definitely interested in root causing this and fixing it, but my time
the next few weeks is extremely short. If somebody could at least create a
test case that reliably fails, that would be a big step in fixing it.

On Sat, Jul 11, 2015 at 4:21 PM, jdejong notifications@github.com wrote:

On top of that we added a new lock to our code and often the lock is
getting left behind in Redis. We are stuck on a slightly older version of
Redis gem due to some dependencies (3.0.7) and we use RedisLabs MultiAZ
cluster. Any thoughts?


Reply to this email directly or view it on GitHub
#163 (comment)
.

@mooncowboy
Copy link

@jdejong having the same problem as you, also using RedisLabs. Stuck on redis 3.2.2 and redis-objects 0.7.0 gems because of dependencies. @nateware working on getting a test case but can't make it work. Only fails in production and only sometimes. The lock gets left behind.

@byronalfonso
Copy link

@theplastictoy were you able to resolve this issue?

@mooncowboy
Copy link

@byronalfonso no. We're migrating to a newer stack version, so in the meantime we're doing it differently. I would say this is a coincidence of old versions of redis-objects and rails.

@byronalfonso
Copy link

@theplastictoy I see.. Sounds like a good route to go to.. Thanks for the reply 👍
Btw, it turned out that my problem wasn't really the timeout but that the services are needed be unlocked using RDM.. It's working well on my end now.. thanks and credits to my seniors who helped me figure this out :)

@jdejong
Copy link
Author

jdejong commented Oct 16, 2016

I think I might have finally figured out the issue. In cases where the class locks are using used, it doesn't seem like the options are reaching the Lock class from the Object:Lock class. The reason I think this is the issue is any class lock always sets the key's value to 1 but when I just the Lock class directly it sets the timestamp as expected. Do the service locks just not work?

@nateware
Copy link
Owner

nateware commented Oct 7, 2018

Can you see if redis-objects v1.4.3 just released to rubygems improves locking, for those of you still interested?

@nateware
Copy link
Owner

CC @tmsrjs to potentially verify the tests are actually working

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants