Pub/Sub silently unsubscribes? #4

Closed
hakusaro opened this Issue Jul 3, 2012 · 15 comments

Comments

Projects
None yet
3 participants

hakusaro commented Jul 3, 2012

This is an issue I've experienced with another Java based Redis client before (namely Jedis), where in effect the subscription dies silently. It doesn't call the unsubscribe method, and it doesn't cause any exceptions - it seemingly just stops listening for certain channels.

I'm just calling the subscribe method as shown in the README example, and while it will work reliably for a short period of time, it'll just stop later.

Lettuce 2.1.0 with Netty 3.3.0 Final.

Owner

wg commented Jul 3, 2012

Hmm, that's worrying! Do you see anything unusual in the redis server logs? If you could capture a set of netty debug logs that might be very helpful.

hakusaro commented Jul 3, 2012

The Redis server has no logging of the issue - it's completely silent when the subscription ends.

Netty debug logs are a pain to replicate in the environment that I'm working in, but I'm doing verification to see if it's our end or yours.

Is it possible to implement some way to detect what channels a client is subscribed to? That would at least allow a sort of keep alive to re-subscribe if it isn't subscribed.

Owner

wg commented Jul 4, 2012

The RedisPubSubConnection does contain two private fields: "channels" and "patterns" which are Sets containing all subscribed channels and patterns. However it is already designed to resubscribe on reconnect.

Have you been able to isolate a reproducible test case?

hakusaro commented Jul 4, 2012

I'm implementing it in a server, and that's where the issue is originating. Using the code outside of said server is operating normally, as far as I can tell. In staging it works fine, in production it dies after 10 or so minutes.

If I can find this to be an actual issue with the library I'll reopen this, but I've convinced myself that it's likely an issue with something I'm doing.

On a side note, thanks for being up to date on Redis and maintaining a development presence, unlike some other libraries I know.

hakusaro closed this Jul 4, 2012

Owner

wg commented Jul 4, 2012

OK, thank you! The pub/sub code is relatively untested compared to the rest as none of my own projects use it, so there may well be bugs that I'd like to fix.

hakusaro commented Jul 4, 2012

Again, in staging it's performed fine, it's the actual production instances that have failed, which leads me to believe that it's a Netty incompatibility issue.

hakusaro reopened this Jul 4, 2012

hakusaro commented Jul 4, 2012

Okay, after doing more testing, if no data is sent to the channel that the client is subscribed to, in 15 minutes, using the implementation on the README, it'll silently fail.

The 15 minute mark seems to be around where it fails for some odd reason, again with indefinite timeouts specified throughout the system (Redis remains connected, but data no longer goes through the subscription path).

Owner

wg commented Jul 4, 2012

And you've verified that the redis server's "timeout" config option is set to zero?

hakusaro closed this Jul 4, 2012

hakusaro reopened this Jul 4, 2012

hakusaro commented Jul 4, 2012

Yep.

I really hate how easy it is to misclick the close issue button.

Owner

wg commented Jul 4, 2012

Interesting, do you know if there's some sort of firewall or other network appliance between the client and server? Sometimes they're configured to drop idle connections after a certain period of time.

Can you send me the output of redis' INFO command from your production server?

hakusaro commented Jul 4, 2012

  1. On staging and production, both of which we've replicated it on, we have a varying firewall configuration. On staging it's virtually non-existent, though. It's worth noting that the Redis server never shows a connection dropping, it always shows the same number of clients connected in correlation to the number of instances we're running.

  2. I'll get that sometime in the next 8-12 hours. Holiday.

wedtm commented Sep 18, 2012

Is there an update on this issue? Would really like to know if Pub/Sub is safe with lettuce.

I can tell you that I solved the issue by just sending data every 5 minutes or so back and forth. Redis info would yield nothing of value, but I can still get that if you like.

Owner

wg commented Sep 19, 2012

Thanks for the followup @nicatronTg! Since the same thing occurs with Jedis I'm going to guess it's an environment-specific issue and close this issue. Sounds very much like something between you and the server is closing the connection if idle. If you ever find out that's not the case do let me know as I'm very eager to fix any bugs on the lettuce side.

wg closed this Sep 19, 2012

It's very much a higgs buggson. I have no timeouts set or anything, but I've not been able to maintain the connection.

Jedis is a generally out of date library, so don't credit that. It makes calls that the Redis server will complain aren't correct or malformed, hence why I ditched it.

If I find more information I'll be happy to let you know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment