PING not allowed while subscribing

[za-creature]

In the current implementation (tested on 2.4.7) redis does not allow ping commands when the client is in subscribe mode.
I believe this should be changed because currently, there is no way for the client to determine if the connection is still alive while subscribing to one or more queues, and no message has been posted for a long time.

[antirez]

Approved, however when you are in this mode PING should not reply with a standard reply but should send you a PONG "message" in the usual Pub/Sub protocol mode.

[rootslab]

@za-creature, I think that a client, that opens up a socket to the redis-server, it's aware if its socket is readable / writable, open / closed; I do not have to send a PING command ( with redis protocol ) to test if the connection is alive.
However, if you still need a response from server you could also re-send a dummy/trivial SUBSCRIBE command.

@antirez Interesting, any idea on what channel ( subscribed by the client ) PONG will be "published"?

[za-creature]

@rootslab
Actually, you are wrong. Most operating systems have TCP timeouts so long that a connection could be literally kept open for hours after the network cable was unplugged. An application may of course override this timeout to a reasonable value, but the timeout is triggered whenever no data has been read for that many seconds (that's the way sockets work; the only reliable way to determine whether a connection is still up is to read from it).
Now this shouldn't be a problem for the most part, because the Redis protocol is designed in a command -> response fashion; so basically it's: "send a request, give it a five count and then report the connection as failed if no response". However, in subscribe mode, because the client no longer sends any packets after subscribing, it will wait (potentially for hours or even days) until it gets a message on any of the subscribed queues. Now if one were to set a socket timeout to 5 seconds, and it was triggered, there's currently no way to determine whether it was triggered because the connection to Redis was severed, or just because no messages were published on any subscribed queues. You can see the problem in action here (redis-py went one step ahead and closed the connection as well):
redis/redis-py#225
The fix I wrote initially involved sending a ping() after a socket timeout to make sure the connection was still up; however Redis does not allow this so I ended up patching the client to ignore Redis errors while in subscribe mode. That is, I'm sure you'll agree, a poor solution.
Sending a dummy subscribe on an unique id followed by an unsubscribe on said id is at best a quick fix; fact of the matter is, a protocol should not stress the developer's creativity, and there really isn't a good reason for which 'ping' commands should be disallowed when in the subscribed state.

[rootslab]

Hello @za-creature, probably I misunderstood this sentence:

'there is no way for the client to determinates if the connection is still alive'

If you're talking about the service connection to REDIS, You are absolutely right. Otherwise, If we talk about testing the liveness of the network connection, you can use the socket (TCP) keepalive as usual.

About the workaround, for now my solutions are :

• to send any dummy command to redis-server. It will respond with an error.
• re-subscribe to an already subscribed channel, I think that a client waiting for days for a message, can also refresh its subscription every n hours.

The proposed solution by @antirez is interesting, it breaks a little the redis client-server Pub/Sub mode; a little question will arise: in your opinion, on what channel the PING reply will be published?

[za-creature]

@rootslab I was talking about the network connection, but you are right about keep-alives; I just found out that setsockopt can be used to configure the keep-alive parameters on a per socket basis. Previously, I thought that they can only be set globally by writing to /proc, and that requires root access (unfeasible imo).

I don't really know what you meant by service connection; as far as I know, Redis uses a single socket for communication. Or are you referring to detecting errors if the Redis server is frozen for some reason? Because crashes are handled by the operating system (all open sockets are closed automatically on process termination, and the remote end receives a SIGHUP or equivalent)

Lastly, yes, both of your solutions work (I've actually implemented the first one), but I strongly believe that workarounds are a bad coding practice, and should be avoided if at all possible. Subscribing to a channel twice again, works, but is counter-intuitive.

As for the PONG reply, I don't really know; maybe a reserved channel name like 'redis', but that would break backward compatibility. Or a "null" channel name that cannot be subscribed to anyway.
@antirez Since the PONG would only happen for clients that are aware of the new PING functionality, is there a good reason that I'm missing for which it can't reply with a "normal" response, or is it related to the server architecture?

This could also be implemented in a heartbeat-fashion (the way AMQP does it) in which the server sends these messages in a timely fashion chosen by the client, but that would break compatibility with existing clients that will have to handle the out of band heartbeat frames.

[rootslab]

@za-creature for service connection, I simply mean : the aliveness of the service which the (network) socket is connected to.

[rootslab]

@za-creature @antirez If redis-server creates a default read-only channel, a client could SUBSCRIBE to that channel, as usual, to test if the service is up.
I think that it doesn't break any compatibility.

[za-creature]

@rootslab Yeah, that's what I thought. It wasn't immediately obvious to me because I usually like to think of the two situations (network error vs service error) as the same one for simplicity, as the code to handle them is almost always the same (close the connection, try again later or log/display an error message)
About the read-only channel, the only compatibility issue that I can think of is that it will reserve the channel name, and any applications that currently use said channel will fail upon upgrading to 2.6. This may be minimized by using a GUID/UUID as the channel name, or, as per my previous suggestion, masquerading the PONG as a message posted on the empty-string "channel". As far as I know, attempting to publish without sending a channel name is a command error.

[rootslab]

@za-creature
.. a SUBSCRIBE command, without arguments, could be a solution to receive a message from this 'unnamed echo' channel.
With previous Redis versions, the client will receive, as usual, an -ERR reply.
Whatever will be, I think that it is not a crucial decision in Redis development.
Bye, for now! ;)

[antirez]

Moved to 3.0 milestone, not a priority for 2.6, we are too near RC1. Thanks. I'll comment on the issue again in a few weeks.

[za-creature]

@rootslab, @antirez

After further investigation, the TCP keep-alive mechanism can only be used on Linux 2.4+
The Keep-Alive feature is optional in a TCP implementation, and if implemented, must default to off.
The default idle timeout before sending a Keep-Alive frame is mandated by the spec to two hours (7200 seconds)
While most major operating systems (Linux, BSD/Darwin and Windows NT) allow an application to override this value, the other two values (Keep-Alive retransmission interval and maximum number of frames) can only be overriden on Linux. On Windows, the number of frames is fixed to 5 (NT/2000/XP) or 10 (Vista/7). On BSD/Darwin, I'm not entirely sure about the number of frames, but it should also be somewhere in the lines of 10 (Linux uses 9 by default). The biggest problem is that there's no way to change the retransmission interval, which defaults to 75 seconds. This adds up to a total time of 'idle_time' + at least 75 seconds on BSD, which in my opinion is unacceptable for real-time purposes. A patch that was supposed to implement this was as far as I could find out, rejected, and the only way to change the timeouts is via sysctl, which obviously, requires root access.
So the bottom line is that this will have to be implemented eventually; using the Keep-Alive mechanism is inherently unreliable and unportable.

[ciphor]

I think the server should allow the client to send PING to detect connection liveness in Pub/Sub mode. Otherwise, the client cannot immediately recover from an occasional connection broken.

[swy19870105]

oh,leaning!

[laino]

bump!

This would solve some troubles I have with redis.

[antirez]

Thanks for raising this again in the issues list, I'll address this before the end of the week. Proposed solution:

• PING allowed in Pub/Sub mode.
• PING results into a message received by the client of special pong type.

In theory it could be cool to automatically send pings to idle connections from time to time, however this has issues with backward compatibility. However at a latter time when most clients will implement PING in the context of Pub/Sub this will be possible.

Please comment ASAP if you are not happy with the proposal.

[antirez]

Implementing the proposed solution (see my last comment) today, please PING me if you have some different idea. Btw before acting I'm going to re-read the whole thread...

[antirez]

Implemented, not closing for lack of tests (work in progress).

[churinga]

This behavioral change breaks hiredis library. Hiredis does not expect such command to return non-error response, causing redisProcessCallbacks() to segfault.

[badboy]

Sounds like a bug in hiredis. Can you fill an issue in https://github.com/redis/hiredis and provide a small test case? I can take care of that then.

[churinga]

@badboy I tested, and in older version of hiredis (0.10.x), it crashes upon receiving "pong" response in subscribing mode (in async.c), and in latest master version of hiredis, it no longer crashes, but it silently discarded the "pong" response, so that the callback of the "ping" command is never called.

I will need to find time to produce a test case, because it involves async I/O, and my current code base is very large. I will custom write a slimmer test case for this, maybe within a couple of weeks.

Thanks!

[antirez]

Implemented into unstable, will be part of Redis 3.2. Closed.

[aidevr]

Why i cannot issue PING or unsubscribe from redis-cli once subscribed to event.

[badboy]

@aidevr Because redis-cli switches to a blocking mode waiting for published messages. It does not take anymore user input at that point (besides C-c to exit)