What does this error mean and how should I handle it?

[vgoklani]

Hi - Quick question, I occasionally get this exception, how should I handle it?

Thanks!

[E 150409 10:21:13 ioloop:588] Exception in callback None
Traceback (most recent call last):
File "/opt/anaconda/lib/python2.7/site-packages/tornado-4.1-py2.7-macosx-10.5-x86_64.egg/tornado/ioloop.py", line 840, in start
handler_func(fd_obj, events)
File "/opt/anaconda/lib/python2.7/site-packages/tornado-4.1-py2.7-macosx-10.5-x86_64.egg/tornado/stack_context.py", line 275, in null_wrapper
return fn(_args, *_kwargs)
File "/opt/anaconda/lib/python2.7/site-packages/tornadis/connection.py", line 189, in _handle_events
self._handle_read()
File "/opt/anaconda/lib/python2.7/site-packages/tornadis/connection.py", line 201, in _handle_read
chunk = self._read(self.read_page_size)
File "/opt/anaconda/lib/python2.7/site-packages/tornadis/connection.py", line 244, in _read
raise ConnectionError("error during socket.recv: %s" % e)
ConnectionError: error during socket.recv: [Errno 60] Operation timed out
Error -> 'StopObject' object is not iterable

[thefab]

It means that you probably have a very slow redis command and you hit a network timeout.

In this case, the yielded command with return an internal StopObject. And you probably try to iterate over it. I reproduce something similar with a BLPOP without timeout and a CLIENT KILL command in another connection.

A ConnectionError exception should be better in this case. We will try to fix it soon.

But... it's not the root cause of your error...

[vgoklani]

could you please give me an example of using the ConnectionError object for wrapping this command.

Thanks!

[thefab]

Just pushed into master some commits to do that. In this case, you should have a ConnectionError exception with no change to your code and no StopObject anymore. Can you confirm this ?

[vgoklani]

I've been able to replicate the error numerous times from the current branch. This is how I replicated the issue: I ran tornado on my laptop, and while the tornado server was running locally, I close the laptop screen (thereby breaking the connection to redis, which was running on my server). Now I open the laptop and hit the endpoint (I never restarted tornado), and I get this error:

Traceback (most recent call last):
  File "/opt/anaconda/lib/python2.7/site-packages/tornado-4.1-py2.7-macosx-10.5-x86_64.egg/tornado/ioloop.py", line 840, in start
    handler_func(fd_obj, events)
  File "/opt/anaconda/lib/python2.7/site-packages/tornado-4.1-py2.7-macosx-10.5-x86_64.egg/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "build/bdist.macosx-10.5-x86_64/egg/tornadis/connection.py", line 189, in _handle_events
    self._handle_read()
  File "build/bdist.macosx-10.5-x86_64/egg/tornadis/connection.py", line 201, in _handle_read
    chunk = self._read(self.read_page_size)
  File "build/bdist.macosx-10.5-x86_64/egg/tornadis/connection.py", line 243, in _read
    self.disconnect()
  File "build/bdist.macosx-10.5-x86_64/egg/tornadis/connection.py", line 173, in disconnect
    self._close_callback()
  File "build/bdist.macosx-10.5-x86_64/egg/tornadis/client.py", line 132, in _close_callback
    raise ConnectionError("closed connection")
ConnectionError: closed connection

Error -> ConnectionError('closed connection',) is not JSON serializableError -> ConnectionError('closed connection',) is not JSON serializable

This all makes sense, BUT, I have this in my code:

if client.is_connected() is False:
    yield client.connect()

The question is, why doesn't the server re-connect to redis when I hit this part of the code? I admit it's a poor way of re-producing the error, but I need our code to be bomb-proof.

Thanks!

[thefab]

very interesting issue @vgoklani ! I'm going to work on this ! thanks for this report

[thefab]

I just pushed into master some changes about this.

I can't reproduce easily your use case. And probably you need the #8 feature to completely fix your issue.

We are very interested to have a "bomb proof" code. So we are very interested by your feedback on this subject.

There is a new option "return_connection_error" in the Client constructor:
if True, return a ConnectionError object if the redis server close the connection. If False, return None (default) in that particular case.

=> and there is no exception raised in that particular case

[vgoklani]

Thank you so much for your efforts!!!

I'm still confused by this:

if client.is_connected() is False:
    yield client.connect()

if the connection is closed, shouldn't this just open it again? why do i have manage all these exceptions. I might be misunderstanding your comment above, what's the best way for me to call redis and manage the connection closed exception? Thanks again for all your help!

[thefab]

In your use case, tornadis detects the connection drop after the first command (and not before)

So you have to do something like:

client = Client(return_connection_error=True)
[...]

result = yield client.call("...")
if isinstance(result, ConnectionError):
    # you should be disconnected here
    yield client.connect()
    result = yield client.call("...") # retry your command

It's far from perfect. But resend automatically (at the tornadis level) the same command after a connection drop does not sound good to me.

[vgoklani]

This is exactly what I needed! You should add this example to the documents, so it's clear how you want to us to handle this case. It would be cool if there were a way to manage this directly with the client.call(), but I'll just use this for now. Thanks again for all your help!

[vgoklani]

I'm still getting timeout errors. It's a very difficult issue to test/replicate, as you have to wait for the connection to timeout. The "isinstance(result, ConnectionError)" isn't catching the exception, and it's propagating all the way up the stack and getting caught by the main exception handler in the AlertsData class. Here's my code snippet and the error below:

import sys
import tornado
import tornadis
from bson import json_util
from tornadis.exceptions import ConnectionError
import settings

client = tornadis.Client(host=settings.hostname1, port=6384, return_connection_error=True)
client.connect()

@tornado.gen.coroutine
def process_alerts_batch_async(channel='test', queries='test', start=-1500, end=-1, instance_id='processing'):
    pipeline = tornadis.Pipeline()
    for _query in queries:
        redis_key=channel + '_' + instance_id + '_' + _query
        pipeline.stack_call("ZRANGE", redis_key, start, end)

    results = yield client.call(pipeline)
    if isinstance(results, ConnectionError):
        yield client.connect()
        results = yield client.call(pipeline)

    raise tornado.gen.Return( results )

class AlertsData(tornado.web.RequestHandler):
    @tornado.gen.coroutine
    def post(self):
        try:
            data = yield process_alerts_batch_async()
            self.write(json_util.dumps(data))
            self.set_header('Content-Type', 'application/json; charset=utf-8')
            self.finish()
        except Exception as e:
            sys.stderr.write('Error -> %s' % e.message)

Error message:

Error -> you are not connected[

Note that the "Error ->" is in AlertsData which is calling the process_alerts_batch_async function. The "isinstance(results, ConnectionError)" isn't handling the error.

[vgoklani]

This is what's in the traceback, how should I handle this error:

  File "/opt/anaconda/lib/python2.7/site-packages/tornado-4.1-py2.7-macosx-10.5-x86_64.egg/tornado/gen.py", line 807, in run
    value = future.result()
  File "/opt/anaconda/lib/python2.7/site-packages/tornado-4.1-py2.7-macosx-10.5-x86_64.egg/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "/opt/anaconda/lib/python2.7/site-packages/tornado-4.1-py2.7-macosx-10.5-x86_64.egg/tornado/gen.py", line 212, in wrapper
    yielded = next(result)
  File "/Users/vgoklani/metricle/apps_dev/web_app/dashboard/utils.py", line 105, in get_intraday_prices
    results = yield client7.call(pipeline)
  File "build/bdist.macosx-10.5-x86_64/egg/tornadis/client.py", line 204, in call
    raise ClientError("you are not connected")

[thefab]

ok, very interesting (as usual)

one idea:

• the connect() method is an async one so when you do "client.connect()", you are not sure to be connected just after this line (so there is a risk to call "client.call()" before the connection is done)

one potential solution:

• can you try the autoconnect branch ? (just add "autoconnect=True" to the Client constructor call and remove "client.connect()" lines)

[vgoklani]

Thanks for the quick response! I'll switch branches. Quick Question: does "autoconnect=True" automatically reconnect? How does it handle connection timeouts? Are you resending automatically at the tornadis level?

[thefab]

autoconnect=True and you have automatic connect (or reconnect) at each "client.call()" call

if the disconnection is not seen before the command execution, there is no automatic "call another time" (dangerous) so keep something like this:

results = yield client.call(pipeline)
if isinstance(results, ConnectionError):
    results = yield client.call(pipeline)

[thefab]

@vgoklani : some news ? I would like to merge the autoconnect branch and make a new release. Is it ok for you ?

[thefab]

everything is merged and a new release 0.4 is out
examples are up2date

be careful about API breaks (see CHANGES.md file)

thanks for this discussion

the API is better now

[vgoklani]

Just got back! Thanks for all the updates! :)

Do you have any examples, or patterns, that should follow for using your library. Specifically, I am referring to the connection disconnection issues. What's the preferred method for connecting and then auto-reconnecting if the connection goes stale?

Thanks!

[thefab]

with 0.4, autoconnect=True by default and no exception is raised any more (in case of connection issues)

so don't connect by yourself and just test if the result is a ConnectionError like here:
https://github.com/thefab/tornadis/blob/master/examples/context_manager.py

no try/except block is required any more

[thefab]

and if a Client object returned a ConnectionError, the connection is closed and a new connection will be made at the first "call()" on this object (auto(re)connect))

please open new issues if you find some specific case or this behavior is not rock solid