Graceful reconnection with connection pooling

[miohtama]

I am running py-redis with a Celery application and using a connection pool. I am using Redis as a Celery broker and result backend. I connect o Redis over rediss:// that goes through a TLS terminator. This particular TLS terminator has a habit of regularly disconnecting the underlying TCP/IP connection.

I want to make my stack to do graceful connections to Redis when the connection is terminated.

1. Should redis-py connection pool be able to gracefully connect by itself

2. If graceful reconnection is not built in... in which part of the stack (Celery, redis-py) you see it meaningful to catch disconnect errors and rebuild the connection in the connection pool

3. If graceful reconnection should already happen.... in which part of redis-py this happens for the connection pool and what should I watch if it looks like reconnecting is not working

Currently what happens that after a forceful disconnect the pooled connection (seemingly) gets stuck and does not reconnet:

OSError: Connection closed by server.
  File "redis/connection.py", line 129, in _read_from_socket
    raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)

ConnectionError: Error while reading from socket: ('Connection closed by server.',)
  File "redis/client.py", line 572, in execute_command
    connection.send_command(*args)
  File "redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "redis/connection.py", line 538, in send_packed_command
    self.connect()
  File "redis/connection.py", line 446, in connect
    self.on_connect()
  File "redis/connection.py", line 514, in on_connect
    if nativestr(self.read_response()) != 'OK':
  File "redis/connection.py", line 577, in read_response
    response = self._parser.read_response()
  File "redis/connection.py", line 238, in read_response
    response = self._buffer.readline()
  File "redis/connection.py", line 168, in readline
    self._read_from_socket()
  File "redis/connection.py", line 143, in _read_from_socket
    (e.args,))

OSError: Connection closed by server.
  File "redis/connection.py", line 129, in _read_from_socket
    raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)

ConnectionError: Error while reading from socket: ('Connection closed by server.',)
  File "celery/app/trace.py", line 374, in trace_task
    R = retval = fun(*args, **kwargs)
  File "celery/app/trace.py", line 629, in __protected_call__
    return self.run(*args, **kwargs)
  File "xxx/tasks.py", line 373, in build_price_data
    _build_price_data(self.get_request())
  File "xxx/tasks.py", line 368, in _build_price_data
    rebuild_price_data(request, asset)
  File "xxx/graphs.py", line 393, in rebuild_price_data
    redis.set(key_name, json.dumps(data))
  File "redis/client.py", line 1072, in set
    return self.execute_command('SET', *pieces)
  File "redis/client.py", line 578, in execute_command
    connection.send_command(*args)
  File "redis/connection.py", line 563, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "redis/connection.py", line 538, in send_packed_command
    self.connect()
  File "redis/connection.py", line 446, in connect
    self.on_connect()
  File "redis/connection.py", line 514, in on_connect
    if nativestr(self.read_response()) != 'OK':
  File "redis/connection.py", line 577, in read_response
    response = self._parser.read_response()
  File "redis/connection.py", line 238, in read_response
    response = self._buffer.readline()
  File "redis/connection.py", line 168, in readline
    self._read_from_socket()
  File "redis/connection.py", line 143, in _read_from_socket
    (e.args,))

[lovemyliwu]

oo

[gdubicki]

With redis-py 3.2.1 I do see automatic reconnection of connection pool working at least in the following case: while having an app using redis-py connected to local redis server I stopped it, started again and and my app was working fine again.

I would close this ticket then as it's probably outdated and because of that it is misleadingly suggesting that redis-py is not gracefully reconnecting.

[andymccurdy]

redis-py connection pools ensure that a connection is healthy when the connection is given out via get_connection. If the connection is not healthy, it will attempt to reconnect to the server. If there's an error reconnecting, that error will be raised to the user.