Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Connection management improvements. #886
Specifically, I modified
I also modified the
This was referenced
Nov 22, 2017
This PR has a couple of major issues. I attempted to use it in production and ran into some bugs.
First is line https://github.com/smartfile/redis-py/blob/f9348968dec581830a32e8469228c74ce443bd88/redis/connection.py#L1010 .
Second issue is https://github.com/smartfile/redis-py/blob/f9348968dec581830a32e8469228c74ce443bd88/redis/connection.py#L687 .
I updated the code and have been using it in production for 2 months without issue. Here is the diff for the improved version.
@taylor-cedar Good changes! Counting the connections rather than relying on a counter is better than the original method. Also, I work exclusively with Linux, so I am unable to test any other platform.
I do have a question about line 990 in your diff however,
@taylor-cedar OK, makes sense. Seems like we would want your changes merged into this PR. Do you think you can make a PR against my fork? I can merge them in and issue another PR or update this one.
I could try to add your changes myself, but I would not be able to test the Mac specific parts, and I would probably mess it up.
added a commit
this pull request
Feb 1, 2018
referenced this pull request
Feb 6, 2018
For limiting connections, have you guys had any luck using the
@TCTChadwell for me it is not about limiting connections, but about handling timeouts properly. The default pool reuses the same connections over and over. When load spikes new connections are created to handle load (OK). But then when load reduces these connections stagnate and are timed out by the server. The connection pool never notices and never closes them (it does not attempt to use them again) so they remain in CLOSE_WAIT (NOT OK.)
If load increases again these stale connections throw application errors when they are used again. The pool does not check the connection before returning it for use.
This change causes all connections in the pool to be used, round robin. This keeps timeouts at bay and detects errors. It also culls extras and performs a liveness test (ping) before returning a connection to the application. Broken connections are closed and replaced (errors that arise from CLOSE_WAIT while closing are handled and the connection is fully closed).
I don't need to limit connections, my redis server can handle the load, but by default connections are not properly managed by this pool.