Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection pool won't release its free connection to be used after so many timeout errors #444

Closed
Hesarn opened this issue Feb 16, 2019 · 3 comments

Comments

@Hesarn
Copy link

Hesarn commented Feb 16, 2019

I'm having the same problem as in this issue that you mentioned:
aio-libs-abandoned/aioredis-py#231

and i also saw this issue too:
#196

I just make a burst of requests to test my redis with cached decorator of aiocache library and i just set the timeout to be too short to get timeout when performing too many requests. so after this load test, i'm just getting timeout error for every single request on connecting to redis.

but it sounds that aioredis fixed this issue by patching the asyncio.Lock. But i'm having this issue still. so maybe i'm missing something.

Here is the code to reproduce this:

import logging
from aiocache import cached, RedisCache
from aiocache.serializers import PickleSerializer

from sanic import Sanic
from sanic.response import json

logger = logging.getLogger(__name__)


@cached(
        cache=RedisCache,
        namespace="main",
        serializer=PickleSerializer(),
        ttl=10,
        endpoint='127.0.0.1',
        pool_max_size=2,
        pool_min_size=2,
        timeout=0.1,
        create_connection_timeout=0.1
    )
async def handler_get(req):
    return json('hello')


if __name__ == '__main__':
    app = Sanic(log_config=None, configure_logging=None)

    app.add_route(handler_get, '/', ['GET'])
    app.run(host='0.0.0.0', port=8080)
aiocache==0.10.1
aioredis==1.2.0

And then i start load testing with ab:
ab -n 10000 -c 1000 http://127.0.0.1:8080/

but after this load test, even a single curl is getting timeout error:
curl localhost:8080

As i traced down this issue a little, i found out that after load test, in the ConnecionsPool class of aioredis, the state of connections of the pool is not correct, as there is 2 used connection in _used field of pool instance and there is no free connection at all in _pool(which is a deque).

So i was wondering if you could help me out or if let me know if it's a bug or what.

Thanks

@argaen
Copy link
Member

argaen commented Feb 18, 2019

Hmmm weird, the code for patching the Lock is still in aioredis: https://github.com/aio-libs/aioredis/blob/master/aioredis/locks.py

I'll have a look when I can and try to reproduce

@fanjindong
Copy link

+1 the same problem

@Dreamsorcerer
Copy link
Member

We've migrated to the new redis library now, so maybe not a problem now.
Please reopen if you can still reproduce on master (or 0.12 once released).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants