Is it possible to use a Redis connection pool?

[pieroliviermarquis]

Connecting to a connection pool would be very useful. I have an issue with my application reaching the maximum number of Redis clients.

[VeNoMouS]

if you running this under gunicorn etc... the short answer is no, each thread will be launched under its own application context, so it wont share a GIL ...

What I personally ended up doing , was using nutcracker and having python connect to that, to minimize the impact to the redis server

Edit:

ahhh sorry i came back to edit this, i just remember, due to limits package running "script" cmds to redis i couldnt run limiter through nutcracker

Edit 2:

I haven't tested it yet... but you could try https://github.com/RedisLabs/redis-cluster-proxy

[alisaifee]

When you say connection pool, do you mean being able to do something like this:

from limits.storage import RedisStorage
from limits.strategies import MovingWindowRateLimiter
import redis
pool = redis.ConnectionPool(host='localhost', port=6379, db=0)
storage = RedisStorage(connection_pool = pool)
limiter = MovingWindowRateLimiter(storage)

Unfortunately not at the moment - since the redis implementation delegates the construction of the redis client entirely to the redis.Redis.from_url function.

Having said it does seem reasonable to reuse a shared connection pool in your application and there have been other use-cases where letting the application provide a "Storage" that has been pre-constructed.

[alisaifee]

This issue should be in the limits repository, so I'll transfer it there.

[VeNoMouS]

Well the problem comes when your using flask / gunicorn for example under green threads.. gunicorn isnt going to share the connection / connection pool because of the green thread...

So each thread is going to open up its own connection pool and when you have a cluster of servers it really hits the redis... when you doing thousands of requests a second

you can put nutcracker on the local system as a "middleware" to use just one connection pool from the server itself to the redis server , but due to the exact queries your code is running, it isnt supported in nutcracker

https://github.com/twitter/twemproxy

[alisaifee]

I think you might be able to use redis-proxy in envoy (which supports lua scripting afaik) reference

[VeNoMouS]

When you say connection pool, do you mean being able to do something like this:

from limits.storage import RedisStorage
from limits.strategies import MovingWindowRateLimiter
import redis
pool = redis.ConnectionPool(host='localhost', port=6379, db=0)
storage = RedisStorage(connection_pool = pool)
limiter = MovingWindowRateLimiter(storage)

Unfortunately not at the moment - since the redis implementation delegates the construction of the redis client entirely to the redis.Redis.from_url function.

Having said it does seem reasonable to reuse a shared connection pool in your application and there have been other use-cases where letting the application provide a "Storage" that has been pre-constructed.

HI @alisaifee,

just circling back around to this, to see if this would be possible ? as if you using gunicorn etc, using a redis BlockingConnectionPool and monkey patching gevent can limit the number / rape connections being opened

would be awesome if we could just pass a redis instance to limiter rather than a URI, that way we can share an already created pool instance with the rest of the application and not create an entire new pool...

[VeNoMouS]

could

limits/limits/storage/redis.py

Lines 126 to 129 in b22ccbb

	redis = self.dependencies["redis"]
	uri = uri.replace("redis+unix", "unix")
	self.storage = redis.from_url(uri, **options)

be replaced with something along like the follwoing...

    redis = self.dependencies["redis"]
    if isinstance(uri, redis.client.Redis):
        self.storage = uri
    else:
        uri = uri.replace("redis+unix", "unix")
        self.storage = redis.from_url(uri, **options)

[alisaifee]

@VeNoMouS could you take a look at #110 to see if that'll work for you?

[VeNoMouS]

Will test today for you, i'll have to hack your flask limiter up first to allow the new param

mmmm i'll make some simple test code and test outside of flask first to see if it uses pool connection

[alisaifee]

You shouldn't need to hack anything in flask-limiter, this should work:

from flask_limiter import Limiter
pool = ....
limiter = Limiter(....., storage_uri="redis://", storage_options={"connection_pool": pool})

[VeNoMouS]

You shouldn't need to hack anything in flask-limiter, this should work:

from flask_limiter import Limiter
pool = ....
limiter = Limiter(....., storage_uri="redis://", storage_options={"connection_pool": pool})

ok cheers will test out in a bit and let you know

[alisaifee]

🙇

[VeNoMouS]

Since my stuff is in a docker... this is a dirty trick to pull the code base with the PR as a zip for requirements.txt incase you didnt know...

https://api.github.com/repos/alisaifee/limits/zipball/pull/110/head

[VeNoMouS]

Appears to be working @alisaifee

[VeNoMouS]

Testing it now on a test production server

[alisaifee]

I've cut a 2.4.0 release if you want to give that a shot as well!

[VeNoMouS]

Sure , give me ~hour and ill roll out another test

[VeNoMouS]

2.4.0 appears to be working

[alisaifee]

Awesome! Thanks for helping verify and also the idea!

[VeNoMouS]

thanks for patching ❤️