-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Clustered rate limiters #5800
Comments
This is a harder change than I first thought; I don't have a Redis cluster to test against so I need to create some non-trivial test infrastructure first. |
|
Good blog post on this subject: https://redis.com/blog/redis-clustering-best-practices-with-keys/ |
The way I implemented this in my Crystal Redis library was to have |
@jgaskins Yep. The Pro Web UI already has a sharding support; the customer could set up the Web UI with each cluster member as a shard. |
If it's helpful, I started a cluster for testing with a Procfile containing:
and then running:
The config files mentioned do not need to exist, they are written by redis during the cluster setup command. In CI, we use a docker-compose containing: # This creates a redis cluster with 3 master nodes and 3 replicas, as
# recommended in https://redis.io/docs/manual/scaling.
# It's likely the replicas aren't necessary in test environments but I don't
# see any harm including them just to make CI that little bit closer to
# production.
#
# Clustered redis expects to have a public IP address and port that's used
# by the other nodes as well as clients. This is fiddly to get right in
# docker so we use a third party docker image that does that work for us.
# See https://github.com/bitnami/bitnami-docker-redis-cluster.
redis-node-0:
image: docker.io/bitnami/redis-cluster:7.0
environment:
- 'ALLOW_EMPTY_PASSWORD=true'
- 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
- 'REDIS_CLUSTER_CREATOR=yes'
- 'REDIS_CLUSTER_REPLICAS=1'
depends_on:
- redis-node-1
- redis-node-2
- redis-node-3
- redis-node-4
- redis-node-5
redis-node-1:
image: docker.io/bitnami/redis-cluster:7.0
environment:
- 'ALLOW_EMPTY_PASSWORD=true'
- 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
redis-node-2:
image: docker.io/bitnami/redis-cluster:7.0
environment:
- 'ALLOW_EMPTY_PASSWORD=true'
- 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
redis-node-3:
image: docker.io/bitnami/redis-cluster:7.0
environment:
- 'ALLOW_EMPTY_PASSWORD=true'
- 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
redis-node-4:
image: docker.io/bitnami/redis-cluster:7.0
environment:
- 'ALLOW_EMPTY_PASSWORD=true'
- 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
redis-node-5:
image: docker.io/bitnami/redis-cluster:7.0
environment:
- 'ALLOW_EMPTY_PASSWORD=true'
- 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5' |
There's a separate cluster client which implements it on top of the new redis-client gem: https://github.com/redis-rb/redis-cluster-client (Just interesting, not really relevant to this particular issue.) |
💚 |
If using many thousands or even millions of rate limiters, it can be hugely beneficial to use a Cluster as the rate limiter storage in order to scale beyond one Redis instance. We can modify the rate limiter data model to take advantage of "Hash tags" in the cluster spec1 to ensure all keys for a limiter X resides on the same Redis instance within the cluster. This ensures that any related Lua script will have local data to work with when running within Redis.
Note that the Web UI functionality is not cluster-friendly; it uses SCAN to look for all limiters to list but but this will only scan one Redis instance. Nevertheless, for the largest users this limitation on visibility (who's using the Web UI to look at millions of limiters??) is a reasonable trade off to enable massive scale.
cc @sj26
Footnotes
https://redis.io/docs/reference/cluster-spec/ ↩
The text was updated successfully, but these errors were encountered: