Clustered rate limiters

[mperham]

If using many thousands or even millions of rate limiters, it can be hugely beneficial to use a Cluster as the rate limiter storage in order to scale beyond one Redis instance. We can modify the rate limiter data model to take advantage of "Hash tags" in the cluster spec¹ to ensure all keys for a limiter X resides on the same Redis instance within the cluster. This ensures that any related Lua script will have local data to work with when running within Redis.

Note that the Web UI functionality is not cluster-friendly; it uses SCAN to look for all limiters to list but but this will only scan one Redis instance. Nevertheless, for the largest users this limitation on visibility (who's using the Web UI to look at millions of limiters??) is a reasonable trade off to enable massive scale.

cc @sj26

Footnotes

1. https://redis.io/docs/reference/cluster-spec/ ↩

[mperham]

This is a harder change than I first thought; I don't have a Redis cluster to test against so I need to create some non-trivial test infrastructure first.

[mperham]

• Concurrent - uses multiple keys, needs hash tagging
• Bucket - uses multiple keys, needs hash tagging
• Window - only uses one key, naturally cluster-safe
• Leaky - only uses one key, naturally cluster-safe
• Points - only uses one key, naturally cluster-safe

[mperham]

Good blog post on this subject: https://redis.com/blog/redis-clustering-best-practices-with-keys/

[jgaskins]

Note that the Web UI functionality is not cluster-friendly; it uses SCAN to look for all limiters to list but but this will only scan one Redis instance.

The way I implemented this in my Crystal Redis library was to have Cluster#scan_each run scan_each across one replica of each shard. IIRC the Ruby Redis library doesn't have that, and it may not be a great idea at a large-enough scale, but it could still work for this purpose.

[mperham]

@jgaskins Yep. The Pro Web UI already has a sharding support; the customer could set up the Web UI with each cluster member as a shard.

[sj26]

If it's helpful, I started a cluster for testing with a Procfile containing:

redis-node-1: cd tmp/redis-cluster/node-1 && redis-server --port 7001 --cluster-enabled yes --cluster-config-file node.conf --cluster-node-timeout 5000 --appendonly yes
redis-node-2: cd tmp/redis-cluster/node-2 && redis-server --port 7002 --cluster-enabled yes --cluster-config-file node.conf --cluster-node-timeout 5000 --appendonly yes
redis-node-3: cd tmp/redis-cluster/node-3 && redis-server --port 7003 --cluster-enabled yes --cluster-config-file node.conf --cluster-node-timeout 5000 --appendonly yes

and then running:

redis-cli --cluster create 127.0.0.1:700{1,2,3} --cluster-replicas 0

The config files mentioned do not need to exist, they are written by redis during the cluster setup command.

In CI, we use a docker-compose containing:

  # This creates a redis cluster with 3 master nodes and 3 replicas, as
  # recommended in https://redis.io/docs/manual/scaling.
  # It's likely the replicas aren't necessary in test environments but I don't
  # see any harm including them just to make CI that little bit closer to
  # production.
  #
  # Clustered redis expects to have a public IP address and port that's used
  # by the other nodes as well as clients. This is fiddly to get right in
  # docker so we use a third party docker image that does that work for us.
  # See https://github.com/bitnami/bitnami-docker-redis-cluster.
  redis-node-0:
    image: docker.io/bitnami/redis-cluster:7.0
    environment:
      - 'ALLOW_EMPTY_PASSWORD=true'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
      - 'REDIS_CLUSTER_CREATOR=yes'
      - 'REDIS_CLUSTER_REPLICAS=1'
    depends_on:
      - redis-node-1
      - redis-node-2
      - redis-node-3
      - redis-node-4
      - redis-node-5

  redis-node-1:
    image: docker.io/bitnami/redis-cluster:7.0
    environment:
      - 'ALLOW_EMPTY_PASSWORD=true'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-2:
    image: docker.io/bitnami/redis-cluster:7.0
    environment:
      - 'ALLOW_EMPTY_PASSWORD=true'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-3:
    image: docker.io/bitnami/redis-cluster:7.0
    environment:
      - 'ALLOW_EMPTY_PASSWORD=true'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-4:
    image: docker.io/bitnami/redis-cluster:7.0
    environment:
      - 'ALLOW_EMPTY_PASSWORD=true'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

  redis-node-5:
    image: docker.io/bitnami/redis-cluster:7.0
    environment:
      - 'ALLOW_EMPTY_PASSWORD=true'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'

[sj26]

IIRC the Ruby Redis library doesn't have that

There's a separate cluster client which implements it on top of the new redis-client gem:

https://github.com/redis-rb/redis-cluster-client

(Just interesting, not really relevant to this particular issue.)

[sj26]

💚

[gstokkink-bex]

@sj26 just wondering, what Redis client are you configuring for Sidekiq::Limiter, with what settings? I know there’s redis-cluster-client (https://github.com/redis-rb/redis-cluster-client) which seems like the new go-to solution. I'm also looking into setting up a Redis Cluster, just to store the rate limiters in.