[Feature Request] Add support for Redis Clusters

[derrickmehaffy]

Hello o/

First off thank you for creating this middleware! I have been testing it for the new Strapi rc versions (working great by the way) and I was wondering if you would be interested to add support for the redis-clustr as I believe this will be very helpful for larger installations of Strapi in a cluster and thus allowing all of them to share the cache.

Please do let me know what you think!

[derrickmehaffy]

Another alternative would be using ioredis which natively supports both Redis clusters and Sentinel HA

[patrixr]

Hey @derrickmehaffy

Thanks for trying it out :)

Definitely open to it, how do you see it being enabled ?

Maybe adding a cluster: true to the configuration? (other options?)

Would also need to see if it plays nicely with the LRU module I'm currently using for redis.

Let me know your thoughts

[derrickmehaffy]

Well my thought is moving over to ioredis you would instead just pass an object over to the ioredis package. (I am prototyping a ratelimit middleware that I'm not sure is really even useful but just playing around).

https://github.com/derrickmehaffy/strapi-middleware-ratelimit/blob/66be2dd9b5c2b666c9fa012e11005845ec3f3b53/lib/index.js#L13

https://github.com/derrickmehaffy/strapi-middleware-ratelimit/blob/66be2dd9b5c2b666c9fa012e11005845ec3f3b53/lib/defaults.json#L5-L11

Which based on how ioredis takes in the settings for normal vs cluster:

Normal: https://github.com/luin/ioredis#connect-to-redis
Sentinel: https://github.com/luin/ioredis#sentinel
Cluster: https://github.com/luin/ioredis#cluster

I think making the connection options as transparent as possible and simply allowing users to use whatever format ioredis allows would be best. Strapi itself actually does this with bookshelf/knex ORMs for the database in that is more or less just passes the contents of the database.json over to them and as such supports things like knex's tarn.js pool management.

Instead of directly referencing each part of the redis connection (host, port, password) just allowing for an object would make it pretty easy. Like you said though, it would need to be tested with your LRU module. The way redis clusters work is by partitioning each member of the cluster with a different set of key values. (If you are familiar with typical RAID hard drive storage, a default Redis cluster would be like a RAID 0 AKA spanning the data) Sentinels would be used for redundancy on top of this.

A normal non-redundant redis cluster would be like:
A1 Node: Hash Slots 0-5500
B1 Node: Hash Slots 5501-11000
C1 Node: Hash Slots 11001-16383

Where as a Sentinel would allow for:
A1(Master) / A2 (Slave): Hash Slots 0-5500
B1(Master) / B2 (Slave): Hash Slots 5501-11000
C1(Master) / C2 (Slave): Hash Slots 11001-16383

Which the Sentinels would watch for a Node 1 failure and automatically switch over to a Node 2.

It can get fairly complex in terms of setup, while most of my skill background is in devops and not actual Node/Javascript development, the key point I think is to open up to the users most or all of the options provided by Redis instead of just the single locally installed option. For the most part you would be interacting with ioredis and it would handle all the background logic for the clusters. I believe you can just do your normal set/find/ect and ioredis should handle it.

[derrickmehaffy]

I had actually touched on the topic of using Redis with Strapi as a caching system a few months ago in this discussion: https://github.com/strapi/strapi/discussions/3866

[patrixr]

@derrickmehaffy I just released version 1.0.2 which now uses the ioredis client

I'm forwarding the following properties from the middleware config to the redis client ['host', 'port', 'url', 'path', 'password', 'db', 'family']

Note: Some valuable bug fixes included

Let me know how that works out for you :)

[derrickmehaffy]

Certainly, I look forward to testing it tonight (I need to spin up a few test Redis clusters for this anyway). Likewise I can certainly do my level best to submit a pull request as well, though as I mentioned my Javascript/Node skills are certainly not quite to high level yet 😉

Thank you for working on this so quickly though, I certainly look forward to promoting this in the Strapi slack (if you aren't on, I would highly recommend it!).

[patrixr]

Always happy receiving PRs :)

I'll close this ticket, If you find any issues we can start categorising them as bugs 👍

See you on slack