Align rate limiter & cache memcached configuration

[guillecro]

Preflight Checklist

• I have completed all Troubleshooting Steps.
• I'm on the latest version of Directus.
• There's no other issue that already describes my problem.

Describe the Bug

We are a NGO that has multiple projects, many of them requires a backend, and we don't have devs for backend, so Directus will help us to achieve to develop great projects setting multiple Directus in our infrastructure.

We are deploying the Directus using docker.
For cache and rate limiting, we will be using 3 hosts of memcached. The deal with this is because creating a cluster of Redis HA takes a lot of work and we know that the load balancing for memcached is done in the client. So that simplifies our infrastructure.

We tried to deploy using this envs:

# ...
CACHE_STORE: 'memcache'
CACHE_MEMCACHE: 'array:memcached-01,memcached-02,memcached-03'
# ...
RATE_LIMITER_STORE: 'memcache'
RATE_LIMITER_MEMCACHE: 'array:memcached-01,memcached-02,memcached-03'
# ...

But we had this problem:

So, We investigated..
According to the log, it uses Client.create() with memjs but it seems it doesn't support the array sintax.
A set of memcached servers should be set as STRING and not array, so no arrays

Client.create = function(serversStr, options) {
  serversStr = serversStr || process.env.MEMCACHIER_SERVERS ||
                             process.env.MEMCACHE_SERVERS || 'localhost:11211';
  var serverUris = serversStr.split(',');
  var servers = serverUris.map(function(uri) {
    var uriParts = uri.split('@');
    var hostPort = uriParts[uriParts.length - 1].split(':');
    var userPass = (uriParts[uriParts.length - 2] || '').split(':');
    return new Server(hostPort[0], parseInt(hostPort[1] || 11211, 10), userPass[0], userPass[1], options);
  });
  return new Client(servers, options);
};

So we changed to:

# ...
CACHE_STORE: 'memcache'
CACHE_MEMCACHE: 'memcached-01,memcached-02,memcached-03'
# ...
RATE_LIMITER_STORE: 'memcache'
RATE_LIMITER_MEMCACHE: 'memcached-01,memcached-02,memcached-03'
# ...

And it works! It seems it creates the client!
But when I go to https://localhost:8055 it keeps loading and then we get this error:

We tried to just use one instance of memcached, and it worked:

# ...
CACHE_STORE: 'memcache'
CACHE_MEMCACHE: 'memcached-01'
# ...
RATE_LIMITER_STORE: 'memcache'
RATE_LIMITER_MEMCACHE: 'memcached-01'
# ...

We believe we should be able to use multiple hosts for memcached, but we don't know why it fails, or if it just the rate-limiter failing to achieve it.

I know that in the documentation it mentions a singular memcached instance, but we believe we should be able to set up multiple hosts for Cache and Rate Limiting specially if we are considering multiple Directus instances and/or high availability.

Variable	Description
CACHE_MEMCACHE	Location of your memcache instance
RATE_LIMITER_MEMCACHE	Location of your memcache instance

Any ideas or explanation would be very appreciated!

To Reproduce

Here is a docker-compose.yaml you can run in your local environment.

version: '3'
services:
  database:
    container_name: database
    image: mysql:5.7
    volumes:
      - ./data/database:/var/lib/mysql
    ports:
      - "3306:3306"
    networks:
      - directus
    environment:
      MYSQL_ROOT_PASSWORD: 'root'
      MYSQL_DATABASE: 'directus'

  memcached-alpha:
    container_name: memcached-alpha
    image: memcached:1.6-alpine
    networks:
      - directus

  memcached-beta:
    container_name: memcached-beta
    image: memcached:1.6-alpine
    networks:
      - directus

  memcached-gamma:
    container_name: memcached-gamma
    image: memcached:1.6-alpine
    networks:
      - directus


  directus:
    container_name: directus
    image: directus/directus:9.0.0-rc.91
    ports:
      - 8055:8055
    volumes:
      # By default, uploads are stored in /directus/uploads
      # Always make sure your volumes matches the storage root when using
      # local driver
      - ./uploads:/directus/uploads
      # Make sure to also mount the volume when using SQLite
      # - ./database:/directus/database
      # If you want to load extensions from the host
      # - ./extensions:/directus/extensions
    networks:
      - directus
    depends_on:
      - memcached-alpha
      - memcached-beta
      - memcached-gamma
      - database
    environment:
      KEY: 'uih123-iubp-r98r-4nf0n213'
      SECRET: '9812hd22-sioj-9j33-9ejnlkn5'

      DB_CLIENT: 'mysql'
      DB_HOST: 'database'
      DB_PORT: '3306'
      DB_DATABASE: 'directus'
      DB_USER: 'root'
      DB_PASSWORD: 'root'

      RATE_LIMITER_ENABLED: 'true'
      RATE_LIMITER_POINTS: '50'
      RATE_LIMITER_DURATION: '5'
      RATE_LIMITER_STORE: 'memcache'
      RATE_LIMITER_MEMCACHE: 'memcached-alpha,memcached-beta,memcached-gamma'

      EMAIL_TRANSPORT: 'smtp'
      EMAIL_SMTP_HOST: 'smtp.mailtrap.io'
      EMAIL_SMTP_PORT: ''
      EMAIL_SMTP_USER: ''
      EMAIL_SMTP_PASSWORD: ''
      EMAIL_SMTP_POOL: 'false'
      EMAIL_SMTP_SECURE: 'false'
      EMAIL_SMTP_IGNORE_TLS: 'false'
      
      TELEMETRY: 'false'

      CACHE_ENABLED: 'true'
      CACHE_STORE: 'memcache'
      CACHE_MEMCACHE: 'memcached-alpha,memcached-beta,memcached-gamma'


      ADMIN_EMAIL: 'admin@gmail.com'
      ADMIN_PASSWORD: 'd1r3ctu5'

      # Make sure to set this in production
      # (see https://docs.directus.io/reference/environment-variables/#general)
      # PUBLIC_URL: 'https://directus.example.com'

networks:
  directus:

Then go to http://localhost:8055
You will see that it fails after a little bit

Now stop the services, and change the yaml to

# ...
CACHE_STORE: 'memcache'
CACHE_MEMCACHE: 'memcached-01'
# ...
RATE_LIMITER_STORE: 'memcache'
RATE_LIMITER_MEMCACHE: 'memcached-01'
# ...

Run the services, boom! It works! But only one instance.

What version of Directus are you using?

v9.0.0-rc.91

What version of Node.js are you using?

Version shipped with directus/directus:9.0.0-rc.91 (npm 7.20.3)

What database are you using?

mysql:5.7

What browser are you using?

Chome and Firefox (latest)

What operating system are you using?

Ubuntu

How are you deploying Directus?

Docker (directus/directus:9.0.0-rc.91)

[rijkvanzanten]

Odd that the array syntax didn't work! The memcached docs seem to indicate that's exactly how you're supposed to configure clusters 🤔

I'm wondering if this is a compatibility problem within the rate limiter package we rely on instead of the underlying memcached itself 🤔

[rijkvanzanten]

Ahh I think I know where the problem is.. The cache provider (keyv) relies on memjs, while the rate limiter package relies on memcached. Memcached uses an array, while memjs uses a comma separated string 🙈

[rijkvanzanten]

Yup, that seems to be it. This is positively stupid, but try:

      RATE_LIMITER_MEMCACHE: 'array:memcached-alpha,memcached-beta,memcached-gamma'

      ...

      CACHE_MEMCACHE: 'memcached-alpha,memcached-beta,memcached-gamma'

that seems to make it go on my end

[guillecro]

I can confirm it works with this configuration, locally and in our production environment!
I haven't confirmed what happens with the Sessions management cause we are not using it in our case but might would be great to give it a look too :)

      RATE_LIMITER_MEMCACHE: 'array:memcached-alpha,memcached-beta,memcached-gamma'
      ...
      CACHE_MEMCACHE: 'memcached-alpha,memcached-beta,memcached-gamma'

I don't know what else I can do to help, An improvement would be for the documentation, mentioning the format for multiple instances of memcached, but I don't know if the team will plan something different in this case to normalize the env vars here.

Thank you @rijkvanzanten :)

[rijkvanzanten]

but I don't know if the team will plan something different in this case to normalize the env vars here.

Yeah, I think the ideal solution here is to programmatically account for the fact that keyv relies on a string. A simple "if array -> join with ','" would help tremendously in this case, as we could simply say that you'd use an array across the board 🙂