storage for tracking rate limits

[static-moonlight]

Airflow gives me this warning:

Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.

When I follow the link, I was expecting to find something I could just configure. Instead I see a lot of python code and I am completely overwhelmed and have now idea what to do.

I don't even understand what that means. I don't understand why the default is in a state that requires explicit action, instead of using the already-connected database. I mean, there is persistent storage right there. I don't understand why this has to be done by writing some python code, instead of configuring some settings in the airflow.cfg. I don't even know if this warning is bad or not.

I'm totally confused and could need some help. What is the easiest (but also production-ready) way to solve this?

[starkmarkus]

It is not a broker in the task-execution sense. This is just storage for the rate limiter state, so the backend keeps short-lived counters such as request counts in the current window and when the window resets.

It is not meant for metrics or Airflow metadata.

For Valkey vs Redis: Valkey is broadly Redis-compatible. If Redis is already available, that is the safest default. If your deployment already uses Valkey and the limiter backend supports it, that should usually work too.

The “multiple webserver/API instances” part just means shared state matters if you run more than one Airflow web-facing process. In that case, a shared backend like Redis or Valkey keeps the rate limiting consistent across instances.

[static-moonlight]

1. Is that the broker thing I saw in some example deployments? Or is that a different thing I would need Redis for?
2. What kind of counters are handled by this? Only internal stuff like those rate limits? Or something else, as well?
3. Only recently I learned about the Redis/Valkey situation. Is Valkey fully API compatible to Redis? Or should I stick with Redis, just to be safe?
4. You mentioned multiple webserver instances. Did you mean the Airflow API server by that? Is it save to have 2 or more of those running?

Sorry for asking so many questions.

[starkmarkus]

No, this is separate from the task-execution broker. The rate limiter backend only stores short-lived counters for the limiter itself, not the task queue or other Airflow internals.

It does not handle metrics. It is just shared storage for rate-limit state.

On Valkey vs Redis: Valkey is a Redis fork and is generally used as a Redis-compatible replacement for common use cases. If you already have Redis, that is the safest default. If your setup already uses Valkey, it should usually work too, but I would still test it in your exact deployment.

And yes, having multiple webserver/API processes is normal in production. The important part is that they share the same limiter backend, otherwise each process would track limits independently.

[static-moonlight]

I checked the Helm-chart deployment for reference. It doesn't seem to have a deployment for that. I also checked the generated default config (airflow config list --defaults). It doesn't seem to contain a configuration option for the limiter URI, you mentioned earlier.

[starkmarkus]

Yes, that matches what I found.

This is not an airflow.cfg setting. RATELIMIT_STORAGE_URI belongs in webserver_config.py, for example:

RATELIMIT_STORAGE_URI = "redis://redis_host:6379/0"

For the Helm chart, the usual hook is to provide a custom webserver_config.py via webserver.webserverConfig / webserver.webserverConfigConfigMapName. For Airflow 3 API server setups, there is also an apiServer.apiServerConfig equivalent.

So the production path is:

1. use Redis or Valkey as shared rate-limit storage
2. set RATELIMIT_STORAGE_URI in webserver_config.py
3. mount that file through the Helm chart values

The docs could definitely make this clearer for Airflow users.

[static-moonlight]

I put it in, just like that, with a wrong URI, just to see what happens. The warning is gone, so I think I'm on the right track. I still have to cleanup the configuration. Since this is a python file, I should be able to utilize os.getenv() to get the configuration in (config map -> container env variables -> os.getenv()).

Interestingly, I don't see any errors yet, even when the ratelimit-Redis server doesn't exist yet.

Aside from that, your input was very helpful. Thank you so much!