diff --git a/providers/celery/docs/celery_executor.rst b/providers/celery/docs/celery_executor.rst index 6017c68c83c04..a2637b7decfa5 100644 --- a/providers/celery/docs/celery_executor.rst +++ b/providers/celery/docs/celery_executor.rst @@ -95,6 +95,27 @@ Some caveats: - Tasks can consume resources. Make sure your worker has enough resources to run ``worker_concurrency`` tasks - Queue names are limited to 256 characters, but each broker backend might have its own restrictions +Redis maintenance +----------------- + +When Redis is used as the Celery broker, Airflow uses it for transient Celery messages: queued task +commands, task acknowledgement data, and other broker-side state. Airflow task history, Dag runs, +connections, Variables, and XCom records are stored in the Airflow metadata database and should be +maintained with database tooling such as ``airflow db clean``. Cleaning the metadata database does not +clean Redis, and cleaning Redis does not clean Airflow's metadata database. + +Do not flush or delete keys from the Redis database used by Airflow while schedulers or Celery workers +are running. Removing broker keys while tasks are queued, running, or being acknowledged can lose +pending work or make task state temporarily inconsistent. If you need to discard stale broker data +after an outage, first stop Airflow schedulers and Celery workers, make sure there are no tasks that +you intend to preserve in ``queued`` or ``running`` state, back up the Redis database if it is +persistent, and then remove only the Redis database or keyspace configured in ``[celery] broker_url``. + +If you configured Redis as the Celery ``result_backend``, the same maintenance window should also +account for result backend keys. Deleting those keys can remove Celery task result information that the +scheduler may still query. For production deployments, Airflow recommends a database-backed result +backend so broker cleanup and task-result storage are handled separately. + See :doc:`apache-airflow:administration-and-deployment/modules_management` for details on how Python and Airflow manage modules. Architecture