Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions providers/celery/docs/celery_executor.rst
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,27 @@ Some caveats:
- Tasks can consume resources. Make sure your worker has enough resources to run ``worker_concurrency`` tasks
- Queue names are limited to 256 characters, but each broker backend might have its own restrictions

Redis maintenance
-----------------

When Redis is used as the Celery broker, Airflow uses it for transient Celery messages: queued task
commands, task acknowledgement data, and other broker-side state. Airflow task history, Dag runs,
connections, Variables, and XCom records are stored in the Airflow metadata database and should be
maintained with database tooling such as ``airflow db clean``. Cleaning the metadata database does not
clean Redis, and cleaning Redis does not clean Airflow's metadata database.

Do not flush or delete keys from the Redis database used by Airflow while schedulers or Celery workers
are running. Removing broker keys while tasks are queued, running, or being acknowledged can lose
pending work or make task state temporarily inconsistent. If you need to discard stale broker data
after an outage, first stop Airflow schedulers and Celery workers, make sure there are no tasks that
you intend to preserve in ``queued`` or ``running`` state, back up the Redis database if it is
persistent, and then remove only the Redis database or keyspace configured in ``[celery] broker_url``.

If you configured Redis as the Celery ``result_backend``, the same maintenance window should also
account for result backend keys. Deleting those keys can remove Celery task result information that the
scheduler may still query. For production deployments, Airflow recommends a database-backed result
backend so broker cleanup and task-result storage are handled separately.

See :doc:`apache-airflow:administration-and-deployment/modules_management` for details on how Python and Airflow manage modules.

Architecture
Expand Down
Loading