High CPU and database load caused by Airflow 3 dag-processor and scheduler

[michaelosthege]

Apache Airflow version

3.0.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Yesterday I upgraded my small home setup (Raspberry Pi, 2 dags, 3 tasks) to Airflow 3.

In comparison to Airflow 2.9.0 the CPU load has increased dramatically:

A few minutes later I also stopped the scheduler and database CPU load went down too.

What you think should happen instead?

Airflow 3 should cause the same or lower CPU load to Airflow 2.

How to reproduce

My docker-compose setup uses this Dockerfile

FROM apache/airflow:3.0.2

ENV AIRFLOW__CORE__EXECUTOR=LocalExecutor
ENV AIRFLOW__API__BASE_URL="http://junas.fritz.box:5000"
ENV AIRFLOW__CORE__AUTH_MANAGER="airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager"
# Note: AIRFLOW__API_AUTH__JWT_SECRET is set in the .env file
ENV AIRFLOW__DATABASE__SQL_ALCHEMY_CONN="postgresql+psycopg2://airflowu:xxxxx@raspi.fritz.box:5001/airflowdb"
ENV AIRFLOW__CORE__LOAD_EXAMPLES="False"
ENV AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK="True"


# Settings to reduce CPU load
ENV AIRFLOW__API__WORKERS=1
ENV AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL=120
ENV AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES=1
ENV AIRFLOW__DAG_PROCESSOR__BUNDLE_REFRESH_CHECK_INTERVAL=60

COPY requirements.txt /tmp/requirements.txt  # one entry: paho-mqtt
RUN pip install -r /tmp/requirements.txt

The docker-compose services:

  airflow-db:
    container_name: airflow-db
    build:
      context: ./dockerfiles/airflow-postgres
      dockerfile: Dockerfile
    restart: unless-stopped
    ports:
      - "5001:5432"
    volumes:
      - af_pgdata:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: airflowdb
      POSTGRES_USER: airflowu
      POSTGRES_PASSWORD: xxxxxxx
  airflow-api-server:
    container_name: airflow-api-server
    build:
      context: ./dockerfiles/airflow
      dockerfile: Dockerfile
    depends_on:
      - airflow-db
    restart: unless-stopped
    env_file: ./dockerfiles/airflow/.env
    ports:
      - "5000:8080"
    user: "1000:0"
    volumes:
      - /home/admin/airflow/dags:/opt/airflow/dags
      - /home/admin/airflow/logs:/opt/airflow/logs
    command: bash -c "airflow db migrate && airflow api-server"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/api/v2/version"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s
  airflow-dag-processor:
    container_name: airflow-dag-processor
    build:
      context: ./dockerfiles/airflow
      dockerfile: Dockerfile
    depends_on:
      - airflow-db
      - airflow-api-server
    restart: unless-stopped
    env_file: ./dockerfiles/airflow/.env
    user: "1000:0"
    volumes:
      - /home/admin/airflow/dags:/opt/airflow/dags
      - /home/admin/airflow/logs:/opt/airflow/logs
    command: bash -c "airflow dag-processor"
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type DagProcessorJob']
      interval: 30s
      timeout: 20s
      retries: 5
      start_period: 30s
  airflow-scheduler:
    container_name: airflow-scheduler
    build:
      context: ./dockerfiles/airflow
      dockerfile: Dockerfile
    depends_on:
      - airflow-db
      - airflow-api-server
      - airflow-dag-processor
    restart: unless-stopped
    env_file: ./dockerfiles/airflow/.env
    user: "1000:0"
    volumes:
      - /home/admin/airflow/dags:/opt/airflow/dags
      - /home/admin/airflow/logs:/opt/airflow/logs
    command: bash -c "airflow scheduler"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8974/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 30s

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Airflow 3 should cause the same or lower CPU load to Airflow 2.

This is by no means guaranteed or expected. The characteristics of Airflow 3 should be different, but "lower" CPU usage in all scenarios is not guaranteed.

I think - in order for this one to be actionable, you need to provide a LOT more information. Airflow has never been targetted for Raspbery PI environment, and while it **might ** work there, there might be many reasons why some resources (not necesarily CPU) will not be enough. It could for example mean that you started to run out of memory and this caused CPU usage while some processes are being swapped out. Or accidentally you might have started using (depend how you build it) Amd images and emulation kicked in (that would certainly explain extremely high CPU usage).

So in order to be able to help you with your problem you need to do a lot more investigation - there is absolutely no way we can help you with that little information that you provided.

You could do a lot of investigation:

1. check other resources (memory especially) - and possibly using memray to see what is using memory
2. check the architecture your images are using
3. run airflow py-spy to see what is really taking a lot of time and provide py-spy flamegraphs

For now - this is an interesting discussion, so I convert it to a discussion, but if you provide more evidences and details we might turn it into a real issue.

[ashb]

That while True in the manager is not a tight loop -- we call to self._service_processor_sockets(timeout=poll_time) which does a select/epoll/kqueue etc, so should be yielding CPU time, unless we can't get around the loop on under a second.

[michaelosthege]

According to the documentation, BaseSelector.select is supposed to wait until some registered file objects become ready, or the timeout expires.
So it's going to continue before that timeout if it can.

Maybe I'm missing something here, but in my understanding that's what a timeout is supposed to do.
What this loop needs IMO is a sleep(max(0, interval - loop_duration)).

[michaelosthege]

Turns out the dag-processor loop is fine, but the healthcheck was the issue!

My dag-processor healthcheck (copied from the docker-compose example) was:

healthcheck:
  test: ["CMD-SHELL", 'airflow jobs check --job-type DagProcessorJob']
  interval: 30s
  timeout: 20s
  retries: 5
  start_period: 30s

I disabled it yesterday and the container's CPU load went down to ~1 %.
Now I just ran

watch -n 5 docker exec -it airflow-dag-processor airflow jobs check --job-type DagProcessorJob

and the CPU of the dag-processor, as well as the database reads go 🚀 🔥

---

The CPU load by the Postgres container remained high regardless. Here it's definitely the scheduler, but I don't know why.

[ashb]

I wonder if the DB load is caused by #53401 -- a missing index

[michaelosthege]

I'll be happy to test the next release candidate.

---

But the inefficient airflow jobs check --job-type DagProcessorJob healthcheck is also concerning, don't you think? That CPU load is not in the database.

[michaelosthege]

Looks like Airflow 3.0.5 finally fixed it

I did not check wheter airflow jobs check --job-type DagProcessorJob still causes high CPU load, but I'm happy again regardless.