Airflow Triggerer memory leak

[deepak4babu]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.2.0

What happened and how to reproduce it?

Airflow trigger memory is increasing continously

What you think should happen instead?

Memory should drop after certain period

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Other

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.17.1
apache-airflow-providers-cncf-kubernetes==10.14.0
apache-airflow-providers-common-compat==1.14.1
apache-airflow-providers-common-io==1.7.1
apache-airflow-providers-common-sql==1.33.0
apache-airflow-providers-fab==3.6.1
apache-airflow-providers-smtp==2.4.3
apache-airflow-providers-standard==1.12.1

Official Helm Chart version

Not Applicable

Kubernetes Version

Not applicable

Helm Chart configuration

Not applicable

Docker Image customizations

FROM apache/airflow:slim-3.2.0-python3.12

USER root

RUN apt-get update -y || true
RUN apt-get install --fix-missing -y \
    apt-transport-https=2.6.1 \
    build-essential=12.9 \
    default-libmysqlclient-dev=1.1.0 \
    g++=4:12.2.0-3 \
    gcc=4:12.2.0-3 \
    gnupg2=2.2.40-1.1+deb12u2 \
    htop=3.2.2-2 \
    libavif15=0.11.1-1+deb12u1 \
    libexpat1=2.5.0-1+deb12u2 \
    libffi-dev=3.4.4-1 \
    libgdk-pixbuf-2.0-0=2.42.10+dfsg-1+deb12u4 \
    libgl1-mesa-glx=22.3.6-1+deb12u1 \
    libgl1=1.6.0-1 \
    libglib2.0-0=2.74.6-2+deb12u8 \
    libgnutls30=3.7.9-2+deb12u6 \
    libgssrpc4=1.20.1-2+deb12u4 \
    libicu-dev=72.1-3+deb12u1 \
    libjxl0.7=0.7.0-10+deb12u1 \
    libkrb5-3=1.20.1-2+deb12u4 \
    libleptonica-dev=1.82.0-3+b3 \
    libmunge2=0.5.15-2+deb12u1 \
    libnss3=2:3.87.1-1+deb12u2 \
    libopencv-dev=4.6.0+dfsg-12 \
    libpam0g=1.5.2-6+deb12u2 \
    libperl5.36=5.36.0-7+deb12u3 \
    libpng-dev=1.6.39-2+deb12u4 \
    libsm6=2:1.2.3-1 \
    libssh-gcrypt-4=0.10.6-0+deb12u2 \
    libssl-dev=3.0.19-1~deb12u2 \
    libtbb-dev=2021.8.0-2+deb12u1 \
    libtbbmalloc2=2021.8.0-2+deb12u1 \
    libtiff6=4.5.0-6+deb12u4 \
    libtiffxx6=4.5.0-6+deb12u4 \
    libtk8.6=8.6.13-2 \
    libxext6=2:1.3.4-1+b1 \
    libxml2=2.9.14+dfsg-1.3~deb12u5 \
    libxrender-dev=1:0.9.10-1.1 \
    openssh-client=1:9.2p1-2+deb12u9 \
    procps=2:4.0.2-3 \
    python3-opencv=4.6.0+dfsg-12 \
    python3.11-minimal=3.11.2-6+deb12u6 \
    python3.11=3.11.2-6+deb12u6 \
    sqlite3=3.40.1-2+deb12u2 \
    sudo=1.9.13p3-1+deb12u3 \
    vim=2:9.0.1378-2+deb12u2 \
    wget=1.21.3-1+deb12u1

# separate run command for unpinned packages for caching
RUN apt-get update  && apt-get install --fix-missing -y libpq-dev libgl1 libxslt1.1

# Add tesseract apt url
RUN echo "deb https://notesalexp.org/tesseract-ocr5/$(lsb_release -cs)/ $(lsb_release -cs) main"\
    | sudo tee /etc/apt/sources.list.d/notesalexp.list > /dev/null
RUN wget -O - https://notesalexp.org/debian/alexp_key.asc | sudo apt-key add -

# Install tesseract
RUN apt-get update -y || true
RUN apt-get install -y \
    tesseract-ocr=5.5.2-1 \
    tesseract-ocr-eng=1:5.0.0~git39-6572757-3 \
    tesseract-ocr-bul=1:5.0.0~git39-6572757-3 tesseract-ocr-cat=1:5.0.0~git39-6572757-3 \
    tesseract-ocr-chi-sim=1:5.0.0~git39-6572757-3 tesseract-ocr-chi-tra=1:5.0.0~git39-6572757-3 \
    tesseract-ocr-ces=1:5.0.0~git39-6572757-3 tesseract-ocr-dan=1:5.0.0~git39-6572757-3 \
    tesseract-ocr-deu=1:5.0.0~git39-6572757-3 tesseract-ocr-ell=1:5.0.0~git39-6572757-3 \
    tesseract-ocr-eus=1:5.0.0~git39-6572757-3 tesseract-ocr-fin=1:5.0.0~git39-6572757-3 \
    tesseract-ocr-fra=1:5.0.0~git39-6572757-3 tesseract-ocr-glg=1:5.0.0~git39-6572757-3 \
    tesseract-ocr-hrv=1:5.0.0~git39-6572757-3 tesseract-ocr-hun=1:5.0.0~git39-6572757-3 \
    tesseract-ocr-ind=1:5.0.0~git39-6572757-3 tesseract-ocr-isl=1:5.0.0~git39-6572757-3 \
    tesseract-ocr-ita=1:5.0.0~git39-6572757-3 tesseract-ocr-nld=1:5.0.0~git39-6572757-3 \
    tesseract-ocr-pol=1:5.0.0~git39-6572757-3 tesseract-ocr-por=1:5.0.0~git39-6572757-3 \
    tesseract-ocr-ron=1:5.0.0~git39-6572757-3 tesseract-ocr-rus=1:5.0.0~git39-6572757-3 \
    tesseract-ocr-slk=1:5.0.0~git39-6572757-3 tesseract-ocr-spa=1:5.0.0~git39-6572757-3 \
    tesseract-ocr-swe=1:5.0.0~git39-6572757-3 

RUN wget https://dot.net/v1/dotnet-install.sh -O dotnet-install.sh
RUN chmod +x ./dotnet-install.sh
RUN ./dotnet-install.sh --version 8.0.8 --runtime dotnet --install-dir /home/airflow/.dotnet

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Please test it with memray and provide us flamegraphs https://airflow.apache.org/docs/apache-airflow/stable/howto/memory-profiling.html

[wjddn279]

@potiuk

I've open two PRs to support memray tracing in triggerer (not now) #65994 and add option to provide deeper tracing in memray #65996. Can you see it? ,

[potiuk]

Oh absolutely :)

[potiuk]

I just fif not get to it yet with all the PR flood and security things going on. Maybe in the meantime (befor we merge and release it - you could help @deepak4babu if they have some questions about it) @wjddn279 - you seem to have the most experience with it :)

[potiuk]

BTW. @deepak4babu -> we can convert it back to issue when we have more information and we can figure out some root causes for it from your tests.

[wjddn279]

@potiuk sure!

[potiuk]

Ideally - analyse it before to se if this is not issue in one of your deps.

[wjddn279]

@deepak4babu
The memory profiling described in this guide doesn't yet cover the triggerer. I've opened a PR for it, and it'll likely ship in the next release.

If you'd like to try it before then, feel free to apply the patch from that PR #65994. Happy to help if you run into any trouble :)

[arkadiuszbach]

May be the same reason as for dag processor memory leak, see 2nd point here: #50708 (comment)

Seems like structlog.BytesLogger is also called for each trigger and it leaks file handles:

airflow/airflow-core/src/airflow/jobs/triggerer_job_runner.py

Line 351 in 3bc3ccf

underlying_logger = structlog.BytesLogger(self._filehandle)