Building Docker image seems to ignore constraints

[kurtrwall]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.10.3

What happened?

I built an Airflow Docker image using this repo as the docker context on the 2.10.3 tag and I'm running into incompatibility issues with the version of the Celery provider package. The constraints.txt file at /home/airflow/constraints.txt baked in the image pins celery at 3.7.2 but pip says I have 3.10.2.

Docker build configuration of the airflow image:

    build:
      context: https://github.com/apache/airflow.git#2.10.3
      dockerfile: Dockerfile
      args:
        ADDITIONAL_DEV_APT_DEPS: gcc
        AIRFLOW_BRANCH: 2.10.3
        AIRFLOW_EXTRAS: apache-hdfs,apache-kafka,apache-spark,atlassian-jira,celery,docker,elasticsearch,http,ldap,postgres,redis,papermill,virtualenv
        AIRFLOW_UID: "2060"
        AIRFLOW_VERSION_SPECIFICATION: ==2.10.3
        INSTALL_MSSQL_CLIENT: "false"
        INSTALL_MYSQL_CLIENT: "false"
        INSTALL_POSTGRES_CLIENT: "true"
        PYTHON_BASE_IMAGE: python:3.12-slim-bookworm

Thanks for any assistance!

What you think should happen instead?

I expect that the constraints file used would constrain the versions of the libraries installed, but that doesn't seem to be the case.

How to reproduce

Create a compose file with a service that has a build configuration like so:

airflow:
    build:
      context: https://github.com/apache/airflow.git#2.10.3
      dockerfile: Dockerfile
      args:
        ADDITIONAL_DEV_APT_DEPS: gcc
        AIRFLOW_BRANCH: 2.10.3
        AIRFLOW_EXTRAS: apache-hdfs,apache-kafka,apache-spark,atlassian-jira,celery,docker,elasticsearch,http,ldap,postgres,redis,papermill,virtualenv
        AIRFLOW_UID: "2060"
        AIRFLOW_VERSION_SPECIFICATION: ==2.10.3
        INSTALL_MSSQL_CLIENT: "false"
        INSTALL_MYSQL_CLIENT: "false"
        INSTALL_POSTGRES_CLIENT: "true"
        PYTHON_BASE_IMAGE: python:3.12-slim-bookworm

Operating System

OSX

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

This is interesting discussion, but not clear if this is isssue.

The way how you should build image and what build args to pass is described here with docker build commands - https://airflow.apache.org/docs/docker-stack/build.html#examples-of-image-customizing - se also arguments reference

You miss AIRFLOW_VERSION to be specified likely (but hard to say). If you need help with debugging stuff, please take a look at detailed output of the build process where you will see the exact steps and what commands are run at every step and see if you can figure out what is wrong. Generally, self analyse the output. If still in doubt, please copy && paste relevant output fragments showing the wrong constraints being used.

[potiuk]

Specifically please take a close look at the exact pip install command that are used and which files are downloaded as constraints and what packages are installed after every step.

[potiuk]

If you get that information out, I might be able to help you

[potiuk]

Also I think likely AIRFLOW_VERSION_SPECIFICATION should not be used. This is what might cause problems - it's optional and I am not sure what effect it has on constraints, we do not use it in any of the examples )