Breeze build-image with dry-run-docker flag command

[Bowrna]

Apache Airflow version

main (development)

What happened

I tried to execute the ./breeze build-image --dry-run-docker

Good version of docker 20.10.10.
Dry run mode

Build image


docker buildx build --builder default --progress=tty --pull --build-arg PYTHON_BASE_IMAGE=python:3.7-slim-buster --build-arg AIRFLOW_VERSION=2.3.0.dev0 --build-arg AIRFLOW_BRANCH=main --build-arg AIRFLOW_EXTRAS=devel_ci --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES=true --build-arg ADDITIONAL_AIRFLOW_EXTRAS= --build-arg ADDITIONAL_PYTHON_DEPS= --build-arg ADDITIONAL_DEV_APT_COMMAND= --build-arg ADDITIONAL_DEV_APT_DEPS= --build-arg ADDITIONAL_DEV_APT_ENV= --build-arg ADDITIONAL_RUNTIME_APT_COMMAND= --build-arg ADDITIONAL_RUNTIME_APT_DEPS= --build-arg ADDITIONAL_RUNTIME_APT_ENV= --build-arg UPGRADE_TO_NEWER_DEPENDENCIES=false --build-arg CONSTRAINTS_GITHUB_REPOSITORY=apache/airflow --build-arg AIRFLOW_CONSTRAINTS_REFERENCE=constraints-main --build-arg AIRFLOW_CONSTRAINTS=constraints-source-providers --build-arg AIRFLOW_IMAGE_REPOSITORY=https://github.com/apache/airflow --build-arg AIRFLOW_IMAGE_DATE_CREATED=2022-02-09T09:24:11Z --build-arg BUILD_ID=0 --build-arg COMMIT_SHA=ebf760871470446fb5e0cd58edf8fec32547e65e --cache-from=ghcr.io/apache/airflow/main/ci/python3.7:cache -t ghcr.io/apache/airflow/main/ci/python3.7 --target main . -f Dockerfile.ci


   CI image.

   Branch name:             main
   Docker image:            ghcr.io/apache/airflow/main/ci/python3.7


   Airflow source version:  2.3.0.dev0
   Python version:          3.7
   Backend:                 sqlite

I tried to execute the docker command alone by extracting it out. But it failed to execute in the cache apt-get command while the breeze command worked properly

[+] Building 26.8s (8/38)
 => [internal] load build definition from Dockerfile.ci                                                                                                                                                0.0s
 => => transferring dockerfile: 36B                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 35B                                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/python:3.7-slim-buster                                                                                                                              1.5s
 => importing cache manifest from ghcr.io/apache/airflow/main/ci/python3.7:cache                                                                                                                       1.6s
 => [internal] load build context                                                                                                                                                                      0.6s
 => => transferring context: 417.24kB                                                                                                                                                                  0.6s
 => [ 1/33] FROM docker.io/library/python:3.7-slim-buster@sha256:66fbe736b614879d760c5569a775f2d8e6c3d60f9f33c6d39b1048a54f2d4cc0                                                                      0.0s
 => CACHED [ 2/33] RUN echo "Base image version: python:3.7-slim-buster"                                                                                                                               0.0s
 => ERROR [ 3/33] RUN apt-get update     && apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1     && apt-get install -y --no-install-recommends curl gnupg2     && mkdir -pv /u  23.6s
------
 > [ 3/33] RUN apt-get update     && apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1     && apt-get install -y --no-install-recommends curl gnupg2     && mkdir -pv /usr/share/man/man1     && mkdir -pv /usr/share/man/man7     && export      && bash -o pipefail -o errexit -o nounset -o nolog -c "    curl --silent --fail --location https://deb.nodesource.com/setup_14.x | bash -     && curl --silent --fail https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - >/dev/null 2>&1     && echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list"     && bash -o pipefail -o errexit -o nounset -o nolog -c ""     && apt-get update     && apt-get install -y --no-install-recommends            apt-utils            build-essential            dirmngr            dumb-init            freetds-bin            freetds-dev            git            graphviz            gosu            libffi-dev            libldap2-dev            libkrb5-dev            libpq-dev            libsasl2-2            libsasl2-dev            libsasl2-modules            libssl-dev            libenchant-dev            locales             netcat            nodejs            rsync            sasl2-bin            sudo            unixodbc            unixodbc-dev            yarn                 && apt-get autoremove -yqq --purge     && apt-get clean     && rm -rf /var/lib/apt/lists/*:
#7 0.603 Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
#7 0.740 Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
#7 0.878 Get:3 http://security.debian.org/debian-security buster/updates/main arm64 Packages [308 kB]
#7 0.992 Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
#7 1.031 Get:5 http://deb.debian.org/debian buster/main arm64 Packages [7735 kB]
#7 2.133 Get:6 http://deb.debian.org/debian buster-updates/main arm64 Packages [8784 B]

What you expected to happen

I expected the docker command to run successfully like breeze build-image command

How to reproduce

By executing the below docker command in AIRFLOW_HOME dir

docker buildx build --builder default --progress=tty --pull --build-arg PYTHON_BASE_IMAGE=python:3.7-slim-buster --build-arg AIRFLOW_VERSION=2.3.0.dev0 --build-arg AIRFLOW_BRANCH=main --build-arg AIRFLOW_EXTRAS=devel_ci --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES=true --build-arg ADDITIONAL_AIRFLOW_EXTRAS= --build-arg ADDITIONAL_PYTHON_DEPS= --build-arg ADDITIONAL_DEV_APT_COMMAND= --build-arg ADDITIONAL_DEV_APT_DEPS= --build-arg ADDITIONAL_DEV_APT_ENV= --build-arg ADDITIONAL_RUNTIME_APT_COMMAND= --build-arg ADDITIONAL_RUNTIME_APT_DEPS= --build-arg ADDITIONAL_RUNTIME_APT_ENV= --build-arg UPGRADE_TO_NEWER_DEPENDENCIES=false --build-arg CONSTRAINTS_GITHUB_REPOSITORY=apache/airflow --build-arg AIRFLOW_CONSTRAINTS_REFERENCE=constraints-main --build-arg AIRFLOW_CONSTRAINTS=constraints-source-providers --build-arg AIRFLOW_IMAGE_REPOSITORY=https://github.com/apache/airflow --build-arg AIRFLOW_IMAGE_DATE_CREATED=2022-02-09T09:24:11Z --build-arg BUILD_ID=0 --build-arg COMMIT_SHA=ebf760871470446fb5e0cd58edf8fec32547e65e --cache-from=ghcr.io/apache/airflow/main/ci/python3.7:cache -t ghcr.io/apache/airflow/main/ci/python3.7 --target main . -f Dockerfile.ci

Operating System

macOS

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Bowrna]

@potiuk While I could reproduce the issue, I am not sure about the root cause of this issue. Could you tell me how i can start with debugging this issue?

[potiuk]

This should be fixed now. It was a temporary error.

[Bowrna]

@potiuk This issue still persists.

[potiuk]

Hmmm. Very strange . IT works for mre when I run it and even when I copied the docker command...

Can you please rebase and check in the latest main ?

[potiuk]

[jarek:~/code/airflow] [breeze] remove-wrapper-script-from-static-checks+ 7s ± ./breeze build-image --dry-run-docker
Good version of docker 20.10.7.
Dry run mode

Build image


docker buildx build --builder default --progress=tty --pull --build-arg PYTHON_BASE_IMAGE=python:3.7-slim-buster --build-arg AIRFLOW_VERSION=2.3.0.dev0 --build-arg AIRFLOW_BRANCH=main --build-arg AIRFLOW_EXTRAS=devel_ci --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES=true --build-arg ADDITIONAL_AIRFLOW_EXTRAS= --build-arg ADDITIONAL_PYTHON_DEPS= --build-arg ADDITIONAL_DEV_APT_COMMAND= --build-arg ADDITIONAL_DEV_APT_DEPS= --build-arg ADDITIONAL_DEV_APT_ENV= --build-arg ADDITIONAL_RUNTIME_APT_COMMAND= --build-arg ADDITIONAL_RUNTIME_APT_DEPS= --build-arg ADDITIONAL_RUNTIME_APT_ENV= --build-arg UPGRADE_TO_NEWER_DEPENDENCIES=false --build-arg CONSTRAINTS_GITHUB_REPOSITORY=apache/airflow --build-arg AIRFLOW_CONSTRAINTS_REFERENCE=constraints-main --build-arg AIRFLOW_CONSTRAINTS=constraints-source-providers --build-arg AIRFLOW_IMAGE_REPOSITORY=https://github.com/apache/airflow --build-arg AIRFLOW_IMAGE_DATE_CREATED=2022-02-17T13:57:46Z --build-arg BUILD_ID=0 --build-arg COMMIT_SHA=5d1857dae1d3a6d3f712be2043d2aeb99d8c7b12 --cache-from=ghcr.io/apache/airflow/main/ci/python3.7:cache -t ghcr.io/apache/airflow/main/ci/python3.7 --target main . -f Dockerfile.ci 


   CI image.

   Branch name:             main
   Docker image:            ghcr.io/apache/airflow/main/ci/python3.7


   Airflow source version:  2.3.0.dev0
   Python version:          3.7
   Backend:                 mssql 

[jarek:~/code/airflow] [breeze] remove-wrapper-script-from-static-checks+ ± docker buildx build --builder default --progress=tty --pull --build-arg PYTHON_BASE_IMAGE=python:3.7-slim-buster --build-arg AIRFLOW_VERSION=2.3.0.dev0 --build-arg AIRFLOW_BRANCH=main --build-arg AIRFLOW_EXTRAS=devel_ci --build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES=true --build-arg ADDITIONAL_AIRFLOW_EXTRAS= --build-arg ADDITIONAL_PYTHON_DEPS= --build-arg ADDITIONAL_DEV_APT_COMMAND= --build-arg ADDITIONAL_DEV_APT_DEPS= --build-arg ADDITIONAL_DEV_APT_ENV= --build-arg ADDITIONAL_RUNTIME_APT_COMMAND= --build-arg ADDITIONAL_RUNTIME_APT_DEPS= --build-arg ADDITIONAL_RUNTIME_APT_ENV= --build-arg UPGRADE_TO_NEWER_DEPENDENCIES=false --build-arg CONSTRAINTS_GITHUB_REPOSITORY=apache/airflow --build-arg AIRFLOW_CONSTRAINTS_REFERENCE=constraints-main --build-arg AIRFLOW_CONSTRAINTS=constraints-source-providers --build-arg AIRFLOW_IMAGE_REPOSITORY=https://github.com/apache/airflow --build-arg AIRFLOW_IMAGE_DATE_CREATED=2022-02-17T13:57:46Z --build-arg BUILD_ID=0 --build-arg COMMIT_SHA=5d1857dae1d3a6d3f712be2043d2aeb99d8c7b12 --cache-from=ghcr.io/apache/airflow/main/ci/python3.7:cache -t ghcr.io/apache/airflow/main/ci/python3.7 --target main . -f Dockerfile.ci

[+] Building 146.9s (43/43) FINISHED                                                                                                                                                                                                                                                   
 => [internal] load build definition from Dockerfile.ci                                                                                                                                                                                                                           0.9s
 => => transferring dockerfile: 36B                                                                                                                                                                                                                                               0.8s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                 0.9s
 => => transferring context: 35B                                                                                                                                                                                                                                                  0.8s
 => [internal] load metadata for docker.io/library/python:3.7-slim-buster                                                                                                                                                                                                         1.9s
 => [auth] library/python:pull token for registry-1.docker.io                                                                                                                                                                                                                     0.0s
 => importing cache manifest from ghcr.io/apache/airflow/main/ci/python3.7:cache                                                                                                                                                                                                  1.5s
 => [internal] load build context                                                                                                                                                                                                                                                 1.9s
 => => transferring context: 1.83MB                                                                                                                                                                                                                                               1.9s
 => [auth] apache/airflow/main/ci/python3.7:pull token for ghcr.io                                                                                                                                                                                                                0.0s
 => [ 1/35] FROM docker.io/library/python:3.7-slim-buster@sha256:66fbe736b614879d760c5569a775f2d8e6c3d60f9f33c6d39b1048a54f2d4cc0                                                                                                                                                 0.0s
 => => resolve docker.io/library/python:3.7-slim-buster@sha256:66fbe736b614879d760c5569a775f2d8e6c3d60f9f33c6d39b1048a54f2d4cc0                                                                                                                                                   0.0s
 => CACHED [ 2/35] RUN echo "Base image version: python:3.7-slim-buster"                                                                                                                                                                                                          0.0s
 => CACHED [ 3/35] RUN apt-get update     && apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1     && apt-get install -y --no-install-recommends curl gnupg2     && mkdir -pv /usr/share/man/man1     && mkdir -pv /usr/share/man/man7     && export      &  0.0s
 => CACHED [ 4/35] COPY scripts/docker/install_mysql.sh scripts/docker/install_mssql.sh /scripts/docker/                                                                                                                                                                          0.0s
 => CACHED [ 5/35] RUN bash /scripts/docker/install_mysql.sh prod      && bash /scripts/docker/install_mysql.sh dev      && bash /scripts/docker/install_mssql.sh     && adduser --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password               --quiet "  0.0s
 => CACHED [ 6/35] RUN SYSTEM=$(uname -s | tr '[:upper:]' '[:lower:]')     && HELM_URL="https://get.helm.sh/helm-v3.6.3-${SYSTEM}-amd64.tar.gz"     && curl --silent --location "${HELM_URL}" | tar -xz -O "${SYSTEM}"-amd64/helm > /usr/local/bin/helm     && chmod +x /usr/loc  0.0s
 => CACHED [ 7/35] RUN mkdir -pv /usr/share/man/man1     && mkdir -pv /usr/share/man/man7     && export      && export      && bash -o pipefail -o errexit -o nounset -o nolog -c ""     && bash -o pipefail -o errexit -o nounset -o nolog -c ""     && apt-get update     && a  0.0s
 => CACHED [ 8/35] WORKDIR /opt/airflow                                                                                                                                                                                                                                           0.0s
 => CACHED [ 9/35] RUN mkdir -pv /root/airflow &&     mkdir -pv /root/airflow/dags &&     mkdir -pv /root/airflow/logs                                                                                                                                                            0.0s
 => CACHED [10/35] RUN echo "Airflow version: 2.3.0.dev0"                                                                                                                                                                                                                         0.0s
 => CACHED [11/35] COPY scripts/docker/install_pip_version.sh scripts/docker/install_airflow_dependencies_from_branch_tip.sh      scripts/docker/common.sh      /scripts/docker/                                                                                                  0.0s
 => CACHED [12/35] RUN echo -e "\n\e[32mThe 'Running pip as the root user' warnings below are not valid but we can't disable them :(\e[0m\n";     echo -e "\n\e[34mSee https://github.com/pypa/pip/issues/10556 for details.\e[0m\n" ;     bash /scripts/docker/install_pip_vers  0.0s
 => CACHED [13/35] COPY scripts/docker/install_pipx_tools.sh /scripts/docker/                                                                                                                                                                                                     0.0s
 => CACHED [14/35] RUN bash /scripts/docker/install_pipx_tools.sh                                                                                                                                                                                                                 0.0s
 => CACHED [15/35] COPY airflow/www/package.json airflow/www/yarn.lock /opt/airflow/airflow/www/                                                                                                                                                                                  0.0s
 => CACHED [16/35] COPY scripts/docker/prepare_node_modules.sh /scripts/docker/                                                                                                                                                                                                   0.0s
 => CACHED [17/35] RUN bash /scripts/docker/prepare_node_modules.sh                                                                                                                                                                                                               0.0s
 => CACHED [18/35] COPY airflow/www/webpack.config.js /opt/airflow/airflow/www/                                                                                                                                                                                                   0.0s
 => CACHED [19/35] COPY airflow/www/static /opt/airflow/airflow/www/static/                                                                                                                                                                                                       0.0s
 => CACHED [20/35] COPY scripts/docker/compile_www_assets.sh /scripts/docker/                                                                                                                                                                                                     0.0s
 => CACHED [21/35] RUN REMOVE_ARTIFACTS="false" BUILD_TYPE="build" bash /scripts/docker/compile_www_assets.sh                                                                                                                                                                     0.0s
 => CACHED [22/35] COPY setup.py /opt/airflow/setup.py                                                                                                                                                                                                                            0.0s
 => [23/35] COPY setup.cfg /opt/airflow/setup.cfg                                                                                                                                                                                                                                 0.0s
 => [24/35] COPY airflow/__init__.py /opt/airflow/airflow/__init__.py                                                                                                                                                                                                             0.0s
 => [25/35] COPY scripts/docker/install_airflow.sh /scripts/docker/                                                                                                                                                                                                               0.0s
 => [26/35] RUN if [[ true == "true" ]]; then         bash /scripts/docker/install_airflow.sh;     fi                                                                                                                                                                           134.3s
 => [27/35] COPY scripts/in_container/entrypoint_ci.sh /entrypoint                                                                                                                                                                                                                0.0s 
 => [28/35] RUN chmod a+x /entrypoint                                                                                                                                                                                                                                             0.4s 
 => [29/35] COPY scripts/docker/install_pip_version.sh scripts/docker/install_additional_dependencies.sh /scripts/docker/                                                                                                                                                         0.0s 
 => [30/35] RUN bash /scripts/docker/install_pip_version.sh;     if [[ -n "" ]]; then         bash /scripts/docker/install_additional_dependencies.sh;     fi                                                                                                                     3.7s 
 => [31/35] RUN if command -v airflow; then         register-python-argcomplete airflow >> ~/.bashrc ;     fi                                                                                                                                                                     0.5s 
 => [32/35] RUN echo "source /etc/bash_completion" >> ~/.bashrc                                                                                                                                                                                                                   0.4s 
 => [33/35] COPY . /opt/airflow/                                                                                                                                                                                                                                                  0.4s 
 => [34/35] WORKDIR /opt/airflow                                                                                                                                                                                                                                                  0.0s 
 => [35/35] RUN ln -sf /usr/bin/dumb-init /usr/local/bin/dumb-init                                                                                                                                                                                                                0.3s
 => exporting to image                                                                                                                                                                                                                                                            0.6s
 => => exporting layers                                                                                                                                                                                                                                                           0.6s
 => => writing image sha256:d9200196fc6c668e98ca889643197d3c5d466b708d02a11b632fc9d24e5bf100                                                                                                                                                                                      0.0s
 => => naming to ghcr.io/apache/airflow/main/ci/python3.7                

[potiuk]

Just for the record - that was a problem with Mac OS handling of platform the failure only happened on MacOS because buildkit approach defaults to host platform and not Linux. this is now fixed in the upcoming PRs.