Switch to hardened Docker images in Dockerfiles

[shahar1]

Description

Following the fresh new statement from Docker, hardened container images are now free and open source - which might mean that we could eventually utilize it in Airflow to enhance security.

The main challenge is that it requires authentication, which could be problematic legally & practically for local builds. If we get to overcome this one, implementation will be a great security improvement.

Use case/motivation

Improve container's security

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[alessandro-peyrachia-rnc]

Hi,
coming in from the Slack channel.
This comes with an exceptional timing for us.
We are very new to airflow, we literally started 1 month ago, and because of the market we operate in, we need secured/hardened images.
I was starting to understand how to do it properly, then I saw your message.
We are on airflow 3 in AKS, I'm building experience on it, please count me in to help out on testing.

[potiuk]

will do

[suii2210]

Hi! I’m interested in working on this. For hardened images, should we aim for XYZ base image? Are there specific guidelines we must follow?

[shahar1]

Hi! I’m interested in working on this. For hardened images, should we aim for XYZ base image? Are there specific guidelines we must follow?

Thank you for your will to help in this area! I assigned you to this task.
Basically, it should be simple as switching the base image in the Docker file to the hardened version - see instructions in the link.
Building this image requires logging it to Dockerhub using a Personal Access Token (PAT) before the build step. In the GitHub Actions, the step should be:

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          registry: dhi.io
          username: ${{ vars.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_PAT }}

I tested the above in another repository and it works - but it will be problematic for now to configure it in Airflow's CI/CD, because I cannot use my personal user for the credentials (and AFAIK, we currently don't have a user dedicated for ASF organization).
Therefore, you could start by testing locally and see that nothing breaks in their image, and if it all works - create a documentation for instructing others how to make it work.
In parallel I could try to get in touch with Airflow's infra. team to see if we could maybe arrange a Dockerhub user for using it in Airflow's CI/CD.

[potiuk]

Hi! I’m interested in working on this. For hardened images, should we aim for XYZ base image? Are there specific guidelines we must follow?

Basically the same images we have now (debian:bookworm) - but ideally we should use "python images" and get rid of building python on our own (We recently switched from community "official" python images in #53150 so we basically need to revert back - but using the "hardened" images.

The images itself (https://www.docker.com/products/hardened-images/) are already based on debian - and there seem to be all python versions available in https://hub.docker.com/hardened-images/catalog/dhi/python/images. We want to use debian 12 (bookwrorm) based ones, and we do not want to use the FIPS version becuase FIPS variants are not available under free licence and require paid enterprise plan from Docker. However - it should be possible for those users who want to rebuild their images using fips-compliant images, so we should likely test it and describe as an option for users in "Build images" docs.

@shahar1 mentioned that they require sign-in, and you will need to do docker login ghi.io with your account to work on those, but I think when we decide to merge them we will have to setup a separte process to mirror the images to "ASF" managed images where we will make them publicly available so that they can be used by anyildone during development without the need to login do ghi.io with docker credentials. When you got to the point when you want to test in our CI, let us know and we will make manual mirroring and once it works we will automatet the mirroring.

Few caveats:

• I think we should be able to simplify both Dockerfile.ci and Docker - because the images come in two variants: dev (with -dev suffix) and runtime (without suffix) and we can utilise it to simplify our image buildin. Currently what we are doing is:

For CI image -> we use a single stage image (except scripts that are coming from it's own stage) -> and the image is bassed on "debian-slim" image and then we install build-essentials and other development tools needed. Likely the -dev image already has pretty much everything we neeed, so for CI image we should use the -dev image and strip out installing most development dependencies.

For PROD image, this is a bit more complex (but also a nice optimisation for us) - because we use multi-stage images. Stage 1 (build) install (and potentially builds) all Python packages, and then we copy those packages from build stage to the main stage from .local folder - because we install those packages with pip install --user. In the main stage we use the .local folder as venv (works nicely) and we use "airflow" user - we also make the image openshift-compatible by using group 0 that all users (including airflow) share and automatically create a user if random uid is used by openshift.

So for PROD image we want to:

• use -dev image for build stage - and also remove unnecessary installing of build-essentials and similar dependencies.
• use runtime image for main stage

The little difficulty here is the question of users: the -dev images are root, but the runtime ones are nonroot which means that some user mapping will have to happen, and (because Python installation is not relocatable) - some things with user management might need fixing.

Of course entrypoints might need some adjustments (we are using dumb-init) - also we need to come back to passing "base image" through build args as it was before #59517 - and this shoudl be a way for our users to pass fips image as build arg if they want to make the image fips-compliant.

The images as well have something calls sfw - secure firewall - which apparently is providing some (proxy based?) security when installing packages, I do not know details of it - but it also might have some impact.

There is also an issue we discovered recently - the "official" images had .pyc CPython files removed from installation of Python - which had very little savings for size of the image, but caused an unexpected memory leak -> so we need to check if the hardened images do not have the same problem (and we might even report that issue to Docker if they do) - fix to it was #58944 - for that we might need to compile all CPython files - both in build and main segment - but this might be somewhat more complicated for nonroot main image (we woudl need to switch to root temporarily).

Ideally - we shoudl follow exactly what Hardened images do when it comes to security - signing, sboms etc. but this could ba a follow up - we've never done that for our images, but it's great opportunity to just follow what docker does with their images.

We should also compare sizes of images before and after the change to avoid balooning the size of the image.

That's about all guidelines I can come up with from the top of my head.

[potiuk]

@shahar1 -> our messages crosed :). It's a bit more (see my message) - there will be quite a few changes needed because those images are much better suited for our build and we can come back to using pre-built Python hardened images which simplifies the build process (+ "dev/runtime" split will also simplify our PROD Dockerfile - as we will be able to use dev image directly for "build" segment.

In parallel I could try to get in touch with Airflow's infra. team to see if we could maybe arrange a Dockerhub user for using it in Airflow's CI/CD.

We will just mirror the base images to "airflow" managed dockerHub (and make it public). The images are licenced with Apache Licence 2.0 so we can redistribute them.

[suii2210]

Hi! I’m interested in working on this. For hardened images, should we aim for XYZ base image? Are there specific guidelines we must follow?

Thank you for your will to help in this area! I assigned you to this task. Basically, it should be simple as switching the base image in the Docker file to the hardened version - see instructions in the link. Building this image requires logging it to Dockerhub using a Personal Access Token (PAT) before the build step. In the GitHub Actions, the step should be:

  - name: Login to Docker Hub
    uses: docker/login-action@v3
    with:
      registry: dhi.io
      username: ${{ vars.DOCKERHUB_USERNAME }}
      password: ${{ secrets.DOCKERHUB_PAT }}

I tested the above in another repository and it works - but it will be problematic for now to configure it in Airflow's CI/CD, because I cannot use my personal user for the credentials (and AFAIK, we currently don't have a user dedicated for ASF organization). Therefore, you could start by testing locally and see that nothing breaks in their image, and if it all works - create a documentation for instructing others how to make it work. In parallel I could try to get in touch with Airflow's infra. team to see if we could maybe arrange a Dockerhub user for using it in Airflow's CI/CD.

Got it - I’ll start by testing the hardened base image locally (with Docker Hub PAT login) and validate that Airflow builds and runs correctly without regressions. Once that’s confirmed, I’ll prepare clear documentation covering local usage, authentication requirements, and any caveats.

[suii2210]

Hi! I’m interested in working on this. For hardened images, should we aim for XYZ base image? Are there specific guidelines we must follow?

Basically the same images we have now (debian:bookworm) - but ideally we should use "python images" and get rid of building python on our own (We recently switched from community "official" python images in #53150 so we basically need to revert back - but using the "hardened" images.

The images itself (https://www.docker.com/products/hardened-images/) are already based on debian - and there seem to be all python versions available in https://hub.docker.com/hardened-images/catalog/dhi/python/images. We want to use debian 12 (bookwrorm) based ones, and we do not want to use the FIPS version becuase FIPS variants are not available under free licence and require paid enterprise plan from Docker. However - it should be possible for those users who want to rebuild their images using fips-compliant images, so we should likely test it and describe as an option for users in "Build images" docs.

@shahar1 mentioned that they require sign-in, and you will need to do docker login ghi.io with your account to work on those, but I think when we decide to merge them we will have to setup a separte process to mirror the images to "ASF" managed images where we will make them publicly available so that they can be used by anyildone during development without the need to login do ghi.io with docker credentials. When you got to the point when you want to test in our CI, let us know and we will make manual mirroring and once it works we will automatet the mirroring.

Few caveats:

• I think we should be able to simplify both Dockerfile.ci and Docker - because the images come in two variants: dev (with -dev suffix) and runtime (without suffix) and we can utilise it to simplify our image buildin. Currently what we are doing is:

For CI image -> we use a single stage image (except scripts that are coming from it's own stage) -> and the image is bassed on "debian-slim" image and then we install build-essentials and other development tools needed. Likely the -dev image already has pretty much everything we neeed, so for CI image we should use the -dev image and strip out installing most development dependencies.

For PROD image, this is a bit more complex (but also a nice optimisation for us) - because we use multi-stage images. Stage 1 (build) install (and potentially builds) all Python packages, and then we copy those packages from build stage to the main stage from .local folder - because we install those packages with pip install --user. In the main stage we use the .local folder as venv (works nicely) and we use "airflow" user - we also make the image openshift-compatible by using group 0 that all users (including airflow) share and automatically create a user if random uid is used by openshift.

So for PROD image we want to:

• use -dev image for build stage - and also remove unnecessary installing of build-essentials and similar dependencies.
• use runtime image for main stage

The little difficulty here is the question of users: the -dev images are root, but the runtime ones are nonroot which means that some user mapping will have to happen, and (because Python installation is not relocatable) - some things with user management might need fixing.

Of course entrypoints might need some adjustments (we are using dumb-init) - also we need to come back to passing "base image" through build args as it was before #59517 - and this shoudl be a way for our users to pass fips image as build arg if they want to make the image fips-compliant.

The images as well have something calls sfw - secure firewall - which apparently is providing some (proxy based?) security when installing packages, I do not know details of it - but it also might have some impact.

There is also an issue we discovered recently - the "official" images had .pyc CPython files removed from installation of Python - which had very little savings for size of the image, but caused an unexpected memory leak -> so we need to check if the hardened images do not have the same problem (and we might even report that issue to Docker if they do) - fix to it was #58944 - for that we might need to compile all CPython files - both in build and main segment - but this might be somewhat more complicated for nonroot main image (we woudl need to switch to root temporarily).

Ideally - we shoudl follow exactly what Hardened images do when it comes to security - signing, sboms etc. but this could ba a follow up - we've never done that for our images, but it's great opportunity to just follow what docker does with their images.

We should also compare sizes of images before and after the change to avoid balooning the size of the image.

That's about all guidelines I can come up with from the top of my head.

Thanks for the detailed explanations - this clarifies the direction really well.

I’ll start by switching to the hardened Python images based on Debian 12 (bookworm), using the -dev variant for the CI image and the build stage in the PROD Dockerfile, and the runtime (non-root) image for the final PROD stage. I’ll avoid FIPS images by default, but reintroduce the base image as a build arg (as before #59517) so users can optionally build FIPS-compliant images if needed.
I’ll begin with local builds only (using docker login dhi.io) and validate that nothing breaks. For CI, I’ll simplify the Dockerfile by relying on the -dev image and removing redundant build dependencies. For the PROD image, I’ll validate the multi-stage flow, including copying .local user-installed packages and handling user/root differences between the -dev and runtime stages.
I’ll also check entrypoints (dumb-init), OpenShift compatibility, and user/group handling for the non-root runtime image. In parallel, I’ll verify whether the hardened images include compiled .pyc files so we don’t reintroduce the memory leak fixed in #58944, and compare image sizes before and after the change to avoid regressions.
I’ll document local usage, PAT authentication, optional FIPS builds, and any caveats I find. Once local validation looks good, I’ll sync back before touching CI so we can coordinate mirroring the images to ASF-managed repositories.

I’ll post updates as I make progress.