Skip to content
Permalink
Browse files
Add slim images to release process (#23391)
This PR adds slim images to release process of Airflow.

Those images are small as they do not contain any extras.

Fixes: #20849
  • Loading branch information
potiuk committed May 7, 2022
1 parent efed15b commit 6e5955831672c71bfc0424dd50c8e72f6fd5b2a7
Showing 14 changed files with 635 additions and 248 deletions.
@@ -324,30 +324,30 @@ repos:
files: ^setup\.cfg$|^setup\.py$
pass_filenames: false
entry: ./scripts/ci/pre_commit/pre_commit_check_order_setup.py
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.2.0']
- id: check-extra-packages-references
name: Checks setup extra packages
description: Checks if all the libraries in setup.py are listed in extra-packages-ref.rst file
language: python
files: ^setup\.py$|^docs/apache-airflow/extra-packages-ref\.rst$
pass_filenames: false
entry: ./scripts/ci/pre_commit/pre_commit_check_setup_extra_packages_ref.py
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
# This check might be removed when min-airflow-version in providers is 2.2
- id: check-airflow-2-1-compatibility
name: Check that providers are 2.1 compatible.
entry: ./scripts/ci/pre_commit/pre_commit_check_2_1_compatibility.py
language: python
pass_filenames: true
files: ^airflow/providers/.*\.py$
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
- id: update-breeze-file
name: Update output of breeze commands in BREEZE.rst
entry: ./scripts/ci/pre_commit/pre_commit_breeze_cmd_line.py
language: python
files: ^BREEZE\.rst$|^dev/breeze/.*$
pass_filenames: false
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
- id: update-local-yml-file
name: Update mounts in the local yml file
entry: ./scripts/ci/pre_commit/pre_commit_local_yml_mounts.sh
@@ -379,7 +379,7 @@ repos:
language: python
files: ^Dockerfile$
pass_filenames: false
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
- id: update-supported-versions
name: Updates supported versions in documentation
entry: ./scripts/ci/pre_commit/pre_commit_supported_versions.py
@@ -582,15 +582,15 @@ repos:
- 'jsonschema>=3.2.0,<5.0.0'
- 'tabulate==0.8.8'
- 'jsonpath-ng==1.5.3'
- 'rich>=12.3.0'
- 'rich==12.3.0'
- id: check-pre-commit-information-consistent
name: Update information about pre-commit hooks and verify ids and names
entry: ./scripts/ci/pre_commit/pre_commit_check_pre_commit_hooks.py
args:
- --max-length=70
language: python
files: ^\.pre-commit-config\.yaml$|^scripts/ci/pre_commit/pre_commit_check_pre_commit_hook_names\.py$
additional_dependencies: ['pyyaml', 'jinja2', 'black==22.3.0', 'tabulate', 'rich>=12.3.0']
additional_dependencies: ['pyyaml', 'jinja2', 'black==22.3.0', 'tabulate', 'rich==12.3.0']
require_serial: true
pass_filenames: false
- id: check-airflow-providers-have-extras
@@ -600,7 +600,7 @@ repos:
files: ^setup\.py$|^airflow/providers/.*\.py$
pass_filenames: false
require_serial: true
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
- id: update-breeze-readme-config-hash
name: Update Breeze README.md with config files hash
language: python
@@ -615,15 +615,15 @@ repos:
files: ^dev/breeze/.*$
pass_filenames: false
require_serial: true
additional_dependencies: ['click', 'rich>=12.3.0']
additional_dependencies: ['click', 'rich==12.3.0']
- id: check-system-tests-present
name: Check if system tests have required segments of code
entry: ./scripts/ci/pre_commit/pre_commit_check_system_tests.py
language: python
files: ^tests/system/.*/example_[^/]*.py$
exclude: ^tests/system/providers/google/bigquery/example_bigquery_queries\.py$
pass_filenames: true
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
- id: lint-markdown
name: Run markdownlint
description: Checks the style of Markdown files.
@@ -714,15 +714,15 @@ repos:
language: python
pass_filenames: true
files: ^\.github/workflows/.*\.yml$
additional_dependencies: ['PyYAML', 'rich>=12.3.0']
additional_dependencies: ['PyYAML', 'rich==12.3.0']
- id: check-docstring-param-types
name: Check that docstrings do not specify param types
entry: ./scripts/ci/pre_commit/pre_commit_docstring_param_type.py
language: python
pass_filenames: true
files: \.py$
exclude: ^airflow/_vendor/
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
- id: lint-chart-schema
name: Lint chart/values.schema.json file
entry: ./scripts/ci/pre_commit/pre_commit_chart_schema.py
@@ -757,59 +757,59 @@ repos:
entry: ./scripts/ci/pre_commit/pre_commit_mypy.py
files: ^dev/.*\.py$
require_serial: true
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
- id: run-mypy
name: Run mypy for core
language: python
entry: ./scripts/ci/pre_commit/pre_commit_mypy.py --namespace-packages
files: \.py$
exclude: ^provider_packages|^docs|^airflow/_vendor/|^airflow/providers|^airflow/migrations|^dev
require_serial: true
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
- id: run-mypy
name: Run mypy for providers
language: python
entry: ./scripts/ci/pre_commit/pre_commit_mypy.py --namespace-packages
files: ^airflow/providers/.*\.py$
require_serial: true
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
- id: run-mypy
name: Run mypy for /docs/ folder
language: python
entry: ./scripts/ci/pre_commit/pre_commit_mypy.py
files: ^docs/.*\.py$
exclude: ^docs/rtd-deprecation
require_serial: true
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
- id: run-flake8
name: Run flake8
language: python
entry: ./scripts/ci/pre_commit/pre_commit_flake8.py
files: \.py$
pass_filenames: true
exclude: ^airflow/_vendor/
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
- id: lint-javascript
name: ESLint against airflow/ui
language: python
'types_or': [javascript, tsx, ts]
files: ^airflow/ui/
entry: ./scripts/ci/pre_commit/pre_commit_ui_lint.py
pass_filenames: false
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
- id: lint-javascript
name: ESLint against current UI JavaScript files
language: python
'types_or': [javascript]
files: ^airflow/www/static/js/
entry: ./scripts/ci/pre_commit/pre_commit_www_lint.py
pass_filenames: false
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
- id: update-migration-references
name: Update migration ref doc
language: python
entry: ./scripts/ci/pre_commit/pre_commit_migration_reference.py
pass_filenames: false
files: ^airflow/migrations/versions/.*\.py$|^docs/apache-airflow/migrations-ref\.rst$
additional_dependencies: ['rich>=12.3.0']
additional_dependencies: ['rich==12.3.0']
## ONLY ADD PRE-COMMITS HERE THAT REQUIRE CI IMAGE
@@ -556,6 +556,7 @@ do not need or have no access to run). Those are usually connected with releasin
``breeze prepare-provider-packages`` (when releasing provider packages)
* Finding the updated dependencies since the last successful build when we have conflict with
``breeze find-newer-dependencies`` command
* Release production images to DockerHub with ``breeze release-prod-images`` command


Details of Breeze usage
@@ -947,6 +948,34 @@ Those are all available flags of ``verify-prod-image`` command:
:width: 100%
:alt: Breeze verify-prod-image

Releasing Production images to DockerHub
----------------------------------------

The **Production image** can be released by release managers who have permissions to push the image. This
happens only when there is an RC candidate or final version of Airflow released.

You release "regular" and "slim" images as separate steps.

Releasing "regular" images:

.. code-block:: bash
breeze release-prod-images --airflow-version 2.4.0
Or "slim" images:

.. code-block:: bash
breeze release-prod-images --airflow-version 2.4.0 --slim-images
By default when you are releasing the "final" image, we also tag image with "latest" tags but this
step can be skipped if you pass the ``--skip-latest`` flag.

These are all of the available flags for the ``release-prod-images`` command:

.. image:: ./images/breeze/output-release-prod-images.svg
:width: 100%
:alt: Release prod images


Running static checks
@@ -133,7 +133,7 @@ for the PRs that are already correctly assigned to the milestone. You can also a
question with `--assume-yes` flag.

You cn review the list of PRs cherry-picked and produce a nice summary with `--print-summary` (this flag
assumes `--skip-assigned` so that the summary can be produced without questions:
assumes the `--skip-assigned` flag, so that the summary can be produced without questions:

```shell
,/dev/assign_cherry_picked_prs_with_milestone.py assign-prs --previous-release v2-2-stable \
@@ -493,10 +493,13 @@ to have an environment prepared to build multi-platform images. You can achieve

## Prerequisites

You need to have buildx plugin installed to run the build. Also you need to have regctl
You need to have buildx plugin installed to run the build. Also, you need to have regctl
installed from https://github.com/regclient/regclient in order to tag the multi-platform images in
DockerHub. The script to build images will refuse to work if you do not have those two installed.

You also need to have the right permissions to push the images, so you should run
`docker login` before and authenticate with your DockerHub token.

## Setting environment with emulation

According to the [official installation instructions](https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images)
@@ -540,9 +543,16 @@ docker buildx ls
airflow_cache1 tcp://127.0.0.1:2375
```

Preparing regular images:

```shell script
breeze release-prod-images --airflow-version "${VERSION}"
```

Preparing slim images:

```shell script
./dev/prepare_prod_docker_images.sh ${VERSION}
breeze release-prod-images --airflow-version "${VERSION}" --slim-images
```

This will wipe Breeze cache and docker-context-files in order to make sure the build is "clean". It
@@ -998,19 +1008,34 @@ At this point we release an official package:

## Manually prepare production Docker Image


Note that this scripts prepares multi-platform image, so you need to fulfill prerequisites as
described above in the preparation of RC images.

Note that by default the `latest` images tagged are aliased to the just released image which is the usual
way we release. For example when you are releasing 2.3.N image and 2.3 is our latest branch the new image is
marked as "latest".

In case we are releasing (which almost never happens so far) a critical bugfix release in one of
the older branches, you should add the `--skip-latest` flag.

Preparing regular images:

```shell script
breeze release-prod-images --airflow-version "${VERSION}"
```

Preparing slim images:

```shell script
breeze release-prod-images --airflow-version "${VERSION}" --slim-images
```

Preparing a release that is not in the latest branch:

```shell script
./dev/prepare_prod_docker_images.sh ${VERSION}
breeze release-prod-images --airflow-version "${VERSION}" --slim-images --skip-latest
```

Note! When you release the 'final' (non-rc) version you will be asked if you want to
tag the images as latest - if you are releasing the latest stable branch, you
should answer y and tags will be created and pushed. If you are releasing a
patch release from an older branch, you should answer n and creating tags will
be skipped.

## Publish documentation

@@ -52,6 +52,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.

---------------------------------------------------------------------------------------------------------

Package config hash: 6ab00f80228fe4392cdb1263a52c55416c153052010bd5baca25add84da7475d8601cc399342c72ad95df0e1f2de99e2a99e3d0b19631ff0480554360a9578b4
Package config hash: dfdac0e241f273bbe3ae4a3d5e9c9f39426d792c2468753d94fd0dfa6b5d46331703c5d419e5e8bf3c199129acde63cfa61e30c6a3b9a920047e0d115fd1d57c

---------------------------------------------------------------------------------------------------------
@@ -63,7 +63,7 @@ install_requires =
pytest-xdist
pyyaml
requests
rich>=12.3.0
rich==12.3.0
rich_click

[options.packages.find]

0 comments on commit 6e59558

Please sign in to comment.