Skip to content

Commit

Permalink
Add changes from 1.10.14 (#12993)
Browse files Browse the repository at this point in the history 
This commit adds Changelog, Updating.md and replaces 1.10.13 to 1.10.14
across the codebase

(cherry picked from commit 969d3ea)
  • Loading branch information
@kaxil
kaxil committed Jan 22, 2021
1 parent 5671964 commit fdd971a
Show file tree
Hide file tree
Showing 7 changed files with 181 additions and 60 deletions.
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,7 @@ jobs:
with:
name: airflow-documentation
path: "./files/documentation"
retention-days: 7

tests-helm:
timeout-minutes: 20
Expand Down
117 changes: 110 additions & 7 deletions BREEZE.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -541,6 +541,74 @@ dependencies). If you work offline and do not want to rebuild the images when ne
``FORCE_ANSWER_TO_QUESTIONS`` variable to ``no`` as described in the
`Setting default behaviour for user interaction <#setting-default-behaviour-for-user-interaction>`_ section.

Preparing packages
------------------

Breeze can also be used to prepare airflow packages - both "apache-airflow" main package and
provider packages.

You can read more about testing provider packages in
`TESTING.rst <TESTING.rst#running-tests-with-packages>`_

There are several commands that you can run in Breeze to manage and build packages:

* preparing Provider Readme files
* preparing Airflow packages
* preparing Provider packages

Preparing provider readme files is part of the release procedure by the release managers
and it is described in detail in `dev <dev/README.md>`_ .

You can prepare provider packages - by default regular provider packages are prepared, but with
``--backport`` flag you can prepare backport packages.

The packages are prepared in ``dist`` folder. Note, that this command cleans up the ``dist`` folder
before running, so you should run it before generating airflow package below as it will be removed.

The below example builds provider packages in the wheel format.

.. code-block:: bash
./breeze prepare-provider-packages
If you run this command without packages, you will prepare all packages, you can however specify
providers that you would like to build. By default only ``wheel`` packages are prepared,
but you can change it providing optional --package-format flag.


.. code-block:: bash
./breeze prepare-provider-packages --package-format=both google amazon
You can also prepare backport provider packages, if you specify ``--backport`` flag. You can read more
about backport packages in `dev <dev/README.md>`_

.. code-block:: bash
./breeze prepare-provider-packages --backports --package-format=both google amazon
You can see all providers available by running this command:

.. code-block:: bash
./breeze prepare-provider-packages -- --help
You can also prepare airflow packages using breeze:

.. code-block:: bash
./breeze prepare-airflow-packages
This prepares airflow .whl package in the dist folder.

Again, you can specify optional ``--package-format`` flag to build airflow packages.

.. code-block:: bash
./breeze prepare-airflow-packages --package-format=bot
Building Production images
--------------------------

Expand Down Expand Up @@ -1067,6 +1135,7 @@ This is the current syntax for `./breeze <./breeze>`_:
generate-constraints Generates pinned constraint files
push-image Pushes images to registry
initialize-local-virtualenv Initializes local virtualenv
prepare-airflow-packages Prepares airflow packages
setup-autocomplete Sets up autocomplete for breeze
start-airflow Starts Scheduler and Webserver and enters the shell
stop Stops the docker-compose environment
Expand Down Expand Up @@ -1142,16 +1211,14 @@ This is the current syntax for `./breeze <./breeze>`_:
Detailed usage for command: build-docs
breeze build-docs [-- <EXTRA_ARGS>]
breeze build-docs
Builds Airflow documentation. The documentation is build inside docker container - to
maintain the same build environment for everyone. Appropriate sources are mapped from
the host to the container so that latest sources are used. The folders where documentation
is generated ('docs/_build') are also mounted to the container - this way results of
the documentation build is available in the host.
The possible extra args are: --docs-only, --spellcheck-only, --help
####################################################################################################
Expand Down Expand Up @@ -1187,8 +1254,7 @@ This is the current syntax for `./breeze <./breeze>`_:
If specified, installs Airflow directly from PIP released version. This happens at
image building time in production image and at container entering time for CI image. One of:
1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3
1.10.2 wheel none
1.10.14 1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 wheel none
When 'none' is used, you can install airflow from local packages. When building image,
airflow package should be added to 'docker-context-files' and
Expand Down Expand Up @@ -1553,6 +1619,44 @@ This is the current syntax for `./breeze <./breeze>`_:
####################################################################################################
Detailed usage for command: prepare-airflow-packages
breeze prepare-airflow-packages [FLAGS]
Prepares airflow packages (sdist and wheel) in dist folder. Note that
prepare-provider-packages command cleans up the dist folder, so if you want also
to generate provider packages, make sure you run prepare-provider-packages first,
and prepare-airflow-packages second.
General form:
'breeze prepare-airflow-packages
Flags:
--package-format PACKAGE_FORMAT
Chooses format of packages to prepare.
One of:
wheel,sdist,both
Default: 
-v, --verbose
Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
debugging - when you run breeze with --verbose flags you will be able to see the commands
executed under the hood and copy&paste them to your terminal to debug them more easily.
Note that you can further increase verbosity and see all the commands executed by breeze
by running 'export VERBOSE_COMMANDS="true"' before running breeze.
####################################################################################################
Detailed usage for command: setup-autocomplete
Expand Down Expand Up @@ -2105,8 +2209,7 @@ This is the current syntax for `./breeze <./breeze>`_:
If specified, installs Airflow directly from PIP released version. This happens at
image building time in production image and at container entering time for CI image. One of:
1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 1.10.8 1.10.7 1.10.6 1.10.5 1.10.4 1.10.3
1.10.2 wheel none
1.10.14 1.10.13 1.10.12 1.10.11 1.10.10 1.10.9 wheel none
When 'none' is used, you can install airflow from local packages. When building image,
airflow package should be added to 'docker-context-files' and
Expand Down
12 changes: 0 additions & 12 deletions CONTRIBUTING.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -818,18 +818,6 @@ You can only run one of the steps via ``--spellcheck-only`` or ``--docs-only``.
./breeze build-docs
or just to run spell-check

.. code-block:: bash
./breeze build-docs -- --spellcheck-only
or just to run documentation building

.. code-block:: bash
./breeze build-docs
Also documentation is available as downloadable artifact in GitHub Actions after the CI builds your PR.

**Known issues:**
Expand Down
67 changes: 43 additions & 24 deletions IMAGES.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ Airflow has two images (build from Dockerfiles):

* Production image (Dockerfile) - that can be used to build your own production-ready Airflow installation
You can read more about building and using the production image in the
`Production Deployments <docs/production-deployment.rst>`_ document. The image is built using
`Dockerfile <Dockerfile>`_
`Production Deployments <https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html>`_ document.
The image is built using `Dockerfile <Dockerfile>`_

* CI image (Dockerfile.ci) - used for running tests and local development. The image is built using
`Dockerfile.ci <Dockerfile.ci>`_
Expand Down Expand Up @@ -64,7 +64,7 @@ The easiest way to build those images is to use `<BREEZE.rst>`_.

Note! Breeze by default builds production image from local sources. You can change it's behaviour by
providing ``--install-airflow-version`` parameter, where you can specify the
tag/branch used to download Airflow package from in github repository. You can
tag/branch used to download Airflow package from in GitHub repository. You can
also change the repository itself by adding ``--dockerhub-user`` and ``--dockerhub-repo`` flag values.

You can build the CI image using this command:
Expand Down Expand Up @@ -117,6 +117,16 @@ parameter to Breeze:
./breeze build-image --python 3.7 --additional-extras=presto \
--production-image --install-airflow-version=1.10.14
.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.


This will build the image using command similar to:

.. code-block:: bash
Expand All @@ -125,14 +135,6 @@ This will build the image using command similar to:
apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,mysql,postgres,redis,slack,ssh,statsd,virtualenv,presto]==1.10.14 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt"
.. note::
On 30th of November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver.
This resolver does not yet work with Apache Airflow and might leads to errors in installation -
depends on your choice of extras. In order to install Airflow you need to either downgrade
pip to version 20.2.4 ``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.


You can also build production images from specific Git version via providing ``--install-airflow-reference``
parameter to Breeze (this time constraints are taken from the ``constraints-master`` branch which is the
HEAD of development for constraints):
Expand All @@ -142,6 +144,15 @@ HEAD of development for constraints):
pip install "https://github.com/apache/airflow/archive/<tag>.tar.gz#egg=apache-airflow" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.6.txt"
You can also skip installing airflow by providing ``--install-airflow-version none`` parameter to Breeze:

.. code-block:: bash
./breeze build-image --python 3.7 --additional-extras=presto \
--production-image --install-airflow-version=none --install-from-local-files-when-building
In this case you usually install airflow and all packages in ``docker-context-files`` folder.

Using cache during builds
=========================

Expand Down Expand Up @@ -276,8 +287,8 @@ production image from the local sources.

The image is primarily optimised for size of the final image, but also for speed of rebuilds - the
'airflow-build-image' segment uses the same technique as the CI builds for pre-installing PIP dependencies.
It first pre-installs them from the right github branch and only after that final airflow installation is
done from either local sources or remote location (PIP or github repository).
It first pre-installs them from the right GitHub branch and only after that final airflow installation is
done from either local sources or remote location (PIP or GitHub repository).

Customizing the image
.....................
Expand Down Expand Up @@ -409,7 +420,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| | | file has to be in docker context so |
| | | it's best to place such file in |
| | | one of the folders included in |
| | | dockerignore . for example in the |
| | | .dockerignore. for example in the |
| | | 'docker-context-files'. Note that the |
| | | location does not work for the first |
| | | stage of installation when the |
Expand All @@ -418,21 +429,31 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| | | set to true. Default location from |
| | | GitHub is used in this case. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_LOCAL_PIP_WHEELS`` | ``false`` | If set to true, Airflow and it's |
| ``AIRFLOW_CONSTRAINTS_REFERENCE`` | ``constraints-master`` | reference (branch or tag) from GitHub |
| | | repository from which constraints are |
| | | used. By default it is set to |
| | | ``constraints-master`` but can be |
| | | ``constraints-1-10`` for 1.10.* versions |
| | | or it could point to specific version |
| | | for example ``constraints-1.10.12`` |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``INSTALL_FROM_DOCKER_CONTEXT_FILES`` | ``false`` | If set to true, Airflow and it's |
| | | dependencies are installed from locally |
| | | downloaded .whl files placed in the |
| | | ``docker-context-files``. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_EXTRAS`` | ``all`` | extras to install |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``INSTALL_AIRFLOW_VIA_PIP`` | ``false`` | If set to true, Airflow is installed via |
| | | pip install. if you want to install |
| ``INSTALL_FROM_PYPI`` | ``true`` | If set to true, Airflow is installed |
| | | from pypi. If you want to install |
| | | Airflow from externally provided binary |
| | | package you can set it to false, place |
| | | the package in ``docker-context-files`` |
| | | and set ``AIRFLOW_LOCAL_PIP_WHEELS`` to |
| | | true. You have to also set to true the |
| | | and set |
| | | ``INSTALL_FROM_DOCKER_CONTEXT_FILES`` to |
| | | true. For this you have to also set the |
| | | ``AIRFLOW_PRE_CACHED_PIP_PACKAGES`` flag |
| | | to false |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_PRE_CACHED_PIP_PACKAGES`` | ``true`` | Allows to pre-cache airflow PIP packages |
| | | from the GitHub of Apache Airflow |
Expand Down Expand Up @@ -530,7 +551,7 @@ Production images
-----------------

You can find details about using, building, extending and customising the production images in the
`Latest documentation <https://airflow.readthedocs.io/en/latest/production-deployment.html>`_
`Latest documentation <https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html>`_


Image manifests
Expand Down Expand Up @@ -615,10 +636,8 @@ The entrypoint performs those operations:
Using, customising, and extending the production image
======================================================

You can read more about using, customising, and extending the production image in the documentation:

* [Stable docs](https://airflow.apache.org/docs/stable/production-deployment.html)
* [Latest docs from master branch](https://airflow.readthedocs.io/en/latest/production-deployment.html
You can read more about using, customising, and extending the production image in the
`documentation <https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html>`_.

Alpha versions of 1.10.10 production-ready images
=================================================
Expand Down

0 comments on commit fdd971a

Please sign in to comment.