Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions BREEZE.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1314,7 +1314,7 @@ This is the current syntax for `./breeze <./breeze>`_:
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
Uses differen version of Airflow when building PROD image.

2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist
2.0.2 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist

-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
Installs Airflow directly from reference in GitHub when building PROD image.
Expand Down Expand Up @@ -1827,7 +1827,7 @@ This is the current syntax for `./breeze <./breeze>`_:
airflow is just removed. In this case airflow package should be added to dist folder
and --use-packages-from-dist flag should be used.

2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist none
2.0.2 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist none

--use-packages-from-dist
In CI image, if specified it will look for packages placed in dist folder and
Expand Down Expand Up @@ -2529,7 +2529,7 @@ This is the current syntax for `./breeze <./breeze>`_:
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
Uses differen version of Airflow when building PROD image.

2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist
2.0.2 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist

-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
Installs Airflow directly from reference in GitHub when building PROD image.
Expand Down Expand Up @@ -2557,7 +2557,7 @@ This is the current syntax for `./breeze <./breeze>`_:
airflow is just removed. In this case airflow package should be added to dist folder
and --use-packages-from-dist flag should be used.

2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist none
2.0.2 2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist none

--use-packages-from-dist
In CI image, if specified it will look for packages placed in dist folder and
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
# all the build essentials. This makes the image
# much smaller.
#
ARG AIRFLOW_VERSION="2.0.1"
ARG AIRFLOW_VERSION="2.0.2"
ARG AIRFLOW_EXTRAS="async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv"
ARG ADDITIONAL_AIRFLOW_EXTRAS=""
ARG ADDITIONAL_PYTHON_DEPS=""
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Airflow is not a streaming solution, but it is often used to process real-time d

Apache Airflow is tested with:

| | Master version (dev) | Stable version (2.0.1) | Previous version (1.10.15) |
| | Master version (dev) | Stable version (2.0.2) | Previous version (1.10.15) |
| ------------ | ------------------------- | ------------------------ | ------------------------- |
| Python | 3.6, 3.7, 3.8 | 3.6, 3.7, 3.8 | 2.7, 3.5, 3.6, 3.7, 3.8 |
| PostgreSQL | 9.6, 10, 11, 12, 13 | 9.6, 10, 11, 12, 13 | 9.6, 10, 11, 12, 13 |
Expand Down Expand Up @@ -168,15 +168,15 @@ them to appropriate format and workflow that your tool requires.


```bash
pip install apache-airflow==2.0.1 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.7.txt"
pip install apache-airflow==2.0.2 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt"
```

2. Installing with extras (for example postgres,google)

```bash
pip install apache-airflow[postgres,google]==2.0.1 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.7.txt"
pip install apache-airflow[postgres,google]==2.0.2 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt"
```

For information on installing provider packages check
Expand Down
1 change: 1 addition & 0 deletions breeze-complete
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ _breeze_allowed_installation_methods=". apache-airflow"
}

_breeze_allowed_install_airflow_versions=$(cat <<-EOF
2.0.2
2.0.1
2.0.0
1.10.15
Expand Down
8 changes: 4 additions & 4 deletions docs/apache-airflow/extra-packages-ref.rst
Original file line number Diff line number Diff line change
Expand Up @@ -84,14 +84,14 @@ For example the below command will install:
* apache-airflow-providers-google
* apache-airflow-providers-apache-spark

with a consistent set of dependencies based on constraint files provided by Airflow Community at the time 2.0.1 version was released.
with a consistent set of dependencies based on constraint files provided by Airflow Community at the time 2.0.2 version was released.

.. code-block:: bash

pip install apache-airflow[google,amazon,apache.spark]==2.0.1 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.6.txt"
pip install apache-airflow[google,amazon,apache.spark]==2.0.2 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.6.txt"

Note, that this will install providers in the versions that were released at the time of Airflow 2.0.1 release. You can later
Note, that this will install providers in the versions that were released at the time of Airflow 2.0.2 release. You can later
upgrade those providers manually if you want to use latest versions of the providers.


Expand Down
12 changes: 6 additions & 6 deletions docs/apache-airflow/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ not work or will produce unusable Airflow installation.
In order to have repeatable installation, starting from **Airflow 1.10.10** and updated in
**Airflow 1.10.13** we also keep a set of "known-to-be-working" constraint files in the
``constraints-master``, ``constraints-2-0`` and ``constraints-1-10`` orphan branches and then we create tag
for each released version e.g. ``constraints-2.0.1``. This way, when we keep a tested and working set of dependencies.
for each released version e.g. ``constraints-2.0.2``. This way, when we keep a tested and working set of dependencies.

Those "known-to-be-working" constraints are per major/minor Python version. You can use them as constraint
files when installing Airflow from PyPI. Note that you have to specify correct Airflow version
Expand All @@ -176,7 +176,7 @@ You can create the URL to the file substituting the variables in the template be

where:

- ``AIRFLOW_VERSION`` - Airflow version (e.g. ``2.0.1``) or ``master``, ``2-0``, ``1-10`` for latest development version
- ``AIRFLOW_VERSION`` - Airflow version (e.g. ``2.0.2``) or ``master``, ``2-0``, ``1-10`` for latest development version
- ``PYTHON_VERSION`` Python version e.g. ``3.8``, ``3.7``

There is also a no-providers constraint file, which contains just constraints required to install Airflow core. This allows
Expand All @@ -202,7 +202,7 @@ postgres and google provider, as well as ``async`` extra.

.. code-block:: bash

AIRFLOW_VERSION=2.0.1
AIRFLOW_VERSION=2.0.2
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install "apache-airflow[async,postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
Expand All @@ -220,7 +220,7 @@ being installed.

.. code-block:: bash

AIRFLOW_VERSION=2.0.1
AIRFLOW_VERSION=2.0.2
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install --upgrade "apache-airflow[postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
Expand Down Expand Up @@ -256,11 +256,11 @@ If you don't want to install any extra providers, initially you can use the comm

.. code-block:: bash

AIRFLOW_VERSION=2.0.1
AIRFLOW_VERSION=2.0.2
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
# For example: 3.6
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt"
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-no-providers-2.0.1/constraints-3.6.txt
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-no-providers-2.0.2/constraints-3.6.txt
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"


Expand Down
4 changes: 2 additions & 2 deletions docs/apache-airflow/start/local.rst
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,11 @@ constraint files to enable reproducible installation, so using ``pip`` and const
# (optional)
export AIRFLOW_HOME=~/airflow

AIRFLOW_VERSION=2.0.1
AIRFLOW_VERSION=2.0.2
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
# For example: 3.6
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.6.txt
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.6.txt
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

# initialize the database
Expand Down
4 changes: 2 additions & 2 deletions docs/docker-stack/build-arg-ref.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Those are the most common arguments that you use when you want to build a custom
+==========================================+==========================================+==========================================+
| ``PYTHON_BASE_IMAGE`` | ``python:3.6-slim-buster`` | Base python image. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_VERSION`` | ``2.0.1`` | version of Airflow. |
| ``AIRFLOW_VERSION`` | ``2.0.2`` | version of Airflow. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_EXTRAS`` | (see Dockerfile) | Default extras with which airflow is |
| | | installed. |
Expand Down Expand Up @@ -64,7 +64,7 @@ Those are the most common arguments that you use when you want to build a custom
| | | 2.0.* installation. In case of building |
| | | specific version you want to point it |
| | | to specific tag, for example |
| | | ``constraints-2.0.1``. |
| | | ``constraints-2.0.2``. |
| | | Auto-detected if empty. |
+------------------------------------------+------------------------------------------+------------------------------------------+

Expand Down
14 changes: 7 additions & 7 deletions docs/docker-stack/build.rst
Original file line number Diff line number Diff line change
Expand Up @@ -245,8 +245,8 @@ Additional explanation is needed for the last point. Airflow uses constraints to
that it can be predictably installed, even if some new versions of Airflow dependencies are
released (or even dependencies of our dependencies!). The docker image and accompanying scripts
usually determine automatically the right versions of constraints to be used based on the Airflow
version installed and Python version. For example 2.0.1 version of Airflow installed from PyPI
uses constraints from ``constraints-2.0.1`` tag). However in some cases - when installing airflow from
version installed and Python version. For example 2.0.2 version of Airflow installed from PyPI
uses constraints from ``constraints-2.0.2`` tag). However in some cases - when installing airflow from
GitHub for example - you have to manually specify the version of constraints used, otherwise
it will default to the latest version of the constraints which might not be compatible with the
version of Airflow you use.
Expand All @@ -270,23 +270,23 @@ Building from PyPI packages
This is the basic way of building the custom images from sources.

The following example builds the production image in version ``3.6`` with latest PyPI-released Airflow,
with default set of Airflow extras and dependencies. The ``2.0.1`` constraints are used automatically.
with default set of Airflow extras and dependencies. The ``2.0.2`` constraints are used automatically.

.. exampleinclude:: docker-examples/customizing/stable-airflow.sh
:language: bash
:start-after: [START build]
:end-before: [END build]

The following example builds the production image in version ``3.7`` with default extras from ``2.0.1`` PyPI
package. The ``2.0.1`` constraints are used automatically.
The following example builds the production image in version ``3.7`` with default extras from ``2.0.2`` PyPI
package. The ``2.0.2`` constraints are used automatically.

.. exampleinclude:: docker-examples/customizing/pypi-selected-version.sh
:language: bash
:start-after: [START build]
:end-before: [END build]

The following example builds the production image in version ``3.8`` with additional airflow extras
(``mssql,hdfs``) from ``2.0.1`` PyPI package, and additional dependency (``oauth2client``).
(``mssql,hdfs``) from ``2.0.2`` PyPI package, and additional dependency (``oauth2client``).

.. exampleinclude:: docker-examples/customizing/pypi-extras-and-deps.sh
:language: bash
Expand All @@ -312,7 +312,7 @@ have more complex dependencies to build.
Building optimized images
.........................

The following example the production image in version ``3.6`` with additional airflow extras from ``2.0.1``
The following example the production image in version ``3.6`` with additional airflow extras from ``2.0.2``
PyPI package but it includes additional apt dev and runtime dependencies.

The dev dependencies are those that require ``build-essential`` and usually need to involve recompiling
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ cd "${AIRFLOW_SOURCES}"
# [START build]
docker build . \
--build-arg PYTHON_BASE_IMAGE="python:3.6-slim-buster" \
--build-arg AIRFLOW_VERSION="2.0.1" \
--build-arg AIRFLOW_VERSION="2.0.2" \
--build-arg ADDITIONAL_PYTHON_DEPS="mpi4py" \
--build-arg ADDITIONAL_DEV_APT_DEPS="libopenmpi-dev" \
--build-arg ADDITIONAL_RUNTIME_APT_DEPS="openmpi-common" \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ cd "${AIRFLOW_SOURCES}"
# [START build]
docker build . -f Dockerfile \
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg AIRFLOW_VERSION="2.0.1" \
--build-arg AIRFLOW_VERSION="2.0.2" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="slack,odbc" \
--build-arg ADDITIONAL_PYTHON_DEPS=" \
azure-storage-blob \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ cd "${AIRFLOW_SOURCES}"
# [START build]
docker build . \
--build-arg PYTHON_BASE_IMAGE="python:3.6-slim-buster" \
--build-arg AIRFLOW_VERSION="2.0.1" \
--build-arg AIRFLOW_VERSION="2.0.2" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="jdbc" \
--build-arg ADDITIONAL_PYTHON_DEPS="pandas" \
--build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++" \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ cd "${AIRFLOW_SOURCES}"
# [START build]
docker build . \
--build-arg PYTHON_BASE_IMAGE="python:3.8-slim-buster" \
--build-arg AIRFLOW_VERSION="2.0.1" \
--build-arg AIRFLOW_VERSION="2.0.2" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="mssql,hdfs" \
--build-arg ADDITIONAL_PYTHON_DEPS="oauth2client" \
--tag "$(basename "$0")"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ cd "${AIRFLOW_SOURCES}"
# [START build]
docker build . \
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg AIRFLOW_VERSION="2.0.1" \
--build-arg AIRFLOW_VERSION="2.0.2" \
--tag "$(basename "$0")"
# [END build]
docker rmi --force "$(basename "$0")"
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
FROM apache/airflow:2.0.1
FROM apache/airflow:2.0.2
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
FROM apache/airflow:2.0.1
FROM apache/airflow:2.0.2
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,6 @@

# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
FROM apache/airflow:2.0.1
FROM apache/airflow:2.0.2
RUN pip install --no-cache-dir lxml
# [END Dockerfile]
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
FROM apache/airflow:2.0.1
FROM apache/airflow:2.0.2

COPY --chown=airflow:root test_dag.py /opt/airflow/dags

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

# This is an example Dockerfile. It is not intended for PRODUCTION use
# [START Dockerfile]
FROM apache/airflow:2.0.1
FROM apache/airflow:2.0.2
RUN umask 0002; \
mkdir -p ~/writeable-directory
# [END Dockerfile]
Original file line number Diff line number Diff line change
Expand Up @@ -25,18 +25,18 @@ cd "${AIRFLOW_SOURCES}"
rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true

curl -Lo "docker-context-files/constraints-3.7.txt" \
https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.7.txt
https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt

pip download --dest docker-context-files \
--constraint docker-context-files/constraints-3.7.txt \
"apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==2.0.1"
"apache-airflow[async,aws,azure,celery,dask,elasticsearch,gcp,kubernetes,postgres,redis,slack,ssh,statsd,virtualenv]==2.0.2"
# [END download]

# [START build]
docker build . \
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-buster" \
--build-arg AIRFLOW_INSTALLATION_METHOD="apache-airflow" \
--build-arg AIRFLOW_VERSION="2.0.1" \
--build-arg AIRFLOW_VERSION="2.0.2" \
--build-arg INSTALL_MYSQL_CLIENT="false" \
--build-arg AIRFLOW_PRE_CACHED_PIP_PACKAGES="false" \
--build-arg INSTALL_FROM_DOCKER_CONTEXT_FILES="true" \
Expand Down
4 changes: 2 additions & 2 deletions docs/docker-stack/recipes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Then build a new image.
.. code-block:: bash

docker build . \
--build-arg BASE_AIRFLOW_IMAGE="apache/airflow:2.0.1" \
--build-arg BASE_AIRFLOW_IMAGE="apache/airflow:2.0.2" \
-t my-airflow-image


Expand All @@ -66,5 +66,5 @@ Then build a new image.
.. code-block:: bash

docker build . \
--build-arg BASE_AIRFLOW_IMAGE="apache/airflow:2.0.1" \
--build-arg BASE_AIRFLOW_IMAGE="apache/airflow:2.0.2" \
-t my-airflow-image
2 changes: 1 addition & 1 deletion docs/helm-chart/manage-dags-files.rst
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ The recommended way to update your DAGs with this chart is to build a new docker
.. code-block:: bash

docker build --tag "my-company/airflow:8a0da78" . -f - <<EOF
FROM apache/airflow:2.0.1
FROM apache/airflow:2.0.2

USER root

Expand Down