Thank you for finding the time to report the problem!
- We really appreciate the community efforts to improve Airflow.
+ We really appreciate the community's efforts to improve Airflow.
- Note, that you do not need to create an issue if you have a change ready to submit!
+ Note, you do not need to create an issue if you have a change ready to submit!
- You can open [Pull Request](https://github.com/apache/airflow/pulls) immediately instead.
+ You can open a [pull request](https://github.com/apache/airflow/pulls) immediately instead.
@@ -305,6 +364,9 @@ Breeze helps with running tests in the same environment/way as CI tests are run.
types of tests while you enter Breeze CI interactive environment - this is described in detail
in ``_
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command and it is not yet available in the new ``breeze`` command):
+
.. raw:: html
`_
+.. image:: ./images/breeze/output-tests.svg
+ :width: 100%
+ :alt: Breeze tests
+
+Kubernetes tests
+----------------
+
+* Manage KinD Kubernetes cluster and deploy Airflow to KinD cluster ``./breeze-legacy kind-cluster`` commands
+* Run Kubernetes tests specified with ``./breeze-legacy kind-cluster tests`` command
+* Enter the interactive kubernetes test environment with ``./breeze-legacy kind-cluster shell`` command
+
+CI Image tasks
+--------------
+
+The image building is usually run for users automatically when needed,
+but sometimes Breeze users might want to manually build, pull or verify the CI images.
+
+* Build CI docker image with ``breeze build-image`` command
+* Pull CI images in parallel ``breeze pull-image`` command
+* Verify CI image ``breeze verify-image`` command
-Maintainer tasks
+PROD Image tasks
----------------
+Users can also build Production images when they are developing them. However when you want to
+use the PROD image, the regular docker build commands are recommended. See
+`building the image `_
+
+* Build PROD image with ``breeze build-prod-image`` command
+* Pull PROD image in parallel ``breeze pull-prod-image`` command
+* Verify CI image ``breeze verify-prod-image`` command
+
+Configuration and maintenance
+-----------------------------
+
+* Cleanup breeze with ``breeze cleanup`` command
+* Self-upgrade breeze with ``breeze self-upgrade`` command
+* Setup autocomplete for Breeze with ``breeze setup-autocomplete`` command
+* Checking available resources for docker with ``breeze resource-check`` command
+* Freeing space needed to run CI tests with ``breeze free-space`` command
+* Fixing ownership of files in your repository with ``breeze fix-ownership`` command
+* Print Breeze version with ``breeze version`` command
+* Outputs hash of commands defined by ``breeze`` with ``command-hash-export`` (useful to avoid needless
+ regeneration of Breeze images)
+
+Release tasks
+-------------
+
Maintainers also can use Breeze for other purposes (those are commands that regular contributors likely
-do not need):
+do not need or have no access to run). Those are usually connected with releasing Airflow:
-* Prepare cache for CI: ``breeze prepare-build-cache`` (needs buildx plugin and write access to cache ghcr.io)
+* Prepare cache for CI: ``breeze build-image --prepare-build-cache`` and
+ ``breeze build-prod image --prepare-build-cache``(needs buildx plugin and write access to registry ghcr.io)
* Generate constraints with ``breeze generate-constraints`` (needed when conflicting changes are merged)
-* Prepare airflow packages: ``breeze prepare-airflow-packages`` (when releasing Airflow)
+* Prepare airflow packages: ``breeze prepare-airflow-package`` (when releasing Airflow)
+* Verify providers: ``breeze verify-provider-packages`` (when releasing provider packages) - including importing
+ the providers in an earlier airflow version.
* Prepare provider documentation ``breeze prepare-provider-documentation`` and prepare provider packages
``breeze prepare-provider-packages`` (when releasing provider packages)
+* Finding the updated dependencies since the last successful build when we have conflict with
+ ``breeze find-newer-dependencies`` command
+* Release production images to DockerHub with ``breeze release-prod-images`` command
+
Details of Breeze usage
=======================
@@ -430,11 +587,30 @@ Database volumes in Breeze
--------------------------
Breeze keeps data for all it's integration in named docker volumes. Each backend and integration
-keeps data in their own volume. Those volumes are persisted until ``./breeze stop`` command or
-``./breeze restart`` command is run. You can also preserve the volumes by adding flag
-``--preserve-volumes`` when you run either of those commands. Then, next time when you start
-``Breeze``, it will have the data pre-populated. You can always delete the volumes by
-running ``./breeze stop`` without the ``--preserve-volumes`` flag.
+keeps data in their own volume. Those volumes are persisted until ``breeze stop`` command.
+You can also preserve the volumes by adding flag ``--preserve-volumes`` when you run the command.
+Then, next time when you start Breeze, it will have the data pre-populated.
+
+Those are all available flags of ``stop`` command:
+
+.. image:: ./images/breeze/output-stop.svg
+ :width: 100%
+ :alt: Breeze stop
+
+Image cleanup
+--------------
+
+Breeze uses docker images heavily and those images are rebuild periodically. This might cause extra
+disk usage by the images. If you need to clean-up the images periodically you can run
+``breeze cleanup`` command (by default it will skip removing your images before cleaning up but you
+can also remove the images to clean-up everything by adding ``--all``).
+
+Those are all available flags of ``cleanup`` command:
+
+
+.. image:: ./images/breeze/output-cleanup.svg
+ :width: 100%
+ :alt: Breeze cleanup
Launching multiple terminals
----------------------------
@@ -448,6 +624,8 @@ capability of creating multiple virtual terminals and multiplex between them. Mo
found at `tmux GitHub wiki page `_ . Tmux has several useful shortcuts
that allow you to split the terminals, open new tabs etc - it's pretty useful to learn it.
+Here is the part of Breeze video which is relevant:
+
.. raw:: html
@@ -321,19 +383,34 @@ You can use additional ``breeze`` flags to choose your environment. You can spec
version to use, and backend (the meta-data database). Thanks to that, with Breeze, you can recreate the same
environments as we have in matrix builds in the CI.
-For example, you can choose to run Python 3.7 tests with MySQL as backend and in the Docker environment as
-follows:
+For example, you can choose to run Python 3.7 tests with MySQL as backend and with mysql version 8
+as follows:
.. code-block:: bash
- ./breeze --python 3.7 --backend mysql
+ breeze --python 3.7 --backend mysql --mysql-version 8
The choices you make are persisted in the ``./.build/`` cache directory so that next time when you use the
``breeze`` script, it could use the values that were used previously. This way you do not have to specify
them when you run the script. You can delete the ``.build/`` directory in case you want to restore the
default settings.
-The defaults when you run the Breeze environment are Python 3.7 version and SQLite database.
+You can see which value of the parameters that can be stored persistently in cache marked with >VALUE<
+in the help of the commands.
+
+Another part of configuration is enabling/disabling cheatsheet, asciiart. The cheatsheet and asciiart can
+be disabled - they are "nice looking" and cheatsheet
+contains useful information for first time users but eventually you might want to disable both if you
+find it repetitive and annoying.
+
+With the config setting colour-blind-friendly communication for Breeze messages. By default we communicate
+with the users about information/errors/warnings/successes via colour-coded messages, but we can switch
+it off by passing ``--no-colour`` to config in which case the messages to the user printed by Breeze
+will be printed using different schemes (italic/bold/underline) to indicate different kind of messages
+rather than colours.
+
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
.. raw:: html
@@ -344,6 +421,41 @@ The defaults when you run the Breeze environment are Python 3.7 version and SQLi
+Those are all available flags of ``config`` command:
+
+.. image:: ./images/breeze/output-config.svg
+ :width: 100%
+ :alt: Breeze config
+
+
+You can also dump hash of the configuration options used - this is mostly use to generate the dump
+of help of the commands only when they change.
+
+.. image:: ./images/breeze/output-command-hash-export.svg
+ :width: 100%
+ :alt: Breeze command-hash-export
+
+
+Starting complete Airflow installation
+======================================
+
+For testing Airflow oyou often want to start multiple components (in multiple terminals). Breeze has
+built-in ``start-airflow`` command that start breeze container, launches multiple terminals using tmux
+and launches all Airflow necessary components in those terminals.
+
+You can also use it to start any released version of Airflow from ``PyPI`` with the
+``--use-airflow-version`` flag.
+
+.. code-block:: bash
+
+ breeze --python 3.7 --backend mysql --use-airflow-version 2.2.5 start-airflow
+
+Those are all available flags of ``start-airflow`` command:
+
+.. image:: ./images/breeze/output-start-airflow.svg
+ :width: 100%
+ :alt: Breeze start-airflow
+
Troubleshooting
===============
@@ -351,13 +463,13 @@ Troubleshooting
If you are having problems with the Breeze environment, try the steps below. After each step you
can check whether your problem is fixed.
-1. If you are on macOS, check if you have enough disk space for Docker.
-2. Restart Breeze with ``./breeze restart``.
-3. Delete the ``.build`` directory and run ``./breeze build-image``.
-4. Clean up Docker images via ``breeze cleanup-image`` command.
+1. If you are on macOS, check if you have enough disk space for Docker (Breeze will warn you if not).
+2. Stop Breeze with ``breeze stop``.
+3. Delete the ``.build`` directory and run ``breeze build-image``.
+4. Clean up Docker images via ``breeze cleanup`` command.
5. Restart your Docker Engine and try again.
6. Restart your machine and try again.
-7. Re-install Docker CE and try again.
+7. Re-install Docker Desktop and try again.
In case the problems are not solved, you can set the VERBOSE_COMMANDS variable to "true":
@@ -377,51 +489,96 @@ Airflow Breeze is a bash script serving as a "swiss-army-knife" of Airflow testi
hood it uses other scripts that you can also run manually if you have problem with running the Breeze
environment. Breeze script allows performing the following tasks:
-Airflow developers tasks
-------------------------
+Development tasks
+-----------------
-Regular development tasks:
+Those are commands mostly used by contributors:
-* Setup autocomplete for Breeze with ``breeze setup-autocomplete`` command
+* Execute arbitrary command in the test environment with ``breeze shell`` command
* Enter interactive shell in CI container when ``shell`` (or no command) is specified
* Start containerised, development-friendly airflow installation with ``breeze start-airflow`` command
* Build documentation with ``breeze build-docs`` command
-* Initialize local virtualenv with ``breeze initialize-local-virtualenv`` command
-* Build CI docker image with ``breeze build-image`` command
-* Cleanup CI docker image with ``breeze cleanup-image`` command
-* Run static checks with autocomplete support ``breeze static-check`` command
+* Initialize local virtualenv with ``./scripts/tools/initialize_virtualenv.py`` command
+* Run static checks with autocomplete support ``breeze static-checks`` command
* Run test specified with ``breeze tests`` command
+* Build CI docker image with ``breeze build-image`` command
+* Cleanup breeze with ``breeze cleanup`` command
Additional management tasks:
* Join running interactive shell with ``breeze exec`` command
* Stop running interactive environment with ``breeze stop`` command
-* Restart running interactive environment with ``breeze restart`` command
-* Execute arbitrary command in the test environment with ``breeze shell`` command
-* Execute arbitrary docker-compose command with ``breeze docker-compose`` command
+* Execute arbitrary docker-compose command with ``./breeze-legacy docker-compose`` command
-Kubernetes tests related:
+Tests
+-----
-* Manage KinD Kubernetes cluster and deploy Airflow to KinD cluster ``breeze kind-cluster`` commands
-* Run Kubernetes tests specified with ``breeze kind-cluster tests`` command
-* Enter the interactive kubernetes test environment with ``breeze kind-cluster shell`` command
+* Run docker-compose tests with ``breeze docker-compose-tests`` command.
+* Run test specified with ``breeze tests`` command.
-Airflow can also be used for managing Production images (with ``--production-image`` flag added for image
-related command) - this is a development-only feature, regular users of Airflow should use ``docker build``
-commands to manage the images as described in the user documentation about
-`building the image
@@ -464,6 +642,8 @@ to enter the running container. It's as easy as launching ``breeze exec`` while
Breeze environment. You will be dropped into bash and environment variables will be read in the same
way as when you enter the environment. You can do it multiple times and open as many terminals as you need.
+Here is the part of Breeze video which is relevant:
+
.. raw:: html
`__
* ``install_terraform.sh`` - installs `Terraform `__
Launching Breeze integrations
@@ -521,6 +708,9 @@ Once integration is started, it will continue to run until the environment is st
Note that running integrations uses significant resources - CPU and memory.
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
+
.. raw:: html
`_
+ to clone the repo locally (you can also do it in your IDE - see the `Using your IDE `_
+ chapter below.
+Typical development tasks
+#########################
-3. Open your IDE or source code editor and select the option to clone the repository
-
- .. raw:: html
-
-
@@ -474,6 +654,12 @@ way as when you enter the environment. You can do it multiple times and open as
+Those are all available flags of ``exec`` command:
+
+.. image:: ./images/breeze/output-exec.svg
+ :width: 100%
+ :alt: Breeze exec
+
Additional tools
----------------
@@ -500,6 +686,7 @@ Currently available scripts:
for iTerm2 (Mac OS only)
* ``install_java.sh`` - installs `the OpenJDK 8u41
@@ -530,7 +720,7 @@ Note that running integrations uses significant resources - CPU and memory.
-Building CI images
+Managing CI images
------------------
With Breeze you can build images that are used by Airflow CI and production ones.
@@ -543,6 +733,9 @@ The CI image is built automatically as needed, however it can be rebuilt manuall
image should be built manually - but also a variant of this image is built automatically when
kubernetes tests are executed see `Running Kubernetes tests <#running-kubernetes-tests>`_
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
+
.. raw:: html
@@ -562,10 +755,50 @@ latest image build on CI. This might happen when for example latest patches have
Python images or when significant changes are made in the Dockerfile. In such cases, Breeze will
download the latest images before rebuilding because this is usually faster than rebuilding the image.
-In most cases, rebuilding an image requires network connectivity (for example, to download new
-dependencies). If you work offline and do not want to rebuild the images when needed, you can set the
-``FORCE_ANSWER_TO_QUESTIONS`` variable to ``no`` as described in the
-`Setting default behaviour for user interaction <#setting-default-behaviour-for-user-interaction>`_ section.
+Those are all available flags of ``build-image`` command:
+
+.. image:: ./images/breeze/output-build-image.svg
+ :width: 100%
+ :alt: Breeze build-image
+
+You can also pull the CI images locally in parallel with optional verification.
+
+Those are all available flags of ``pull-image`` command:
+
+.. image:: ./images/breeze/output-pull-image.svg
+ :width: 100%
+ :alt: Breeze pull-image
+
+Finally, you can verify CI image by running tests - either with the pulled/built images or
+with an arbitrary image.
+
+Those are all available flags of ``verify-image`` command:
+
+.. image:: ./images/breeze/output-verify-image.svg
+ :width: 100%
+ :alt: Breeze verify-image
+
+Verifying providers
+-------------------
+
+Breeze can also be used to verify if provider classes are importable and if they are following the
+right naming conventions. This happens automatically on CI but you can also run it manually.
+
+.. code-block:: bash
+
+ breeze verify-provider-packages
+
+You can also run the verification with an earlier airflow version to check for compatibility.
+
+.. code-block:: bash
+
+ breeze verify-provider-packages --use-airflow-version 2.1.0
+
+All the command parameters are here:
+
+.. image:: ./images/breeze/output-verify-provider-packages.svg
+ :width: 100%
+ :alt: Breeze verify-provider-packages
Preparing packages
------------------
@@ -578,12 +811,31 @@ You can read more about testing provider packages in
There are several commands that you can run in Breeze to manage and build packages:
-* preparing Provider Readme files
+* preparing Provider documentation files
* preparing Airflow packages
* preparing Provider packages
-Preparing provider readme files is part of the release procedure by the release managers
-and it is described in detail in `dev `_ .
+Preparing provider documentation files is part of the release procedure by the release managers
+and it is described in detail in `dev `_ .
+
+The below example perform documentation preparation for provider packages.
+
+.. code-block:: bash
+
+ breeze prepare-provider-documentation
+
+By default, the documentation preparation runs package verification to check if all packages are
+importable, but you can add ``--skip-package-verification`` to skip it.
+
+.. code-block:: bash
+
+ breeze prepare-provider-documentation --skip-package-verification
+
+You can also add ``--answer yes`` to perform non-interactive build.
+
+.. image:: ./images/breeze/output-prepare-provider-documentation.svg
+ :width: 100%
+ :alt: Breeze prepare-provider-documentation
The packages are prepared in ``dist`` folder. Note, that this command cleans up the ``dist`` folder
before running, so you should run it before generating airflow package below as it will be removed.
@@ -592,7 +844,7 @@ The below example builds provider packages in the wheel format.
.. code-block:: bash
- ./breeze prepare-provider-packages
+ breeze prepare-provider-packages
If you run this command without packages, you will prepare all packages, you can however specify
providers that you would like to build. By default ``both`` types of packages are prepared (
@@ -600,20 +852,23 @@ providers that you would like to build. By default ``both`` types of packages ar
.. code-block:: bash
- ./breeze prepare-provider-packages google amazon
+ breeze prepare-provider-packages google amazon
You can see all providers available by running this command:
.. code-block:: bash
- ./breeze prepare-provider-packages -- --help
+ breeze prepare-provider-packages --help
+.. image:: ./images/breeze/output-prepare-provider-packages.svg
+ :width: 100%
+ :alt: Breeze prepare-provider-packages
-You can also prepare airflow packages using breeze:
+You can prepare airflow packages using breeze:
.. code-block:: bash
- ./breeze prepare-airflow-packages
+ breeze prepare-airflow-package
This prepares airflow .whl package in the dist folder.
@@ -622,10 +877,13 @@ default is to build ``both`` type of packages ``sdist`` and ``wheel``.
.. code-block:: bash
- ./breeze prepare-airflow-packages --package-format=wheel
+ breeze prepare-airflow-package --package-format=wheel
+.. image:: ./images/breeze/output-prepare-airflow-package.svg
+ :width: 100%
+ :alt: Breeze prepare-airflow-package
-Building Production images
+Managing Production images
--------------------------
The **Production image** is also maintained in GitHub Container Registry for Caching
@@ -636,30 +894,30 @@ However in many cases you want to add your own custom version of the image - wit
python dependencies, additional Airflow extras. Breeze's ``build-image`` command helps to build your own,
customized variant of the image that contains everything you need.
-You can switch to building the production image by adding ``--production-image`` flag to the ``build_image``
-command. Note, that the images can also be built using ``docker build`` command by passing appropriate
+You can switch to building the production image by using ``build-prod-image`` command.
+Note, that the images can also be built using ``docker build`` command by passing appropriate
build-args as described in `IMAGES.rst `_ , but Breeze provides several flags that
-makes it easier to do it. You can see all the flags by running ``./breeze build-image --help``,
+makes it easier to do it. You can see all the flags by running ``breeze build-prod-image --help``,
but here typical examples are presented:
.. code-block:: bash
- ./breeze build-image --production-image --additional-extras "jira"
+ breeze build-prod-image --additional-extras "jira"
This installs additional ``jira`` extra while installing airflow in the image.
.. code-block:: bash
- ./breeze build-image --production-image --additional-python-deps "torchio==0.17.10"
+ breeze build-prod-image --additional-python-deps "torchio==0.17.10"
This install additional pypi dependency - torchio in specified version.
.. code-block:: bash
- ./breeze build-image --production-image --additional-dev-apt-deps "libasound2-dev" \
- --additional-runtime-apt-deps "libasound2"
+ breeze build-prod-image --additional-dev-apt-deps "libasound2-dev" \
+ --additional-runtime-apt-deps "libasound2"
This installs additional apt dependencies - ``libasound2-dev`` in the build image and ``libasound`` in the
final image. Those are development dependencies that might be needed to build and use python packages added
@@ -671,11 +929,20 @@ suffix and they need to also be paired with corresponding runtime dependency add
.. code-block:: bash
- ./breeze build-image --production-image --python 3.7 --additional-dev-deps "libasound2-dev" \
+ breeze build-prod-image --python 3.7 --additional-dev-deps "libasound2-dev" \
--additional-runtime-apt-deps "libasound2"
Same as above but uses python 3.7.
+Those are all available flags of ``build-prod-image`` command:
+
+.. image:: ./images/breeze/output-build-prod-image.svg
+ :width: 100%
+ :alt: Breeze commands
+
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
+
.. raw:: html
``. The package name has to be one of the providers or ``apache-airflow``. For
@@ -753,12 +1099,21 @@ instance, for using it with Amazon, the command would be:
.. code-block:: bash
- ./breeze build-docs -- --package-filter apache-airflow-providers-amazon
+ breeze build-docs --package-filter apache-airflow-providers-amazon
Often errors during documentation generation come from the docstrings of auto-api generated classes.
During the docs building auto-api generated files are stored in the ``docs/_api`` folder. This helps you
easily identify the location the problems with documentation originated from.
+Those are all available flags of ``build-docs`` command:
+
+.. image:: ./images/breeze/output-build-docs.svg
+ :width: 100%
+ :alt: Breeze build documentation
+
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
+
.. raw:: html
+3. Follow `Cloning a repository
@@ -685,6 +952,53 @@ Same as above but uses python 3.7.
+You can also pull PROD images in parallel with optional verification.
+
+Those are all available flags of ``pull-prod-image`` command:
+
+.. image:: ./images/breeze/output-pull-prod-image.svg
+ :width: 100%
+ :alt: Breeze pull-prod-image
+
+Finally, you can verify PROD image by running tests - either with the pulled/built images or
+with an arbitrary image.
+
+Those are all available flags of ``verify-prod-image`` command:
+
+.. image:: ./images/breeze/output-verify-prod-image.svg
+ :width: 100%
+ :alt: Breeze verify-prod-image
+
+Releasing Production images to DockerHub
+----------------------------------------
+
+The **Production image** can be released by release managers who have permissions to push the image. This
+happens only when there is an RC candidate or final version of Airflow released.
+
+You release "regular" and "slim" images as separate steps.
+
+Releasing "regular" images:
+
+.. code-block:: bash
+
+ breeze release-prod-images --airflow-version 2.4.0
+
+Or "slim" images:
+
+.. code-block:: bash
+
+ breeze release-prod-images --airflow-version 2.4.0 --slim-images
+
+By default when you are releasing the "final" image, we also tag image with "latest" tags but this
+step can be skipped if you pass the ``--skip-latest`` flag.
+
+These are all of the available flags for the ``release-prod-images`` command:
+
+.. image:: ./images/breeze/output-release-prod-images.svg
+ :width: 100%
+ :alt: Release prod images
+
+
Running static checks
---------------------
@@ -694,18 +1008,51 @@ you have auto-complete setup you should see auto-completable list of all checks
.. code-block:: bash
- ./breeze static-check mypy
+ breeze static-checks -t mypy
The above will run mypy check for currently staged files.
-You can also add arbitrary pre-commit flag after ``--``
+You can also pass specific pre-commit flags for example ``--all-files`` :
.. code-block:: bash
- ./breeze static-check mypy -- --all-files
+ breeze static-checks -t mypy --all-files
The above will run mypy check for all files.
+There is a convenience ``--last-commit`` flag that you can use to run static check on last commit only:
+
+.. code-block:: bash
+
+ breeze static-checks -t mypy --last-commit
+
+The above will run mypy check for all files in the last commit.
+
+There is another convenience ``--commit-ref`` flag that you can use to run static check on specific commit:
+
+.. code-block:: bash
+
+ breeze static-checks -t mypy --commit-ref 639483d998ecac64d0fef7c5aa4634414065f690
+
+The above will run mypy check for all files in the 639483d998ecac64d0fef7c5aa4634414065f690 commit.
+Any ``commit-ish`` reference from Git will work here (branch, tag, short/long hash etc.)
+
+If you ever need to get a list of the files that will be checked (for troubleshooting) use these commands:
+
+.. code-block:: bash
+
+ breeze static-checks -t identity --verbose # currently staged files
+ breeze static-checks -t identity --verbose --from-ref $(git merge-base main HEAD) --to-ref HEAD # branch updates
+
+Those are all available flags of ``static-checks`` command:
+
+.. image:: ./images/breeze/output-static-checks.svg
+ :width: 100%
+ :alt: Breeze static checks
+
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
+
.. raw:: html
@@ -715,13 +1062,12 @@ The above will run mypy check for all files.
-If you ever need to get a list of the files that will be checked (for troubleshooting when playing with
-``--from-ref`` and ``--to-ref``), use these commands:
+.. note::
-.. code-block:: bash
+ When you run static checks, some of the artifacts (mypy_cache) is stored in docker-compose volume
+ so that it can speed up static checks execution significantly. However, sometimes, the cache might
+ get broken, in which case you should run ``breeze stop`` to clean up the cache.
- ./breeze static-check identity --verbose # currently staged files
- ./breeze static-check identity --verbose -- --from-ref $(git merge-base main HEAD) --to-ref HEAD # branch updates
Building the Documentation
--------------------------
@@ -730,7 +1076,7 @@ To build documentation in Breeze, use the ``build-docs`` command:
.. code-block:: bash
- ./breeze build-docs
+ breeze build-docs
Results of the build can be found in the ``docs/_build`` folder.
@@ -745,7 +1091,7 @@ extra ``--`` flag.
.. code-block:: bash
- ./breeze build-docs -- --spellcheck-only
+ breeze build-docs --spellcheck-only
This process can take some time, so in order to make it shorter you can filter by package, using the flag
``--package-filter
@@ -777,34 +1132,45 @@ files are stored in separated orphan branches: ``constraints-main``, ``constrain
Those are constraint files as described in detail in the
``_ contributing documentation.
-You can use ``./breeze generate-constraints`` command to manually generate constraints for a single python
-version and single constraint mode like this:
-.. code-block:: bash
+You can use ``breeze generate-constraints`` command to manually generate constraints for
+all or selected python version and single constraint mode like this:
+
+.. warning::
- ./breeze generate-constraints --generate-constraints-mode pypi-providers
+ In order to generate constraints, you need to build all images with ``--upgrade-to-newer-dependencies``
+ flag - for all python versions.
+.. code-block:: bash
+
+ breeze generate-constraints --airflow-constraints-mode constraints
+
Constraints are generated separately for each python version and there are separate constraints modes:
* 'constraints' - those are constraints generated by matching the current airflow version from sources
and providers that are installed from PyPI. Those are constraints used by the users who want to
- install airflow with pip. Use ``pypi-providers`` mode for that.
+ install airflow with pip.
* "constraints-source-providers" - those are constraints generated by using providers installed from
current sources. While adding new providers their dependencies might change, so this set of providers
is the current set of the constraints for airflow and providers from the current main sources.
- Those providers are used by CI system to keep "stable" set of constraints. Use
- ``source-providers`` mode for that.
+ Those providers are used by CI system to keep "stable" set of constraints.
* "constraints-no-providers" - those are constraints generated from only Apache Airflow, without any
providers. If you want to manage airflow separately and then add providers individually, you can
- use those. Use ``no-providers`` mode for that.
+ use those.
+
+Those are all available flags of ``generate-constraints`` command:
+
+.. image:: ./images/breeze/output-generate-constraints.svg
+ :width: 100%
+ :alt: Breeze generate-constraints
In case someone modifies setup.py, the scheduled CI Tests automatically upgrades and
pushes changes to the constraint files, however you can also perform test run of this locally using
-the procedure described in ``_ which utilises
-multiple processors on your local machine to generate such constraints faster.
+the procedure described in `Refreshing CI Cache `_
+which utilises multiple processors on your local machine to generate such constraints faster.
This bumps the constraint files to latest versions and stores hash of setup.py. The generated constraint
and setup.py hash files are stored in the ``files`` folder and while generating the constraints diff
@@ -833,7 +1199,7 @@ To use your host IDE with Breeze:
.. code-block:: bash
- ./breeze initialize-local-virtualenv --python 3.8
+ ./scripts/tools/initialize_virtualenv.py
.. warning::
Make sure that you use the right Python version in this command - matching the Python version you have
@@ -846,6 +1212,9 @@ This is a lightweight solution that has its own limitations.
More details on using the local virtualenv are available in the `LOCAL_VIRTUALENV.rst `_.
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+but it is not available in the ``breeze`` command):
+
.. raw:: html
`_.
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command and it is not yet available in the current ``breeze`` command):
+
.. raw:: html
`_.
+
+Those are all available flags of ``find-newer-dependencies`` command:
+
+.. image:: ./images/breeze/output-find-newer-dependencies.svg
+ :width: 100%
+ :alt: Breeze find-newer-dependencies
Internal details of Breeze
==========================
@@ -938,11 +1372,17 @@ Running Arbitrary commands in the Breeze environment
----------------------------------------------------
To run other commands/executables inside the Breeze Docker-based environment, use the
-``./breeze shell`` command. You should add your command as -c "command" after ``--`` as extra arguments.
+``breeze shell`` command.
.. code-block:: bash
- ./breeze shell -- -c "ls -la"
+ breeze shell "ls -la"
+
+Those are all available flags of ``shell`` command:
+
+.. image:: ./images/breeze/output-shell.svg
+ :width: 100%
+ :alt: Breeze shell
Running "Docker Compose" commands
---------------------------------
@@ -953,48 +1393,22 @@ after ``--`` as extra arguments.
.. code-block:: bash
- ./breeze docker-compose pull -- --ignore-pull-failures
-
-Restarting Breeze environment
------------------------------
-
-You can also restart the environment and enter it via:
-
-.. code-block:: bash
-
- ./breeze restart
-
+ ./breeze-legacy docker-compose pull -- --ignore-pull-failures
Setting default answers for user interaction
--------------------------------------------
Sometimes during the build, you are asked whether to perform an action, skip it, or quit. This happens
-when rebuilding or removing an image - actions that take a lot of time and could be potentially destructive.
-
-For automation scripts, you can export one of the three variables to control the default
-interaction behaviour:
-
-.. code-block::
-
- export FORCE_ANSWER_TO_QUESTIONS="yes"
-
-If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``yes``, the images are automatically rebuilt when needed.
-Images are deleted without asking.
-
-.. code-block::
-
- export FORCE_ANSWER_TO_QUESTIONS="no"
+when rebuilding or removing an image and in few other cases - actions that take a lot of time
+or could be potentially destructive. You can force answer to the questions by providing an
+``--answer`` flag in the commands that support it.
-If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``no``, the old images are used even if rebuilding is needed.
-This is useful when you work offline. Deleting images is aborted.
+For automation scripts, you can export the ``ANSWER`` variable (and set it to
+``y``, ``n``, ``q``, ``yes``, ``no``, ``quit`` - in all case combinations).
.. code-block::
- export FORCE_ANSWER_TO_QUESTIONS="quit"
-
-If ``FORCE_ANSWER_TO_QUESTIONS`` is set to ``quit``, the whole script is aborted. Deleting images is aborted.
-
-If more than one variable is set, ``yes`` takes precedence over ``no``, which takes precedence over ``quit``.
+ export ANSWER="yes"
Fixing File/Directory Ownership
-------------------------------
@@ -1003,11 +1417,18 @@ On Linux, there is a problem with propagating ownership of created files (a know
files and directories created in the container are not owned by the host user (but by the root user in our
case). This may prevent you from switching branches, for example, if files owned by the root user are
created within your sources. In case you are on a Linux host and have some files in your sources created
-by the root user, you can fix the ownership of those files by running this script:
+by the root user, you can fix the ownership of those files by running :
.. code-block::
- ./scripts/ci/tools/fix_ownership.sh
+ breeze fix-ownership
+
+Those are all available flags of ``fix-ownership`` command:
+
+.. image:: ./images/breeze/output-fix-ownership.svg
+ :width: 100%
+ :alt: Breeze fix-ownership
+
Mounting Local Sources to Breeze
--------------------------------
@@ -1048,6 +1469,7 @@ You can connect to these ports/databases using:
* Flower: http://127.0.0.1:25555
* Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
* Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
+* MSSQL: jdbc:sqlserver://127.0.0.1:21433;databaseName=airflow;user=sa;password=Airflow123
* Redis: redis://127.0.0.1:26379/0
If you do not use ``start-airflow`` command, you can start the webserver manually with
@@ -1120,1683 +1542,25 @@ of the ``Dockerfile.ci``. This way dependencies will be added incrementally.
Before merge, these dependencies should be moved to the appropriate ``apt-get install`` command,
which is already in the ``Dockerfile.ci``.
+Recording command output
+========================
-Breeze Command-Line Interface Reference
-=======================================
-
-Airflow Breeze Syntax
----------------------
-
-This is the current syntax for `./breeze <./breeze>`_:
-
- .. START BREEZE HELP MARKER
-
-.. code-block:: text
-
-
- ####################################################################################################
-
- usage: breeze [FLAGS] [COMMAND] --
-
- By default the script enters the CI container and drops you to bash shell, but you can choose
- one of the commands to run specific actions instead.
-
- Add --help after each command to see details:
+Breeze uses built-in capability of ``rich`` to record and print the command help as an ``svg`` file.
+It's enabled by setting ``RECORD_BREEZE_OUTPUT_FILE`` to a file name where it will be recorded.
+By default it records the screenshots with default characters width and with "Breeze screenshot" title,
+but you can override it with ``RECORD_BREEZE_WIDTH`` and ``RECORD_BREEZE_TITLE`` variables respectively.
- Commands without arguments:
+Uninstalling Breeze
+===================
+Breeze was installed with ``pipx``, with ``pipx list``, you can list the installed packages.
+Once you have the name of ``breeze`` package you can proceed to uninstall it.
- shell [Default] Enters interactive shell in the container
- build-docs Builds documentation in the container
- build-image Builds CI or Production docker image
- prepare-build-cache Prepares CI or Production build cache
- cleanup-image Cleans up the container image created
- exec Execs into running breeze container in new terminal
- generate-constraints Generates pinned constraint files
- initialize-local-virtualenv Initializes local virtualenv
- prepare-airflow-packages Prepares airflow packages
- setup-autocomplete Sets up autocomplete for breeze
- start-airflow Starts Scheduler and Webserver and enters the shell
- stop Stops the docker-compose environment
- restart Stops the docker-compose environment including DB cleanup
- toggle-suppress-cheatsheet Toggles on/off cheatsheet
- toggle-suppress-asciiart Toggles on/off asciiart
-
- Commands with arguments:
-
- docker-compose Executes specified docker-compose command
- kind-cluster Manages KinD cluster on the host
- prepare-provider-documentation Prepares provider packages documentation
- prepare-provider-packages Prepares provider packages
- static-check Performs selected static check for changed files
- tests Runs selected tests in the container
-
- Help commands:
-
- flags Shows all breeze's flags
- help Shows this help message
- help-all Shows detailed help for all commands and flags
-
- ####################################################################################################
-
- Detailed usage
-
- ####################################################################################################
-
-
- Detailed usage for command: shell
-
-
- breeze shell [FLAGS] [-- ]
-
- This is default subcommand if no subcommand is used.
-
- Enters interactive shell where you can run all tests, start Airflow webserver, scheduler,
- workers, interact with the database, run DAGs etc. It is the default command if no command
- is selected. The shell is executed in the container and in case integrations are chosen,
- the integrations will be started as separated docker containers - under the docker-compose
- supervision. Local sources are by default mounted to within the container so you can edit
- them locally and run tests immediately in the container. Several folders ('files', 'dist')
- are also mounted so that you can exchange files between the host and container.
-
- The 'files/airflow-breeze-config/variables.env' file can contain additional variables
- and setup. This file is automatically sourced when you enter the container. Database
- and webserver ports are forwarded to appropriate database/webserver so that you can
- connect to it from your host environment.
-
- You can also pass after -- they will be passed as bash parameters, this is
- especially useful to pass bash options, for example -c to execute command:
+.. code-block:: bash
- 'breeze shell -- -c "ls -la"'
- 'breeze -- -c "ls -la"'
+ pipx list
- For GitHub repository, the --github-repository flag can be used to specify the repository
- to pull and push images. You can also use --github-image-id in case
- you want to pull the image with specific COMMIT_SHA tag.
+This will also remove breeze from the folder: ``${HOME}.local/bin/``
- 'breeze shell \
- --github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
- 'breeze \
- --github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e' - pull/use image with SHA
-
- Most flags are applicable to the shell command as it will run build when needed.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: build-docs
-
-
- breeze build-docs [-- ]
-
- Builds Airflow documentation. The documentation is build inside docker container - to
- maintain the same build environment for everyone. Appropriate sources are mapped from
- the host to the container so that latest sources are used. The folders where documentation
- is generated ('docs/_build') are also mounted to the container - this way results of
- the documentation build is available in the host.
-
- The possible extra args are: --docs-only, --spellcheck-only, --package-filter, --help
-
-
- ####################################################################################################
-
-
- Detailed usage for command: build-image
-
-
- breeze build-image [FLAGS]
-
- Builds docker image (CI or production) without entering the container. You can pass
- additional options to this command, such as:
-
- Choosing python version:
- '--python'
-
- Choosing cache option:
- '--build-cache-local' or '-build-cache-pulled', or '--build-cache-none'
-
- Choosing whether to force pull images or force build the image:
- '--force-build-image'
-
- You can also pass '--production-image' flag to build production image rather than CI image.
-
- For GitHub repository, the '--github-repository' can be used to choose repository
- to pull/push images.
-
- Flags:
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- -a, --install-airflow-version INSTALL_AIRFLOW_VERSION
- Uses different version of Airflow when building PROD image.
-
- 2.0.2 2.0.1 2.0.0 wheel sdist
-
- -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
- Installs Airflow directly from reference in GitHub when building PROD image.
- This can be a GitHub branch like main or v2-2-test, or a tag like 2.2.0rc1.
-
- --installation-method INSTALLATION_METHOD
- Method of installing Airflow in PROD image - either from the sources ('.')
- or from package 'apache-airflow' to install from PyPI.
- Default in Breeze is to install from sources. One of:
-
- . apache-airflow
-
- --upgrade-to-newer-dependencies
- Upgrades PIP packages to latest versions available without looking at the constraints.
-
- -I, --production-image
- Use production image for entering the environment and builds (not for tests).
-
- -F, --force-build-images
- Forces building of the local docker images. The images are rebuilt
- automatically for the first time or when changes are detected in
- package-related files, but you can force it using this flag.
-
- --cleanup-docker-context-files
- Removes whl and tar.gz files created in docker-context-files before running the command.
- In case there are some files there it unnecessarily increases the context size and
- makes the COPY . always invalidated - if you happen to have those files when you build your
- image.
-
- Customization options:
-
- -E, --extras EXTRAS
- Extras to pass to build images The default are different for CI and production images:
-
- CI image:
- devel_ci
-
- Production image:
- amazon,async,celery,cncf.kubernetes,dask,docker,elasticsearch,ftp,google,google_auth,
- grpc,hashicorp,http,ldap,microsoft.azure,mysql,odbc,pandas,postgres,redis,sendgrid,
- sftp,slack,ssh,statsd,virtualenv
-
- --image-tag TAG
- Additional tag in the image.
-
- --skip-installing-airflow-providers-from-sources
- By default 'pip install' in Airflow 2.0 installs only the provider packages that
- are needed by the extras. When you build image during the development (which is
- default in Breeze) all providers are installed by default from sources.
- You can disable it by adding this flag but then you have to install providers from
- wheel packages via --use-packages-from-dist flag.
-
- --disable-pypi-when-building
- Disable installing Airflow from pypi when building. If you use this flag and want
- to install Airflow, you have to install it from packages placed in
- 'docker-context-files' and use --install-from-docker-context-files flag.
-
- --additional-extras ADDITIONAL_EXTRAS
- Additional extras to pass to build images The default is no additional extras.
-
- --additional-python-deps ADDITIONAL_PYTHON_DEPS
- Additional python dependencies to use when building the images.
-
- --dev-apt-command DEV_APT_COMMAND
- The basic command executed before dev apt deps are installed.
-
- --additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
- Additional command executed before dev apt deps are installed.
-
- --additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
- Additional apt dev dependencies to use when building the images.
-
- --dev-apt-deps DEV_APT_DEPS
- The basic apt dev dependencies to use when building the images.
-
- --additional-dev-apt-deps ADDITIONAL_DEV_DEPS
- Additional apt dev dependencies to use when building the images.
-
- --additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
- Additional environment variables set when adding dev dependencies.
-
- --runtime-apt-command RUNTIME_APT_COMMAND
- The basic command executed before runtime apt deps are installed.
-
- --additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
- Additional command executed before runtime apt deps are installed.
-
- --runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
- The basic apt runtime dependencies to use when building the images.
-
- --additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
- Additional apt runtime dependencies to use when building the images.
-
- --additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
- Additional environment variables set when adding runtime dependencies.
-
- Build options:
-
- --disable-mysql-client-installation
- Disables installation of the mysql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
- --disable-mssql-client-installation
- Disables installation of the mssql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
- --constraints-location
- Url to the constraints file. In case of the production image it can also be a path to the
- constraint file placed in 'docker-context-files' folder, in which case it has to be
- in the form of '/docker-context-files/'
-
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
-
- --install-from-docker-context-files
- This flag is used during image building. If it is used additionally to installing
- Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
- in the 'docker-context-files' folder. The same flag can be used during entering the image in
- the CI image - in this case also the .whl and .tar.gz files will be installed automatically
-
- -C, --force-clean-images
- Force build images with cache disabled. This will remove the pulled or build images
- and start building images from scratch. This might take a long time.
-
- -r, --skip-rebuild-check
- Skips checking image for rebuilds. It will use whatever image is available locally/pulled.
-
- -L, --build-cache-local
- Uses local cache to build images. No pulled images will be used, but results of local
- builds in the Docker cache are used instead. This will take longer than when the pulled
- cache is used for the first time, but subsequent '--build-cache-local' builds will be
- faster as they will use mostly the locally build cache.
-
- This is default strategy used by the Production image builds.
-
- -U, --build-cache-pulled
- Uses images pulled from GitHub Container Registry to build images.
- Those builds are usually faster than when ''--build-cache-local'' with the exception if
- the registry images are not yet updated. The images are updated after successful merges
- to main.
-
- This is default strategy used by the CI image builds.
-
- -X, --build-cache-disabled
- Disables cache during docker builds. This is useful if you want to make sure you want to
- rebuild everything from scratch.
-
- This strategy is used by default for both Production and CI images for the scheduled
- (nightly) builds in CI.
-
- -g, --github-repository GITHUB_REPOSITORY
- GitHub repository used to pull, push images.
- Default: apache/airflow.
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: prepare-build-cache
-
-
- breeze prepare-build-cache [FLAGS]
-
- Prepares build cache (CI or production) without entering the container. You can pass
- additional options to this command, such as:
-
- Choosing python version:
- '--python'
-
- You can also pass '--production-image' flag to build production image rather than CI image.
-
- For GitHub repository, the '--github-repository' can be used to choose repository
- to pull/push images. Cleanup docker context files and pull cache are forced. This command
- requires buildx to be installed.
-
- Flags:
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- -a, --install-airflow-version INSTALL_AIRFLOW_VERSION
- Uses different version of Airflow when building PROD image.
-
- 2.0.2 2.0.1 2.0.0 wheel sdist
-
- -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
- Installs Airflow directly from reference in GitHub when building PROD image.
- This can be a GitHub branch like main or v2-2-test, or a tag like 2.2.0rc1.
-
- --installation-method INSTALLATION_METHOD
- Method of installing Airflow in PROD image - either from the sources ('.')
- or from package 'apache-airflow' to install from PyPI.
- Default in Breeze is to install from sources. One of:
-
- . apache-airflow
-
- --upgrade-to-newer-dependencies
- Upgrades PIP packages to latest versions available without looking at the constraints.
-
- -I, --production-image
- Use production image for entering the environment and builds (not for tests).
-
- -g, --github-repository GITHUB_REPOSITORY
- GitHub repository used to pull, push images.
- Default: apache/airflow.
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: cleanup-image
-
-
- breeze cleanup-image [FLAGS]
-
- Removes the breeze-related images created in your local docker image cache. This will
- not reclaim space in docker cache. You need to 'docker system prune' (optionally
- with --all) to reclaim that space.
-
- Flags:
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- -I, --production-image
- Use production image for entering the environment and builds (not for tests).
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: exec
-
-
- breeze exec [-- ]
-
- Execs into interactive shell to an already running container. The container mus be started
- already by breeze shell command. If you are not familiar with tmux, this is the best
- way to run multiple processes in the same container at the same time for example scheduler,
- webserver, workers, database console and interactive terminal.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: generate-constraints
-
-
- breeze generate-constraints [FLAGS]
-
- Generates pinned constraint files with all extras from setup.py. Those files are generated in
- files folder - separate files for different python version. Those constraint files when
- pushed to orphan constraints-main, constraints-2-0 branches are used
- to generate repeatable CI test runs as well as run repeatable production image builds and
- upgrades when you want to include installing or updating some of the released providers
- released at the time particular airflow version was released. You can use those
- constraints to predictably install released Airflow versions. This is mainly used to test
- the constraint generation or manually fix them - constraints are pushed to the orphan
- branches by a successful scheduled CRON job in CI automatically, but sometimes manual fix
- might be needed.
-
- Flags:
-
- --generate-constraints-mode GENERATE_CONSTRAINTS_MODE
- Mode of generating constraints - determines whether providers are installed when generating
- constraints and which version of them (either the ones from sources are used or the ones
- from pypi.
-
- One of:
-
- source-providers pypi-providers no-providers
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: initialize-local-virtualenv
-
-
- breeze initialize-local-virtualenv [FLAGS]
-
- Initializes locally created virtualenv installing all dependencies of Airflow
- taking into account the constraints for the version specified.
- This local virtualenv can be used to aid auto-completion and IDE support as
- well as run unit tests directly from the IDE. You need to have virtualenv
- activated before running this command.
-
- Flags:
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
-
- ####################################################################################################
-
-
- Detailed usage for command: prepare-airflow-packages
-
-
- breeze prepare-airflow-packages [FLAGS]
-
- Prepares airflow packages (sdist and wheel) in dist folder. Note that
- prepare-provider-packages command cleans up the dist folder, so if you want also
- to generate provider packages, make sure you run prepare-provider-packages first,
- and prepare-airflow-packages second. You can specify optional
- --version-suffix-for-pypi flag to generate rc candidates for PyPI packages.
- The packages are prepared in dist folder
-
- Examples:
-
- 'breeze prepare-airflow-packages --package-format wheel' or
- 'breeze prepare-airflow-packages --version-suffix-for-pypi rc1'
-
- Flags:
-
- --package-format PACKAGE_FORMAT
-
- Chooses format of packages to prepare.
-
- One of:
-
- both,sdist,wheel
-
- Default: both
-
- -S, --version-suffix-for-pypi SUFFIX
- Adds optional suffix to the version in the generated provider package. It can be used
- to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-
- -N, --version-suffix-for-svn SUFFIX
- Adds optional suffix to the generated names of package. It can be used to generate
- rc1/rc2 ... versions of the packages to be uploaded to SVN.
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: setup-autocomplete
-
-
- breeze setup-autocomplete
-
- Sets up autocomplete for breeze commands. Once you do it you need to re-enter the bash
- shell and when typing breeze command will provide autocomplete for
- parameters and values.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: start-airflow
-
-
- breeze start-airflow
-
- Like the Shell command this will enter the interactive shell, but it will also start
- automatically the Scheduler and the Webserver. It will leave you in a tmux session where you
- can also observe what is happening in your Airflow.
-
- This is a convenient way to setup a development environment. Your dags will be loaded from the
- folder 'files/dags' on your host machine (it could take some times).
-
- If you want to load default connections and example dags you can use the dedicated flags.
-
- Flags:
-
- --use-airflow-version AIRFLOW_SPECIFICATION
- In CI image, installs Airflow at runtime from PIP released version or using
- the installation method specified (sdist, wheel, none). When 'none' is used,
- airflow is just removed. In this case airflow package should be added to dist folder
- and --use-packages-from-dist flag should be used.
-
- 2.0.2 2.0.1 2.0.0 wheel sdist none
-
- --use-packages-from-dist
- In CI image, if specified it will look for packages placed in dist folder and
- it will install the packages after entering the image.
- This is useful for testing provider packages.
-
- --load-example-dags
- Include Airflow example dags.
-
- --load-default-connections
- Include Airflow Default Connections.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: stop
-
-
- breeze stop
-
- Brings down running docker compose environment. When you start the environment, the docker
- containers will continue running so that startup time is shorter. But they take quite a lot of
- memory and CPU. This command stops all running containers from the environment.
-
- Flags:
-
- --preserve-volumes
- Use this flag if you would like to preserve data volumes from the databases used
- by the integrations. By default, those volumes are deleted, so when you run 'stop'
- or 'restart' commands you start from scratch, but by using this flag you can
- preserve them. If you want to delete those volumes after stopping Breeze, just
- run the 'breeze stop' again without this flag.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: restart
-
-
- breeze restart [FLAGS]
-
- Restarts running docker compose environment. When you restart the environment, the docker
- containers will be restarted. That includes cleaning up the databases. This is
- especially useful if you switch between different versions of Airflow.
-
- Flags:
-
- --preserve-volumes
- Use this flag if you would like to preserve data volumes from the databases used
- by the integrations. By default, those volumes are deleted, so when you run 'stop'
- or 'restart' commands you start from scratch, but by using this flag you can
- preserve them. If you want to delete those volumes after stopping Breeze, just
- run the 'breeze stop' again without this flag.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: toggle-suppress-cheatsheet
-
-
- breeze toggle-suppress-cheatsheet
-
- Toggles on/off cheatsheet displayed before starting bash shell.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: toggle-suppress-asciiart
-
-
- breeze toggle-suppress-asciiart
-
- Toggles on/off asciiart displayed before starting bash shell.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: docker-compose
-
-
- breeze docker-compose [FLAGS] COMMAND [-- ]
-
- Run docker-compose command instead of entering the environment. Use 'help' as command
- to see available commands. The passed after -- are treated
- as additional options passed to docker-compose. For example
-
- 'breeze docker-compose pull -- --ignore-pull-failures'
-
- Flags:
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- -b, --backend BACKEND
- Backend to use for tests - it determines which database is used.
- One of:
-
- sqlite mysql postgres mssql
-
- Default: sqlite
-
- --postgres-version POSTGRES_VERSION
- Postgres version used. One of:
-
- 10 11 12 13
-
- --mysql-version MYSQL_VERSION
- MySql version used. One of:
-
- 5.7 8
-
- --mssql-version MSSQL_VERSION
- MSSql version used. One of:
-
- 2017-latest 2019-latest
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: kind-cluster
-
-
- breeze kind-cluster [FLAGS] OPERATION
-
- Manages host-side Kind Kubernetes cluster that is used to run Kubernetes integration tests.
- It allows to start/stop/restart/status the Kind Kubernetes cluster and deploy Airflow to it.
- This enables you to run tests inside the breeze environment with latest airflow images.
- Note that in case of deploying airflow, the first step is to rebuild the image and loading it
- to the cluster so you can also pass appropriate build image flags that will influence
- rebuilding the production image. Operation is one of:
-
- start stop restart status deploy test shell k9s
-
- The last two operations - shell and k9s allow you to perform interactive testing with
- kubernetes tests. You can enter the shell from which you can run kubernetes tests and in
- another terminal you can start the k9s CLI to debug kubernetes instance. It is an easy
- way to debug the kubernetes deployments.
-
- You can read more about k9s at https://k9scli.io/
-
- Flags:
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- -F, --force-build-images
- Forces building of the local docker images. The images are rebuilt
- automatically for the first time or when changes are detected in
- package-related files, but you can force it using this flag.
-
- --cleanup-docker-context-files
- Removes whl and tar.gz files created in docker-context-files before running the command.
- In case there are some files there it unnecessarily increases the context size and
- makes the COPY . always invalidated - if you happen to have those files when you build your
- image.
-
- Customization options:
-
- -E, --extras EXTRAS
- Extras to pass to build images The default are different for CI and production images:
-
- CI image:
- devel_ci
-
- Production image:
- amazon,async,celery,cncf.kubernetes,dask,docker,elasticsearch,ftp,google,google_auth,
- grpc,hashicorp,http,ldap,microsoft.azure,mysql,odbc,pandas,postgres,redis,sendgrid,
- sftp,slack,ssh,statsd,virtualenv
-
- --image-tag TAG
- Additional tag in the image.
-
- --skip-installing-airflow-providers-from-sources
- By default 'pip install' in Airflow 2.0 installs only the provider packages that
- are needed by the extras. When you build image during the development (which is
- default in Breeze) all providers are installed by default from sources.
- You can disable it by adding this flag but then you have to install providers from
- wheel packages via --use-packages-from-dist flag.
-
- --disable-pypi-when-building
- Disable installing Airflow from pypi when building. If you use this flag and want
- to install Airflow, you have to install it from packages placed in
- 'docker-context-files' and use --install-from-docker-context-files flag.
-
- --additional-extras ADDITIONAL_EXTRAS
- Additional extras to pass to build images The default is no additional extras.
-
- --additional-python-deps ADDITIONAL_PYTHON_DEPS
- Additional python dependencies to use when building the images.
-
- --dev-apt-command DEV_APT_COMMAND
- The basic command executed before dev apt deps are installed.
-
- --additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
- Additional command executed before dev apt deps are installed.
-
- --additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
- Additional apt dev dependencies to use when building the images.
-
- --dev-apt-deps DEV_APT_DEPS
- The basic apt dev dependencies to use when building the images.
-
- --additional-dev-apt-deps ADDITIONAL_DEV_DEPS
- Additional apt dev dependencies to use when building the images.
-
- --additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
- Additional environment variables set when adding dev dependencies.
-
- --runtime-apt-command RUNTIME_APT_COMMAND
- The basic command executed before runtime apt deps are installed.
-
- --additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
- Additional command executed before runtime apt deps are installed.
-
- --runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
- The basic apt runtime dependencies to use when building the images.
-
- --additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
- Additional apt runtime dependencies to use when building the images.
-
- --additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
- Additional environment variables set when adding runtime dependencies.
-
- Build options:
-
- --disable-mysql-client-installation
- Disables installation of the mysql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
- --disable-mssql-client-installation
- Disables installation of the mssql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
- --constraints-location
- Url to the constraints file. In case of the production image it can also be a path to the
- constraint file placed in 'docker-context-files' folder, in which case it has to be
- in the form of '/docker-context-files/'
-
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
-
- --install-from-docker-context-files
- This flag is used during image building. If it is used additionally to installing
- Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
- in the 'docker-context-files' folder. The same flag can be used during entering the image in
- the CI image - in this case also the .whl and .tar.gz files will be installed automatically
-
- -C, --force-clean-images
- Force build images with cache disabled. This will remove the pulled or build images
- and start building images from scratch. This might take a long time.
-
- -r, --skip-rebuild-check
- Skips checking image for rebuilds. It will use whatever image is available locally/pulled.
-
- -L, --build-cache-local
- Uses local cache to build images. No pulled images will be used, but results of local
- builds in the Docker cache are used instead. This will take longer than when the pulled
- cache is used for the first time, but subsequent '--build-cache-local' builds will be
- faster as they will use mostly the locally build cache.
-
- This is default strategy used by the Production image builds.
-
- -U, --build-cache-pulled
- Uses images pulled from GitHub Container Registry to build images.
- Those builds are usually faster than when ''--build-cache-local'' with the exception if
- the registry images are not yet updated. The images are updated after successful merges
- to main.
-
- This is default strategy used by the CI image builds.
-
- -X, --build-cache-disabled
- Disables cache during docker builds. This is useful if you want to make sure you want to
- rebuild everything from scratch.
-
- This strategy is used by default for both Production and CI images for the scheduled
- (nightly) builds in CI.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: prepare-provider-documentation
-
-
- breeze prepare-provider-documentation [FLAGS] [PACKAGE_ID ...]
-
- Prepares documentation files for provider packages.
-
- The command is optionally followed by the list of packages to generate readme for.
- If the first parameter is not formatted as a date, then today is regenerated.
- If no packages are specified, readme for all packages are generated.
- If no date is specified, current date + 3 days is used (allowing for PMC votes to pass).
-
- Examples:
-
- 'breeze prepare-provider-documentation' or
- 'breeze prepare-provider-documentation --version-suffix-for-pypi rc1'
-
- General form:
-
- 'breeze prepare-provider-documentation ...'
-
- * is usually directory in the airflow/providers folder (for example
- 'google' but in several cases, it might be one level deeper separated with
- '.' for example 'apache.hive'
-
- Flags:
-
- -S, --version-suffix-for-pypi SUFFIX
- Adds optional suffix to the version in the generated provider package. It can be used
- to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-
- -N, --version-suffix-for-svn SUFFIX
- Adds optional suffix to the generated names of package. It can be used to generate
- rc1/rc2 ... versions of the packages to be uploaded to SVN.
-
- --package-format PACKAGE_FORMAT
-
- Chooses format of packages to prepare.
-
- One of:
-
- both,sdist,wheel
-
- Default: both
-
- --non-interactive
-
- Runs the command in non-interactive mode.
-
- --generate-providers-issue
-
- Generate providers issue that should be created.
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: prepare-provider-packages
-
-
- breeze prepare-provider-packages [FLAGS] [PACKAGE_ID ...]
-
- Prepares provider packages. You can provide (after --) optional list of packages to prepare.
- If no packages are specified, readme for all packages are generated. You can specify optional
- --version-suffix-for-svn flag to generate rc candidate packages to upload to SVN or
- --version-suffix-for-pypi flag to generate rc candidates for PyPI packages. You can also
- provide both suffixes in case you prepare alpha/beta versions. The packages are prepared in
- dist folder. Note that this command also cleans up dist folder before generating the packages
- so that you do not have accidental files there. This will delete airflow package if it is
- prepared there so make sure you run prepare-provider-packages first,
- and prepare-airflow-packages second.
-
- Examples:
-
- 'breeze prepare-provider-packages' or
- 'breeze prepare-provider-packages google' or
- 'breeze prepare-provider-packages --package-format wheel google' or
- 'breeze prepare-provider-packages --version-suffix-for-svn rc1 http google amazon' or
- 'breeze prepare-provider-packages --version-suffix-for-pypi rc1 http google amazon'
- 'breeze prepare-provider-packages --version-suffix-for-pypi a1
- --version-suffix-for-svn a1 http google amazon'
-
- General form:
-
- 'breeze prepare-provider-packages [--package-format PACKAGE_FORMAT] \
- [--version-suffix-for-svn|--version-suffix-for-pypi] ...'
-
- * is usually directory in the airflow/providers folder (for example
- 'google'), but in several cases, it might be one level deeper separated with '.'
- for example 'apache.hive'
-
- Flags:
-
- --package-format PACKAGE_FORMAT
-
- Chooses format of packages to prepare.
-
- One of:
-
- both,sdist,wheel
-
- Default: both
-
- -S, --version-suffix-for-pypi SUFFIX
- Adds optional suffix to the version in the generated provider package. It can be used
- to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-
- -N, --version-suffix-for-svn SUFFIX
- Adds optional suffix to the generated names of package. It can be used to generate
- rc1/rc2 ... versions of the packages to be uploaded to SVN.
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: static-check
-
-
- breeze static-check [FLAGS] static_check [-- ]
-
- Run selected static checks for currently changed files. You should specify static check that
- you would like to run or 'all' to run all checks. One of:
-
- all airflow-config-yaml airflow-providers-available airflow-provider-yaml-files-ok
- autoflake base-operator black blacken-docs boring-cyborg build
- build-providers-dependencies chart-schema-lint capitalized-breeze
- changelog-duplicates check-apache-license check-builtin-literals
- check-executables-have-shebangs check-extras-order check-hooks-apply
- check-integrations check-merge-conflict check-xml daysago-import-check
- debug-statements detect-private-key docstring-params doctoc dont-use-safe-filter
- end-of-file-fixer fix-encoding-pragma flake8 flynt forbidden-xcom-get-value
- codespell forbid-tabs helm-lint identity incorrect-use-of-LoggingMixin
- insert-license isort json-schema language-matters lint-dockerfile lint-openapi
- markdownlint mermaid migration-reference mixed-line-ending mypy mypy-helm
- no-providers-in-core-examples no-relative-imports persist-credentials-disabled
- pre-commit-descriptions pre-commit-hook-names pretty-format-json
- provide-create-sessions providers-changelogs providers-init-file
- providers-subpackages-init-file provider-yamls pydevd pydocstyle python-no-log-warn
- pyupgrade restrict-start_date rst-backticks setup-order setup-extra-packages
- shellcheck sort-in-the-wild sort-spelling-wordlist stylelint trailing-whitespace
- ui-lint update-breeze-file update-extras update-local-yml-file update-setup-cfg-file
- update-supported-versions update-versions vendor-k8s-json-schema
- verify-db-migrations-documented version-sync www-lint yamllint yesqa
-
- You can pass extra arguments including options to the pre-commit framework as
- passed after --. For example:
-
- 'breeze static-check mypy' or
- 'breeze static-check mypy -- --files tests/core.py'
- 'breeze static-check mypy -- --all-files'
-
- To check all files that differ between you current branch and main run:
-
- 'breeze static-check all -- --from-ref $(git merge-base main HEAD) --to-ref HEAD'
-
- To check all files that are in the HEAD commit run:
-
- 'breeze static-check mypy -- --from-ref HEAD^ --to-ref HEAD'
-
-
- You can see all the options by adding --help EXTRA_ARG:
-
- 'breeze static-check mypy -- --help'
-
-
- ####################################################################################################
-
-
- Detailed usage for command: tests
-
-
- breeze tests [FLAGS] [TEST_TARGET ..] [-- ]
-
- Run the specified unit test target. There might be multiple
- targets specified separated with comas. The passed after -- are treated
- as additional options passed to pytest. You can pass 'tests' as target to
- run all tests. For example:
-
- 'breeze tests tests/core/test_core.py -- --logging-level=DEBUG'
- 'breeze tests tests
-
- Flags:
-
- --test-type TEST_TYPE
- Type of the test to run. One of:
-
- All,Always,Core,Providers,API,CLI,Integration,Other,WWW,Postgres,MySQL,Helm,
- Quarantined
-
- Default: All
-
-
- ####################################################################################################
-
-
- Detailed usage for command: flags
-
-
- Explains in detail all the flags that can be used with breeze.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: help
-
-
- breeze help
-
- Shows general help message for all commands.
-
-
- ####################################################################################################
-
-
- Detailed usage for command: help-all
-
-
- breeze help-all
-
- Shows detailed help for all commands and flags.
-
-
- ####################################################################################################
-
-
- ####################################################################################################
-
- Summary of all flags supported by Breeze:
-
- ****************************************************************************************************
- Choose Airflow variant
-
- -p, --python PYTHON_MAJOR_MINOR_VERSION
- Python version used for the image. This is always major/minor version.
-
- One of:
-
- 3.7 3.8 3.9 3.10
-
- --platform PLATFORM
- Builds image for the platform specified.
-
- One of:
-
- linux/amd64 linux/arm64 linux/amd64,linux/arm64
-
-
- -d, --debian DEBIAN_VERSION
- Debian version used for the image. This is always name of the debian distribution version.
-
- One of:
-
- bullseye buster
-
- ****************************************************************************************************
- Choose backend to run for Airflow
-
- -b, --backend BACKEND
- Backend to use for tests - it determines which database is used.
- One of:
-
- sqlite mysql postgres mssql
-
- Default: sqlite
-
- --postgres-version POSTGRES_VERSION
- Postgres version used. One of:
-
- 10 11 12 13
-
- --mysql-version MYSQL_VERSION
- MySql version used. One of:
-
- 5.7 8
-
- --mssql-version MSSQL_VERSION
- MSSql version used. One of:
-
- 2017-latest 2019-latest
-
- ****************************************************************************************************
- Enable production image
-
- -I, --production-image
- Use production image for entering the environment and builds (not for tests).
-
- ****************************************************************************************************
- Additional actions executed while entering breeze
-
- -d, --db-reset
- Resets the database at entry to the environment. It will drop all the tables
- and data and recreate the DB from scratch even if 'restart' command was not used.
- Combined with 'restart' command it enters the environment in the state that is
- ready to start Airflow webserver/scheduler/worker. Without the switch, the database
- does not have any tables and you need to run reset db manually.
-
- -i, --integration INTEGRATION
- Integration to start during tests - it determines which integrations are started
- for integration tests. There can be more than one integration started, or all to
- start all integrations. Selected integrations are not saved for future execution.
- One of:
-
- cassandra kerberos mongo openldap pinot rabbitmq redis statsd trino all
-
- --init-script INIT_SCRIPT_FILE
- Initialization script name - Sourced from files/airflow-breeze-config. Default value
- init.sh. It will be executed after the environment is configured and started.
-
- ****************************************************************************************************
- Additional actions executed while starting Airflow
-
- --load-example-dags
- Include Airflow example dags.
-
- --load-default-connections
- Include Airflow Default Connections.
-
- ****************************************************************************************************
- Cleanup options when stopping Airflow
-
- --preserve-volumes
- Use this flag if you would like to preserve data volumes from the databases used
- by the integrations. By default, those volumes are deleted, so when you run 'stop'
- or 'restart' commands you start from scratch, but by using this flag you can
- preserve them. If you want to delete those volumes after stopping Breeze, just
- run the 'breeze stop' again without this flag.
-
- ****************************************************************************************************
- Kind kubernetes and Kubernetes tests configuration(optional)
-
- Configuration for the KinD Kubernetes cluster and tests:
-
- -K, --kubernetes-mode KUBERNETES_MODE
- Kubernetes mode - only used in case one of kind-cluster commands is used.
- One of:
-
- image
-
- Default: image
-
- -V, --kubernetes-version KUBERNETES_VERSION
- Kubernetes version - only used in case one of kind-cluster commands is used.
- One of:
-
- v1.21.1 v1.20.2
-
- Default: v1.21.1
-
- --kind-version KIND_VERSION
- Kind version - only used in case one of kind-cluster commands is used.
- One of:
-
- v0.11.1
-
- Default: v0.11.1
-
- --helm-version HELM_VERSION
- Helm version - only used in case one of kind-cluster commands is used.
- One of:
-
- v3.6.3
-
- Default: v3.6.3
-
- --executor EXECUTOR
- Executor to use in a kubernetes cluster.
- One of:
-
- KubernetesExecutor CeleryExecutor LocalExecutor CeleryKubernetesExecutor
- LocalKubernetesExecutor
-
- Default: KubernetesExecutor
-
- ****************************************************************************************************
- Manage mounting local files
-
- -l, --skip-mounting-local-sources
- Skips mounting local volume with sources - you get exactly what is in the
- docker image rather than your current local sources of Airflow.
-
- ****************************************************************************************************
- Assume answers to questions
-
- -y, --assume-yes
- Assume 'yes' answer to all questions.
-
- -n, --assume-no
- Assume 'no' answer to all questions.
-
- -q, --assume-quit
- Assume 'quit' answer to all questions.
-
- ****************************************************************************************************
- Install different Airflow version during PROD image build
-
- -a, --install-airflow-version INSTALL_AIRFLOW_VERSION
- Uses different version of Airflow when building PROD image.
-
- 2.0.2 2.0.1 2.0.0 wheel sdist
-
- -t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
- Installs Airflow directly from reference in GitHub when building PROD image.
- This can be a GitHub branch like main or v2-2-test, or a tag like 2.2.0rc1.
-
- --installation-method INSTALLATION_METHOD
- Method of installing Airflow in PROD image - either from the sources ('.')
- or from package 'apache-airflow' to install from PyPI.
- Default in Breeze is to install from sources. One of:
-
- . apache-airflow
-
- --upgrade-to-newer-dependencies
- Upgrades PIP packages to latest versions available without looking at the constraints.
-
- ****************************************************************************************************
- Use different Airflow version at runtime in CI image
-
- --use-airflow-version AIRFLOW_SPECIFICATION
- In CI image, installs Airflow at runtime from PIP released version or using
- the installation method specified (sdist, wheel, none). When 'none' is used,
- airflow is just removed. In this case airflow package should be added to dist folder
- and --use-packages-from-dist flag should be used.
-
- 2.0.2 2.0.1 2.0.0 wheel sdist none
-
- --use-packages-from-dist
- In CI image, if specified it will look for packages placed in dist folder and
- it will install the packages after entering the image.
- This is useful for testing provider packages.
-
- ****************************************************************************************************
- Credentials
-
- -f, --forward-credentials
- Forwards host credentials to docker container. Use with care as it will make
- your credentials available to everything you install in Docker.
-
- ****************************************************************************************************
- Flags for building Docker images (both CI and production)
-
- -F, --force-build-images
- Forces building of the local docker images. The images are rebuilt
- automatically for the first time or when changes are detected in
- package-related files, but you can force it using this flag.
-
- --cleanup-docker-context-files
- Removes whl and tar.gz files created in docker-context-files before running the command.
- In case there are some files there it unnecessarily increases the context size and
- makes the COPY . always invalidated - if you happen to have those files when you build your
- image.
-
- Customization options:
-
- -E, --extras EXTRAS
- Extras to pass to build images The default are different for CI and production images:
-
- CI image:
- devel_ci
-
- Production image:
- amazon,async,celery,cncf.kubernetes,dask,docker,elasticsearch,ftp,google,google_auth,
- grpc,hashicorp,http,ldap,microsoft.azure,mysql,odbc,pandas,postgres,redis,sendgrid,
- sftp,slack,ssh,statsd,virtualenv
-
- --image-tag TAG
- Additional tag in the image.
-
- --skip-installing-airflow-providers-from-sources
- By default 'pip install' in Airflow 2.0 installs only the provider packages that
- are needed by the extras. When you build image during the development (which is
- default in Breeze) all providers are installed by default from sources.
- You can disable it by adding this flag but then you have to install providers from
- wheel packages via --use-packages-from-dist flag.
-
- --disable-pypi-when-building
- Disable installing Airflow from pypi when building. If you use this flag and want
- to install Airflow, you have to install it from packages placed in
- 'docker-context-files' and use --install-from-docker-context-files flag.
-
- --additional-extras ADDITIONAL_EXTRAS
- Additional extras to pass to build images The default is no additional extras.
-
- --additional-python-deps ADDITIONAL_PYTHON_DEPS
- Additional python dependencies to use when building the images.
-
- --dev-apt-command DEV_APT_COMMAND
- The basic command executed before dev apt deps are installed.
-
- --additional-dev-apt-command ADDITIONAL_DEV_APT_COMMAND
- Additional command executed before dev apt deps are installed.
-
- --additional-dev-apt-deps ADDITIONAL_DEV_APT_DEPS
- Additional apt dev dependencies to use when building the images.
-
- --dev-apt-deps DEV_APT_DEPS
- The basic apt dev dependencies to use when building the images.
-
- --additional-dev-apt-deps ADDITIONAL_DEV_DEPS
- Additional apt dev dependencies to use when building the images.
-
- --additional-dev-apt-envs ADDITIONAL_DEV_APT_ENVS
- Additional environment variables set when adding dev dependencies.
-
- --runtime-apt-command RUNTIME_APT_COMMAND
- The basic command executed before runtime apt deps are installed.
-
- --additional-runtime-apt-command ADDITIONAL_RUNTIME_APT_COMMAND
- Additional command executed before runtime apt deps are installed.
-
- --runtime-apt-deps ADDITIONAL_RUNTIME_APT_DEPS
- The basic apt runtime dependencies to use when building the images.
-
- --additional-runtime-apt-deps ADDITIONAL_RUNTIME_DEPS
- Additional apt runtime dependencies to use when building the images.
-
- --additional-runtime-apt-envs ADDITIONAL_RUNTIME_APT_DEPS
- Additional environment variables set when adding runtime dependencies.
-
- Build options:
-
- --disable-mysql-client-installation
- Disables installation of the mysql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
- --disable-mssql-client-installation
- Disables installation of the mssql client which might be problematic if you are building
- image in controlled environment. Only valid for production image.
-
- --constraints-location
- Url to the constraints file. In case of the production image it can also be a path to the
- constraint file placed in 'docker-context-files' folder, in which case it has to be
- in the form of '/docker-context-files/'
-
- --disable-pip-cache
- Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
-
- --install-from-docker-context-files
- This flag is used during image building. If it is used additionally to installing
- Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
- in the 'docker-context-files' folder. The same flag can be used during entering the image in
- the CI image - in this case also the .whl and .tar.gz files will be installed automatically
-
- -C, --force-clean-images
- Force build images with cache disabled. This will remove the pulled or build images
- and start building images from scratch. This might take a long time.
-
- -r, --skip-rebuild-check
- Skips checking image for rebuilds. It will use whatever image is available locally/pulled.
-
- -L, --build-cache-local
- Uses local cache to build images. No pulled images will be used, but results of local
- builds in the Docker cache are used instead. This will take longer than when the pulled
- cache is used for the first time, but subsequent '--build-cache-local' builds will be
- faster as they will use mostly the locally build cache.
-
- This is default strategy used by the Production image builds.
-
- -U, --build-cache-pulled
- Uses images pulled from GitHub Container Registry to build images.
- Those builds are usually faster than when ''--build-cache-local'' with the exception if
- the registry images are not yet updated. The images are updated after successful merges
- to main.
-
- This is default strategy used by the CI image builds.
-
- -X, --build-cache-disabled
- Disables cache during docker builds. This is useful if you want to make sure you want to
- rebuild everything from scratch.
-
- This strategy is used by default for both Production and CI images for the scheduled
- (nightly) builds in CI.
-
- ****************************************************************************************************
- Flags for pulling/pushing Docker images (both CI and production)
-
- -g, --github-repository GITHUB_REPOSITORY
- GitHub repository used to pull, push images.
- Default: apache/airflow.
-
-
-
-
- -s, --github-image-id COMMIT_SHA
- of the image. Images in GitHub registry are stored with those
- to be able to easily find the image for particular CI runs. Once you know the
- , you can specify it in github-image-id flag and Breeze will
- automatically pull and use that image so that you can easily reproduce a problem
- that occurred in CI.
-
- Default: latest.
-
- ****************************************************************************************************
- Flags for running tests
-
- --test-type TEST_TYPE
- Type of the test to run. One of:
-
- All,Always,Core,Providers,API,CLI,Integration,Other,WWW,Postgres,MySQL,Helm,
- Quarantined
-
- Default: All
-
- ****************************************************************************************************
- Flags for generation of the provider packages
-
- -S, --version-suffix-for-pypi SUFFIX
- Adds optional suffix to the version in the generated provider package. It can be used
- to generate rc1/rc2 ... versions of the packages to be uploaded to PyPI.
-
- -N, --version-suffix-for-svn SUFFIX
- Adds optional suffix to the generated names of package. It can be used to generate
- rc1/rc2 ... versions of the packages to be uploaded to SVN.
-
- ****************************************************************************************************
- Increase verbosity of the scripts
-
- -v, --verbose
- Show verbose information about executed docker, kind, kubectl, helm commands. Useful for
- debugging - when you run breeze with --verbose flags you will be able to see the commands
- executed under the hood and copy&paste them to your terminal to debug them more easily.
-
- Note that you can further increase verbosity and see all the commands executed by breeze
- by running 'export VERBOSE_COMMANDS="true"' before running breeze.
-
- --dry-run-docker
- Only show docker commands to execute instead of actually executing them. The docker
- commands are printed in yellow color.
-
- ****************************************************************************************************
- Print detailed help message
-
- -h, --help
- Shows detailed help message for the command specified.
+.. code-block:: bash
- .. END BREEZE HELP MARKER
+ pipx uninstall apache-airflow-breeze
diff --git a/Breeze2 b/Breeze2
deleted file mode 100755
index 9591f6ba7d748..0000000000000
--- a/Breeze2
+++ /dev/null
@@ -1,83 +0,0 @@
-#!/usr/bin/env python3
-# isort: skip
-import os
-import sys
-
-# Python <3.4 does not have pathlib
-from venv import EnvBuilder
-
-if sys.version_info.major != 3 or sys.version_info.minor < 7:
- print("ERROR! Make sure you use Python 3.7+ !!")
- sys.exit(1)
-
-import subprocess
-from os import execv
-from pathlib import Path
-
-if getattr(sys, 'frozen', False):
- # If the application is run as a bundle, the PyInstaller bootloader
- # extends the sys module by a flag frozen=True and sets the temporary app
- # path into variable _MEIPASS' and sys.executable is Breeze's executable path.
- AIRFLOW_SOURCES_DIR = Path(sys.executable).parent.resolve()
-else:
- AIRFLOW_SOURCES_DIR = Path(__file__).parent.resolve()
-BUILD_DIR = AIRFLOW_SOURCES_DIR / ".build"
-BUILD_BREEZE_DIR = BUILD_DIR / "breeze2"
-BUILD_BREEZE_CFG_SAVED = BUILD_BREEZE_DIR / "setup.cfg.saved"
-BUILD_BREEZE_VENV_DIR = BUILD_BREEZE_DIR / "venv"
-BUILD_BREEZE_VENV_BIN_DIR = BUILD_BREEZE_VENV_DIR / ("Scripts" if os.name == 'nt' else "bin")
-BUILD_BREEZE_VENV_PYTHON = BUILD_BREEZE_VENV_BIN_DIR / "python"
-BUILD_BREEZE_VENV_BREEZE = BUILD_BREEZE_VENV_BIN_DIR / "Breeze2"
-
-BREEZE_SOURCE_PATH = AIRFLOW_SOURCES_DIR / "dev" / "breeze"
-BREEZE_SETUP_CFG_PATH = BREEZE_SOURCE_PATH / "setup.cfg"
-
-BUILD_BREEZE_DIR.mkdir(parents=True, exist_ok=True)
-
-
-def needs_installation() -> bool:
- """Returns true if Breeze's virtualenv needs (re)installation"""
- if not BUILD_BREEZE_VENV_DIR.exists() or not BUILD_BREEZE_CFG_SAVED.exists():
- return True
- return BREEZE_SETUP_CFG_PATH.read_text() != BUILD_BREEZE_CFG_SAVED.read_text()
-
-
-def save_config():
- """Saves cfg file to virtualenv to check if there is a need for reinstallation of the virtualenv"""
- BUILD_BREEZE_CFG_SAVED.write_text(BREEZE_SETUP_CFG_PATH.read_text())
-
-
-if needs_installation():
- print(f"(Re)Installing Breeze's virtualenv in {BUILD_BREEZE_VENV_DIR}")
- try:
- EnvBuilder(system_site_packages=False, upgrade=True, with_pip=True, prompt="breeze").create(
- str(BUILD_BREEZE_VENV_DIR)
- )
- except Exception as e:
- # in some cases (mis-configured python) the venv creation might not work via API
- # (ensurepip missing). This is the case in case of default MacOS Python and Python executable
- # Bundled in Windows executable, In this case we fallback to running venv as a tool using default
- # Python3 found on path (in case of Windows Bundled exe, you don't even have a current
- # interpreted executable available, because Python interpreter is executed through a library.
- # and sys.executable points to the Bundled exe file.
- BUILD_BREEZE_VENV_DIR.mkdir(parents=True, exist_ok=True)
- subprocess.run(["python3", "-m", "venv", f"{BUILD_BREEZE_VENV_DIR}"], check=True)
- if os.name == 'nt':
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PYTHON}", "-m", "pip", "install", "--upgrade", "-e", "."],
- cwd=BREEZE_SOURCE_PATH,
- check=True,
- )
- else:
- subprocess.run(
- [f"{BUILD_BREEZE_VENV_PYTHON}", "-m", "pip", "install", "--upgrade", "-e", "."],
- cwd=BREEZE_SOURCE_PATH,
- check=True,
- )
- save_config()
-
-if os.name == 'nt':
- # This is the best way of running it on Windows, though it leaves the original process hanging around
- subprocess.run([f"{BUILD_BREEZE_VENV_BREEZE}"] + sys.argv[1:], check=True)
-else:
- execv(f"{BUILD_BREEZE_VENV_BREEZE}", [f"{BUILD_BREEZE_VENV_BREEZE}"] + sys.argv[1:])
diff --git a/CI.rst b/CI.rst
index 43ea218a69a7e..f24639271e977 100644
--- a/CI.rst
+++ b/CI.rst
@@ -59,14 +59,14 @@ Container Registry used as cache
We are using GitHub Container Registry to store the results of the ``Build Images``
workflow which is used in the ``Tests`` workflow.
-Currently in main version of Airflow we run tests in 4 different versions of Python (3.6, 3.7, 3.8, 3.9)
+Currently in main version of Airflow we run tests in 4 different versions of Python (3.7, 3.8, 3.9, 3.10)
which means that we have to build 8 images (4 CI ones and 4 PROD ones). Yet we run around 12 jobs
with each of the CI images. That is a lot of time to just build the environment to run. Therefore
we are utilising ``pull_request_target`` feature of GitHub Actions.
This feature allows to run a separate, independent workflow, when the main workflow is run -
this separate workflow is different than the main one, because by default it runs using ``main`` version
-of the sources but also - and most of all - that it has WRITE access to the Github Container Image registry.
+of the sources but also - and most of all - that it has WRITE access to the GitHub Container Image registry.
This is especially important in our case where Pull Requests to Airflow might come from any repository,
and it would be a huge security issue if anyone from outside could
@@ -158,7 +158,7 @@ You can use those variables when you try to reproduce the build locally.
| | | | | builds it forces rebuild, regardless if it |
| | | | | is determined to be needed. |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
-| ``FORCE_ANSWER_TO_QUESTIONS`` | | yes | yes | This variable determines if answer to questions |
+| ``ANSWER`` | | yes | yes | This variable determines if answer to questions |
| | | | | during the build process should be |
| | | | | automatically given. For local development, |
| | | | | the user is occasionally asked to provide |
@@ -186,13 +186,6 @@ You can use those variables when you try to reproduce the build locally.
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``HOST_OS`` | | Linux | Linux | OS of the Host (Darwin/Linux). |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
-| ``HOST_HOME`` | | | | Home directory on the host. |
-+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
-| Version suffix variables |
-+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
-| ``VERSION_SUFFIX_FOR_PYPI`` | | | | Version suffix used during provider |
-| | | | | package preparation for PyPI builds. |
-+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| Git variables |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``COMMIT_SHA`` | | GITHUB_SHA | GITHUB_SHA | SHA of the commit of the build is run |
@@ -491,7 +484,7 @@ running, GitHub Actions will cancel the old workflow run automatically.
Build Images Workflow
---------------------
-This workflow builds images for the CI Workflow.
+This workflow builds images for the CI Workflow for Pull Requests coming from forks.
It's a special type of workflow: ``pull_request_target`` which means that it is triggered when a pull request
is opened. This also means that the workflow has Write permission to push to the GitHub registry the images
@@ -499,8 +492,14 @@ used by CI jobs which means that the images can be built only once and reused by
(including the matrix jobs). We've implemented it so that the ``Tests`` workflow waits
until the images are built by the ``Build Images`` workflow before running.
-This workflow is also triggered on normal pushes to our "main" branches, i.e. after a
-pull request is merged and whenever ``scheduled`` run is triggered.
+Those "Build Image" steps are skipped in case Pull Requests do not come from "forks" (i.e. those
+are internal PRs for Apache Airflow repository. This is because in case of PRs coming from
+Apache Airflow (only committers can create those) the "pull_request" workflows have enough
+permission to push images to GitHub Registry.
+
+This workflow is not triggered on normal pushes to our "main" branches, i.e. after a
+pull request is merged and whenever ``scheduled`` run is triggered. Again in this case the "CI" workflow
+has enough permissions to push the images. In this case we simply do not run this workflow.
The workflow has the following jobs:
@@ -662,9 +661,9 @@ For example knowing that the CI job was for commit ``cd27124534b46c9688a1d89e75f
.. code-block:: bash
- docker pull ghcr.io/apache/airflow/main/ci/python3.6:cd27124534b46c9688a1d89e75fcd137ab5137e3
+ docker pull ghcr.io/apache/airflow/main/ci/python3.7:cd27124534b46c9688a1d89e75fcd137ab5137e3
- docker run -it ghcr.io/apache/airflow/main/ci/python3.6:cd27124534b46c9688a1d89e75fcd137ab5137e3
+ docker run -it ghcr.io/apache/airflow/main/ci/python3.7:cd27124534b46c9688a1d89e75fcd137ab5137e3
But you usually need to pass more variables and complex setup if you want to connect to a database or
@@ -674,7 +673,7 @@ cd27124534b46c9688a1d89e75fcd137ab5137e3, in python 3.8 environment you can run:
.. code-block:: bash
- ./breeze --github-image-id cd27124534b46c9688a1d89e75fcd137ab5137e3 --python 3.8
+ ./breeze-legacy --github-image-id cd27124534b46c9688a1d89e75fcd137ab5137e3 --python 3.8
You will be dropped into a shell with the exact version that was used during the CI run and you will
be able to run pytest tests manually, easily reproducing the environment that was used in CI. Note that in
@@ -701,13 +700,7 @@ In order to add a new version the following operations should be done (example u
.. code-block:: bash
- ./breeze build-image --python 3.10
-
-* push image as cache to GitHub:
-
-.. code-block:: bash
-
- ./breeze push-image --python 3.10
+ breeze build-image --python 3.10
* Find the 2 new images (prod, ci) created in
`GitHub Container registry `_
diff --git a/CI_DIAGRAMS.md b/CI_DIAGRAMS.md
index 1bbee13ae2aa2..3b228b13c71b7 100644
--- a/CI_DIAGRAMS.md
+++ b/CI_DIAGRAMS.md
@@ -33,15 +33,13 @@ sequenceDiagram
activate Tests
Tests -->> Build Images: Trigger 'pull_request_target'
activate Build Images
- Note over Build Images: Build info
- par 3.6, [3.7, 3.8, 3.9]
- activate GitHub Registry
- GitHub Registry ->> Build Images: Pull CI Images from Cache
- deactivate GitHub Registry
- Note over Build Images: Build CI Images
[COMMIT_SHA] - end - par No CI image - Note over Tests: Build info
Which tests?
Which Python? + Note over Build Images: Build info
Decide which Python + Note over Tests: Build info
Decide on tests
Decide on Matrix (selective) + Note over Tests: Skip Build
(Runs in 'Build Images')
CI Images + Note over Tests: Skip Build
(Runs in 'Build Images')
PROD Images + par + GitHub Registry ->> Build Images: Pull CI Images
[latest] + Note over Build Images: Build CI Images
[COMMIT_SHA]
Use latest constraints
or upgrade if setup changed and Note over Tests: OpenAPI client gen and @@ -49,77 +47,135 @@ sequenceDiagram and Note over Tests: Test examples
PROD image building end - par 3.6, [3.7, 3.8, 3.9] - activate GitHub Registry - Build Images ->> GitHub Registry: Push CI Images - Note over GitHub Registry: Tagged CI Images
[COMMIT_SHA] - end - par 3.6, [3.7, 3.8, 3.9] - GitHub Registry ->> Build Images: Pull PROD Images from Cache - Note over Build Images: Build PROD Images
[COMMIT_SHA] - end + Build Images ->> GitHub Registry: Push CI Images
[COMMIT_SHA] loop Wait for CI images - par 3.6, [3.7, 3.8, 3.9] - Tests ->> Tests: Check CI Images - Note over Tests: Wait for
[COMMIT_SHA] - end + GitHub Registry ->> Tests: Pull CI Images
[COMMIT_SHA] end - par 3.6, [3.7, 3.8, 3.9] - GitHub Registry ->> Tests: Pull CI Image - Note over Tests: Verify CI Image - end - deactivate GitHub Registry - par 3.6, [3.7, 3.8, 3.9] - opt Needed? + Note over Tests: Verify CI Images
[COMMIT_SHA] + par + GitHub Registry ->> Build Images: Pull PROD Images
[latest] + Note over Build Images: Build PROD Images
[COMMIT_SHA] + and + opt Note over Tests: Run static checks end and - opt Needed? + opt Note over Tests: Run basic
static checks end and - opt Needed? + opt Note over Tests: Build docs end and - opt Needed? + opt Note over Tests: Tests end and - opt Needed? + opt Note over Tests: Test provider
packages build end and - opt Needed? + opt Note over Tests: Helm tests end end - par 3.6, [3.7, 3.8, 3.9] - Build Images ->> GitHub Registry: Push PROD Images - activate GitHub Registry - end + Build Images ->> GitHub Registry: Push PROD Images
[COMMIT_SHA] deactivate Build Images - Note over GitHub Registry: Tagged PROD Images
[COMMIT_SHA] loop Wait for PROD images - par 3.6, [3.7, 3.8, 3.9] - Tests ->> Tests: Check PROD Images - Note over Tests: Wait for
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull PROD Images
[COMMIT_SHA] + end + Note over Tests: Verify PROD Image
[COMMIT_SHA] + par + opt + Note over Tests: Run Kubernetes
tests + end + and + opt + Note over Tests: Run Kubernetes
upgrade tests end end - par 3.6, [3.7, 3.8, 3.9] - GitHub Registry ->> Tests: Pull PROD Image - Note over Tests: Verify PROD Image + opt + Note over Tests: Generate constraints end - deactivate GitHub Registry - par 3.6, [3.7, 3.8, 3.9] - opt Needed? + Tests -->> Airflow Repo: Status update + deactivate Airflow Repo + deactivate Tests +``` + +## Pull request flow from "apache/airflow" repo + +```mermaid +sequenceDiagram + Note over Airflow Repo: pull request + Note over Tests: pull_request
[Read Token] + Note over Build Images: pull_request_target
[Write Token] + activate Airflow Repo + Airflow Repo -->> Tests: Trigger 'pull_request' + activate Tests + Tests -->> Build Images: Trigger 'pull_request_target' + activate Build Images + Note over Build Images: Build info + Note over Build Images: Skip Build
(Runs in 'Tests')
CI Images + Note over Build Images: Skip Build
(Runs in 'Tests')
PROD Images + deactivate Build Images + Note over Tests: Build info
Decide on tests
Decide on Matrix (selective) + par + GitHub Registry ->> Tests: Pull CI Images
[latest] + Note over Tests: Build CI Images
[COMMIT_SHA]
Use latest constraints
or upgrade if setup changed + and + Note over Tests: OpenAPI client gen + and + Note over Tests: Test UI + and + Note over Tests: Test examples
PROD image building + end + Tests ->> GitHub Registry: Push CI Images
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull CI Images
[COMMIT_SHA] + Note over Tests: Verify CI Image
[COMMIT_SHA] + par + GitHub Registry ->> Tests: Pull PROD Images
[latest] + Note over Tests: Build PROD Images
[COMMIT_SHA] + and + opt + Note over Tests: Run static checks + end + and + opt + Note over Tests: Run basic
static checks + end + and + opt + Note over Tests: Build docs + end + and + opt + Note over Tests: Tests + end + and + opt + Note over Tests: Test provider
packages build + end + and + opt + Note over Tests: Helm tests + end + end + Tests ->> GitHub Registry: Push PROD Images
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull PROD Image
[COMIT_SHA] + Note over Tests: Verify PROD Image
[COMMIT_SHA] + par + opt Note over Tests: Run Kubernetes
tests end and - opt Needed? + opt Note over Tests: Run Kubernetes
upgrade tests end end + opt + Note over Tests: Generate constraints + end Tests -->> Airflow Repo: Status update deactivate Airflow Repo deactivate Tests @@ -129,23 +185,15 @@ sequenceDiagram ```mermaid sequenceDiagram - Note over Airflow Repo: merge + Note over Airflow Repo: pull request Note over Tests: push
[Write Token] - Note over Build Images: push
[Write Token] activate Airflow Repo Airflow Repo -->> Tests: Trigger 'push' activate Tests - Airflow Repo -->> Build Images: Trigger 'push' - activate Build Images - Note over Build Images: Build info - par 3.6, 3.7, 3.8, 3.9 - activate GitHub Registry - GitHub Registry ->> Build Images: Pull CI Images from Cache - deactivate GitHub Registry - Note over Build Images: Build CI Images
[COMMIT_SHA] - end - par No CI image - Note over Tests: Build info
All tests
All python + Note over Tests: Build info
All tests
Full matrix + par + GitHub Registry ->> Tests: Pull CI Images
[latest] + Note over Tests: Build CI Images
[COMMIT_SHA]
Always upgrade deps and Note over Tests: OpenAPI client gen and @@ -153,72 +201,60 @@ sequenceDiagram and Note over Tests: Test examples
PROD image building end - par 3.6, 3.7, 3.8, 3.9 - Build Images ->> GitHub Registry: Push CI Images - activate GitHub Registry - Note over GitHub Registry: Tagged CI Images
[COMMIT_SHA] - end - par 3.6, 3.7, 3.8, 3.9 - GitHub Registry ->> Build Images: Pull PROD Images from Cache - Note over Build Images: Build PROD Images
[COMMIT_SHA] - end - loop Wait for CI images - par 3.6, 3.7, 3.8, 3.9 - Tests ->> Tests: Check CI Images - Note over Tests: Wait for
[COMMIT_SHA] + Tests ->> GitHub Registry: Push CI Images
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull CI Images
[COMMIT_SHA] + Note over Tests: Verify CI Image
[COMMIT_SHA] + par + GitHub Registry ->> Tests: Pull PROD Images
[latest] + Note over Tests: Build PROD Images
[COMMIT_SHA] + and + opt + Note over Tests: Run static checks end - end - par 3.6, 3.7, 3.8, 3.9 - GitHub Registry ->> Tests: Pull CI Image [COMMIT_SHA] - Note over Tests: Verify CI Image - end - deactivate GitHub Registry - par 3.6, 3.7, 3.8, 3.9 - Note over Tests: Run static checks and - Note over Tests: Build docs + opt + Note over Tests: Run basic
static checks + end and - Note over Tests: Tests + opt + Note over Tests: Build docs + end and - Note over Tests: Test provider
packages build + opt + Note over Tests: Tests + end and - Note over Tests: Helm tests - end - par 3.6, 3.7, 3.8, 3.9 - Build Images ->> GitHub Registry: Push PROD Images - Note over GitHub Registry: Tagged PROD Images
[COMMIT_SHA] - activate GitHub Registry - end - deactivate Build Images - loop Wait for PROD images - par 3.6, 3.7, 3.8, 3.9 - Tests ->> Tests: Check PROD Images - Note over Tests: Wait for
[COMMIT_SHA] + opt + Note over Tests: Test provider
packages build end - end - par 3.6, 3.7, 3.8, 3.9 - GitHub Registry ->> Tests: Pull PROD Image [COMMIT_SHA] - Note over Tests: Verify PROD Image - end - deactivate GitHub Registry - par 3.6, 3.7, 3.8, 3.9 - Note over Tests: Run Kubernetes
tests and - Note over Tests: Run Kubernetes
upgrade tests + opt + Note over Tests: Helm tests + end end - Note over Tests: Merge Coverage - Tests -->> Coverage.io: Upload Coverage - par 3.6, 3.7, 3.8, 3.9 - Tests ->> GitHub Registry: Push CI Images to Cache - activate GitHub Registry + Tests ->> GitHub Registry: Push PROD Images
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull PROD Image
[COMMIT_SHA] + Note over Tests: Verify PROD Image
[COMMIT_SHA] + par + opt + Note over Tests: Run Kubernetes
tests + end and - Tests ->> GitHub Registry: Push PROD Images to Cache + opt + Note over Tests: Run Kubernetes
upgrade tests + end end - Note over GitHub Registry: Tagged Images
[latest] - deactivate GitHub Registry - par 3.6, 3.7, 3.8, 3.9 - Note over Tests: Generate constraints - Tests ->> Airflow Repo: Push constraints + Note over Tests: Generate constraints + opt In merge run? + Tests ->> Airflow Repo: Push constraints if changed + end + opt In merge run? + GitHub Registry ->> Tests: Pull CI Image
[latest] + Note over Tests: Build CI Images
[latest]
Use latest constraints + Tests ->> GitHub Registry: Push CI Image
[latest] + GitHub Registry ->> Tests: Pull PROD Image
[latest] + Note over Tests: Build PROD Images
[latest] + Tests ->> GitHub Registry: Push PROD Image
[latest] end Tests -->> Airflow Repo: Status update deactivate Airflow Repo @@ -230,19 +266,14 @@ sequenceDiagram ```mermaid sequenceDiagram Note over Airflow Repo: scheduled - Note over Tests: schedule
[Write Token] - Note over Build Images: schedule
[Write Token] + Note over Tests: push
[Write Token] activate Airflow Repo Airflow Repo -->> Tests: Trigger 'schedule' activate Tests - Airflow Repo -->> Build Images: Trigger 'schedule' - activate Build Images - Note over Build Images: Build info - par 3.6, 3.7, 3.8, 3.9 - Note over Build Images: Build CI Images
Cache disabled
[COMMIT_SHA] - end - par No CI image - Note over Tests: Build info
All tests
All python + Note over Tests: Build info
All tests
Full matrix + par + GitHub Registry ->> Tests: Pull CI Images
[latest] + Note over Tests: Build CI Images
[COMMIT_SHA]
Always upgrade deps and Note over Tests: OpenAPI client gen and @@ -250,62 +281,57 @@ sequenceDiagram and Note over Tests: Test examples
PROD image building end - par 3.6, 3.7, 3.8, 3.9 - Build Images ->> GitHub Registry: Push CI Images - activate GitHub Registry - Note over GitHub Registry: Tagged CI Images
[COMMIT_SHA] - end - par 3.6, 3.7, 3.8, 3.9 - Note over Build Images: Build PROD Images
Cache disabled
[COMMIT_SHA] - end - loop Wait for CI images - par 3.6, 3.7, 3.8, 3.9 - Tests ->> Tests: Check CI Images - Note over Tests: Wait for
[COMMIT_SHA] + Tests ->> GitHub Registry: Push CI Images
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull CI Images
[COMMIT_SHA] + Note over Tests: Verify CI Image
[COMMIT_SHA] + par + GitHub Registry ->> Tests: Pull PROD Images
[latest] + Note over Tests: Build PROD Images
[COMMIT_SHA] + and + opt + Note over Tests: Run static checks end - end - par 3.6, 3.7, 3.8, 3.9 - GitHub Registry ->> Tests: Pull CI Image [COMMIT_SHA] - Note over Tests: Verify CI Image - end - deactivate GitHub Registry - par 3.6, 3.7, 3.8, 3.9 - Note over Tests: Run static checks and - Note over Tests: Build docs + opt + Note over Tests: Run basic
static checks + end and - Note over Tests: Tests + opt + Note over Tests: Build docs + end and - Note over Tests: Test provider
packages build + opt + Note over Tests: Tests + end and - Note over Tests: Helm tests - end - par 3.6, 3.7, 3.8, 3.9 - Build Images ->> GitHub Registry: Push PROD Images - activate GitHub Registry - Note over GitHub Registry: Tagged PROD Images
[COMMIT_SHA] - end - deactivate Build Images - loop Wait for PROD images - par 3.6, 3.7, 3.8, 3.9 - Tests ->> Tests: Check PROD Images - Note over Tests: Wait for
[COMMIT_SHA] + opt + Note over Tests: Test provider
packages build end - end - par 3.6, 3.7, 3.8, 3.9 - GitHub Registry ->> Tests: Pull PROD Image [COMMIT_SHA] - Note over Tests: Verify PROD Image - end - deactivate GitHub Registry - par 3.6, 3.7, 3.8, 3.9 - Note over Tests: Run Kubernetes
tests and - Note over Tests: Run Kubernetes
upgrade tests + opt + Note over Tests: Helm tests + end end - par 3.6, 3.7, 3.8, 3.9 - Note over Tests: Generate constraints - Tests ->> Airflow Repo: Push constraints + Tests ->> GitHub Registry: Push PROD Images
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull PROD Image
[COMMIT_SHA] + Note over Tests: Verify PROD Image
[COMMIT_SHA] + par + opt + Note over Tests: Run Kubernetes
tests + end + and + opt + Note over Tests: Run Kubernetes
upgrade tests + end end + Note over Tests: Generate constraints + Tests ->> Airflow Repo: Push constraints if changed + GitHub Registry ->> Tests: Pull CI Image
[latest] + Note over Tests: Build CI Images
[latest]
Use latest constraints + Tests ->> GitHub Registry: Push CI Image
[latest] + GitHub Registry ->> Tests: Pull PROD Image
[latest] + Note over Tests: Build PROD Images
[latest] + Tests ->> GitHub Registry: Push PROD Image
[latest] Tests -->> Airflow Repo: Status update deactivate Airflow Repo deactivate Tests diff --git a/COMMITTERS.rst b/COMMITTERS.rst index e9ca3eefa37ad..054988407cb60 100644 --- a/COMMITTERS.rst +++ b/COMMITTERS.rst @@ -176,13 +176,13 @@ become active again you can simply email the PMC and ask to be reinstated. The PMC also can mark committers as inactive after they have not been involved in the community for more than 12 months. -Github configuration for committers +GitHub configuration for committers ----------------------------------- -To be able to merge PRs, committers have to integrate their Github ID with Apache systems. To do that follow steps: +To be able to merge PRs, committers have to integrate their GitHub ID with Apache systems. To do that follow steps: -1. Verify you have a Github ID `enabled with 2FA`__.
+5. After accepting the GitHub Invitation verify that you are a member of the `Airflow committers team on GitHub `__.
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 34b220405e6b8..8bab15352577a 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -66,6 +66,18 @@ implement it.
Issue reporting and resolution process
--------------------------------------
+An unusual element of the Apache Airflow project is that you can open a PR to
+fix an issue or make an enhancement, without needing to open an issue first.
+This is intended to make it as easy as possible to contribute to the project.
+
+If you however feel the need to open an issue (usually a bug or feature request)
+consider starting with a `GitHub Discussion `_ instead.
+In the vast majority of cases discussions are better than issues - you should only open
+issues if you are sure you found a bug and have a reproducible case,
+or when you want to raise a feature request that will not require a lot of discussion.
+If you have a very important topic to discuss, start a discussion on the
+`Devlist `_ instead.
+
The Apache Airflow project uses a set of labels for tracking and triaging issues, as
well as a set of priorities and milestones to track how and when the enhancements and bug
fixes make it into an Airflow release. This is documented as part of
@@ -203,27 +215,16 @@ also have support for popular remote development environments: GitHub Codespaces
You can see the differences between the various environments
`here `__.
-The local env instructions can be found in full in the `LOCAL_VIRTUALENV.rst`_ file.
-
-.. _LOCAL_VIRTUALENV.rst:
-https://github.com/apache/airflow/blob/main/LOCAL_VIRTUALENV.rst
+The local env instructions can be found in full in the `LOCAL_VIRTUALENV.rst `_ file.
The Breeze Docker Compose env is to maintain a consistent and common development environment so that you
can replicate CI failures locally and work on solving them locally rather by pushing to CI.
-The Breeze instructions can be found in full in the `BREEZE.rst`_ file.
-
-.. _BREEZE.rst:
-https://github.com/apache/airflow/blob/main/BREEZE.rst
+The Breeze instructions can be found in full in the `BREEZE.rst `_ file.
You can configure the Docker-based Breeze development environment as follows:
-1. Install the latest versions of the `Docker Community Edition`_ and `Docker Compose`_ and add them to the PATH.
-
-.. _Docker Community Edition:
-https://github.com/apache/airflow/blob/main/BREEZE.rst#docker-community-edition
-
-.. _Docker Compose: https://github.com/apache/airflow/blob/main/BREEZE.rst#docker-compose
+1. Install the latest versions of the `Docker Community Edition `_ and add them to the PATH.
2. Install `jq`_ on your machine. The exact command depends on the operating system (or Linux distribution) you use.
@@ -245,7 +246,7 @@ or on macOS with `Homebrew `_
.. code-block:: bash
- ./breeze
+ breeze
Breeze starts with downloading the Airflow CI image from
the Docker Hub and installing all required dependencies.
@@ -257,13 +258,14 @@ to make them immediately visible in the environment.
.. code-block:: bash
- mkvirtualenv myenv --python=python3.7
+ mkvirtualenv myenv --python=python3.9
5. Initialize the created environment:
.. code-block:: bash
- ./breeze initialize-local-virtualenv --python 3.7
+ ./scripts/tools/initialize_virtualenv.py
+
6. Open your IDE (for example, PyCharm) and select the virtualenv you created
as the project's default virtualenv in your IDE.
@@ -329,6 +331,24 @@ Step 4: Prepare PR
this step is automatically run while you are committing your code. If not, you can do it manually
via ``git add`` and then ``pre-commit run``.
+ * Consider adding a newsfragment to your PR so you can add an entry in the release notes.
+ The following newsfragment types are supported:
+
+ * `significant`
+ * `feature`
+ * `improvement`
+ * `bugfix`
+ * `doc`
+ * `misc`
+
+ To add a newsfragment, simply create an rst file named ``{pr_number}.{type}.rst`` (e.g. ``1234.bugfix.rst``)
+ and place in either `newsfragments `__ for core newsfragments,
+ or `chart/newsfragments `__ for helm chart newsfragments.
+
+ For significant newsfragments, similar to git commits, the first line is the summary and optionally a
+ body can be added with an empty line separating it.
+ For other newsfragment types, only use a single summary line.
+
2. Rebase your fork, squash commits, and resolve all conflicts. See `How to rebase PR <#how-to-rebase-pr>`_
if you need help with rebasing your change. Remember to rebase often if your PR takes a lot of time to
review/fix. This will make rebase process much easier and less painful and the more often you do it,
@@ -472,10 +492,10 @@ Development Environments
There are two environments, available on Linux and macOS, that you can use to
develop Apache Airflow:
-- `Local virtualenv development environment <#local-virtualenv-development-environment>`_
+- `Local virtualenv development environment `_
that supports running unit tests and can be used in your IDE.
-- `Breeze Docker-based development environment <#breeze-development-environment>`_ that provides
+- `Breeze Docker-based development environment `_ that provides
an end-to-end CI solution with all software dependencies covered.
The table below summarizes differences between the environments:
@@ -533,7 +553,7 @@ Limitations:
real unit tests. Technically, to run integration tests, you can configure
and install the dependencies on your own, but it is usually complex.
Instead, you are recommended to use
- `Breeze development environment <#breeze-development-environment>`__ with all required packages
+ `Breeze development environment `__ with all required packages
pre-installed.
- You need to make sure that your local environment is consistent with other
@@ -554,6 +574,13 @@ All details about using and running Airflow Breeze can be found in
The Airflow Breeze solution is intended to ease your local development as "*It's
a Breeze to develop Airflow*".
+.. note::
+
+ We are in a process of switching to the new Python-based Breeze from a legacy Bash
+ Breeze. Not all functionality has been ported yet and the old Breeze is still available
+ until then as ``./breeze-legacy`` script. The documentation mentions when the old ./breeze-legacy
+ should be still used.
+
Benefits:
- Breeze is a complete environment that includes external components, such as
@@ -581,9 +608,14 @@ Limitations:
disk space and CPU. You can stop the environment manually after you use it
or even use a ``bare`` environment to decrease resource usage.
-**NOTE:** Breeze CI images are not supposed to be used in production environments.
-They are optimized for repeatability of tests, maintainability and speed of building rather
-than production performance. The production images are not yet officially published.
+
+
+.. note::
+
+ Breeze CI images are not supposed to be used in production environments.
+ They are optimized for repeatability of tests, maintainability and speed of building rather
+ than production performance. The production images are not yet officially published.
+
Airflow dependencies
@@ -614,20 +646,18 @@ all dependencies needed in the CI environment.
This is the full list of those extras:
.. START EXTRAS HERE
-
airbyte, alibaba, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra, apache.drill,
apache.druid, apache.hdfs, apache.hive, apache.kylin, apache.livy, apache.pig, apache.pinot,
-apache.spark, apache.sqoop, apache.webhdfs, asana, async, atlas, aws, azure, cassandra, celery,
-cgroups, cloudant, cncf.kubernetes, crypto, dask, databricks, datadog, dbt.cloud, deprecated_api,
-devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc, docker, druid, elasticsearch,
-exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google, google_auth, grpc,
-hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, jira, kerberos, kubernetes, ldap,
+apache.spark, apache.sqoop, apache.webhdfs, arangodb, asana, async, atlas, aws, azure, cassandra,
+celery, cgroups, cloudant, cncf.kubernetes, crypto, dask, databricks, datadog, dbt.cloud,
+deprecated_api, devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc, docker, druid,
+elasticsearch, exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google, google_auth,
+grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, jira, kerberos, kubernetes, ldap,
leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo, mssql, mysql,
neo4j, odbc, openfaas, opsgenie, oracle, pagerduty, pandas, papermill, password, pinot, plexus,
postgres, presto, qds, qubole, rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentry,
sftp, singularity, slack, snowflake, spark, sqlite, ssh, statsd, tableau, telegram, trino, vertica,
virtualenv, webhdfs, winrm, yandex, zendesk
-
.. END EXTRAS HERE
Provider packages
@@ -644,7 +674,7 @@ and not packaged together with the core, unless you set ``INSTALL_PROVIDERS_FROM
variable to ``true``.
In Breeze - which is a development environment, ``INSTALL_PROVIDERS_FROM_SOURCES`` variable is set to true,
-but you can add ``--skip-installing-airflow-providers-from-sources`` flag to Breeze to skip installing providers when
+but you can add ``--install-providers-from-sources=false`` flag to Breeze to install providers from PyPI instead of source files when
building the images.
One watch-out - providers are still always installed (or rather available) if you install airflow from
@@ -853,47 +883,42 @@ There are several sets of constraints we keep:
providers. If you want to manage airflow separately and then add providers individually, you can
use those. Those constraints are named ``constraints-no-providers-.txt``.
-We also have constraints with "source-providers" but they are used i
-
-The first ones can be used as constraints file when installing Apache Airflow in a repeatable way.
+The first two can be used as constraints file when installing Apache Airflow in a repeatable way.
It can be done from the sources:
+from the PyPI package:
+
.. code-block:: bash
- pip install -e . \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.7.txt"
+ pip install apache-airflow[google,amazon,async]==2.2.5 \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.5/constraints-3.7.txt"
+The last one can be used to install Airflow in "minimal" mode - i.e when bare Airflow is installed without
+extras.
-or from the PyPI package:
+When you install airflow from sources (in editable mode) you should use "constraints-source-providers"
+instead (this accounts for the case when some providers have not yet been released and have conflicting
+requirements).
.. code-block:: bash
- pip install apache-airflow \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.7.txt"
+ pip install -e . \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
This works also with extras - for example:
.. code-block:: bash
- pip install .[ssh] \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.7.txt"
+ pip install ".[ssh]" \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
-As of apache-airflow 1.10.12 it is also possible to use constraints directly from GitHub using specific
-tag/hash name. We tag commits working for particular release with constraints- tag. So for example
-fixed valid constraints 1.10.12 can be used by using ``constraints-1.10.12`` tag:
-
-.. code-block:: bash
-
- pip install apache-airflow[ssh]==1.10.12 \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt"
-
There are different set of fixed constraint files for different python major/minor versions and you should
use the right file for the right python version.
If you want to update just airflow dependencies, without paying attention to providers, you can do it using
--no-providers constraint files as well.
+``constraints-no-providers`` constraint files as well.
.. code-block:: bash
@@ -909,7 +934,9 @@ if the tests are successful.
Documentation
=============
-Documentation for ``apache-airflow`` package and other packages that are closely related to it ie. providers packages are in ``/docs/`` directory. For detailed information on documentation development, see: `docs/README.rst `_
+Documentation for ``apache-airflow`` package and other packages that are closely related to it ie.
+providers packages are in ``/docs/`` directory. For detailed information on documentation development,
+see: `docs/README.rst `_
Static code checks
==================
@@ -1140,8 +1167,20 @@ development machine before continuing with migration.
$ cd airflow
$ alembic revision -m "add new field to db"
Generating
- ~/airflow/airflow/migrations/versions/12341123_add_new_field_to_db.py
+ ~/airflow/airflow/migrations/versions/a1e23c41f123_add_new_field_to_db.py
+
+Note that migration file names are standardized by pre-commit hook ``update-migration-references``, so that they sort alphabetically and indicate
+the Airflow version in which they first appear (the alembic revision ID is removed). As a result you should expect to see a pre-commit failure
+on the first attempt. Just stage the modified file and commit again
+(or run the hook manually before committing).
+
+After your new migration file is run through pre-commit it will look like this:
+
+.. code-block::
+
+ 1234_A_B_C_add_new_field_to_db.py
+This represents that your migration is the 1234th migration and expected for release in Airflow version A.B.C.
Node.js Environment Setup
=========================
@@ -1214,6 +1253,7 @@ commands:
yarn run prod
# Starts a web server that manages and updates your assets as you modify them
+ # You'll need to run the webserver in debug mode too: `airflow webserver -d`
yarn run dev
diff --git a/CONTRIBUTORS_QUICK_START.rst b/CONTRIBUTORS_QUICK_START.rst
index 2b9d6367c2e32..10ff2cc9516d9 100644
--- a/CONTRIBUTORS_QUICK_START.rst
+++ b/CONTRIBUTORS_QUICK_START.rst
@@ -16,7 +16,7 @@
under the License.
*************************
-Contributor's Quick Guide
+Contributor's Quick Start
*************************
.. contents:: :local:
@@ -24,37 +24,41 @@ Contributor's Quick Guide
Note to Starters
################
-There are two ways you can run the Airflow dev env on your machine:
- 1. With a Docker Container
- 2. With a local virtual environment
-Before deciding which method to choose, there are a couple factors to consider:
-Running Airflow in a container is the most reliable way: it provides a more consistent environment and allows integration tests with a number of integrations (cassandra, mongo, mysql, etc.). However it also requires **4GB RAM, 40GB disk space and at least 2 cores**.
-If you are working on a basic feature, installing Airflow on a local environment might be sufficient.
+Airflow is quite a complex project, and setting up a working environment, but we made it rather simple if
+you follow the guide.
+
+There are three ways you can run the Airflow dev env:
-- |Virtual Env Guide|
+1. With a Docker Containers and Docker Compose (on your local machine). This environment is managed
+ with `Breeze `_ tool written in Python that makes the environment management, yeah you
+ guessed it - a breeze.
+2. With a local virtual environment (on your local machine).
+3. With a remote, managed environment (via remote development environment)
+
+Before deciding which method to choose, there are a couple factors to consider:
-.. |Virtual Env Guide| raw:: html
+* Running Airflow in a container is the most reliable way: it provides a more consistent environment
+ and allows integration tests with a number of integrations (cassandra, mongo, mysql, etc.).
+ However it also requires **4GB RAM, 40GB disk space and at least 2 cores**.
+* If you are working on a basic feature, installing Airflow on a local environment might be sufficient.
+ For a comprehensive venv tutorial - visit
+ `Virtual Env guide `_
+* You need to have usually a paid account to access managed, remote virtual environment.
- For a comprehensive venv tutorial - visit Virtual Env Guide
+Local machine development
+#########################
-Prerequisites
-#############
+If you do not work with remote development environment, you need those prerequisites.
1. Docker Community Edition
2. Docker Compose
3. pyenv (you can also use pyenv-virtualenv or virtualenvwrapper)
-4. jq
-
-
-Installing Prerequisites on Ubuntu
-##################################
+The below setup describe Ubuntu installation. It might be slightly different on different machines.
Docker Community Edition
------------------------
-
1. Installing required packages for Docker and setting up docker repo
.. code-block:: bash
@@ -97,9 +101,6 @@ Note : After adding user to docker group Logout and Login again for group member
$ docker run hello-world
-
-
-
Docker Compose
--------------
@@ -123,10 +124,9 @@ Docker Compose
$ docker-compose --version
-
-
Pyenv and setting up virtual-env
--------------------------------
+ Note: You might have issues with pyenv if you have a Mac with an M1 chip. Consider using virtualenv as an alternative.
1. Install pyenv and configure your shell's environment for Pyenv as suggested in Pyenv `README `_
@@ -165,50 +165,8 @@ Pyenv and setting up virtual-env
$ pyenv activate airflow-env
-
-Installing jq
---------------------------------
-
-``jq`` is a lightweight and flexible command-line JSON processor.
-
-Install ``jq`` with the following command:
-
-.. code-block:: bash
-
- $ sudo apt install jq
-
-
-
-Setup and develop using PyCharm
-###############################
-
-.. raw:: html
-
-
-2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+2. Goto your github account's fork of airflow click on ``Code`` you will find the link to your repo.
.. raw:: html
@@ -232,49 +190,41 @@ Forking and cloning Project
alt="Cloning github fork of Apache airflow">
@@ -855,6 +1224,17 @@ More details on using the local virtualenv are available in the `LOCAL_VIRTUALEN
+Running docker-compose tests
+----------------------------
+
+You can use Breeze to run docker-compose tests. Those tests are run using Production image
+and they are running test with the Quick-start docker compose we have.
+
+.. image:: ./images/breeze/output-docker-compose-tests.svg
+ :width: 100%
+ :alt: Breeze generate-constraints
+
+
Running Kubernetes tests
------------------------
@@ -864,6 +1244,9 @@ automatically to run the tests.
This is described in detail in `Testing Kubernetes
@@ -881,8 +1264,16 @@ You can always stop it via:
.. code-block:: bash
- ./breeze stop
+ breeze stop
+Those are all available flags of ``stop`` command:
+
+.. image:: ./images/breeze/output-stop.svg
+ :width: 100%
+ :alt: Breeze stop
+
+Here is the part of Breeze video which is relevant (note that it refers to the old ``./breeze-legacy``
+command but it is very similar to current ``breeze`` command):
.. raw:: html
@@ -893,6 +1284,49 @@ You can always stop it via:
+Resource check
+==============
+
+Breeze requires certain resources to be available - disk, memory, CPU. When you enter Breeze's shell,
+the resources are checked and information if there is enough resources is displayed. However you can
+manually run resource check any time by ``breeze resource-check`` command.
+
+Those are all available flags of ``resource-check`` command:
+
+.. image:: ./images/breeze/output-resource-check.svg
+ :width: 100%
+ :alt: Breeze resource-check
+
+
+Freeing the space
+=================
+
+When our CI runs a job, it needs all memory and disk it can have. We have a Breeze command that frees
+the memory and disk space used. You can also use it clear space locally but it performs a few operations
+that might be a bit invasive - such are removing swap file and complete pruning of docker disk space used.
+
+Those are all available flags of ``free-space`` command:
+
+.. image:: ./images/breeze/output-free-space.svg
+ :width: 100%
+ :alt: Breeze free-space
+
+
+Tracking backtracking issues for CI builds
+==========================================
+
+When our CI runs a job, we automatically upgrade our dependencies in the ``main`` build. However, this might
+lead to conflicts and ``pip`` backtracking for a long time (possibly forever) for dependency resolution.
+Unfortunately those issues are difficult to diagnose so we had to invent our own tool to help us with
+diagnosing them. This tool is ``find-newer-dependencies`` and it works in the way that it helps to guess
+which new dependency might have caused the backtracking. The whole process is described in
+`tracking backtracking issues [COMMIT_SHA] - end - par No CI image - Note over Tests: Build info
Which tests?
Which Python? + Note over Build Images: Build info
Decide which Python + Note over Tests: Build info
Decide on tests
Decide on Matrix (selective) + Note over Tests: Skip Build
(Runs in 'Build Images')
CI Images + Note over Tests: Skip Build
(Runs in 'Build Images')
PROD Images + par + GitHub Registry ->> Build Images: Pull CI Images
[latest] + Note over Build Images: Build CI Images
[COMMIT_SHA]
Use latest constraints
or upgrade if setup changed and Note over Tests: OpenAPI client gen and @@ -49,77 +47,135 @@ sequenceDiagram and Note over Tests: Test examples
PROD image building end - par 3.6, [3.7, 3.8, 3.9] - activate GitHub Registry - Build Images ->> GitHub Registry: Push CI Images - Note over GitHub Registry: Tagged CI Images
[COMMIT_SHA] - end - par 3.6, [3.7, 3.8, 3.9] - GitHub Registry ->> Build Images: Pull PROD Images from Cache - Note over Build Images: Build PROD Images
[COMMIT_SHA] - end + Build Images ->> GitHub Registry: Push CI Images
[COMMIT_SHA] loop Wait for CI images - par 3.6, [3.7, 3.8, 3.9] - Tests ->> Tests: Check CI Images - Note over Tests: Wait for
[COMMIT_SHA] - end + GitHub Registry ->> Tests: Pull CI Images
[COMMIT_SHA] end - par 3.6, [3.7, 3.8, 3.9] - GitHub Registry ->> Tests: Pull CI Image - Note over Tests: Verify CI Image - end - deactivate GitHub Registry - par 3.6, [3.7, 3.8, 3.9] - opt Needed? + Note over Tests: Verify CI Images
[COMMIT_SHA] + par + GitHub Registry ->> Build Images: Pull PROD Images
[latest] + Note over Build Images: Build PROD Images
[COMMIT_SHA] + and + opt Note over Tests: Run static checks end and - opt Needed? + opt Note over Tests: Run basic
static checks end and - opt Needed? + opt Note over Tests: Build docs end and - opt Needed? + opt Note over Tests: Tests end and - opt Needed? + opt Note over Tests: Test provider
packages build end and - opt Needed? + opt Note over Tests: Helm tests end end - par 3.6, [3.7, 3.8, 3.9] - Build Images ->> GitHub Registry: Push PROD Images - activate GitHub Registry - end + Build Images ->> GitHub Registry: Push PROD Images
[COMMIT_SHA] deactivate Build Images - Note over GitHub Registry: Tagged PROD Images
[COMMIT_SHA] loop Wait for PROD images - par 3.6, [3.7, 3.8, 3.9] - Tests ->> Tests: Check PROD Images - Note over Tests: Wait for
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull PROD Images
[COMMIT_SHA] + end + Note over Tests: Verify PROD Image
[COMMIT_SHA] + par + opt + Note over Tests: Run Kubernetes
tests + end + and + opt + Note over Tests: Run Kubernetes
upgrade tests end end - par 3.6, [3.7, 3.8, 3.9] - GitHub Registry ->> Tests: Pull PROD Image - Note over Tests: Verify PROD Image + opt + Note over Tests: Generate constraints end - deactivate GitHub Registry - par 3.6, [3.7, 3.8, 3.9] - opt Needed? + Tests -->> Airflow Repo: Status update + deactivate Airflow Repo + deactivate Tests +``` + +## Pull request flow from "apache/airflow" repo + +```mermaid +sequenceDiagram + Note over Airflow Repo: pull request + Note over Tests: pull_request
[Read Token] + Note over Build Images: pull_request_target
[Write Token] + activate Airflow Repo + Airflow Repo -->> Tests: Trigger 'pull_request' + activate Tests + Tests -->> Build Images: Trigger 'pull_request_target' + activate Build Images + Note over Build Images: Build info + Note over Build Images: Skip Build
(Runs in 'Tests')
CI Images + Note over Build Images: Skip Build
(Runs in 'Tests')
PROD Images + deactivate Build Images + Note over Tests: Build info
Decide on tests
Decide on Matrix (selective) + par + GitHub Registry ->> Tests: Pull CI Images
[latest] + Note over Tests: Build CI Images
[COMMIT_SHA]
Use latest constraints
or upgrade if setup changed + and + Note over Tests: OpenAPI client gen + and + Note over Tests: Test UI + and + Note over Tests: Test examples
PROD image building + end + Tests ->> GitHub Registry: Push CI Images
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull CI Images
[COMMIT_SHA] + Note over Tests: Verify CI Image
[COMMIT_SHA] + par + GitHub Registry ->> Tests: Pull PROD Images
[latest] + Note over Tests: Build PROD Images
[COMMIT_SHA] + and + opt + Note over Tests: Run static checks + end + and + opt + Note over Tests: Run basic
static checks + end + and + opt + Note over Tests: Build docs + end + and + opt + Note over Tests: Tests + end + and + opt + Note over Tests: Test provider
packages build + end + and + opt + Note over Tests: Helm tests + end + end + Tests ->> GitHub Registry: Push PROD Images
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull PROD Image
[COMIT_SHA] + Note over Tests: Verify PROD Image
[COMMIT_SHA] + par + opt Note over Tests: Run Kubernetes
tests end and - opt Needed? + opt Note over Tests: Run Kubernetes
upgrade tests end end + opt + Note over Tests: Generate constraints + end Tests -->> Airflow Repo: Status update deactivate Airflow Repo deactivate Tests @@ -129,23 +185,15 @@ sequenceDiagram ```mermaid sequenceDiagram - Note over Airflow Repo: merge + Note over Airflow Repo: pull request Note over Tests: push
[Write Token] - Note over Build Images: push
[Write Token] activate Airflow Repo Airflow Repo -->> Tests: Trigger 'push' activate Tests - Airflow Repo -->> Build Images: Trigger 'push' - activate Build Images - Note over Build Images: Build info - par 3.6, 3.7, 3.8, 3.9 - activate GitHub Registry - GitHub Registry ->> Build Images: Pull CI Images from Cache - deactivate GitHub Registry - Note over Build Images: Build CI Images
[COMMIT_SHA] - end - par No CI image - Note over Tests: Build info
All tests
All python + Note over Tests: Build info
All tests
Full matrix + par + GitHub Registry ->> Tests: Pull CI Images
[latest] + Note over Tests: Build CI Images
[COMMIT_SHA]
Always upgrade deps and Note over Tests: OpenAPI client gen and @@ -153,72 +201,60 @@ sequenceDiagram and Note over Tests: Test examples
PROD image building end - par 3.6, 3.7, 3.8, 3.9 - Build Images ->> GitHub Registry: Push CI Images - activate GitHub Registry - Note over GitHub Registry: Tagged CI Images
[COMMIT_SHA] - end - par 3.6, 3.7, 3.8, 3.9 - GitHub Registry ->> Build Images: Pull PROD Images from Cache - Note over Build Images: Build PROD Images
[COMMIT_SHA] - end - loop Wait for CI images - par 3.6, 3.7, 3.8, 3.9 - Tests ->> Tests: Check CI Images - Note over Tests: Wait for
[COMMIT_SHA] + Tests ->> GitHub Registry: Push CI Images
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull CI Images
[COMMIT_SHA] + Note over Tests: Verify CI Image
[COMMIT_SHA] + par + GitHub Registry ->> Tests: Pull PROD Images
[latest] + Note over Tests: Build PROD Images
[COMMIT_SHA] + and + opt + Note over Tests: Run static checks end - end - par 3.6, 3.7, 3.8, 3.9 - GitHub Registry ->> Tests: Pull CI Image [COMMIT_SHA] - Note over Tests: Verify CI Image - end - deactivate GitHub Registry - par 3.6, 3.7, 3.8, 3.9 - Note over Tests: Run static checks and - Note over Tests: Build docs + opt + Note over Tests: Run basic
static checks + end and - Note over Tests: Tests + opt + Note over Tests: Build docs + end and - Note over Tests: Test provider
packages build + opt + Note over Tests: Tests + end and - Note over Tests: Helm tests - end - par 3.6, 3.7, 3.8, 3.9 - Build Images ->> GitHub Registry: Push PROD Images - Note over GitHub Registry: Tagged PROD Images
[COMMIT_SHA] - activate GitHub Registry - end - deactivate Build Images - loop Wait for PROD images - par 3.6, 3.7, 3.8, 3.9 - Tests ->> Tests: Check PROD Images - Note over Tests: Wait for
[COMMIT_SHA] + opt + Note over Tests: Test provider
packages build end - end - par 3.6, 3.7, 3.8, 3.9 - GitHub Registry ->> Tests: Pull PROD Image [COMMIT_SHA] - Note over Tests: Verify PROD Image - end - deactivate GitHub Registry - par 3.6, 3.7, 3.8, 3.9 - Note over Tests: Run Kubernetes
tests and - Note over Tests: Run Kubernetes
upgrade tests + opt + Note over Tests: Helm tests + end end - Note over Tests: Merge Coverage - Tests -->> Coverage.io: Upload Coverage - par 3.6, 3.7, 3.8, 3.9 - Tests ->> GitHub Registry: Push CI Images to Cache - activate GitHub Registry + Tests ->> GitHub Registry: Push PROD Images
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull PROD Image
[COMMIT_SHA] + Note over Tests: Verify PROD Image
[COMMIT_SHA] + par + opt + Note over Tests: Run Kubernetes
tests + end and - Tests ->> GitHub Registry: Push PROD Images to Cache + opt + Note over Tests: Run Kubernetes
upgrade tests + end end - Note over GitHub Registry: Tagged Images
[latest] - deactivate GitHub Registry - par 3.6, 3.7, 3.8, 3.9 - Note over Tests: Generate constraints - Tests ->> Airflow Repo: Push constraints + Note over Tests: Generate constraints + opt In merge run? + Tests ->> Airflow Repo: Push constraints if changed + end + opt In merge run? + GitHub Registry ->> Tests: Pull CI Image
[latest] + Note over Tests: Build CI Images
[latest]
Use latest constraints + Tests ->> GitHub Registry: Push CI Image
[latest] + GitHub Registry ->> Tests: Pull PROD Image
[latest] + Note over Tests: Build PROD Images
[latest] + Tests ->> GitHub Registry: Push PROD Image
[latest] end Tests -->> Airflow Repo: Status update deactivate Airflow Repo @@ -230,19 +266,14 @@ sequenceDiagram ```mermaid sequenceDiagram Note over Airflow Repo: scheduled - Note over Tests: schedule
[Write Token] - Note over Build Images: schedule
[Write Token] + Note over Tests: push
[Write Token] activate Airflow Repo Airflow Repo -->> Tests: Trigger 'schedule' activate Tests - Airflow Repo -->> Build Images: Trigger 'schedule' - activate Build Images - Note over Build Images: Build info - par 3.6, 3.7, 3.8, 3.9 - Note over Build Images: Build CI Images
Cache disabled
[COMMIT_SHA] - end - par No CI image - Note over Tests: Build info
All tests
All python + Note over Tests: Build info
All tests
Full matrix + par + GitHub Registry ->> Tests: Pull CI Images
[latest] + Note over Tests: Build CI Images
[COMMIT_SHA]
Always upgrade deps and Note over Tests: OpenAPI client gen and @@ -250,62 +281,57 @@ sequenceDiagram and Note over Tests: Test examples
PROD image building end - par 3.6, 3.7, 3.8, 3.9 - Build Images ->> GitHub Registry: Push CI Images - activate GitHub Registry - Note over GitHub Registry: Tagged CI Images
[COMMIT_SHA] - end - par 3.6, 3.7, 3.8, 3.9 - Note over Build Images: Build PROD Images
Cache disabled
[COMMIT_SHA] - end - loop Wait for CI images - par 3.6, 3.7, 3.8, 3.9 - Tests ->> Tests: Check CI Images - Note over Tests: Wait for
[COMMIT_SHA] + Tests ->> GitHub Registry: Push CI Images
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull CI Images
[COMMIT_SHA] + Note over Tests: Verify CI Image
[COMMIT_SHA] + par + GitHub Registry ->> Tests: Pull PROD Images
[latest] + Note over Tests: Build PROD Images
[COMMIT_SHA] + and + opt + Note over Tests: Run static checks end - end - par 3.6, 3.7, 3.8, 3.9 - GitHub Registry ->> Tests: Pull CI Image [COMMIT_SHA] - Note over Tests: Verify CI Image - end - deactivate GitHub Registry - par 3.6, 3.7, 3.8, 3.9 - Note over Tests: Run static checks and - Note over Tests: Build docs + opt + Note over Tests: Run basic
static checks + end and - Note over Tests: Tests + opt + Note over Tests: Build docs + end and - Note over Tests: Test provider
packages build + opt + Note over Tests: Tests + end and - Note over Tests: Helm tests - end - par 3.6, 3.7, 3.8, 3.9 - Build Images ->> GitHub Registry: Push PROD Images - activate GitHub Registry - Note over GitHub Registry: Tagged PROD Images
[COMMIT_SHA] - end - deactivate Build Images - loop Wait for PROD images - par 3.6, 3.7, 3.8, 3.9 - Tests ->> Tests: Check PROD Images - Note over Tests: Wait for
[COMMIT_SHA] + opt + Note over Tests: Test provider
packages build end - end - par 3.6, 3.7, 3.8, 3.9 - GitHub Registry ->> Tests: Pull PROD Image [COMMIT_SHA] - Note over Tests: Verify PROD Image - end - deactivate GitHub Registry - par 3.6, 3.7, 3.8, 3.9 - Note over Tests: Run Kubernetes
tests and - Note over Tests: Run Kubernetes
upgrade tests + opt + Note over Tests: Helm tests + end end - par 3.6, 3.7, 3.8, 3.9 - Note over Tests: Generate constraints - Tests ->> Airflow Repo: Push constraints + Tests ->> GitHub Registry: Push PROD Images
[COMMIT_SHA] + GitHub Registry ->> Tests: Pull PROD Image
[COMMIT_SHA] + Note over Tests: Verify PROD Image
[COMMIT_SHA] + par + opt + Note over Tests: Run Kubernetes
tests + end + and + opt + Note over Tests: Run Kubernetes
upgrade tests + end end + Note over Tests: Generate constraints + Tests ->> Airflow Repo: Push constraints if changed + GitHub Registry ->> Tests: Pull CI Image
[latest] + Note over Tests: Build CI Images
[latest]
Use latest constraints + Tests ->> GitHub Registry: Push CI Image
[latest] + GitHub Registry ->> Tests: Pull PROD Image
[latest] + Note over Tests: Build PROD Images
[latest] + Tests ->> GitHub Registry: Push PROD Image
[latest] Tests -->> Airflow Repo: Status update deactivate Airflow Repo deactivate Tests diff --git a/COMMITTERS.rst b/COMMITTERS.rst index e9ca3eefa37ad..054988407cb60 100644 --- a/COMMITTERS.rst +++ b/COMMITTERS.rst @@ -176,13 +176,13 @@ become active again you can simply email the PMC and ask to be reinstated. The PMC also can mark committers as inactive after they have not been involved in the community for more than 12 months. -Github configuration for committers +GitHub configuration for committers ----------------------------------- -To be able to merge PRs, committers have to integrate their Github ID with Apache systems. To do that follow steps: +To be able to merge PRs, committers have to integrate their GitHub ID with Apache systems. To do that follow steps: -1. Verify you have a Github ID `enabled with 2FA
-
Setup and develop using PyCharm
- - - -Setup Airflow with Breeze -------------------------- - - - -.. note:: - - Only ``pip`` installation is currently officially supported. - - While they are some successes with using other tools like `poetry
- 
-
-
-
-4. Paste the copied clone link in the URL field and submit.
-
- .. raw:: html
-
-
-
- 
-
-
+For many of the development tasks you will need ``Breeze`` to be configured. ``Breeze`` is a development
+environment which uses docker and docker-compose and it's main purpose is to provide a consistent
+and repeatable environment for all the contributors and CI. When using ``Breeze`` you avoid the "works for me"
+syndrome - because not only others can reproduce easily what you do, but also the CI of Airflow uses
+the same environment to run all tests - so you should be able to easily reproduce the same failures you
+see in CI in your local environment.
Setting up Breeze
-~~~~~~~~~~~~~~~~~
-1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
+-----------------
-.. code-block:: bash
+1. Install ``pipx`` - follow the instructions in `Install pipx
-
@@ -483,19 +419,11 @@ Following are some of important topics of Breeze documentation:
Additional tools to the Docker Image
-- |Internal details of Breeze|
-
-.. |Internal details of Breeze| raw:: html
-
-
- Internal details of Breeze
-
-
- |Breeze Command-Line Interface Reference|
.. |Breeze Command-Line Interface Reference| raw:: html
- Breeze Command-Line Interface Reference
@@ -507,113 +435,154 @@ Following are some of important topics of Breeze documentation:
Cleaning the environment
-- |Other uses of the Airflow Breeze environment|
-.. |Other uses of the Airflow Breeze environment| raw:: html
+Configuring Pre-commit
+----------------------
+
+Before committing changes to github or raising a pull request, code needs to be checked for certain quality standards
+such as spell check, code syntax, code formatting, compatibility with Apache License requirements etc. This set of
+tests are applied when you commit your code.
- Other uses of the Airflow Breeze environment
+.. raw:: html
+
+ 
+
-Setting up Debug
-~~~~~~~~~~~~~~~~
+To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be run locally before committing changes.
-1. Configuring Airflow database connection
+1. Installing required packages
-- Airflow is by default configured to use SQLite database. Configuration can be seen on local machine
- ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
+.. code-block:: bash
-- Installing required dependency for MySQL connection in ``airflow-env`` on local machine.
+ $ sudo apt install libxml2-utils
- .. code-block:: bash
+2. Installing required Python packages
- $ pyenv activate airflow-env
- $ pip install PyMySQL
+.. code-block:: bash
-- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
- ``~/airflow/airflow.cfg`` on local machine.
+ $ pyenv activate airflow-env
+ $ pip install pre-commit
-1. Debugging an example DAG
+3. Go to your project directory
-- Add Interpreter to PyCharm pointing interpreter path to ``~/.pyenv/versions/airflow-env/bin/python``, which is virtual
- environment ``airflow-env`` created with pyenv earlier. For adding an Interpreter go to ``File -> Setting -> Project:
- airflow -> Python Interpreter``.
+.. code-block:: bash
- .. raw:: html
+ $ cd ~/Projects/airflow
-
+
- 
-
-- In PyCharm IDE open airflow project, directory ``/files/dags`` of local machine is by default mounted to docker
- machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
- scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
+1. Running pre-commit hooks
+
+.. code-block:: bash
+
+ $ pre-commit run --all-files
+ No-tabs checker......................................................Passed
+ Add license for all SQL files........................................Passed
+ Add license for all other files......................................Passed
+ Add license for all rst files........................................Passed
+ Add license for all JS/CSS/PUML files................................Passed
+ Add license for all JINJA template files.............................Passed
+ Add license for all shell files......................................Passed
+ Add license for all python files.....................................Passed
+ Add license for all XML files........................................Passed
+ Add license for all yaml files.......................................Passed
+ Add license for all md files.........................................Passed
+ Add license for all mermaid files....................................Passed
+ Add TOC for md files.................................................Passed
+ Add TOC for upgrade documentation....................................Passed
+ Check hooks apply to the repository..................................Passed
+ black................................................................Passed
+ Check for merge conflicts............................................Passed
+ Debug Statements (Python)............................................Passed
+ Check builtin type constructor use...................................Passed
+ Detect Private Key...................................................Passed
+ Fix End of Files.....................................................Passed
+ ...........................................................................
+
+5. Running pre-commit for selected files
-- Copy any example DAG present in the ``/airflow/example_dags`` directory to ``/files/dags/``.
+.. code-block:: bash
-- Add a ``__main__`` block at the end of your DAG file to make it runnable. It will run a ``back_fill`` job:
+ $ pre-commit run --files airflow/decorators.py tests/utils/test_task_group.py
- .. code-block:: python
- if __name__ == "__main__":
- dag.clear()
- dag.run()
-- Add ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` to Environment variable of Run Configuration.
+6. Running specific hook for selected files
- - Click on Add configuration
+.. code-block:: bash
- .. raw:: html
+ $ pre-commit run black --files airflow/decorators.py tests/utils/test_task_group.py
+ black...............................................................Passed
+ $ pre-commit run flake8 --files airflow/decorators.py tests/utils/test_task_group.py
+ Run flake8..........................................................Passed
-
-
- 
-
- - Add Script Path and Environment Variable to new Python configuration
- .. raw:: html
+7. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
-
-
- 
-
+.. code-block:: bash
-- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in MySQL Workbench.
+ $ cd ~/Projects/airflow
+ $ pre-commit install
+ $ git commit -m "Added xyz"
+8. To disable Pre-commit
+.. code-block:: bash
-Starting development
---------------------
+ $ cd ~/Projects/airflow
+ $ pre-commit uninstall
-Creating a branch
-~~~~~~~~~~~~~~~~~
+- For more information on visit |STATIC_CODE_CHECKS.rst|
-1. Click on the branch symbol in the status bar
+.. |STATIC_CODE_CHECKS.rst| raw:: html
- .. raw:: html
+
+ STATIC_CODE_CHECKS.rst
-
-
- 
-
+- Following are some of the important links of STATIC_CODE_CHECKS.rst
-2. Give a name to a branch and checkout
+ - |Pre-commit Hooks|
- .. raw:: html
+ .. |Pre-commit Hooks| raw:: html
-
-
- 
-
+
+ Pre-commit Hooks
+
+ - |Running Static Code Checks via Breeze|
+
+ .. |Running Static Code Checks via Breeze| raw:: html
+
+ Running Static Code Checks via Breeze
+
+
+Installing airflow in the local venv
+------------------------------------
+
+1. It may require some packages to be installed; watch the output of the command to see which ones are missing.
+
+.. code-block:: bash
+
+ $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
+ $ ./scripts/tools/initialize_virtualenv.py
+
+
+2. Add following line to ~/.bashrc in order to call breeze command from anywhere.
+
+.. code-block:: bash
+ export PATH=${PATH}:"/home/${USER}/Projects/airflow"
+ source ~/.bashrc
+Running tests with Breeze
+-------------------------
-Testing
-~~~~~~~
+You can usually conveniently run tests in your IDE (see IDE below) using virtualenv but with Breeze you
+can be sure that all the tests are run in the same environment as tests in CI.
All Tests are inside ./tests directory.
@@ -623,18 +592,21 @@ All Tests are inside ./tests directory.
.. code-block:: bash
- root@51d89409f7a2:/opt/airflow# pytest tests/utils/test_trigger_rule.py
- ================================================ test session starts ================================================
- platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/bin/python
- cachedir: .pytest_cache
- rootdir: /opt/airflow, configfile: pytest.ini
- plugins: forked-1.4.0, rerunfailures-9.1.1, requests-mock-1.9.3, asyncio-0.18.1, cov-3.0.0, httpx-0.20.0, xdist-2.5.0, flaky-3.7.0, timeouts-1.2.1, anyio-3.5.0, instafail-0.4.2
- asyncio: mode=strict
- setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
- collected 1 item
+ root@63528318c8b1:/opt/airflow# pytest tests/utils/test_decorators.py
+ ======================================= test session starts =======================================
+ platform linux -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
+ cachedir: .pytest_cache
+ rootdir: /opt/airflow, configfile: pytest.ini
+ plugins: celery-4.4.7, requests-mock-1.8.0, xdist-1.34.0, flaky-3.7.0, rerunfailures-9.0, instafail
+ -0.4.2, forked-1.3.0, timeouts-1.2.1, cov-2.10.0
+ setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
+ collected 3 items
+
+ tests/utils/test_decorators.py::TestApplyDefault::test_apply PASSED [ 33%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_default_args PASSED [ 66%]
+ tests/utils/test_decorators.py::TestApplyDefault::test_incorrect_default_args PASSED [100%]
- tests/utils/test_trigger_rule.py::TestTriggerRule::test_valid_trigger_rules PASSED [100%]
- =========================================== 1 passed, 1 warning in 0.66s ============================================
+ ======================================== 3 passed in 1.49s ========================================
- Running All the test with Breeze by specifying required python version, backend, backend version
@@ -643,26 +615,6 @@ All Tests are inside ./tests directory.
$ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
-- Running specific test in container using shell scripts. Testing in container scripts are located in
- ``./scripts/in_container`` directory.
-
-.. code-block:: bash
-
- root@df8927308887:/opt/airflow# ./scripts/in_container/
- bin/ run_flake8.sh*
- check_environment.sh* run_generate_constraints.sh*
- entrypoint_ci.sh* run_init_script.sh*
- entrypoint_exec.sh* run_install_and_test_provider_packages.sh*
- _in_container_script_init.sh* run_mypy.sh*
- prod/ run_prepare_provider_packages.sh*
- run_ci_tests.sh* run_prepare_provider_documentation.sh*
- run_clear_tmp.sh* run_system_tests.sh*
- run_docs_build.sh* run_tmux_welcome.sh*
- run_extract_tests.sh* stop_tmux_airflow.sh*
- run_fix_ownership.sh* update_quarantined_test_status.py*
-
- root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
-
- Running specific type of test
- Types of tests
@@ -746,1003 +698,44 @@ All Tests are inside ./tests directory.
Local and Remote Debugging in IDE
+Contribution guide
+##################
-Pre-commit
-~~~~~~~~~~
+- To know how to contribute to the project visit |CONTRIBUTING.rst|
-Before committing changes to github or raising a pull request, code needs to be checked for certain quality standards
-such as spell check, code syntax, code formatting, compatibility with Apache License requirements etc. This set of
-tests are applied when you commit your code.
+.. |CONTRIBUTING.rst| raw:: html
-.. raw:: html
+ CONTRIBUTING.rst
-
-
-
-
+- Following are some of important links of CONTRIBUTING.rst
+ - |Types of contributions|
-To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be run locally before committing changes.
+ .. |Types of contributions| raw:: html
-1. Installing required packages
+
+ Types of contributions
-.. code-block:: bash
- $ sudo apt install libxml2-utils
+ - |Roles of contributor|
-2. Installing required Python packages
+ .. |Roles of contributor| raw:: html
-.. code-block:: bash
+ Roles of
+ contributor
- $ pyenv activate airflow-env
- $ pip install pre-commit
-3. Go to your project directory
+ - |Workflow for a contribution|
-.. code-block:: bash
+ .. |Workflow for a contribution| raw:: html
- $ cd ~/Projects/airflow
+
+ Workflow for a contribution
-1. Running pre-commit hooks
-.. code-block:: bash
-
- $ pre-commit run --all-files
- No-tabs checker......................................................Passed
- Add license for all SQL files........................................Passed
- Add license for all other files......................................Passed
- Add license for all rst files........................................Passed
- Add license for all JS/CSS/PUML files................................Passed
- Add license for all JINJA template files.............................Passed
- Add license for all shell files......................................Passed
- Add license for all python files.....................................Passed
- Add license for all XML files........................................Passed
- Add license for all yaml files.......................................Passed
- Add license for all md files.........................................Passed
- Add license for all mermaid files....................................Passed
- Add TOC for md files.................................................Passed
- Add TOC for upgrade documentation....................................Passed
- Check hooks apply to the repository..................................Passed
- black................................................................Passed
- Check for merge conflicts............................................Passed
- Debug Statements (Python)............................................Passed
- Check builtin type constructor use...................................Passed
- Detect Private Key...................................................Passed
- Fix End of Files.....................................................Passed
- ...........................................................................
-
-5. Running pre-commit for selected files
-
-.. code-block:: bash
-
- $ pre-commit run --files airflow/decorators.py tests/utils/test_task_group.py
-
-
-
-6. Running specific hook for selected files
-
-.. code-block:: bash
-
- $ pre-commit run black --files airflow/decorators.py tests/utils/test_task_group.py
- black...............................................................Passed
- $ pre-commit run flake8 --files airflow/decorators.py tests/utils/test_task_group.py
- Run flake8..........................................................Passed
-
-
-
-
-7. Running specific checks in container using shell scripts. Scripts are located in ``./scripts/in_container``
- directory.
-
-.. code-block:: bash
-
- root@df8927308887:/opt/airflow# ./scripts/in_container/
- bin/ run_flake8.sh*
- check_environment.sh* run_generate_constraints.sh*
- entrypoint_ci.sh* run_init_script.sh*
- entrypoint_exec.sh* run_install_and_test_provider_packages.sh*
- _in_container_script_init.sh* run_mypy.sh*
- prod/ run_prepare_provider_packages.sh*
- run_ci_tests.sh* run_prepare_provider_documentation.sh*
- run_clear_tmp.sh* run_system_tests.sh*
- run_docs_build.sh* run_tmux_welcome.sh*
- run_extract_tests.sh* stop_tmux_airflow.sh*
- run_fix_ownership.sh* update_quarantined_test_status.py*
-
-
- root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
-
-
-
-
-8. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
-
-.. code-block:: bash
-
- $ cd ~/Projects/airflow
- $ pre-commit install
- $ git commit -m "Added xyz"
-
-9. To disable Pre-commit
-
-.. code-block:: bash
-
- $ cd ~/Projects/airflow
- $ pre-commit uninstall
-
-
-- For more information on visit |STATIC_CODE_CHECKS.rst|
-
-.. |STATIC_CODE_CHECKS.rst| raw:: html
-
-
- STATIC_CODE_CHECKS.rst
-
-- Following are some of the important links of STATIC_CODE_CHECKS.rst
-
- - |Pre-commit Hooks|
-
- .. |Pre-commit Hooks| raw:: html
-
-
- Pre-commit Hooks
-
- - |Running Static Code Checks via Breeze|
-
- .. |Running Static Code Checks via Breeze| raw:: html
-
- Running Static Code Checks via Breeze
-
-
-
-
-
-Contribution guide
-~~~~~~~~~~~~~~~~~~
-
-- To know how to contribute to the project visit |CONTRIBUTING.rst|
-
-.. |CONTRIBUTING.rst| raw:: html
-
- CONTRIBUTING.rst
-
-- Following are some of important links of CONTRIBUTING.rst
-
- - |Types of contributions|
-
- .. |Types of contributions| raw:: html
-
-
- Types of contributions
-
-
- - |Roles of contributor|
-
- .. |Roles of contributor| raw:: html
-
- Roles of
- contributor
-
-
- - |Workflow for a contribution|
-
- .. |Workflow for a contribution| raw:: html
-
-
- Workflow for a contribution
-
-
-
-Raising Pull Request
-~~~~~~~~~~~~~~~~~~~~
-
-1. Go to your GitHub account and open your fork project and click on Branches
-
- .. raw:: html
-
-
-
- 
-
-
-2. Click on ``New pull request`` button on branch from which you want to raise a pull request.
-
- .. raw:: html
-
-
-
- 
-
-
-3. Add title and description as per Contributing guidelines and click on ``Create pull request``.
-
- .. raw:: html
-
-
-
- 
-
-
-
-Syncing Fork and rebasing Pull request
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Often it takes several days or weeks to discuss and iterate with the PR until it is ready to merge.
-In the meantime new commits are merged, and you might run into conflicts, therefore you should periodically
-synchronize main in your fork with the ``apache/airflow`` main and rebase your PR on top of it. Following
-describes how to do it.
-
-
-- |Syncing fork|
-
-.. |Syncing fork| raw:: html
-
-
- Update new changes made to apache:airflow project to your fork
-
-
-- |Rebasing pull request|
-
-.. |Rebasing pull request| raw:: html
-
-
- Rebasing pull request
-
-.. raw:: html
-
-
-
-
-
-Setup and develop using Visual Studio Code
-##########################################
-
-.. raw:: html
-
-
-
-
-
-
-Setup and develop using Gitpod online workspaces
-################################################
-
-.. raw:: html
-
- Setup and develop using Visual Studio Code
- - - -Setup Airflow with Breeze -------------------------- - - - -.. note:: - - Only ``pip`` installation is currently officially supported. - - While they are some successes with using other tools like `poetry
- 
-
-
-2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
-
- .. raw:: html
-
-
-
- 
-
-
-
-
-3. Open your IDE or source code editor and select the option to clone the repository
-
- .. raw:: html
-
-
-
- 
-
-
-
-4. Paste the copied clone link in the URL field and submit.
-
- .. raw:: html
-
-
-
- 
-
-
-
-Setting up Breeze
-~~~~~~~~~~~~~~~~~
-1. Open terminal and enter into virtual environment ``airflow-env`` and goto project directory
-
-.. code-block:: bash
-
- $ pyenv activate airflow-env
- $ cd ~/Projects/airflow/
-
-2. Initializing breeze autocomplete
-
-.. code-block:: bash
-
- $ ./breeze setup-autocomplete
- $ source ~/.bash_completion.d/breeze-complete
-
-3. Initialize breeze environment with required python version and backend. This may take a while for first time.
-
-.. code-block:: bash
-
- $ ./breeze --python 3.8 --backend mysql
-
-.. note::
- If you encounter an error like "docker.credentials.errors.InitializationError:
- docker-credential-secretservice not installed or not available in PATH", you may execute the following command to fix it:
-
- .. code-block:: bash
-
- $ sudo apt install golang-docker-credential-helper
-
- Once the package is installed, execute the breeze command again to resume image building.
-
-4. Once the breeze environment is initialized, create airflow tables and users from the breeze CLI. ``airflow db reset``
- is required to execute at least once for Airflow Breeze to get the database/tables created.
-
-.. code-block:: bash
-
- root@b76fcb399bb6:/opt/airflow# airflow db reset
- root@b76fcb399bb6:/opt/airflow# airflow users create --role Admin --username admin --password admin \
- --email admin@example.com --firstname foo --lastname bar
-
-
-5. Closing Breeze environment. After successfully finishing above command will leave you in container,
- type ``exit`` to exit the container
-
-.. code-block:: bash
-
- root@b76fcb399bb6:/opt/airflow#
- root@b76fcb399bb6:/opt/airflow# exit
-
-.. code-block:: bash
-
- $ ./breeze stop
-
-Installing airflow in the local virtual environment ``airflow-env`` with breeze.
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-1. It may require some packages to be installed; watch the output of the command to see which ones are missing.
-
-.. code-block:: bash
-
- $ sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql
- $ ./breeze initialize-local-virtualenv --python 3.8
-
-
-2. Add following line to ~/.bashrc in order to call breeze command from anywhere.
-
-.. code-block:: bash
-
- export PATH=${PATH}:"/home/${USER}/Projects/airflow"
- source ~/.bashrc
-
-Using Breeze
-~~~~~~~~~~~~
-
-1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
- In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
- It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
- and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
-
-.. code-block:: bash
-
- $ breeze start-airflow
-
- Use CI image.
-
- Branch name: main
- Docker image: apache/airflow:main-python3.8-ci
- Airflow source version: 2.0.0b2
- Python version: 3.8
- Backend: mysql 5.7
-
-
- Port forwarding:
-
- Ports are forwarded to the running docker containers for webserver and database
- * 28080 -> forwarded to Airflow webserver -> airflow:8080
- * 25555 -> forwarded to Flower dashboard -> airflow:5555
- * 25433 -> forwarded to Postgres database -> postgres:5432
- * 23306 -> forwarded to MySQL database -> mysql:3306
- * 26379 -> forwarded to Redis broker -> redis:6379
-
- Here are links to those services that you can use on host:
- * Webserver: http://127.0.0.1:28080
- * Flower: http://127.0.0.1:25555
- * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
- * Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
- * Redis: redis://127.0.0.1:26379/0
-
-
-.. raw:: html
-
-
-
- 
-
-
-
-- Alternatively you can start the same using following commands
-
- 1. Start Breeze
-
- .. code-block:: bash
-
- $ breeze --python 3.8 --backend mysql
-
- 2. Open tmux
-
- .. code-block:: bash
-
- $ root@0c6e4ff0ab3d:/opt/airflow# tmux
-
- 3. Press Ctrl + B and "
-
- .. code-block:: bash
-
- $ root@0c6e4ff0ab3d:/opt/airflow# airflow scheduler
-
-
- 4. Press Ctrl + B and %
-
- .. code-block:: bash
-
- $ root@0c6e4ff0ab3d:/opt/airflow# airflow webserver
-
-
-
-
-2. Now you can access airflow web interface on your local machine at |http://127.0.0.1:28080| with user name ``admin``
- and password ``admin``.
-
- .. |http://127.0.0.1:28080| raw:: html
-
- http://127.0.0.1:28080
-
- .. raw:: html
-
-
-
- 
-
-
-3. Setup mysql database in
- MySQL Workbench with Host ``127.0.0.1``, port ``23306``, user ``root`` and password
- blank(leave empty), default schema ``airflow``.
-
- .. raw:: html
-
-
-
- 
-
-
-4. Stopping breeze
-
-.. code-block:: bash
-
- root@f3619b74c59a:/opt/airflow# stop_airflow
- root@f3619b74c59a:/opt/airflow# exit
- $ breeze stop
-
-5. Knowing more about Breeze
-
-.. code-block:: bash
-
- $ breeze --help
-
-
-For more information visit : |Breeze documentation|
-
-.. |Breeze documentation| raw:: html
-
- Breeze documentation
-
-Following are some of important topics of Breeze documentation:
-
-
-- |Choosing different Breeze environment configuration|
-
-.. |Choosing different Breeze environment configuration| raw:: html
-
- Choosing different Breeze environment configuration
-
-
-- |Troubleshooting Breeze environment|
-
-.. |Troubleshooting Breeze environment| raw:: html
-
- Troubleshooting
- Breeze environment
-
-
-- |Installing Additional tools to the Docker Image|
-
-.. |Installing Additional tools to the Docker Image| raw:: html
-
- Installing
- Additional tools to the Docker Image
-
-
-- |Internal details of Breeze|
-
-.. |Internal details of Breeze| raw:: html
-
-
- Internal details of Breeze
-
-
-- |Breeze Command-Line Interface Reference|
-
-.. |Breeze Command-Line Interface Reference| raw:: html
-
- Breeze Command-Line Interface Reference
-
-
-- |Cleaning the environment|
-
-.. |Cleaning the environment| raw:: html
-
-
- Cleaning the environment
-
-
-- |Other uses of the Airflow Breeze environment|
-
-.. |Other uses of the Airflow Breeze environment| raw:: html
-
- Other uses of the Airflow Breeze environment
-
-
-
-Setting up Debug
-~~~~~~~~~~~~~~~~
-
-1. Configuring Airflow database connection
-
-- Airflow is by default configured to use SQLite database. Configuration can be seen on local machine
- ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
-
-- Installing required dependency for MySQL connection in ``airflow-env`` on local machine.
-
- .. code-block:: bash
-
- $ pyenv activate airflow-env
- $ pip install PyMySQL
-
-- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
- ``~/airflow/airflow.cfg`` on local machine.
-
-1. Debugging an example DAG
-
-- In Visual Studio Code open airflow project, directory ``/files/dags`` of local machine is by default mounted to docker
- machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
- scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
-
-- Copy any example DAG present in the ``/airflow/example_dags`` directory to ``/files/dags/``.
-
-- Add a ``__main__`` block at the end of your DAG file to make it runnable. It will run a ``back_fill`` job:
-
- .. code-block:: python
-
-
- if __name__ == "__main__":
- dag.clear()
- dag.run()
-
-- Add ``"AIRFLOW__CORE__EXECUTOR": "DebugExecutor"`` to the ``"env"`` field of Debug configuration.
-
- - Using the ``Run`` view click on ``Create a launch.json file``
-
- .. raw:: html
-
-
-
- 
- 
- 
-
-
- - Change ``"program"`` to point to an example dag and add ``"env"`` and ``"python"`` fields to the new Python configuration
-
- .. raw:: html
-
-
-
-
-
- 
-
-
-- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in mysql workbench.
-
-
-
-Starting development
---------------------
-
-
-Creating a branch
-~~~~~~~~~~~~~~~~~
-
-1. Click on the branch symbol in the status bar
-
- .. raw:: html
-
-
-
- 
-
-
-2. Give a name to a branch and checkout
-
- .. raw:: html
-
-
-
- 
-
-
-
-
-Testing
-~~~~~~~
-
-All Tests are inside ./tests directory.
-
-- Running Unit tests inside Breeze environment.
-
- Just run ``pytest filepath+filename`` to run the tests.
-
-.. code-block:: bash
-
- root@63528318c8b1:/opt/airflow# pytest tests/utils/test_decorators.py
- ======================================= test session starts =======================================
- platform linux -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
- cachedir: .pytest_cache
- rootdir: /opt/airflow, configfile: pytest.ini
- plugins: celery-4.4.7, requests-mock-1.8.0, xdist-1.34.0, flaky-3.7.0, rerunfailures-9.0, instafail
- -0.4.2, forked-1.3.0, timeouts-1.2.1, cov-2.10.0
- setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
- collected 3 items
-
- tests/utils/test_decorators.py::TestApplyDefault::test_apply PASSED [ 33%]
- tests/utils/test_decorators.py::TestApplyDefault::test_default_args PASSED [ 66%]
- tests/utils/test_decorators.py::TestApplyDefault::test_incorrect_default_args PASSED [100%]
-
- ======================================== 3 passed in 1.49s ========================================
-
-- Running All the test with Breeze by specifying required python version, backend, backend version
-
-.. code-block:: bash
-
- $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
-
-
-- Running specific test in container using shell scripts. Testing in container scripts are located in
- ``./scripts/in_container`` directory.
-
-.. code-block:: bash
-
- root@df8927308887:/opt/airflow# ./scripts/in_container/
- bin/ run_flake8.sh*
- check_environment.sh* run_generate_constraints.sh*
- entrypoint_ci.sh* run_init_script.sh*
- entrypoint_exec.sh* run_install_and_test_provider_packages.sh*
- _in_container_script_init.sh* run_mypy.sh*
- prod/ run_prepare_provider_packages.sh*
- run_ci_tests.sh* run_prepare_provider_documentation.sh*
- run_clear_tmp.sh* run_system_tests.sh*
- run_docs_build.sh* run_tmux_welcome.sh*
- run_extract_tests.sh* stop_tmux_airflow.sh*
- run_fix_ownership.sh* update_quarantined_test_status.py*
-
- root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
-
-- Running specific type of test
-
- - Types of tests
-
- - Running specific type of test
-
- .. code-block:: bash
-
- $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
-
-
-- Running Integration test for specific test type
-
- - Running an Integration Test
-
- .. code-block:: bash
-
- $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All --integration mongo
-
-
-- For more information on Testing visit : |TESTING.rst|
-
-.. |TESTING.rst| raw:: html
-
- TESTING.rst
-
-- Following are the some of important topics of TESTING.rst
-
- - |Airflow Test Infrastructure|
-
- .. |Airflow Test Infrastructure| raw:: html
-
-
- Airflow Test Infrastructure
-
-
- - |Airflow Unit Tests|
-
- .. |Airflow Unit Tests| raw:: html
-
- Airflow Unit
- Tests
-
-
- - |Helm Unit Tests|
-
- .. |Helm Unit Tests| raw:: html
-
- Helm Unit Tests
-
-
-
- - |Airflow Integration Tests|
-
- .. |Airflow Integration Tests| raw:: html
-
-
- Airflow Integration Tests
-
-
- - |Running Tests with Kubernetes|
-
- .. |Running Tests with Kubernetes| raw:: html
-
-
- Running Tests with Kubernetes
-
-
- - |Airflow System Tests|
-
- .. |Airflow System Tests| raw:: html
-
- Airflow
- System Tests
-
-
- - |Local and Remote Debugging in IDE|
-
- .. |Local and Remote Debugging in IDE| raw:: html
-
- Local and Remote Debugging in IDE
-
-
-Pre-commit
-~~~~~~~~~~
-
-Before committing changes to github or raising a pull request, code needs to be checked for certain quality standards
-such as spell check, code syntax, code formatting, compatibility with Apache License requirements etc. This set of
-tests are applied when you commit your code.
-
-.. raw:: html
-
-
-
-
-
-
-
-To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be run locally before committing changes.
-
-1. Installing required packages
-
-.. code-block:: bash
-
- $ sudo apt install libxml2-utils
-
-2. Installing required Python packages
-
-.. code-block:: bash
-
- $ pyenv activate airflow-env
- $ pip install pre-commit
-
-3. Go to your project directory
-
-.. code-block:: bash
-
- $ cd ~/Projects/airflow
-
-
-1. Running pre-commit hooks
-
-.. code-block:: bash
-
- $ pre-commit run --all-files
- No-tabs checker......................................................Passed
- Add license for all SQL files........................................Passed
- Add license for all other files......................................Passed
- Add license for all rst files........................................Passed
- Add license for all JS/CSS/PUML files................................Passed
- Add license for all JINJA template files.............................Passed
- Add license for all shell files......................................Passed
- Add license for all python files.....................................Passed
- Add license for all XML files........................................Passed
- Add license for all yaml files.......................................Passed
- Add license for all md files.........................................Passed
- Add license for all mermaid files....................................Passed
- Add TOC for md files.................................................Passed
- Add TOC for upgrade documentation....................................Passed
- Check hooks apply to the repository..................................Passed
- black................................................................Passed
- Check for merge conflicts............................................Passed
- Debug Statements (Python)............................................Passed
- Check builtin type constructor use...................................Passed
- Detect Private Key...................................................Passed
- Fix End of Files.....................................................Passed
- ...........................................................................
-
-5. Running pre-commit for selected files
-
-.. code-block:: bash
-
- $ pre-commit run --files airflow/decorators.py tests/utils/test_task_group.py
-
-
-
-6. Running specific hook for selected files
-
-.. code-block:: bash
-
- $ pre-commit run black --files airflow/decorators.py tests/utils/test_task_group.py
- black...............................................................Passed
- $ pre-commit run flake8 --files airflow/decorators.py tests/utils/test_task_group.py
- Run flake8..........................................................Passed
-
-
-
-
-7. Running specific checks in container using shell scripts. Scripts are located in ``./scripts/in_container``
- directory.
-
-.. code-block:: bash
-
- root@df8927308887:/opt/airflow# ./scripts/in_container/
- bin/ run_flake8.sh*
- check_environment.sh* run_generate_constraints.sh*
- entrypoint_ci.sh* run_init_script.sh*
- entrypoint_exec.sh* run_install_and_test_provider_packages.sh*
- _in_container_script_init.sh* run_mypy.sh*
- prod/ run_prepare_provider_packages.sh*
- run_ci_tests.sh* run_prepare_provider_documentation.sh*
- run_clear_tmp.sh* run_system_tests.sh*
- run_docs_build.sh* run_tmux_welcome.sh*
- run_extract_tests.sh* stop_tmux_airflow.sh*
- run_fix_ownership.sh* update_quarantined_test_status.py*
-
-
- root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
-
-
-
-
-8. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
-
-.. code-block:: bash
-
- $ cd ~/Projects/airflow
- $ pre-commit install
- $ git commit -m "Added xyz"
-
-9. To disable Pre-commit
-
-.. code-block:: bash
-
- $ cd ~/Projects/airflow
- $ pre-commit uninstall
-
-
-- For more information on visit |STATIC_CODE_CHECKS.rst|
-
-.. |STATIC_CODE_CHECKS.rst| raw:: html
-
-
- STATIC_CODE_CHECKS.rst
-
-- Following are some of the important links of STATIC_CODE_CHECKS.rst
-
- - |Pre-commit Hooks|
-
- .. |Pre-commit Hooks| raw:: html
-
-
- Pre-commit Hooks
-
- - |Running Static Code Checks via Breeze|
-
- .. |Running Static Code Checks via Breeze| raw:: html
-
- Running Static Code Checks via Breeze
-
-
-
-
-
-Contribution guide
-~~~~~~~~~~~~~~~~~~
-
-- To know how to contribute to the project visit |CONTRIBUTING.rst|
-
-.. |CONTRIBUTING.rst| raw:: html
-
- CONTRIBUTING.rst
-
-- Following are some of important links of CONTRIBUTING.rst
-
- - |Types of contributions|
-
- .. |Types of contributions| raw:: html
-
-
- Types of contributions
-
-
- - |Roles of contributor|
-
- .. |Roles of contributor| raw:: html
-
- Roles of
- contributor
-
-
- - |Workflow for a contribution|
-
- .. |Workflow for a contribution| raw:: html
-
-
- Workflow for a contribution
-
-
-
-Raising Pull Request
-~~~~~~~~~~~~~~~~~~~~
+Raising Pull Request
+--------------------
1. Go to your GitHub account and open your fork project and click on Branches
@@ -1773,7 +766,7 @@ Raising Pull Request
Syncing Fork and rebasing Pull request
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+--------------------------------------
Often it takes several days or weeks to discuss and iterate with the PR until it is ready to merge.
In the meantime new commits are merged, and you might run into conflicts, therefore you should periodically
@@ -1796,265 +789,22 @@ describes how to do it.
Rebasing pull request
-.. raw:: html
-
-
-
- as shown.
-
- .. raw:: html
-
- `_
+* `Visual Studio Code `_
-Starting development
---------------------
-
-
-Creating a branch
-~~~~~~~~~~~~~~~~~
-
-1. Click on the branch symbol in the status bar
-
- .. raw:: html
-
- `_
+* `GitHub Codespaces `_
diff --git a/CONTRIBUTORS_QUICK_START_CODESPACES.rst b/CONTRIBUTORS_QUICK_START_CODESPACES.rst
new file mode 100644
index 0000000000000..70e0a8b3f47cd
--- /dev/null
+++ b/CONTRIBUTORS_QUICK_START_CODESPACES.rst
@@ -0,0 +1,45 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+Setup and develop using GitHub Codespaces
+#########################################
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ https://github.com/apache/airflow/
+
+ .. raw:: html
+
+ `_ to start
+ a new codespace.
+
+3. Once the codespace starts your terminal should be already in ``Breeze`` environment and you should
+ be able to edit and run the tests in VS Code interface.
+
+4. You can use `Quick start quide for Visual Studio Code `_ for details
+ as Codespaces use Visual Studio Code as interface.
+
+
+Follow the `Quick start `_ for typical development tasks.
diff --git a/CONTRIBUTORS_QUICK_START_GITPOD.rst b/CONTRIBUTORS_QUICK_START_GITPOD.rst
new file mode 100644
index 0000000000000..3615d300978b9
--- /dev/null
+++ b/CONTRIBUTORS_QUICK_START_GITPOD.rst
@@ -0,0 +1,81 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. contents:: :local:
+
+Connect your project to Gitpod
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Goto |airflow_github| and fork the project.
+
+ .. |airflow_github| raw:: html
+
+ https://github.com/apache/airflow/
+
+ .. raw:: html
+
+ as shown.
+
+ .. raw:: html
+
+ `_ for typical development tasks.
diff --git a/CONTRIBUTORS_QUICK_START_PYCHARM.rst b/CONTRIBUTORS_QUICK_START_PYCHARM.rst
new file mode 100644
index 0000000000000..88c04c5545036
--- /dev/null
+++ b/CONTRIBUTORS_QUICK_START_PYCHARM.rst
@@ -0,0 +1,132 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. contents:: :local:
+
+Setup your project
+##################
+
+1. Open your IDE or source code editor and select the option to clone the repository
+
+ .. raw:: html
+
+ `_ for typical development tasks.
diff --git a/CONTRIBUTORS_QUICK_START_VSCODE.rst b/CONTRIBUTORS_QUICK_START_VSCODE.rst
new file mode 100644
index 0000000000000..c1baf3191017c
--- /dev/null
+++ b/CONTRIBUTORS_QUICK_START_VSCODE.rst
@@ -0,0 +1,125 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. contents:: :local:
+
+Setup your project
+##################
+
+1. Open your IDE or source code editor and select the option to clone the repository
+
+ .. raw:: html
+
+ `_ for typical development tasks.
diff --git a/Dockerfile b/Dockerfile
index 0149fa861bcba..cab46244ae954 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,3 +1,4 @@
+# syntax=docker/dockerfile:1.4
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -34,7 +35,6 @@
# much smaller.
#
# Use the same builder frontend version for everyone
-# syntax=docker/dockerfile:1.3
ARG AIRFLOW_EXTRAS="amazon,async,celery,cncf.kubernetes,dask,docker,elasticsearch,ftp,google,google_auth,grpc,hashicorp,http,ldap,microsoft.azure,mysql,odbc,pandas,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv"
ARG ADDITIONAL_AIRFLOW_EXTRAS=""
ARG ADDITIONAL_PYTHON_DEPS=""
@@ -44,11 +44,11 @@ ARG AIRFLOW_UID="50000"
ARG AIRFLOW_USER_HOME_DIR=/home/airflow
# latest released version here
-ARG AIRFLOW_VERSION="2.2.4"
+ARG AIRFLOW_VERSION="2.3.1"
ARG PYTHON_BASE_IMAGE="python:3.7-slim-bullseye"
-ARG AIRFLOW_PIP_VERSION=22.0.4
+ARG AIRFLOW_PIP_VERSION=22.1.2
ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
ARG AIRFLOW_IMAGE_README_URL="https://raw.githubusercontent.com/apache/airflow/main/docs/docker-stack/README.md"
@@ -58,6 +58,1034 @@ ARG AIRFLOW_VERSION_SPECIFICATION=""
# By default PIP has progress bar but you can disable it.
ARG PIP_PROGRESS_BAR="on"
+
+##############################################################################################
+# This is the script image where we keep all inlined bash scripts needed in other segments
+##############################################################################################
+FROM scratch as scripts
+
+##############################################################################################
+# Please DO NOT modify the inlined scripts manually. The content of those files will be
+# replaced by pre-commit automatically from the "scripts/docker/" folder.
+# This is done in order to avoid problems with caching and file permissions and in order to
+# make the PROD Dockerfile standalone
+##############################################################################################
+
+# The content below is automatically copied from scripts/docker/determine_debian_version_specific_variables.sh
+COPY <<"EOF" /determine_debian_version_specific_variables.sh
+function determine_debian_version_specific_variables() {
+ local color_red
+ color_red=$'\e[31m'
+ local color_reset
+ color_reset=$'\e[0m'
+
+ local debian_version
+ debian_version=$(lsb_release -cs)
+ if [[ ${debian_version} == "buster" ]]; then
+ export DISTRO_LIBENCHANT="libenchant-dev"
+ export DISTRO_LIBGCC="libgcc-8-dev"
+ export DISTRO_SELINUX="python-selinux"
+ export DISTRO_LIBFFI="libffi6"
+ # Note missing man directories on debian-buster
+ # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
+ mkdir -pv /usr/share/man/man1
+ mkdir -pv /usr/share/man/man7
+ elif [[ ${debian_version} == "bullseye" ]]; then
+ export DISTRO_LIBENCHANT="libenchant-2-2"
+ export DISTRO_LIBGCC="libgcc-10-dev"
+ export DISTRO_SELINUX="python3-selinux"
+ export DISTRO_LIBFFI="libffi7"
+ else
+ echo
+ echo "${color_red}Unknown distro version ${debian_version}${color_reset}"
+ echo
+ exit 1
+ fi
+}
+
+determine_debian_version_specific_variables
+EOF
+
+# The content below is automatically copied from scripts/docker/install_mysql.sh
+COPY <<"EOF" /install_mysql.sh
+set -euo pipefail
+declare -a packages
+
+MYSQL_VERSION="8.0"
+readonly MYSQL_VERSION
+
+COLOR_BLUE=$'\e[34m'
+readonly COLOR_BLUE
+COLOR_RESET=$'\e[0m'
+readonly COLOR_RESET
+
+: "${INSTALL_MYSQL_CLIENT:?Should be true or false}"
+
+install_mysql_client() {
+ if [[ "${1}" == "dev" ]]; then
+ packages=("libmysqlclient-dev" "mysql-client")
+ elif [[ "${1}" == "prod" ]]; then
+ packages=("libmysqlclient21" "mysql-client")
+ else
+ echo
+ echo "Specify either prod or dev"
+ echo
+ exit 1
+ fi
+
+ echo
+ echo "${COLOR_BLUE}Installing mysql client version ${MYSQL_VERSION}: ${1}${COLOR_RESET}"
+ echo
+
+ local key="467B942D3A79BD29"
+ readonly key
+
+ GNUPGHOME="$(mktemp -d)"
+ export GNUPGHOME
+ set +e
+ for keyserver in $(shuf -e ha.pool.sks-keyservers.net hkp://p80.pool.sks-keyservers.net:80 \
+ keyserver.ubuntu.com hkp://keyserver.ubuntu.com:80)
+ do
+ gpg --keyserver "${keyserver}" --recv-keys "${key}" 2>&1 && break
+ done
+ set -e
+ gpg --export "${key}" > /etc/apt/trusted.gpg.d/mysql.gpg
+ gpgconf --kill all
+ rm -rf "${GNUPGHOME}"
+ unset GNUPGHOME
+ echo "deb http://repo.mysql.com/apt/debian/ $(lsb_release -cs) mysql-${MYSQL_VERSION}" > /etc/apt/sources.list.d/mysql.list
+ apt-get update
+ apt-get install --no-install-recommends -y "${packages[@]}"
+ apt-get autoremove -yqq --purge
+ apt-get clean && rm -rf /var/lib/apt/lists/*
+}
+
+if [[ $(uname -m) == "arm64" || $(uname -m) == "aarch64" ]]; then
+ # disable MYSQL for ARM64
+ INSTALL_MYSQL_CLIENT="false"
+fi
+
+if [[ ${INSTALL_MYSQL_CLIENT:="true"} == "true" ]]; then
+ install_mysql_client "${@}"
+fi
+EOF
+
+# The content below is automatically copied from scripts/docker/install_mssql.sh
+COPY <<"EOF" /install_mssql.sh
+set -euo pipefail
+
+: "${INSTALL_MSSQL_CLIENT:?Should be true or false}"
+
+COLOR_BLUE=$'\e[34m'
+readonly COLOR_BLUE
+COLOR_YELLOW=$'\e[33m'
+readonly COLOR_YELLOW
+COLOR_RESET=$'\e[0m'
+readonly COLOR_RESET
+
+function install_mssql_client() {
+ # Install MsSQL client from Microsoft repositories
+ if [[ ${INSTALL_MSSQL_CLIENT:="true"} != "true" ]]; then
+ echo
+ echo "${COLOR_BLUE}Skip installing mssql client${COLOR_RESET}"
+ echo
+ return
+ fi
+ echo
+ echo "${COLOR_BLUE}Installing mssql client${COLOR_RESET}"
+ echo
+ local distro
+ local version
+ distro=$(lsb_release -is | tr '[:upper:]' '[:lower:]')
+ version_name=$(lsb_release -cs | tr '[:upper:]' '[:lower:]')
+ version=$(lsb_release -rs)
+ local driver
+ if [[ ${version_name} == "buster" ]]; then
+ driver=msodbcsql17
+ elif [[ ${version_name} == "bullseye" ]]; then
+ driver=msodbcsql18
+ else
+ echo
+ echo "${COLOR_YELLOW}Only Buster or Bullseye are supported. Skipping MSSQL installation${COLOR_RESET}"
+ echo
+ return
+ fi
+ curl --silent https://packages.microsoft.com/keys/microsoft.asc | apt-key add - >/dev/null 2>&1
+ curl --silent "https://packages.microsoft.com/config/${distro}/${version}/prod.list" > \
+ /etc/apt/sources.list.d/mssql-release.list
+ apt-get update -yqq
+ apt-get upgrade -yqq
+ ACCEPT_EULA=Y apt-get -yqq install -y --no-install-recommends "${driver}"
+ rm -rf /var/lib/apt/lists/*
+ apt-get autoremove -yqq --purge
+ apt-get clean && rm -rf /var/lib/apt/lists/*
+}
+
+if [[ $(uname -m) == "arm64" || $(uname -m) == "aarch64" ]]; then
+ # disable MSSQL for ARM64
+ INSTALL_MSSQL_CLIENT="false"
+fi
+
+install_mssql_client "${@}"
+EOF
+
+# The content below is automatically copied from scripts/docker/install_postgres.sh
+COPY <<"EOF" /install_postgres.sh
+set -euo pipefail
+declare -a packages
+
+COLOR_BLUE=$'\e[34m'
+readonly COLOR_BLUE
+COLOR_RESET=$'\e[0m'
+readonly COLOR_RESET
+
+: "${INSTALL_POSTGRES_CLIENT:?Should be true or false}"
+
+install_postgres_client() {
+ echo
+ echo "${COLOR_BLUE}Installing postgres client${COLOR_RESET}"
+ echo
+
+ if [[ "${1}" == "dev" ]]; then
+ packages=("libpq-dev" "postgresql-client")
+ elif [[ "${1}" == "prod" ]]; then
+ packages=("postgresql-client")
+ else
+ echo
+ echo "Specify either prod or dev"
+ echo
+ exit 1
+ fi
+
+ curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
+ echo "deb https://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list
+ apt-get update
+ apt-get install --no-install-recommends -y "${packages[@]}"
+ apt-get autoremove -yqq --purge
+ apt-get clean && rm -rf /var/lib/apt/lists/*
+}
+
+if [[ ${INSTALL_POSTGRES_CLIENT:="true"} == "true" ]]; then
+ install_postgres_client "${@}"
+fi
+EOF
+
+# The content below is automatically copied from scripts/docker/install_pip_version.sh
+COPY <<"EOF" /install_pip_version.sh
+. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+
+: "${AIRFLOW_PIP_VERSION:?Should be set}"
+
+function install_pip_version() {
+ echo
+ echo "${COLOR_BLUE}Installing pip version ${AIRFLOW_PIP_VERSION}${COLOR_RESET}"
+ echo
+ pip install --disable-pip-version-check --no-cache-dir --upgrade "pip==${AIRFLOW_PIP_VERSION}" &&
+ mkdir -p ${HOME}/.local/bin
+}
+
+common::get_colors
+common::get_airflow_version_specification
+common::override_pip_version_if_needed
+common::show_pip_version_and_location
+
+install_pip_version
+EOF
+
+# The content below is automatically copied from scripts/docker/install_airflow_dependencies_from_branch_tip.sh
+COPY <<"EOF" /install_airflow_dependencies_from_branch_tip.sh
+
+. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+
+: "${AIRFLOW_REPO:?Should be set}"
+: "${AIRFLOW_BRANCH:?Should be set}"
+: "${INSTALL_MYSQL_CLIENT:?Should be true or false}"
+: "${INSTALL_POSTGRES_CLIENT:?Should be true or false}"
+: "${AIRFLOW_PIP_VERSION:?Should be set}"
+
+function install_airflow_dependencies_from_branch_tip() {
+ echo
+ echo "${COLOR_BLUE}Installing airflow from ${AIRFLOW_BRANCH}. It is used to cache dependencies${COLOR_RESET}"
+ echo
+ if [[ ${INSTALL_MYSQL_CLIENT} != "true" ]]; then
+ AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/mysql,}
+ fi
+ if [[ ${INSTALL_POSTGRES_CLIENT} != "true" ]]; then
+ AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/postgres,}
+ fi
+ # Install latest set of dependencies using constraints. In case constraints were upgraded and there
+ # are conflicts, this might fail, but it should be fixed in the following installation steps
+ set -x
+ pip install --root-user-action ignore \
+ "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
+ --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" || true
+ # make sure correct PIP version is used
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ pip freeze | grep apache-airflow-providers | xargs pip uninstall --yes 2>/dev/null || true
+ set +x
+ echo
+ echo "${COLOR_BLUE}Uninstalling just airflow. Dependencies remain. Now target airflow can be reinstalled using mostly cached dependencies${COLOR_RESET}"
+ echo
+ pip uninstall --yes apache-airflow || true
+}
+
+common::get_colors
+common::get_airflow_version_specification
+common::override_pip_version_if_needed
+common::get_constraints_location
+common::show_pip_version_and_location
+
+install_airflow_dependencies_from_branch_tip
+EOF
+
+# The content below is automatically copied from scripts/docker/common.sh
+COPY <<"EOF" /common.sh
+set -euo pipefail
+
+function common::get_colors() {
+ COLOR_BLUE=$'\e[34m'
+ COLOR_GREEN=$'\e[32m'
+ COLOR_RED=$'\e[31m'
+ COLOR_RESET=$'\e[0m'
+ COLOR_YELLOW=$'\e[33m'
+ export COLOR_BLUE
+ export COLOR_GREEN
+ export COLOR_RED
+ export COLOR_RESET
+ export COLOR_YELLOW
+}
+
+
+function common::get_airflow_version_specification() {
+ if [[ -z ${AIRFLOW_VERSION_SPECIFICATION=}
+ && -n ${AIRFLOW_VERSION}
+ && ${AIRFLOW_INSTALLATION_METHOD} != "." ]]; then
+ AIRFLOW_VERSION_SPECIFICATION="==${AIRFLOW_VERSION}"
+ fi
+}
+
+function common::override_pip_version_if_needed() {
+ if [[ -n ${AIRFLOW_VERSION} ]]; then
+ if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; then
+ export AIRFLOW_PIP_VERSION="22.1.2"
+ fi
+ fi
+}
+
+function common::get_constraints_location() {
+ # auto-detect Airflow-constraint reference and location
+ if [[ -z "${AIRFLOW_CONSTRAINTS_REFERENCE=}" ]]; then
+ if [[ ${AIRFLOW_VERSION} =~ v?2.* && ! ${AIRFLOW_VERSION} =~ .*dev.* ]]; then
+ AIRFLOW_CONSTRAINTS_REFERENCE=constraints-${AIRFLOW_VERSION}
+ else
+ AIRFLOW_CONSTRAINTS_REFERENCE=${DEFAULT_CONSTRAINTS_BRANCH}
+ fi
+ fi
+
+ if [[ -z ${AIRFLOW_CONSTRAINTS_LOCATION=} ]]; then
+ local constraints_base="https://raw.githubusercontent.com/${CONSTRAINTS_GITHUB_REPOSITORY}/${AIRFLOW_CONSTRAINTS_REFERENCE}"
+ local python_version
+ python_version="$(python --version 2>/dev/stdout | cut -d " " -f 2 | cut -d "." -f 1-2)"
+ AIRFLOW_CONSTRAINTS_LOCATION="${constraints_base}/${AIRFLOW_CONSTRAINTS_MODE}-${python_version}.txt"
+ fi
+}
+
+function common::show_pip_version_and_location() {
+ echo "PATH=${PATH}"
+ echo "pip on path: $(which pip)"
+ echo "Using pip: $(pip --version)"
+}
+EOF
+
+# The content below is automatically copied from scripts/docker/prepare_node_modules.sh
+COPY <<"EOF" /prepare_node_modules.sh
+set -euo pipefail
+
+COLOR_BLUE=$'\e[34m'
+readonly COLOR_BLUE
+COLOR_RESET=$'\e[0m'
+readonly COLOR_RESET
+
+function prepare_node_modules() {
+ echo
+ echo "${COLOR_BLUE}Preparing node modules${COLOR_RESET}"
+ echo
+ local www_dir
+ if [[ ${AIRFLOW_INSTALLATION_METHOD=} == "." ]]; then
+ # In case we are building from sources in production image, we should build the assets
+ www_dir="${AIRFLOW_SOURCES_TO=${AIRFLOW_SOURCES}}/airflow/www"
+ else
+ www_dir="$(python -m site --user-site)/airflow/www"
+ fi
+ pushd ${www_dir} || exit 1
+ set +e
+ yarn install --frozen-lockfile --no-cache 2>/tmp/out-yarn-install.txt
+ local res=$?
+ if [[ ${res} != 0 ]]; then
+ >&2 echo
+ >&2 echo "Error when running yarn install:"
+ >&2 echo
+ >&2 cat /tmp/out-yarn-install.txt && rm -f /tmp/out-yarn-install.txt
+ exit 1
+ fi
+ rm -f /tmp/out-yarn-install.txt
+ popd || exit 1
+}
+
+prepare_node_modules
+EOF
+
+# The content below is automatically copied from scripts/docker/compile_www_assets.sh
+COPY <<"EOF" /compile_www_assets.sh
+set -euo pipefail
+
+BUILD_TYPE=${BUILD_TYPE="prod"}
+REMOVE_ARTIFACTS=${REMOVE_ARTIFACTS="true"}
+
+COLOR_BLUE=$'\e[34m'
+readonly COLOR_BLUE
+COLOR_RESET=$'\e[0m'
+readonly COLOR_RESET
+
+function compile_www_assets() {
+ echo
+ echo "${COLOR_BLUE}Compiling www assets: running yarn ${BUILD_TYPE}${COLOR_RESET}"
+ echo
+ local www_dir
+ if [[ ${AIRFLOW_INSTALLATION_METHOD=} == "." ]]; then
+ # In case we are building from sources in production image, we should build the assets
+ www_dir="${AIRFLOW_SOURCES_TO=${AIRFLOW_SOURCES}}/airflow/www"
+ else
+ www_dir="$(python -m site --user-site)/airflow/www"
+ fi
+ pushd ${www_dir} || exit 1
+ set +e
+ yarn run "${BUILD_TYPE}" 2>/tmp/out-yarn-run.txt
+ res=$?
+ if [[ ${res} != 0 ]]; then
+ >&2 echo
+ >&2 echo "Error when running yarn run:"
+ >&2 echo
+ >&2 cat /tmp/out-yarn-run.txt && rm -rf /tmp/out-yarn-run.txt
+ exit 1
+ fi
+ rm -f /tmp/out-yarn-run.txt
+ set -e
+ local md5sum_file
+ md5sum_file="static/dist/sum.md5"
+ readonly md5sum_file
+ find package.json yarn.lock static/css static/js -type f | sort | xargs md5sum > "${md5sum_file}"
+ if [[ ${REMOVE_ARTIFACTS} == "true" ]]; then
+ echo
+ echo "${COLOR_BLUE}Removing generated node modules${COLOR_RESET}"
+ echo
+ rm -rf "${www_dir}/node_modules"
+ rm -vf "${www_dir}"/{package.json,yarn.lock,.eslintignore,.eslintrc,.stylelintignore,.stylelintrc,compile_assets.sh,webpack.config.js}
+ else
+ echo
+ echo "${COLOR_BLUE}Leaving generated node modules${COLOR_RESET}"
+ echo
+ fi
+ popd || exit 1
+}
+
+compile_www_assets
+EOF
+
+# The content below is automatically copied from scripts/docker/pip
+COPY <<"EOF" /pip
+#!/usr/bin/env bash
+COLOR_RED=$'\e[31m'
+COLOR_RESET=$'\e[0m'
+COLOR_YELLOW=$'\e[33m'
+
+if [[ $(id -u) == "0" ]]; then
+ echo
+ echo "${COLOR_RED}You are running pip as root. Please use 'airflow' user to run pip!${COLOR_RESET}"
+ echo
+ echo "${COLOR_YELLOW}See: https://airflow.apache.org/docs/docker-stack/build.html#adding-a-new-pypi-package${COLOR_RESET}"
+ echo
+ exit 1
+fi
+exec "${HOME}"/.local/bin/pip "${@}"
+EOF
+
+# The content below is automatically copied from scripts/docker/install_from_docker_context_files.sh
+COPY <<"EOF" /install_from_docker_context_files.sh
+
+. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+
+: "${AIRFLOW_PIP_VERSION:?Should be set}"
+
+function install_airflow_and_providers_from_docker_context_files(){
+ if [[ ${INSTALL_MYSQL_CLIENT} != "true" ]]; then
+ AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/mysql,}
+ fi
+ if [[ ${INSTALL_POSTGRES_CLIENT} != "true" ]]; then
+ AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/postgres,}
+ fi
+
+ if [[ ! -d /docker-context-files ]]; then
+ echo
+ echo "${COLOR_RED}You must provide a folder via --build-arg DOCKER_CONTEXT_FILES= and you missed it!${COLOR_RESET}"
+ echo
+ exit 1
+ fi
+
+ # shellcheck disable=SC2206
+ local pip_flags=(
+ # Don't quote this -- if it is empty we don't want it to create an
+ # empty array element
+ --find-links="file:///docker-context-files"
+ )
+
+ # Find Apache Airflow packages in docker-context files
+ local reinstalling_apache_airflow_package
+ reinstalling_apache_airflow_package=$(ls \
+ /docker-context-files/apache?airflow?[0-9]*.{whl,tar.gz} 2>/dev/null || true)
+ # Add extras when installing airflow
+ if [[ -n "${reinstalling_apache_airflow_package}" ]]; then
+ # When a provider depends on a dev version of Airflow, we need to
+ # specify `apache-airflow==$VER`, otherwise pip will look for it on
+ # pip, and fail to find it
+
+ # This will work as long as the wheel file is correctly named, which it
+ # will be if it was build by wheel tooling
+ local ver
+ ver=$(basename "$reinstalling_apache_airflow_package" | cut -d "-" -f 2)
+ reinstalling_apache_airflow_package="apache-airflow[${AIRFLOW_EXTRAS}]==$ver"
+ fi
+
+ # Find Apache Airflow packages in docker-context files
+ local reinstalling_apache_airflow_providers_packages
+ reinstalling_apache_airflow_providers_packages=$(ls \
+ /docker-context-files/apache?airflow?providers*.{whl,tar.gz} 2>/dev/null || true)
+ if [[ -z "${reinstalling_apache_airflow_package}" && \
+ -z "${reinstalling_apache_airflow_providers_packages}" ]]; then
+ return
+ fi
+
+ echo
+ echo "${COLOR_BLUE}Force re-installing airflow and providers from local files with eager upgrade${COLOR_RESET}"
+ echo
+ # force reinstall all airflow + provider package local files with eager upgrade
+ set -x
+ pip install "${pip_flags[@]}" --root-user-action ignore --upgrade --upgrade-strategy eager \
+ ${reinstalling_apache_airflow_package} ${reinstalling_apache_airflow_providers_packages} \
+ ${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
+ set +x
+
+ # make sure correct PIP version is left installed
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ pip check
+}
+
+function install_all_other_packages_from_docker_context_files() {
+
+ echo
+ echo "${COLOR_BLUE}Force re-installing all other package from local files without dependencies${COLOR_RESET}"
+ echo
+ local reinstalling_other_packages
+ # shellcheck disable=SC2010
+ reinstalling_other_packages=$(ls /docker-context-files/*.{whl,tar.gz} 2>/dev/null | \
+ grep -v apache_airflow | grep -v apache-airflow || true)
+ if [[ -n "${reinstalling_other_packages}" ]]; then
+ set -x
+ pip install --root-user-action ignore --force-reinstall --no-deps --no-index ${reinstalling_other_packages}
+ # make sure correct PIP version is used
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ set -x
+ fi
+}
+
+common::get_colors
+common::get_airflow_version_specification
+common::override_pip_version_if_needed
+common::get_constraints_location
+common::show_pip_version_and_location
+
+install_airflow_and_providers_from_docker_context_files
+
+common::show_pip_version_and_location
+install_all_other_packages_from_docker_context_files
+EOF
+
+# The content below is automatically copied from scripts/docker/install_airflow.sh
+COPY <<"EOF" /install_airflow.sh
+
+. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+
+: "${AIRFLOW_PIP_VERSION:?Should be set}"
+
+function install_airflow() {
+ # Coherence check for editable installation mode.
+ if [[ ${AIRFLOW_INSTALLATION_METHOD} != "." && \
+ ${AIRFLOW_INSTALL_EDITABLE_FLAG} == "--editable" ]]; then
+ echo
+ echo "${COLOR_RED}ERROR! You can only use --editable flag when installing airflow from sources!${COLOR_RESET}"
+ echo "${COLOR_RED} Current installation method is '${AIRFLOW_INSTALLATION_METHOD} and should be '.'${COLOR_RESET}"
+ exit 1
+ fi
+ # Remove mysql from extras if client is not going to be installed
+ if [[ ${INSTALL_MYSQL_CLIENT} != "true" ]]; then
+ AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/mysql,}
+ echo "${COLOR_YELLOW}MYSQL client installation is disabled. Extra 'mysql' installations were therefore omitted.${COLOR_RESET}"
+ fi
+ # Remove postgres from extras if client is not going to be installed
+ if [[ ${INSTALL_POSTGRES_CLIENT} != "true" ]]; then
+ AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/postgres,}
+ echo "${COLOR_YELLOW}Postgres client installation is disabled. Extra 'postgres' installations were therefore omitted.${COLOR_RESET}"
+ fi
+ if [[ "${UPGRADE_TO_NEWER_DEPENDENCIES}" != "false" ]]; then
+ echo
+ echo "${COLOR_BLUE}Installing all packages with eager upgrade${COLOR_RESET}"
+ echo
+ # eager upgrade
+ pip install --root-user-action ignore --upgrade --upgrade-strategy eager \
+ "${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}" \
+ ${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
+ if [[ -n "${AIRFLOW_INSTALL_EDITABLE_FLAG}" ]]; then
+ # Remove airflow and reinstall it using editable flag
+ # We can only do it when we install airflow from sources
+ set -x
+ pip uninstall apache-airflow --yes
+ pip install --root-user-action ignore ${AIRFLOW_INSTALL_EDITABLE_FLAG} \
+ "${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}"
+ set +x
+ fi
+
+ # make sure correct PIP version is used
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ echo
+ echo "${COLOR_BLUE}Running 'pip check'${COLOR_RESET}"
+ echo
+ pip check
+ else \
+ echo
+ echo "${COLOR_BLUE}Installing all packages with constraints and upgrade if needed${COLOR_RESET}"
+ echo
+ set -x
+ pip install --root-user-action ignore ${AIRFLOW_INSTALL_EDITABLE_FLAG} \
+ "${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}" \
+ --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}"
+ # make sure correct PIP version is used
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ # then upgrade if needed without using constraints to account for new limits in setup.py
+ pip install --root-user-action ignore --upgrade --upgrade-strategy only-if-needed \
+ ${AIRFLOW_INSTALL_EDITABLE_FLAG} \
+ "${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}"
+ # make sure correct PIP version is used
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ set +x
+ echo
+ echo "${COLOR_BLUE}Running 'pip check'${COLOR_RESET}"
+ echo
+ pip check
+ fi
+
+}
+
+common::get_colors
+common::get_airflow_version_specification
+common::override_pip_version_if_needed
+common::get_constraints_location
+common::show_pip_version_and_location
+
+install_airflow
+EOF
+
+# The content below is automatically copied from scripts/docker/install_additional_dependencies.sh
+COPY <<"EOF" /install_additional_dependencies.sh
+set -euo pipefail
+
+: "${UPGRADE_TO_NEWER_DEPENDENCIES:?Should be true or false}"
+: "${ADDITIONAL_PYTHON_DEPS:?Should be set}"
+: "${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS:?Should be set}"
+: "${AIRFLOW_PIP_VERSION:?Should be set}"
+
+. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+
+function install_additional_dependencies() {
+ if [[ "${UPGRADE_TO_NEWER_DEPENDENCIES}" != "false" ]]; then
+ echo
+ echo "${COLOR_BLUE}Installing additional dependencies while upgrading to newer dependencies${COLOR_RESET}"
+ echo
+ set -x
+ pip install --root-user-action ignore --upgrade --upgrade-strategy eager \
+ ${ADDITIONAL_PYTHON_DEPS} ${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
+ # make sure correct PIP version is used
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ set +x
+ echo
+ echo "${COLOR_BLUE}Running 'pip check'${COLOR_RESET}"
+ echo
+ pip check
+ else
+ echo
+ echo "${COLOR_BLUE}Installing additional dependencies upgrading only if needed${COLOR_RESET}"
+ echo
+ set -x
+ pip install --root-user-action ignore --upgrade --upgrade-strategy only-if-needed \
+ ${ADDITIONAL_PYTHON_DEPS}
+ # make sure correct PIP version is used
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ set +x
+ echo
+ echo "${COLOR_BLUE}Running 'pip check'${COLOR_RESET}"
+ echo
+ pip check
+ fi
+}
+
+common::get_colors
+common::get_airflow_version_specification
+common::override_pip_version_if_needed
+common::get_constraints_location
+common::show_pip_version_and_location
+
+install_additional_dependencies
+EOF
+
+
+# The content below is automatically copied from scripts/docker/entrypoint_prod.sh
+COPY <<"EOF" /entrypoint_prod.sh
+#!/usr/bin/env bash
+AIRFLOW_COMMAND="${1:-}"
+
+set -euo pipefail
+
+LD_PRELOAD="/usr/lib/$(uname -m)-linux-gnu/libstdc++.so.6"
+export LD_PRELOAD
+
+function run_check_with_retries {
+ local cmd
+ cmd="${1}"
+ local countdown
+ countdown="${CONNECTION_CHECK_MAX_COUNT}"
+
+ while true
+ do
+ set +e
+ local last_check_result
+ local res
+ last_check_result=$(eval "${cmd} 2>&1")
+ res=$?
+ set -e
+ if [[ ${res} == 0 ]]; then
+ echo
+ break
+ else
+ echo -n "."
+ countdown=$((countdown-1))
+ fi
+ if [[ ${countdown} == 0 ]]; then
+ echo
+ echo "ERROR! Maximum number of retries (${CONNECTION_CHECK_MAX_COUNT}) reached."
+ echo
+ echo "Last check result:"
+ echo "$ ${cmd}"
+ echo "${last_check_result}"
+ echo
+ exit 1
+ else
+ sleep "${CONNECTION_CHECK_SLEEP_TIME}"
+ fi
+ done
+}
+
+function run_nc() {
+ # Checks if it is possible to connect to the host using netcat.
+ #
+ # We want to avoid misleading messages and perform only forward lookup of the service IP address.
+ # Netcat when run without -n performs both forward and reverse lookup and fails if the reverse
+ # lookup name does not match the original name even if the host is reachable via IP. This happens
+ # randomly with docker-compose in GitHub Actions.
+ # Since we are not using reverse lookup elsewhere, we can perform forward lookup in python
+ # And use the IP in NC and add '-n' switch to disable any DNS use.
+ # Even if this message might be harmless, it might hide the real reason for the problem
+ # Which is the long time needed to start some services, seeing this message might be totally misleading
+ # when you try to analyse the problem, that's why it's best to avoid it,
+ local host="${1}"
+ local port="${2}"
+ local ip
+ ip=$(python -c "import socket; print(socket.gethostbyname('${host}'))")
+ nc -zvvn "${ip}" "${port}"
+}
+
+
+function wait_for_connection {
+ # Waits for Connection to the backend specified via URL passed as first parameter
+ # Detects backend type depending on the URL schema and assigns
+ # default port numbers if not specified in the URL.
+ # Then it loops until connection to the host/port specified can be established
+ # It tries `CONNECTION_CHECK_MAX_COUNT` times and sleeps `CONNECTION_CHECK_SLEEP_TIME` between checks
+ local connection_url
+ connection_url="${1}"
+ local detected_backend
+ detected_backend=$(python -c "from urllib.parse import urlsplit; import sys; print(urlsplit(sys.argv[1]).scheme)" "${connection_url}")
+ local detected_host
+ detected_host=$(python -c "from urllib.parse import urlsplit; import sys; print(urlsplit(sys.argv[1]).hostname or '')" "${connection_url}")
+ local detected_port
+ detected_port=$(python -c "from urllib.parse import urlsplit; import sys; print(urlsplit(sys.argv[1]).port or '')" "${connection_url}")
+
+ echo BACKEND="${BACKEND:=${detected_backend}}"
+ readonly BACKEND
+
+ if [[ -z "${detected_port=}" ]]; then
+ if [[ ${BACKEND} == "postgres"* ]]; then
+ detected_port=5432
+ elif [[ ${BACKEND} == "mysql"* ]]; then
+ detected_port=3306
+ elif [[ ${BACKEND} == "mssql"* ]]; then
+ detected_port=1433
+ elif [[ ${BACKEND} == "redis"* ]]; then
+ detected_port=6379
+ elif [[ ${BACKEND} == "amqp"* ]]; then
+ detected_port=5672
+ fi
+ fi
+
+ detected_host=${detected_host:="localhost"}
+
+ # Allow the DB parameters to be overridden by environment variable
+ echo DB_HOST="${DB_HOST:=${detected_host}}"
+ readonly DB_HOST
+
+ echo DB_PORT="${DB_PORT:=${detected_port}}"
+ readonly DB_PORT
+ if [[ -n "${DB_HOST=}" ]] && [[ -n "${DB_PORT=}" ]]; then
+ run_check_with_retries "run_nc ${DB_HOST@Q} ${DB_PORT@Q}"
+ else
+ >&2 echo "The connection details to the broker could not be determined. Connectivity checks were skipped."
+ fi
+}
+
+function create_www_user() {
+ local local_password=""
+ # Warning: command environment variables (*_CMD) have priority over usual configuration variables
+ # for configuration parameters that require sensitive information. This is the case for the SQL database
+ # and the broker backend in this entrypoint script.
+ if [[ -n "${_AIRFLOW_WWW_USER_PASSWORD_CMD=}" ]]; then
+ local_password=$(eval "${_AIRFLOW_WWW_USER_PASSWORD_CMD}")
+ unset _AIRFLOW_WWW_USER_PASSWORD_CMD
+ elif [[ -n "${_AIRFLOW_WWW_USER_PASSWORD=}" ]]; then
+ local_password="${_AIRFLOW_WWW_USER_PASSWORD}"
+ unset _AIRFLOW_WWW_USER_PASSWORD
+ fi
+ if [[ -z ${local_password} ]]; then
+ echo
+ echo "ERROR! Airflow Admin password not set via _AIRFLOW_WWW_USER_PASSWORD or _AIRFLOW_WWW_USER_PASSWORD_CMD variables!"
+ echo
+ exit 1
+ fi
+
+ airflow users create \
+ --username "${_AIRFLOW_WWW_USER_USERNAME="admin"}" \
+ --firstname "${_AIRFLOW_WWW_USER_FIRSTNAME="Airflow"}" \
+ --lastname "${_AIRFLOW_WWW_USER_LASTNAME="Admin"}" \
+ --email "${_AIRFLOW_WWW_USER_EMAIL="airflowadmin@example.com"}" \
+ --role "${_AIRFLOW_WWW_USER_ROLE="Admin"}" \
+ --password "${local_password}" || true
+}
+
+function create_system_user_if_missing() {
+ # This is needed in case of OpenShift-compatible container execution. In case of OpenShift random
+ # User id is used when starting the image, however group 0 is kept as the user group. Our production
+ # Image is OpenShift compatible, so all permissions on all folders are set so that 0 group can exercise
+ # the same privileges as the default "airflow" user, this code checks if the user is already
+ # present in /etc/passwd and will create the system user dynamically, including setting its
+ # HOME directory to the /home/airflow so that (for example) the ${HOME}/.local folder where airflow is
+ # Installed can be automatically added to PYTHONPATH
+ if ! whoami &> /dev/null; then
+ if [[ -w /etc/passwd ]]; then
+ echo "${USER_NAME:-default}:x:$(id -u):0:${USER_NAME:-default} user:${AIRFLOW_USER_HOME_DIR}:/sbin/nologin" \
+ >> /etc/passwd
+ fi
+ export HOME="${AIRFLOW_USER_HOME_DIR}"
+ fi
+}
+
+function set_pythonpath_for_root_user() {
+ # Airflow is installed as a local user application which means that if the container is running as root
+ # the application is not available. because Python then only load system-wide applications.
+ # Now also adds applications installed as local user "airflow".
+ if [[ $UID == "0" ]]; then
+ local python_major_minor
+ python_major_minor="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
+ export PYTHONPATH="${AIRFLOW_USER_HOME_DIR}/.local/lib/python${python_major_minor}/site-packages:${PYTHONPATH:-}"
+ >&2 echo "The container is run as root user. For security, consider using a regular user account."
+ fi
+}
+
+function wait_for_airflow_db() {
+ # Wait for the command to run successfully to validate the database connection.
+ run_check_with_retries "airflow db check"
+}
+
+function upgrade_db() {
+ # Runs airflow db upgrade
+ airflow db upgrade || true
+}
+
+function wait_for_celery_broker() {
+ # Verifies connection to Celery Broker
+ local executor
+ executor="$(airflow config get-value core executor)"
+ if [[ "${executor}" == "CeleryExecutor" ]]; then
+ local connection_url
+ connection_url="$(airflow config get-value celery broker_url)"
+ wait_for_connection "${connection_url}"
+ fi
+}
+
+function exec_to_bash_or_python_command_if_specified() {
+ # If one of the commands: 'bash', 'python' is used, either run appropriate
+ # command with exec
+ if [[ ${AIRFLOW_COMMAND} == "bash" ]]; then
+ shift
+ exec "/bin/bash" "${@}"
+ elif [[ ${AIRFLOW_COMMAND} == "python" ]]; then
+ shift
+ exec "python" "${@}"
+ fi
+}
+
+function check_uid_gid() {
+ if [[ $(id -g) == "0" ]]; then
+ return
+ fi
+ if [[ $(id -u) == "50000" ]]; then
+ >&2 echo
+ >&2 echo "WARNING! You should run the image with GID (Group ID) set to 0"
+ >&2 echo " even if you use 'airflow' user (UID=50000)"
+ >&2 echo
+ >&2 echo " You started the image with UID=$(id -u) and GID=$(id -g)"
+ >&2 echo
+ >&2 echo " This is to make sure you can run the image with an arbitrary UID in the future."
+ >&2 echo
+ >&2 echo " See more about it in the Airflow's docker image documentation"
+ >&2 echo " http://airflow.apache.org/docs/docker-stack/entrypoint"
+ >&2 echo
+ # We still allow the image to run with `airflow` user.
+ return
+ else
+ >&2 echo
+ >&2 echo "ERROR! You should run the image with GID=0"
+ >&2 echo
+ >&2 echo " You started the image with UID=$(id -u) and GID=$(id -g)"
+ >&2 echo
+ >&2 echo "The image should always be run with GID (Group ID) set to 0 regardless of the UID used."
+ >&2 echo " This is to make sure you can run the image with an arbitrary UID."
+ >&2 echo
+ >&2 echo " See more about it in the Airflow's docker image documentation"
+ >&2 echo " http://airflow.apache.org/docs/docker-stack/entrypoint"
+ # This will not work so we fail hard
+ exit 1
+ fi
+}
+
+unset PIP_USER
+
+check_uid_gid
+
+umask 0002
+
+CONNECTION_CHECK_MAX_COUNT=${CONNECTION_CHECK_MAX_COUNT:=20}
+readonly CONNECTION_CHECK_MAX_COUNT
+
+CONNECTION_CHECK_SLEEP_TIME=${CONNECTION_CHECK_SLEEP_TIME:=3}
+readonly CONNECTION_CHECK_SLEEP_TIME
+
+create_system_user_if_missing
+set_pythonpath_for_root_user
+if [[ "${CONNECTION_CHECK_MAX_COUNT}" -gt "0" ]]; then
+ wait_for_airflow_db
+fi
+
+if [[ -n "${_AIRFLOW_DB_UPGRADE=}" ]] ; then
+ upgrade_db
+fi
+
+if [[ -n "${_AIRFLOW_WWW_USER_CREATE=}" ]] ; then
+ create_www_user
+fi
+
+if [[ -n "${_PIP_ADDITIONAL_REQUIREMENTS=}" ]] ; then
+ >&2 echo
+ >&2 echo "!!!!! Installing additional requirements: '${_PIP_ADDITIONAL_REQUIREMENTS}' !!!!!!!!!!!!"
+ >&2 echo
+ >&2 echo "WARNING: This is a development/test feature only. NEVER use it in production!"
+ >&2 echo " Instead, build a custom image as described in"
+ >&2 echo
+ >&2 echo " https://airflow.apache.org/docs/docker-stack/build.html"
+ >&2 echo
+ >&2 echo " Adding requirements at container startup is fragile and is done every time"
+ >&2 echo " the container starts, so it is onlny useful for testing and trying out"
+ >&2 echo " of adding dependencies."
+ >&2 echo
+ pip install --root-user-action ignore --no-cache-dir ${_PIP_ADDITIONAL_REQUIREMENTS}
+fi
+
+
+exec_to_bash_or_python_command_if_specified "${@}"
+
+if [[ ${AIRFLOW_COMMAND} == "airflow" ]]; then
+ AIRFLOW_COMMAND="${2:-}"
+ shift
+fi
+
+if [[ ${AIRFLOW_COMMAND} =~ ^(scheduler|celery)$ ]] \
+ && [[ "${CONNECTION_CHECK_MAX_COUNT}" -gt "0" ]]; then
+ wait_for_celery_broker
+fi
+
+exec "airflow" "${@}"
+EOF
+
+# The content below is automatically copied from scripts/docker/clean-logs.sh
+COPY <<"EOF" /clean-logs.sh
+#!/usr/bin/env bash
+
+
+set -euo pipefail
+
+readonly DIRECTORY="${AIRFLOW_HOME:-/usr/local/airflow}"
+readonly RETENTION="${AIRFLOW__LOG_RETENTION_DAYS:-15}"
+
+trap "exit" INT TERM
+
+readonly EVERY=$((15*60))
+
+echo "Cleaning logs every $EVERY seconds"
+
+while true; do
+ echo "Trimming airflow logs to ${RETENTION} days."
+ find "${DIRECTORY}"/logs \
+ -type d -name 'lost+found' -prune -o \
+ -type f -mtime +"${RETENTION}" -name '*.log' -print0 | \
+ xargs -0 rm -f
+
+ seconds=$(( $(date -u +%s) % EVERY))
+ (( seconds < 1 )) || sleep $((EVERY - seconds))
+done
+EOF
+
+# The content below is automatically copied from scripts/docker/airflow-scheduler-autorestart.sh
+COPY <<"EOF" /airflow-scheduler-autorestart.sh
+#!/usr/bin/env bash
+
+while echo "Running"; do
+ airflow scheduler -n 5
+ return_code=$?
+ if (( return_code != 0 )); then
+ echo "Scheduler crashed with exit code $return_code. Respawning.." >&2
+ date >> /tmp/airflow_scheduler_errors.txt
+ fi
+
+ sleep 1
+done
+EOF
+
##############################################################################################
# This is the build image where we build all dependencies
##############################################################################################
@@ -117,8 +1145,8 @@ ENV DEV_APT_DEPS=${DEV_APT_DEPS} \
DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_ENV=${ADDITIONAL_DEV_APT_ENV}
-COPY scripts/docker/determine_debian_version_specific_variables.sh /scripts/docker/
+COPY --from=scripts determine_debian_version_specific_variables.sh /scripts/docker/
# Install basic and additional apt dependencies
RUN apt-get update \
&& apt-get install --no-install-recommends -yqq apt-utils >/dev/null 2>&1 \
@@ -145,7 +1173,7 @@ ARG AIRFLOW_EXTRAS
ARG ADDITIONAL_AIRFLOW_EXTRAS=""
# Allows to override constraints source
ARG CONSTRAINTS_GITHUB_REPOSITORY="apache/airflow"
-ARG AIRFLOW_CONSTRAINTS="constraints"
+ARG AIRFLOW_CONSTRAINTS_MODE="constraints"
ARG AIRFLOW_CONSTRAINTS_REFERENCE=""
ARG AIRFLOW_CONSTRAINTS_LOCATION=""
ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-main"
@@ -177,14 +1205,18 @@ ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
# www to compile the assets but in case of breeze/CI builds we use latest sources and we override those
# those SOURCES_FROM/TO with "airflow/www" and "/opt/airflow/airflow/www" respectively.
# This is to rebuild the assets only when any of the www sources change
-ARG AIRFLOW_SOURCES_WWW_FROM="empty"
-ARG AIRFLOW_SOURCES_WWW_TO="/empty"
+ARG AIRFLOW_SOURCES_WWW_FROM="Dockerfile"
+ARG AIRFLOW_SOURCES_WWW_TO="/Dockerfile"
# By default we install latest airflow from PyPI so we do not need to copy sources of Airflow
# but in case of breeze/CI builds we use latest sources and we override those
# those SOURCES_FROM/TO with "." and "/opt/airflow" respectively
-ARG AIRFLOW_SOURCES_FROM="empty"
-ARG AIRFLOW_SOURCES_TO="/empty"
+ARG AIRFLOW_SOURCES_FROM="Dockerfile"
+ARG AIRFLOW_SOURCES_TO="/Dockerfile"
+
+# By default we do not install from docker context files but if we decide to install from docker context
+# files, we should override those variables to "docker-context-files"
+ARG DOCKER_CONTEXT_FILES="Dockerfile"
ARG AIRFLOW_HOME
ARG AIRFLOW_USER_HOME_DIR
@@ -196,14 +1228,14 @@ ENV INSTALL_MYSQL_CLIENT=${INSTALL_MYSQL_CLIENT} \
# Only copy mysql/mssql installation scripts for now - so that changing the other
# scripts which are needed much later will not invalidate the docker layer here
-COPY scripts/docker/install_mysql.sh scripts/docker/install_mssql.sh scripts/docker/install_postgres.sh /scripts/docker/
+COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/
RUN bash /scripts/docker/install_mysql.sh dev && \
bash /scripts/docker/install_mssql.sh && \
bash /scripts/docker/install_postgres.sh dev
ENV PATH=${PATH}:/opt/mssql-tools/bin
-COPY docker-context-files /docker-context-files
+COPY ${DOCKER_CONTEXT_FILES} /docker-context-files
RUN adduser --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password \
--quiet "airflow" --uid "${AIRFLOW_UID}" --gid "0" --home "${AIRFLOW_USER_HOME_DIR}" && \
@@ -231,7 +1263,7 @@ ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
AIRFLOW_BRANCH=${AIRFLOW_BRANCH} \
AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS} \
CONSTRAINTS_GITHUB_REPOSITORY=${CONSTRAINTS_GITHUB_REPOSITORY} \
- AIRFLOW_CONSTRAINTS=${AIRFLOW_CONSTRAINTS} \
+ AIRFLOW_CONSTRAINTS_MODE=${AIRFLOW_CONSTRAINTS_MODE} \
AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE} \
AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} \
DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} \
@@ -248,9 +1280,8 @@ ENV AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
# Copy all scripts required for installation - changing any of those should lead to
# rebuilding from here
-COPY --chown=airflow:0 scripts/docker/common.sh scripts/docker/install_pip_version.sh \
- /scripts/docker/install_airflow_dependencies_from_branch_tip.sh \
- /scripts/docker/
+COPY --from=scripts common.sh install_pip_version.sh \
+ install_airflow_dependencies_from_branch_tip.sh /scripts/docker/
# In case of Production build image segment we want to pre-install main version of airflow
# dependencies from GitHub so that we do not have to always reinstall it from the scratch.
@@ -265,7 +1296,7 @@ RUN bash /scripts/docker/install_pip_version.sh; \
bash /scripts/docker/install_airflow_dependencies_from_branch_tip.sh; \
fi
-COPY --chown=airflow:0 scripts/docker/compile_www_assets.sh scripts/docker/prepare_node_modules.sh /scripts/docker/
+COPY --from=scripts compile_www_assets.sh prepare_node_modules.sh /scripts/docker/
COPY --chown=airflow:0 ${AIRFLOW_SOURCES_WWW_FROM} ${AIRFLOW_SOURCES_WWW_TO}
# hadolint ignore=SC2086, SC2010
@@ -290,33 +1321,31 @@ RUN if [[ ${AIRFLOW_INSTALLATION_METHOD} == "." ]]; then \
ARG ADDITIONAL_PYTHON_DEPS=""
# We can set this value to true in case we want to install .whl .tar.gz packages placed in the
# docker-context-files folder. This can be done for both - additional packages you want to install
-# and for airflow as well (you have to set INSTALL_FROM_PYPI to false in this case)
-ARG INSTALL_FROM_DOCKER_CONTEXT_FILES=""
-# By default we install latest airflow from PyPI. You can set it to false if you want to install
-# Airflow from the .whl or .tar.gz packages placed in `docker-context-files` folder.
-ARG INSTALL_FROM_PYPI="true"
+# and for airflow as well (you have to set AIRFLOW_IS_IN_CONTEXT to true in this case)
+ARG INSTALL_PACKAGES_FROM_CONTEXT="false"
+# By default we install latest airflow from PyPI or sources. You can set this parameter to false
+# if Airflow is in the .whl or .tar.gz packages placed in `docker-context-files` folder and you want
+# to skip installing Airflow/Providers from PyPI or sources.
+ARG AIRFLOW_IS_IN_CONTEXT="false"
# Those are additional constraints that are needed for some extras but we do not want to
# Force them on the main Airflow package.
-# * certifi<2021.0.0 required to keep snowflake happy
# * dill<0.3.3 required by apache-beam
-# * google-ads<14.0.1 required to prevent updating google-python-api>=2.0.0
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="dill<0.3.3 certifi<2021.0.0 google-ads<14.0.1"
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="dill<0.3.3"
ENV ADDITIONAL_PYTHON_DEPS=${ADDITIONAL_PYTHON_DEPS} \
- INSTALL_FROM_DOCKER_CONTEXT_FILES=${INSTALL_FROM_DOCKER_CONTEXT_FILES} \
- INSTALL_FROM_PYPI=${INSTALL_FROM_PYPI} \
+ INSTALL_PACKAGES_FROM_CONTEXT=${INSTALL_PACKAGES_FROM_CONTEXT} \
+ AIRFLOW_IS_IN_CONTEXT=${AIRFLOW_IS_IN_CONTEXT} \
EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
WORKDIR /opt/airflow
-COPY --chown=airflow:0 scripts/docker/install_from_docker_context_files.sh scripts/docker/install_airflow.sh \
- scripts/docker/install_additional_dependencies.sh \
- /scripts/docker/
+COPY --from=scripts install_from_docker_context_files.sh install_airflow.sh \
+ install_additional_dependencies.sh /scripts/docker/
# hadolint ignore=SC2086, SC2010
-RUN if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then \
+RUN if [[ ${INSTALL_PACKAGES_FROM_CONTEXT} == "true" ]]; then \
bash /scripts/docker/install_from_docker_context_files.sh; \
- elif [[ ${INSTALL_FROM_PYPI} == "true" ]]; then \
+ elif [[ ${AIRFLOW_IS_IN_CONTEXT} == "false" ]]; then \
bash /scripts/docker/install_airflow.sh; \
fi; \
if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then \
@@ -403,7 +1432,7 @@ ENV RUNTIME_APT_DEPS=${RUNTIME_APT_DEPS} \
GUNICORN_CMD_ARGS="--worker-tmp-dir /dev/shm" \
AIRFLOW_INSTALLATION_METHOD=${AIRFLOW_INSTALLATION_METHOD}
-COPY scripts/docker/determine_debian_version_specific_variables.sh /scripts/docker/
+COPY --from=scripts determine_debian_version_specific_variables.sh /scripts/docker/
# Install basic and additional apt dependencies
RUN apt-get update \
@@ -439,7 +1468,7 @@ ENV PATH="${AIRFLOW_USER_HOME_DIR}/.local/bin:${PATH}" \
# Only copy mysql/mssql installation scripts for now - so that changing the other
# scripts which are needed much later will not invalidate the docker layer here.
-COPY scripts/docker/install_mysql.sh /scripts/docker/install_mssql.sh /scripts/docker/install_postgres.sh /scripts/docker/
+COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/
# We run scripts with bash here to make sure we can execute the scripts. Changing to +x might have an
# unexpected result - the cache for Dockerfiles might get invalidated in case the host system
# had different umask set and group x bit was not set. In Azure the bit might be not set at all.
@@ -459,10 +1488,11 @@ RUN bash /scripts/docker/install_mysql.sh prod \
&& find "${AIRFLOW_HOME}" -executable -print0 | xargs --null chmod g+x \
&& find "${AIRFLOW_USER_HOME_DIR}" -executable -print0 | xargs --null chmod g+x
-COPY --chown=airflow:0 --from=airflow-build-image \
+COPY --from=airflow-build-image --chown=airflow:0 \
"${AIRFLOW_USER_HOME_DIR}/.local" "${AIRFLOW_USER_HOME_DIR}/.local"
-COPY --chown=airflow:0 scripts/in_container/prod/entrypoint_prod.sh /entrypoint
-COPY --chown=airflow:0 scripts/in_container/prod/clean-logs.sh /clean-logs
+COPY --from=scripts entrypoint_prod.sh /entrypoint
+COPY --from=scripts clean-logs.sh /clean-logs
+COPY --from=scripts airflow-scheduler-autorestart.sh /airflow-scheduler-autorestart
# Make /etc/passwd root-group-writeable so that user can be dynamically added by OpenShift
# See https://github.com/apache/airflow/issues/9248
@@ -491,7 +1521,7 @@ ENV DUMB_INIT_SETSID="1" \
# Add protection against running pip as root user
RUN mkdir -pv /root/bin
-COPY scripts/docker/pip /root/bin/pip
+COPY --from=scripts pip /root/bin/pip
RUN chmod u+x /root/bin/pip
WORKDIR ${AIRFLOW_HOME}
@@ -528,9 +1558,6 @@ LABEL org.apache.airflow.distro="debian" \
org.opencontainers.image.licenses="Apache-2.0" \
org.opencontainers.image.ref.name="airflow" \
org.opencontainers.image.title="Production Airflow Image" \
- org.opencontainers.image.description="Reference, production-ready Apache Airflow image" \
- io.artifacthub.package.license='Apache-2.0' \
- io.artifacthub.package.readme-url='${AIRFLOW_IMAGE_README_URL}'
-
+ org.opencontainers.image.description="Reference, production-ready Apache Airflow image"
ENTRYPOINT ["/usr/bin/dumb-init", "--", "/entrypoint"]
CMD []
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 18fb20feefb36..c089544f244b3 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -1,3 +1,4 @@
+# syntax=docker/dockerfile:1.4
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -16,6 +17,997 @@
# WARNING: THIS DOCKERFILE IS NOT INTENDED FOR PRODUCTION USE OR DEPLOYMENT.
#
ARG PYTHON_BASE_IMAGE="python:3.7-slim-bullseye"
+
+##############################################################################################
+# This is the script image where we keep all inlined bash scripts needed in other segments
+# We use PYTHON_BASE_IMAGE to make sure that the scripts are different for different platforms.
+##############################################################################################
+FROM ${PYTHON_BASE_IMAGE} as scripts
+
+##############################################################################################
+# Please DO NOT modify the inlined scripts manually. The content of those files will be
+# replaced by pre-commit automatically from the "scripts/docker/" folder.
+# This is done in order to avoid problems with caching and file permissions and in order to
+# make the PROD Dockerfile standalone
+##############################################################################################
+
+# The content below is automatically copied from scripts/docker/determine_debian_version_specific_variables.sh
+COPY <<"EOF" /determine_debian_version_specific_variables.sh
+function determine_debian_version_specific_variables() {
+ local color_red
+ color_red=$'\e[31m'
+ local color_reset
+ color_reset=$'\e[0m'
+
+ local debian_version
+ debian_version=$(lsb_release -cs)
+ if [[ ${debian_version} == "buster" ]]; then
+ export DISTRO_LIBENCHANT="libenchant-dev"
+ export DISTRO_LIBGCC="libgcc-8-dev"
+ export DISTRO_SELINUX="python-selinux"
+ export DISTRO_LIBFFI="libffi6"
+ # Note missing man directories on debian-buster
+ # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199
+ mkdir -pv /usr/share/man/man1
+ mkdir -pv /usr/share/man/man7
+ elif [[ ${debian_version} == "bullseye" ]]; then
+ export DISTRO_LIBENCHANT="libenchant-2-2"
+ export DISTRO_LIBGCC="libgcc-10-dev"
+ export DISTRO_SELINUX="python3-selinux"
+ export DISTRO_LIBFFI="libffi7"
+ else
+ echo
+ echo "${color_red}Unknown distro version ${debian_version}${color_reset}"
+ echo
+ exit 1
+ fi
+}
+
+determine_debian_version_specific_variables
+EOF
+
+# The content below is automatically copied from scripts/docker/install_mysql.sh
+COPY <<"EOF" /install_mysql.sh
+set -euo pipefail
+declare -a packages
+
+MYSQL_VERSION="8.0"
+readonly MYSQL_VERSION
+
+COLOR_BLUE=$'\e[34m'
+readonly COLOR_BLUE
+COLOR_RESET=$'\e[0m'
+readonly COLOR_RESET
+
+: "${INSTALL_MYSQL_CLIENT:?Should be true or false}"
+
+install_mysql_client() {
+ if [[ "${1}" == "dev" ]]; then
+ packages=("libmysqlclient-dev" "mysql-client")
+ elif [[ "${1}" == "prod" ]]; then
+ packages=("libmysqlclient21" "mysql-client")
+ else
+ echo
+ echo "Specify either prod or dev"
+ echo
+ exit 1
+ fi
+
+ echo
+ echo "${COLOR_BLUE}Installing mysql client version ${MYSQL_VERSION}: ${1}${COLOR_RESET}"
+ echo
+
+ local key="467B942D3A79BD29"
+ readonly key
+
+ GNUPGHOME="$(mktemp -d)"
+ export GNUPGHOME
+ set +e
+ for keyserver in $(shuf -e ha.pool.sks-keyservers.net hkp://p80.pool.sks-keyservers.net:80 \
+ keyserver.ubuntu.com hkp://keyserver.ubuntu.com:80)
+ do
+ gpg --keyserver "${keyserver}" --recv-keys "${key}" 2>&1 && break
+ done
+ set -e
+ gpg --export "${key}" > /etc/apt/trusted.gpg.d/mysql.gpg
+ gpgconf --kill all
+ rm -rf "${GNUPGHOME}"
+ unset GNUPGHOME
+ echo "deb http://repo.mysql.com/apt/debian/ $(lsb_release -cs) mysql-${MYSQL_VERSION}" > /etc/apt/sources.list.d/mysql.list
+ apt-get update
+ apt-get install --no-install-recommends -y "${packages[@]}"
+ apt-get autoremove -yqq --purge
+ apt-get clean && rm -rf /var/lib/apt/lists/*
+}
+
+if [[ $(uname -m) == "arm64" || $(uname -m) == "aarch64" ]]; then
+ # disable MYSQL for ARM64
+ INSTALL_MYSQL_CLIENT="false"
+fi
+
+if [[ ${INSTALL_MYSQL_CLIENT:="true"} == "true" ]]; then
+ install_mysql_client "${@}"
+fi
+EOF
+
+# The content below is automatically copied from scripts/docker/install_mssql.sh
+COPY <<"EOF" /install_mssql.sh
+set -euo pipefail
+
+: "${INSTALL_MSSQL_CLIENT:?Should be true or false}"
+
+COLOR_BLUE=$'\e[34m'
+readonly COLOR_BLUE
+COLOR_YELLOW=$'\e[33m'
+readonly COLOR_YELLOW
+COLOR_RESET=$'\e[0m'
+readonly COLOR_RESET
+
+function install_mssql_client() {
+ # Install MsSQL client from Microsoft repositories
+ if [[ ${INSTALL_MSSQL_CLIENT:="true"} != "true" ]]; then
+ echo
+ echo "${COLOR_BLUE}Skip installing mssql client${COLOR_RESET}"
+ echo
+ return
+ fi
+ echo
+ echo "${COLOR_BLUE}Installing mssql client${COLOR_RESET}"
+ echo
+ local distro
+ local version
+ distro=$(lsb_release -is | tr '[:upper:]' '[:lower:]')
+ version_name=$(lsb_release -cs | tr '[:upper:]' '[:lower:]')
+ version=$(lsb_release -rs)
+ local driver
+ if [[ ${version_name} == "buster" ]]; then
+ driver=msodbcsql17
+ elif [[ ${version_name} == "bullseye" ]]; then
+ driver=msodbcsql18
+ else
+ echo
+ echo "${COLOR_YELLOW}Only Buster or Bullseye are supported. Skipping MSSQL installation${COLOR_RESET}"
+ echo
+ return
+ fi
+ curl --silent https://packages.microsoft.com/keys/microsoft.asc | apt-key add - >/dev/null 2>&1
+ curl --silent "https://packages.microsoft.com/config/${distro}/${version}/prod.list" > \
+ /etc/apt/sources.list.d/mssql-release.list
+ apt-get update -yqq
+ apt-get upgrade -yqq
+ ACCEPT_EULA=Y apt-get -yqq install -y --no-install-recommends "${driver}"
+ rm -rf /var/lib/apt/lists/*
+ apt-get autoremove -yqq --purge
+ apt-get clean && rm -rf /var/lib/apt/lists/*
+}
+
+if [[ $(uname -m) == "arm64" || $(uname -m) == "aarch64" ]]; then
+ # disable MSSQL for ARM64
+ INSTALL_MSSQL_CLIENT="false"
+fi
+
+install_mssql_client "${@}"
+EOF
+
+# The content below is automatically copied from scripts/docker/install_postgres.sh
+COPY <<"EOF" /install_postgres.sh
+set -euo pipefail
+declare -a packages
+
+COLOR_BLUE=$'\e[34m'
+readonly COLOR_BLUE
+COLOR_RESET=$'\e[0m'
+readonly COLOR_RESET
+
+: "${INSTALL_POSTGRES_CLIENT:?Should be true or false}"
+
+install_postgres_client() {
+ echo
+ echo "${COLOR_BLUE}Installing postgres client${COLOR_RESET}"
+ echo
+
+ if [[ "${1}" == "dev" ]]; then
+ packages=("libpq-dev" "postgresql-client")
+ elif [[ "${1}" == "prod" ]]; then
+ packages=("postgresql-client")
+ else
+ echo
+ echo "Specify either prod or dev"
+ echo
+ exit 1
+ fi
+
+ curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
+ echo "deb https://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list
+ apt-get update
+ apt-get install --no-install-recommends -y "${packages[@]}"
+ apt-get autoremove -yqq --purge
+ apt-get clean && rm -rf /var/lib/apt/lists/*
+}
+
+if [[ ${INSTALL_POSTGRES_CLIENT:="true"} == "true" ]]; then
+ install_postgres_client "${@}"
+fi
+EOF
+
+# The content below is automatically copied from scripts/docker/install_pip_version.sh
+COPY <<"EOF" /install_pip_version.sh
+. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+
+: "${AIRFLOW_PIP_VERSION:?Should be set}"
+
+function install_pip_version() {
+ echo
+ echo "${COLOR_BLUE}Installing pip version ${AIRFLOW_PIP_VERSION}${COLOR_RESET}"
+ echo
+ pip install --disable-pip-version-check --no-cache-dir --upgrade "pip==${AIRFLOW_PIP_VERSION}" &&
+ mkdir -p ${HOME}/.local/bin
+}
+
+common::get_colors
+common::get_airflow_version_specification
+common::override_pip_version_if_needed
+common::show_pip_version_and_location
+
+install_pip_version
+EOF
+
+# The content below is automatically copied from scripts/docker/install_airflow_dependencies_from_branch_tip.sh
+COPY <<"EOF" /install_airflow_dependencies_from_branch_tip.sh
+
+. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+
+: "${AIRFLOW_REPO:?Should be set}"
+: "${AIRFLOW_BRANCH:?Should be set}"
+: "${INSTALL_MYSQL_CLIENT:?Should be true or false}"
+: "${INSTALL_POSTGRES_CLIENT:?Should be true or false}"
+: "${AIRFLOW_PIP_VERSION:?Should be set}"
+
+function install_airflow_dependencies_from_branch_tip() {
+ echo
+ echo "${COLOR_BLUE}Installing airflow from ${AIRFLOW_BRANCH}. It is used to cache dependencies${COLOR_RESET}"
+ echo
+ if [[ ${INSTALL_MYSQL_CLIENT} != "true" ]]; then
+ AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/mysql,}
+ fi
+ if [[ ${INSTALL_POSTGRES_CLIENT} != "true" ]]; then
+ AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/postgres,}
+ fi
+ # Install latest set of dependencies using constraints. In case constraints were upgraded and there
+ # are conflicts, this might fail, but it should be fixed in the following installation steps
+ set -x
+ pip install --root-user-action ignore \
+ "https://github.com/${AIRFLOW_REPO}/archive/${AIRFLOW_BRANCH}.tar.gz#egg=apache-airflow[${AIRFLOW_EXTRAS}]" \
+ --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}" || true
+ # make sure correct PIP version is used
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ pip freeze | grep apache-airflow-providers | xargs pip uninstall --yes 2>/dev/null || true
+ set +x
+ echo
+ echo "${COLOR_BLUE}Uninstalling just airflow. Dependencies remain. Now target airflow can be reinstalled using mostly cached dependencies${COLOR_RESET}"
+ echo
+ pip uninstall --yes apache-airflow || true
+}
+
+common::get_colors
+common::get_airflow_version_specification
+common::override_pip_version_if_needed
+common::get_constraints_location
+common::show_pip_version_and_location
+
+install_airflow_dependencies_from_branch_tip
+EOF
+
+# The content below is automatically copied from scripts/docker/common.sh
+COPY <<"EOF" /common.sh
+set -euo pipefail
+
+function common::get_colors() {
+ COLOR_BLUE=$'\e[34m'
+ COLOR_GREEN=$'\e[32m'
+ COLOR_RED=$'\e[31m'
+ COLOR_RESET=$'\e[0m'
+ COLOR_YELLOW=$'\e[33m'
+ export COLOR_BLUE
+ export COLOR_GREEN
+ export COLOR_RED
+ export COLOR_RESET
+ export COLOR_YELLOW
+}
+
+
+function common::get_airflow_version_specification() {
+ if [[ -z ${AIRFLOW_VERSION_SPECIFICATION=}
+ && -n ${AIRFLOW_VERSION}
+ && ${AIRFLOW_INSTALLATION_METHOD} != "." ]]; then
+ AIRFLOW_VERSION_SPECIFICATION="==${AIRFLOW_VERSION}"
+ fi
+}
+
+function common::override_pip_version_if_needed() {
+ if [[ -n ${AIRFLOW_VERSION} ]]; then
+ if [[ ${AIRFLOW_VERSION} =~ ^2\.0.* || ${AIRFLOW_VERSION} =~ ^1\.* ]]; then
+ export AIRFLOW_PIP_VERSION="22.1.2"
+ fi
+ fi
+}
+
+function common::get_constraints_location() {
+ # auto-detect Airflow-constraint reference and location
+ if [[ -z "${AIRFLOW_CONSTRAINTS_REFERENCE=}" ]]; then
+ if [[ ${AIRFLOW_VERSION} =~ v?2.* && ! ${AIRFLOW_VERSION} =~ .*dev.* ]]; then
+ AIRFLOW_CONSTRAINTS_REFERENCE=constraints-${AIRFLOW_VERSION}
+ else
+ AIRFLOW_CONSTRAINTS_REFERENCE=${DEFAULT_CONSTRAINTS_BRANCH}
+ fi
+ fi
+
+ if [[ -z ${AIRFLOW_CONSTRAINTS_LOCATION=} ]]; then
+ local constraints_base="https://raw.githubusercontent.com/${CONSTRAINTS_GITHUB_REPOSITORY}/${AIRFLOW_CONSTRAINTS_REFERENCE}"
+ local python_version
+ python_version="$(python --version 2>/dev/stdout | cut -d " " -f 2 | cut -d "." -f 1-2)"
+ AIRFLOW_CONSTRAINTS_LOCATION="${constraints_base}/${AIRFLOW_CONSTRAINTS_MODE}-${python_version}.txt"
+ fi
+}
+
+function common::show_pip_version_and_location() {
+ echo "PATH=${PATH}"
+ echo "pip on path: $(which pip)"
+ echo "Using pip: $(pip --version)"
+}
+EOF
+
+# The content below is automatically copied from scripts/docker/install_pipx_tools.sh
+COPY <<"EOF" /install_pipx_tools.sh
+. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+
+function install_pipx_tools() {
+ echo
+ echo "${COLOR_BLUE}Installing pipx tools${COLOR_RESET}"
+ echo
+ # Make sure PIPX is installed in latest version
+ pip install --root-user-action ignore --upgrade pipx
+ if [[ $(uname -m) != "aarch64" ]]; then
+ # Do not install mssql-cli for ARM
+ # Install all the tools we need available in command line but without impacting the current environment
+ pipx install mssql-cli
+
+ # Unfortunately mssql-cli installed by `pipx` does not work out of the box because it uses
+ # its own execution bash script which is not compliant with the auto-activation of
+ # pipx venvs - we need to manually patch Python executable in the script to fix it: ¯\_(ツ)_/¯
+ sed "s/python /\/root\/\.local\/pipx\/venvs\/mssql-cli\/bin\/python /" -i /root/.local/bin/mssql-cli
+ fi
+}
+
+common::get_colors
+
+install_pipx_tools
+EOF
+
+# The content below is automatically copied from scripts/docker/prepare_node_modules.sh
+COPY <<"EOF" /prepare_node_modules.sh
+set -euo pipefail
+
+COLOR_BLUE=$'\e[34m'
+readonly COLOR_BLUE
+COLOR_RESET=$'\e[0m'
+readonly COLOR_RESET
+
+function prepare_node_modules() {
+ echo
+ echo "${COLOR_BLUE}Preparing node modules${COLOR_RESET}"
+ echo
+ local www_dir
+ if [[ ${AIRFLOW_INSTALLATION_METHOD=} == "." ]]; then
+ # In case we are building from sources in production image, we should build the assets
+ www_dir="${AIRFLOW_SOURCES_TO=${AIRFLOW_SOURCES}}/airflow/www"
+ else
+ www_dir="$(python -m site --user-site)/airflow/www"
+ fi
+ pushd ${www_dir} || exit 1
+ set +e
+ yarn install --frozen-lockfile --no-cache 2>/tmp/out-yarn-install.txt
+ local res=$?
+ if [[ ${res} != 0 ]]; then
+ >&2 echo
+ >&2 echo "Error when running yarn install:"
+ >&2 echo
+ >&2 cat /tmp/out-yarn-install.txt && rm -f /tmp/out-yarn-install.txt
+ exit 1
+ fi
+ rm -f /tmp/out-yarn-install.txt
+ popd || exit 1
+}
+
+prepare_node_modules
+EOF
+
+# The content below is automatically copied from scripts/docker/compile_www_assets.sh
+COPY <<"EOF" /compile_www_assets.sh
+set -euo pipefail
+
+BUILD_TYPE=${BUILD_TYPE="prod"}
+REMOVE_ARTIFACTS=${REMOVE_ARTIFACTS="true"}
+
+COLOR_BLUE=$'\e[34m'
+readonly COLOR_BLUE
+COLOR_RESET=$'\e[0m'
+readonly COLOR_RESET
+
+function compile_www_assets() {
+ echo
+ echo "${COLOR_BLUE}Compiling www assets: running yarn ${BUILD_TYPE}${COLOR_RESET}"
+ echo
+ local www_dir
+ if [[ ${AIRFLOW_INSTALLATION_METHOD=} == "." ]]; then
+ # In case we are building from sources in production image, we should build the assets
+ www_dir="${AIRFLOW_SOURCES_TO=${AIRFLOW_SOURCES}}/airflow/www"
+ else
+ www_dir="$(python -m site --user-site)/airflow/www"
+ fi
+ pushd ${www_dir} || exit 1
+ set +e
+ yarn run "${BUILD_TYPE}" 2>/tmp/out-yarn-run.txt
+ res=$?
+ if [[ ${res} != 0 ]]; then
+ >&2 echo
+ >&2 echo "Error when running yarn run:"
+ >&2 echo
+ >&2 cat /tmp/out-yarn-run.txt && rm -rf /tmp/out-yarn-run.txt
+ exit 1
+ fi
+ rm -f /tmp/out-yarn-run.txt
+ set -e
+ local md5sum_file
+ md5sum_file="static/dist/sum.md5"
+ readonly md5sum_file
+ find package.json yarn.lock static/css static/js -type f | sort | xargs md5sum > "${md5sum_file}"
+ if [[ ${REMOVE_ARTIFACTS} == "true" ]]; then
+ echo
+ echo "${COLOR_BLUE}Removing generated node modules${COLOR_RESET}"
+ echo
+ rm -rf "${www_dir}/node_modules"
+ rm -vf "${www_dir}"/{package.json,yarn.lock,.eslintignore,.eslintrc,.stylelintignore,.stylelintrc,compile_assets.sh,webpack.config.js}
+ else
+ echo
+ echo "${COLOR_BLUE}Leaving generated node modules${COLOR_RESET}"
+ echo
+ fi
+ popd || exit 1
+}
+
+compile_www_assets
+EOF
+
+# The content below is automatically copied from scripts/docker/install_airflow.sh
+COPY <<"EOF" /install_airflow.sh
+
+. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+
+: "${AIRFLOW_PIP_VERSION:?Should be set}"
+
+function install_airflow() {
+ # Coherence check for editable installation mode.
+ if [[ ${AIRFLOW_INSTALLATION_METHOD} != "." && \
+ ${AIRFLOW_INSTALL_EDITABLE_FLAG} == "--editable" ]]; then
+ echo
+ echo "${COLOR_RED}ERROR! You can only use --editable flag when installing airflow from sources!${COLOR_RESET}"
+ echo "${COLOR_RED} Current installation method is '${AIRFLOW_INSTALLATION_METHOD} and should be '.'${COLOR_RESET}"
+ exit 1
+ fi
+ # Remove mysql from extras if client is not going to be installed
+ if [[ ${INSTALL_MYSQL_CLIENT} != "true" ]]; then
+ AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/mysql,}
+ echo "${COLOR_YELLOW}MYSQL client installation is disabled. Extra 'mysql' installations were therefore omitted.${COLOR_RESET}"
+ fi
+ # Remove postgres from extras if client is not going to be installed
+ if [[ ${INSTALL_POSTGRES_CLIENT} != "true" ]]; then
+ AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS/postgres,}
+ echo "${COLOR_YELLOW}Postgres client installation is disabled. Extra 'postgres' installations were therefore omitted.${COLOR_RESET}"
+ fi
+ if [[ "${UPGRADE_TO_NEWER_DEPENDENCIES}" != "false" ]]; then
+ echo
+ echo "${COLOR_BLUE}Installing all packages with eager upgrade${COLOR_RESET}"
+ echo
+ # eager upgrade
+ pip install --root-user-action ignore --upgrade --upgrade-strategy eager \
+ "${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}" \
+ ${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
+ if [[ -n "${AIRFLOW_INSTALL_EDITABLE_FLAG}" ]]; then
+ # Remove airflow and reinstall it using editable flag
+ # We can only do it when we install airflow from sources
+ set -x
+ pip uninstall apache-airflow --yes
+ pip install --root-user-action ignore ${AIRFLOW_INSTALL_EDITABLE_FLAG} \
+ "${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}"
+ set +x
+ fi
+
+ # make sure correct PIP version is used
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ echo
+ echo "${COLOR_BLUE}Running 'pip check'${COLOR_RESET}"
+ echo
+ pip check
+ else \
+ echo
+ echo "${COLOR_BLUE}Installing all packages with constraints and upgrade if needed${COLOR_RESET}"
+ echo
+ set -x
+ pip install --root-user-action ignore ${AIRFLOW_INSTALL_EDITABLE_FLAG} \
+ "${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}" \
+ --constraint "${AIRFLOW_CONSTRAINTS_LOCATION}"
+ # make sure correct PIP version is used
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ # then upgrade if needed without using constraints to account for new limits in setup.py
+ pip install --root-user-action ignore --upgrade --upgrade-strategy only-if-needed \
+ ${AIRFLOW_INSTALL_EDITABLE_FLAG} \
+ "${AIRFLOW_INSTALLATION_METHOD}[${AIRFLOW_EXTRAS}]${AIRFLOW_VERSION_SPECIFICATION}"
+ # make sure correct PIP version is used
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ set +x
+ echo
+ echo "${COLOR_BLUE}Running 'pip check'${COLOR_RESET}"
+ echo
+ pip check
+ fi
+
+}
+
+common::get_colors
+common::get_airflow_version_specification
+common::override_pip_version_if_needed
+common::get_constraints_location
+common::show_pip_version_and_location
+
+install_airflow
+EOF
+
+# The content below is automatically copied from scripts/docker/install_additional_dependencies.sh
+COPY <<"EOF" /install_additional_dependencies.sh
+set -euo pipefail
+
+: "${UPGRADE_TO_NEWER_DEPENDENCIES:?Should be true or false}"
+: "${ADDITIONAL_PYTHON_DEPS:?Should be set}"
+: "${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS:?Should be set}"
+: "${AIRFLOW_PIP_VERSION:?Should be set}"
+
+. "$( dirname "${BASH_SOURCE[0]}" )/common.sh"
+
+function install_additional_dependencies() {
+ if [[ "${UPGRADE_TO_NEWER_DEPENDENCIES}" != "false" ]]; then
+ echo
+ echo "${COLOR_BLUE}Installing additional dependencies while upgrading to newer dependencies${COLOR_RESET}"
+ echo
+ set -x
+ pip install --root-user-action ignore --upgrade --upgrade-strategy eager \
+ ${ADDITIONAL_PYTHON_DEPS} ${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
+ # make sure correct PIP version is used
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ set +x
+ echo
+ echo "${COLOR_BLUE}Running 'pip check'${COLOR_RESET}"
+ echo
+ pip check
+ else
+ echo
+ echo "${COLOR_BLUE}Installing additional dependencies upgrading only if needed${COLOR_RESET}"
+ echo
+ set -x
+ pip install --root-user-action ignore --upgrade --upgrade-strategy only-if-needed \
+ ${ADDITIONAL_PYTHON_DEPS}
+ # make sure correct PIP version is used
+ pip install --disable-pip-version-check "pip==${AIRFLOW_PIP_VERSION}" 2>/dev/null
+ set +x
+ echo
+ echo "${COLOR_BLUE}Running 'pip check'${COLOR_RESET}"
+ echo
+ pip check
+ fi
+}
+
+common::get_colors
+common::get_airflow_version_specification
+common::override_pip_version_if_needed
+common::get_constraints_location
+common::show_pip_version_and_location
+
+install_additional_dependencies
+EOF
+
+# The content below is automatically copied from scripts/docker/entrypoint_ci.sh
+COPY <<"EOF" /entrypoint_ci.sh
+#!/usr/bin/env bash
+if [[ ${VERBOSE_COMMANDS:="false"} == "true" ]]; then
+ set -x
+fi
+
+. /opt/airflow/scripts/in_container/_in_container_script_init.sh
+
+LD_PRELOAD="/usr/lib/$(uname -m)-linux-gnu/libstdc++.so.6"
+export LD_PRELOAD
+
+chmod 1777 /tmp
+
+AIRFLOW_SOURCES=$(cd "${IN_CONTAINER_DIR}/../.." || exit 1; pwd)
+
+PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:=3.7}
+
+export AIRFLOW_HOME=${AIRFLOW_HOME:=${HOME}}
+
+: "${AIRFLOW_SOURCES:?"ERROR: AIRFLOW_SOURCES not set !!!!"}"
+
+if [[ ${SKIP_ENVIRONMENT_INITIALIZATION=} != "true" ]]; then
+
+ if [[ $(uname -m) == "arm64" || $(uname -m) == "aarch64" ]]; then
+ if [[ ${BACKEND:=} == "mysql" || ${BACKEND} == "mssql" ]]; then
+ echo "${COLOR_RED}ARM platform is not supported for ${BACKEND} backend. Exiting.${COLOR_RESET}"
+ exit 1
+ fi
+ fi
+
+ echo
+ echo "${COLOR_BLUE}Running Initialization. Your basic configuration is:${COLOR_RESET}"
+ echo
+ echo " * ${COLOR_BLUE}Airflow home:${COLOR_RESET} ${AIRFLOW_HOME}"
+ echo " * ${COLOR_BLUE}Airflow sources:${COLOR_RESET} ${AIRFLOW_SOURCES}"
+ echo " * ${COLOR_BLUE}Airflow core SQL connection:${COLOR_RESET} ${AIRFLOW__CORE__SQL_ALCHEMY_CONN:=}"
+ echo
+
+ RUN_TESTS=${RUN_TESTS:="false"}
+ CI=${CI:="false"}
+ USE_AIRFLOW_VERSION="${USE_AIRFLOW_VERSION:=""}"
+
+ if [[ ${USE_AIRFLOW_VERSION} == "" ]]; then
+ export PYTHONPATH=${AIRFLOW_SOURCES}
+ echo
+ echo "${COLOR_BLUE}Using airflow version from current sources${COLOR_RESET}"
+ echo
+ if [[ -d "${AIRFLOW_SOURCES}/airflow/www/" ]]; then
+ pushd "${AIRFLOW_SOURCES}/airflow/www/" >/dev/null
+ ./ask_for_recompile_assets_if_needed.sh
+ popd >/dev/null
+ fi
+ # Cleanup the logs, tmp when entering the environment
+ sudo rm -rf "${AIRFLOW_SOURCES}"/logs/*
+ sudo rm -rf "${AIRFLOW_SOURCES}"/tmp/*
+ mkdir -p "${AIRFLOW_SOURCES}"/logs/
+ mkdir -p "${AIRFLOW_SOURCES}"/tmp/
+ elif [[ ${USE_AIRFLOW_VERSION} == "none" ]]; then
+ echo
+ echo "${COLOR_BLUE}Skip installing airflow - only install wheel/tar.gz packages that are present locally.${COLOR_RESET}"
+ echo
+ echo
+ echo "${COLOR_BLUE}Uninstalling airflow and providers"
+ echo
+ uninstall_airflow_and_providers
+ elif [[ ${USE_AIRFLOW_VERSION} == "wheel" ]]; then
+ echo
+ echo "${COLOR_BLUE}Uninstalling airflow and providers"
+ echo
+ uninstall_airflow_and_providers
+ echo "${COLOR_BLUE}Install airflow from wheel package with extras: '${AIRFLOW_EXTRAS}' and constraints reference ${AIRFLOW_CONSTRAINTS_REFERENCE}.${COLOR_RESET}"
+ echo
+ install_airflow_from_wheel "${AIRFLOW_EXTRAS}" "${AIRFLOW_CONSTRAINTS_REFERENCE}"
+ uninstall_providers
+ elif [[ ${USE_AIRFLOW_VERSION} == "sdist" ]]; then
+ echo
+ echo "${COLOR_BLUE}Uninstalling airflow and providers"
+ echo
+ uninstall_airflow_and_providers
+ echo
+ echo "${COLOR_BLUE}Install airflow from sdist package with extras: '${AIRFLOW_EXTRAS}' and constraints reference ${AIRFLOW_CONSTRAINTS_REFERENCE}.${COLOR_RESET}"
+ echo
+ install_airflow_from_sdist "${AIRFLOW_EXTRAS}" "${AIRFLOW_CONSTRAINTS_REFERENCE}"
+ uninstall_providers
+ else
+ echo
+ echo "${COLOR_BLUE}Uninstalling airflow and providers"
+ echo
+ uninstall_airflow_and_providers
+ echo
+ echo "${COLOR_BLUE}Install released airflow from PyPI with extras: '${AIRFLOW_EXTRAS}' and constraints reference ${AIRFLOW_CONSTRAINTS_REFERENCE}.${COLOR_RESET}"
+ echo
+ install_released_airflow_version "${USE_AIRFLOW_VERSION}" "${AIRFLOW_CONSTRAINTS_REFERENCE}"
+ fi
+ if [[ ${USE_PACKAGES_FROM_DIST=} == "true" ]]; then
+ echo
+ echo "${COLOR_BLUE}Install all packages from dist folder${COLOR_RESET}"
+ if [[ ${USE_AIRFLOW_VERSION} == "wheel" ]]; then
+ echo "${COLOR_BLUE}(except apache-airflow)${COLOR_RESET}"
+ fi
+ if [[ ${PACKAGE_FORMAT} == "both" ]]; then
+ echo
+ echo "${COLOR_RED}ERROR:You can only specify 'wheel' or 'sdist' as PACKAGE_FORMAT not 'both'.${COLOR_RESET}"
+ echo
+ exit 1
+ fi
+ echo
+ installable_files=()
+ for file in /dist/*.{whl,tar.gz}
+ do
+ if [[ ${USE_AIRFLOW_VERSION} == "wheel" && ${file} == "/dist/apache?airflow-[0-9]"* ]]; then
+ # Skip Apache Airflow package - it's just been installed above with extras
+ echo "Skipping ${file}"
+ continue
+ fi
+ if [[ ${PACKAGE_FORMAT} == "wheel" && ${file} == *".whl" ]]; then
+ echo "Adding ${file} to install"
+ installable_files+=( "${file}" )
+ fi
+ if [[ ${PACKAGE_FORMAT} == "sdist" && ${file} == *".tar.gz" ]]; then
+ echo "Adding ${file} to install"
+ installable_files+=( "${file}" )
+ fi
+ done
+ if (( ${#installable_files[@]} )); then
+ pip install --root-user-action ignore "${installable_files[@]}"
+ fi
+ fi
+
+ # Added to have run-tests on path
+ export PATH=${PATH}:${AIRFLOW_SOURCES}
+
+ # This is now set in conftest.py - only for pytest tests
+ unset AIRFLOW__CORE__UNIT_TEST_MODE
+
+ mkdir -pv "${AIRFLOW_HOME}/logs/"
+ cp -f "${IN_CONTAINER_DIR}/airflow_ci.cfg" "${AIRFLOW_HOME}/unittests.cfg"
+
+ # Change the default worker_concurrency for tests
+ export AIRFLOW__CELERY__WORKER_CONCURRENCY=8
+
+ set +e
+
+ "${IN_CONTAINER_DIR}/check_environment.sh"
+ ENVIRONMENT_EXIT_CODE=$?
+ set -e
+ if [[ ${ENVIRONMENT_EXIT_CODE} != 0 ]]; then
+ echo
+ echo "Error: check_environment returned ${ENVIRONMENT_EXIT_CODE}. Exiting."
+ echo
+ exit ${ENVIRONMENT_EXIT_CODE}
+ fi
+ # Create symbolic link to fix possible issues with kubectl config cmd-path
+ mkdir -p /usr/lib/google-cloud-sdk/bin
+ touch /usr/lib/google-cloud-sdk/bin/gcloud
+ ln -s -f /usr/bin/gcloud /usr/lib/google-cloud-sdk/bin/gcloud
+
+ if [[ ${SKIP_SSH_SETUP="false"} == "false" ]]; then
+ # Set up ssh keys
+ echo 'yes' | ssh-keygen -t rsa -C your_email@youremail.com -m PEM -P '' -f ~/.ssh/id_rsa \
+ >"${AIRFLOW_HOME}/logs/ssh-keygen.log" 2>&1
+
+ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
+ ln -s -f ~/.ssh/authorized_keys ~/.ssh/authorized_keys2
+ chmod 600 ~/.ssh/*
+
+ # SSH Service
+ sudo service ssh restart >/dev/null 2>&1
+
+ # Sometimes the server is not quick enough to load the keys!
+ while [[ $(ssh-keyscan -H localhost 2>/dev/null | wc -l) != "3" ]] ; do
+ echo "Not all keys yet loaded by the server"
+ sleep 0.05
+ done
+
+ ssh-keyscan -H localhost >> ~/.ssh/known_hosts 2>/dev/null
+ fi
+
+ # shellcheck source=scripts/in_container/configure_environment.sh
+ . "${IN_CONTAINER_DIR}/configure_environment.sh"
+
+ # shellcheck source=scripts/in_container/run_init_script.sh
+ . "${IN_CONTAINER_DIR}/run_init_script.sh"
+
+ cd "${AIRFLOW_SOURCES}"
+
+ if [[ ${START_AIRFLOW:="false"} == "true" || ${START_AIRFLOW} == "True" ]]; then
+ export AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS=${LOAD_DEFAULT_CONNECTIONS}
+ export AIRFLOW__CORE__LOAD_EXAMPLES=${LOAD_EXAMPLES}
+ # shellcheck source=scripts/in_container/bin/run_tmux
+ exec run_tmux
+ fi
+fi
+
+set +u
+if [[ "${RUN_TESTS}" != "true" ]]; then
+ exec /bin/bash "${@}"
+fi
+set -u
+
+export RESULT_LOG_FILE="/files/test_result-${TEST_TYPE}-${BACKEND}.xml"
+
+EXTRA_PYTEST_ARGS=(
+ "--verbosity=0"
+ "--strict-markers"
+ "--durations=100"
+ "--maxfail=50"
+ "--color=yes"
+ "--junitxml=${RESULT_LOG_FILE}"
+ # timeouts in seconds for individual tests
+ "--timeouts-order"
+ "moi"
+ "--setup-timeout=60"
+ "--execution-timeout=60"
+ "--teardown-timeout=60"
+ # Only display summary for non-expected case
+ # f - failed
+ # E - error
+ # X - xpassed (passed even if expected to fail)
+ # The following cases are not displayed:
+ # s - skipped
+ # x - xfailed (expected to fail and failed)
+ # p - passed
+ # P - passed with output
+ "-rfEX"
+)
+
+if [[ "${TEST_TYPE}" == "Helm" ]]; then
+ # Enable parallelism
+ EXTRA_PYTEST_ARGS+=(
+ "-n" "auto"
+ )
+else
+ EXTRA_PYTEST_ARGS+=(
+ "--with-db-init"
+ )
+fi
+
+if [[ ${ENABLE_TEST_COVERAGE:="false"} == "true" ]]; then
+ EXTRA_PYTEST_ARGS+=(
+ "--cov=airflow/"
+ "--cov-config=.coveragerc"
+ "--cov-report=xml:/files/coverage-${TEST_TYPE}-${BACKEND}.xml"
+ )
+fi
+
+declare -a SELECTED_TESTS CLI_TESTS API_TESTS PROVIDERS_TESTS CORE_TESTS WWW_TESTS \
+ ALL_TESTS ALL_PRESELECTED_TESTS ALL_OTHER_TESTS
+
+function find_all_other_tests() {
+ local all_tests_dirs
+ all_tests_dirs=$(find "tests" -type d)
+ all_tests_dirs=$(echo "${all_tests_dirs}" | sed "/tests$/d" )
+ all_tests_dirs=$(echo "${all_tests_dirs}" | sed "/tests\/dags/d" )
+ local path
+ for path in "${ALL_PRESELECTED_TESTS[@]}"
+ do
+ escaped_path="${path//\//\\\/}"
+ all_tests_dirs=$(echo "${all_tests_dirs}" | sed "/${escaped_path}/d" )
+ done
+ for path in ${all_tests_dirs}
+ do
+ ALL_OTHER_TESTS+=("${path}")
+ done
+}
+
+if [[ ${#@} -gt 0 && -n "$1" ]]; then
+ SELECTED_TESTS=("${@}")
+else
+ CLI_TESTS=("tests/cli")
+ API_TESTS=("tests/api" "tests/api_connexion")
+ PROVIDERS_TESTS=("tests/providers")
+ ALWAYS_TESTS=("tests/always")
+ CORE_TESTS=(
+ "tests/core"
+ "tests/executors"
+ "tests/jobs"
+ "tests/models"
+ "tests/serialization"
+ "tests/ti_deps"
+ "tests/utils"
+ )
+ WWW_TESTS=("tests/www")
+ HELM_CHART_TESTS=("tests/charts")
+ ALL_TESTS=("tests")
+ ALL_PRESELECTED_TESTS=(
+ "${CLI_TESTS[@]}"
+ "${API_TESTS[@]}"
+ "${HELM_CHART_TESTS[@]}"
+ "${PROVIDERS_TESTS[@]}"
+ "${CORE_TESTS[@]}"
+ "${ALWAYS_TESTS[@]}"
+ "${WWW_TESTS[@]}"
+ )
+
+ if [[ ${TEST_TYPE:=""} == "CLI" ]]; then
+ SELECTED_TESTS=("${CLI_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "API" ]]; then
+ SELECTED_TESTS=("${API_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "Providers" ]]; then
+ SELECTED_TESTS=("${PROVIDERS_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "Core" ]]; then
+ SELECTED_TESTS=("${CORE_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "Always" ]]; then
+ SELECTED_TESTS=("${ALWAYS_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "WWW" ]]; then
+ SELECTED_TESTS=("${WWW_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "Helm" ]]; then
+ SELECTED_TESTS=("${HELM_CHART_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "Other" ]]; then
+ find_all_other_tests
+ SELECTED_TESTS=("${ALL_OTHER_TESTS[@]}")
+ elif [[ ${TEST_TYPE:=""} == "All" || ${TEST_TYPE} == "Quarantined" || \
+ ${TEST_TYPE} == "Always" || \
+ ${TEST_TYPE} == "Postgres" || ${TEST_TYPE} == "MySQL" || \
+ ${TEST_TYPE} == "Long" || \
+ ${TEST_TYPE} == "Integration" ]]; then
+ SELECTED_TESTS=("${ALL_TESTS[@]}")
+ else
+ echo
+ echo "${COLOR_RED}ERROR: Wrong test type ${TEST_TYPE} ${COLOR_RESET}"
+ echo
+ exit 1
+ fi
+
+fi
+readonly SELECTED_TESTS CLI_TESTS API_TESTS PROVIDERS_TESTS CORE_TESTS WWW_TESTS \
+ ALL_TESTS ALL_PRESELECTED_TESTS
+
+if [[ -n ${LIST_OF_INTEGRATION_TESTS_TO_RUN=} ]]; then
+ # Integration tests
+ for INT in ${LIST_OF_INTEGRATION_TESTS_TO_RUN}
+ do
+ EXTRA_PYTEST_ARGS+=("--integration" "${INT}")
+ done
+elif [[ ${TEST_TYPE:=""} == "Long" ]]; then
+ EXTRA_PYTEST_ARGS+=(
+ "-m" "long_running"
+ "--include-long-running"
+ )
+elif [[ ${TEST_TYPE:=""} == "Postgres" ]]; then
+ EXTRA_PYTEST_ARGS+=(
+ "--backend"
+ "postgres"
+ )
+elif [[ ${TEST_TYPE:=""} == "MySQL" ]]; then
+ EXTRA_PYTEST_ARGS+=(
+ "--backend"
+ "mysql"
+ )
+elif [[ ${TEST_TYPE:=""} == "Quarantined" ]]; then
+ EXTRA_PYTEST_ARGS+=(
+ "-m" "quarantined"
+ "--include-quarantined"
+ )
+fi
+
+echo
+echo "Running tests ${SELECTED_TESTS[*]}"
+echo
+
+ARGS=("${EXTRA_PYTEST_ARGS[@]}" "${SELECTED_TESTS[@]}")
+
+if [[ ${RUN_SYSTEM_TESTS:="false"} == "true" ]]; then
+ "${IN_CONTAINER_DIR}/run_system_tests.sh" "${ARGS[@]}"
+else
+ "${IN_CONTAINER_DIR}/run_ci_tests.sh" "${ARGS[@]}"
+fi
+EOF
+
+# The content below is automatically copied from scripts/docker/entrypoint_exec.sh
+COPY <<"EOF" /entrypoint_exec.sh
+#!/usr/bin/env bash
+. /opt/airflow/scripts/in_container/_in_container_script_init.sh
+
+. /opt/airflow/scripts/in_container/configure_environment.sh
+
+. /opt/airflow/scripts/in_container/run_init_script.sh
+
+exec /bin/bash "${@}"
+EOF
+
+##############################################################################################
+# This is the www image where we keep all inlined files needed to build ui
+# It is copied separately to volume to speed up building and avoid cache miss on changed
+# file permissions.
+# We use PYTHON_BASE_IMAGE to make sure that the scripts are different for different platforms.
+##############################################################################################
+FROM ${PYTHON_BASE_IMAGE} as www
+COPY airflow/www/package.json airflow/www/yarn.lock airflow/www/webpack.config.js /
+COPY airflow/www/static/ /static
+
FROM ${PYTHON_BASE_IMAGE} as main
# Nolog bash flag is currently ignored - but you can replace it with other flags (for example
@@ -51,7 +1043,7 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
-COPY scripts/docker/determine_debian_version_specific_variables.sh /scripts/docker/
+COPY --from=scripts determine_debian_version_specific_variables.sh /scripts/docker/
# Install basic and additional apt dependencies
RUN apt-get update \
@@ -99,7 +1091,8 @@ RUN apt-get update \
# Only copy mysql/mssql installation scripts for now - so that changing the other
# scripts which are needed much later will not invalidate the docker layer here.
-COPY scripts/docker/install_mysql.sh scripts/docker/install_mssql.sh scripts/docker/install_postgres.sh /scripts/docker/
+COPY --from=scripts install_mysql.sh install_mssql.sh install_postgres.sh /scripts/docker/
+
# We run scripts with bash here to make sure we can execute the scripts. Changing to +x might have an
# unexpected result - the cache for Dockerfiles might get invalidated in case the host system
# had different umask set and group x bit was not set. In Azure the bit might be not set at all.
@@ -192,7 +1185,7 @@ ARG AIRFLOW_EXTRAS="all"
ARG ADDITIONAL_AIRFLOW_EXTRAS=""
# Allows to override constraints source
ARG CONSTRAINTS_GITHUB_REPOSITORY="apache/airflow"
-ARG AIRFLOW_CONSTRAINTS="constraints"
+ARG AIRFLOW_CONSTRAINTS_MODE="constraints-source-providers"
ARG AIRFLOW_CONSTRAINTS_REFERENCE=""
ARG AIRFLOW_CONSTRAINTS_LOCATION=""
ARG DEFAULT_CONSTRAINTS_BRANCH="constraints-main"
@@ -202,8 +1195,7 @@ ARG AIRFLOW_CI_BUILD_EPOCH="3"
ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
# By default in the image, we are installing all providers when installing from sources
ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
-ARG INSTALL_FROM_PYPI="true"
-ARG AIRFLOW_PIP_VERSION=22.0.4
+ARG AIRFLOW_PIP_VERSION=22.1.2
# Setup PIP
# By default PIP install run without cache to make image smaller
ARG PIP_NO_CACHE_DIR="true"
@@ -222,14 +1214,13 @@ ENV AIRFLOW_REPO=${AIRFLOW_REPO}\
AIRFLOW_BRANCH=${AIRFLOW_BRANCH} \
AIRFLOW_EXTRAS=${AIRFLOW_EXTRAS}${ADDITIONAL_AIRFLOW_EXTRAS:+,}${ADDITIONAL_AIRFLOW_EXTRAS} \
CONSTRAINTS_GITHUB_REPOSITORY=${CONSTRAINTS_GITHUB_REPOSITORY} \
- AIRFLOW_CONSTRAINTS=${AIRFLOW_CONSTRAINTS} \
+ AIRFLOW_CONSTRAINTS_MODE=${AIRFLOW_CONSTRAINTS_MODE} \
AIRFLOW_CONSTRAINTS_REFERENCE=${AIRFLOW_CONSTRAINTS_REFERENCE} \
AIRFLOW_CONSTRAINTS_LOCATION=${AIRFLOW_CONSTRAINTS_LOCATION} \
DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH} \
AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH} \
AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES} \
INSTALL_PROVIDERS_FROM_SOURCES=${INSTALL_PROVIDERS_FROM_SOURCES} \
- INSTALL_FROM_PYPI=${INSTALL_FROM_PYPI} \
AIRFLOW_VERSION=${AIRFLOW_VERSION} \
AIRFLOW_PIP_VERSION=${AIRFLOW_PIP_VERSION} \
# In the CI image we always:
@@ -253,19 +1244,16 @@ RUN echo "Airflow version: ${AIRFLOW_VERSION}"
# Those are additional constraints that are needed for some extras but we do not want to
# force them on the main Airflow package. Those limitations are:
-# * certifi<2021.0.0: required by snowflake provider
# * dill<0.3.3 required by apache-beam
-# * google-ads<14.0.1 required to prevent updating google-python-api>=2.0.0
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="dill<0.3.3 certifi<2021.0.0 google-ads<14.0.1"
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="dill<0.3.3"
ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
ENV EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS} \
UPGRADE_TO_NEWER_DEPENDENCIES=${UPGRADE_TO_NEWER_DEPENDENCIES}
# Copy all scripts required for installation - changing any of those should lead to
# rebuilding from here
-COPY scripts/docker/install_pip_version.sh scripts/docker/install_airflow_dependencies_from_branch_tip.sh \
- scripts/docker/common.sh \
- /scripts/docker/
+COPY --from=scripts install_pip_version.sh install_airflow_dependencies_from_branch_tip.sh \
+ common.sh /scripts/docker/
# We are first creating a venv where all python packages and .so binaries needed by those are
# installed.
@@ -276,9 +1264,7 @@ COPY scripts/docker/install_pip_version.sh scripts/docker/install_airflow_depend
# are uninstalled, only dependencies remain.
# the cache is only used when "upgrade to newer dependencies" is not set to automatically
# account for removed dependencies (we do not install them in the first place)
-RUN echo -e "\n\e[32mThe 'Running pip as the root user' warnings below are not valid but we can't disable them :(\e[0m\n"; \
- echo -e "\n\e[34mSee https://github.com/pypa/pip/issues/10556 for details.\e[0m\n" ; \
- bash /scripts/docker/install_pip_version.sh; \
+RUN bash /scripts/docker/install_pip_version.sh; \
if [[ ${AIRFLOW_PRE_CACHED_PIP_PACKAGES} == "true" && \
${UPGRADE_TO_NEWER_DEPENDENCIES} == "false" ]]; then \
bash /scripts/docker/install_airflow_dependencies_from_branch_tip.sh; \
@@ -287,7 +1273,7 @@ RUN echo -e "\n\e[32mThe 'Running pip as the root user' warnings below are not v
# The PATH is needed for PIPX to find the tools installed
ENV PATH="/root/.local/bin:${PATH}"
-COPY scripts/docker/install_pipx_tools.sh /scripts/docker/
+COPY --from=scripts install_pipx_tools.sh /scripts/docker/
# Install useful command line tools in their own virtualenv so that they do not clash with
# dependencies installed in Airflow
@@ -296,17 +1282,17 @@ RUN bash /scripts/docker/install_pipx_tools.sh
# Copy package.json and yarn.lock to install node modules
# this way even if other static check files change, node modules will not need to be installed
# we want to keep node_modules so we can do this step separately from compiling assets
-COPY airflow/www/package.json airflow/www/yarn.lock ${AIRFLOW_SOURCES}/airflow/www/
-COPY scripts/docker/prepare_node_modules.sh /scripts/docker/
+COPY --from=www package.json yarn.lock ${AIRFLOW_SOURCES}/airflow/www/
+COPY --from=scripts prepare_node_modules.sh /scripts/docker/
# Package JS/css for production
RUN bash /scripts/docker/prepare_node_modules.sh
# Copy all the needed www/ for assets compilation. Done as two separate COPY
# commands so as otherwise it copies the _contents_ of static/ in to www/
-COPY airflow/www/webpack.config.js ${AIRFLOW_SOURCES}/airflow/www/
-COPY airflow/www/static ${AIRFLOW_SOURCES}/airflow/www/static/
-COPY scripts/docker/compile_www_assets.sh /scripts/docker/
+COPY --from=www webpack.config.js ${AIRFLOW_SOURCES}/airflow/www/
+COPY --from=www static ${AIRFLOW_SOURCES}/airflow/www/static/
+COPY --from=scripts compile_www_assets.sh /scripts/docker/
# Build artifacts without removing temporary artifacts (we will need them for incremental changes)
# in build mode
@@ -318,9 +1304,9 @@ RUN REMOVE_ARTIFACTS="false" BUILD_TYPE="build" bash /scripts/docker/compile_www
COPY setup.py ${AIRFLOW_SOURCES}/setup.py
COPY setup.cfg ${AIRFLOW_SOURCES}/setup.cfg
-COPY airflow/__init__.py ${AIRFLOW_SOURCES}/airflow/__init__.py
+COPY airflow/__init__.py ${AIRFLOW_SOURCES}/airflow/
-COPY scripts/docker/install_airflow.sh /scripts/docker/
+COPY --from=scripts install_airflow.sh /scripts/docker/
# The goal of this line is to install the dependencies from the most current setup.py from sources
# This will be usually incremental small set of packages in CI optimized build, so it will be very fast
@@ -328,15 +1314,13 @@ COPY scripts/docker/install_airflow.sh /scripts/docker/
# Usually we will install versions based on the dependencies in setup.py and upgraded only if needed.
# But in cron job we will install latest versions matching setup.py to see if there is no breaking change
# and push the constraints if everything is successful
-RUN if [[ ${INSTALL_FROM_PYPI} == "true" ]]; then \
- bash /scripts/docker/install_airflow.sh; \
- fi
-
-COPY scripts/in_container/entrypoint_ci.sh /entrypoint
-RUN chmod a+x /entrypoint
+RUN bash /scripts/docker/install_airflow.sh
-COPY scripts/docker/install_pip_version.sh scripts/docker/install_additional_dependencies.sh /scripts/docker/
+COPY --from=scripts entrypoint_ci.sh /entrypoint
+COPY --from=scripts entrypoint_exec.sh /entrypoint-exec
+RUN chmod a+x /entrypoint /entrypoint-exec
+COPY --from=scripts install_pip_version.sh install_additional_dependencies.sh /scripts/docker/
# Additional python deps to install
ARG ADDITIONAL_PYTHON_DEPS=""
diff --git a/IMAGES.rst b/IMAGES.rst
index 5e31e84191d43..58ef0cca54852 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -85,33 +85,32 @@ You can build the CI image using current sources this command:
.. code-block:: bash
- ./breeze build-image
+ breeze build-image
You can build the PROD image using current sources with this command:
.. code-block:: bash
- ./breeze build-image --production-image
+ breeze build-prod-image
By adding ``--python `` parameter you can build the
image version for the chosen Python version.
The images are build with default extras - different extras for CI and production image and you
can change the extras via the ``--extras`` parameters and add new ones with ``--additional-extras``.
-You can see default extras used via ``./breeze flags``.
For example if you want to build Python 3.7 version of production image with
"all" extras installed you should run this command:
.. code-block:: bash
- ./breeze build-image --python 3.7 --extras "all" --production-image
+ breeze build-prod-image --python 3.7 --extras "all"
If you just want to add new extras you can add them like that:
.. code-block:: bash
- ./breeze build-image --python 3.7 --additional-extras "all" --production-image
+ breeze build-prod-image --python 3.7 --additional-extras "all"
The command that builds the CI image is optimized to minimize the time needed to rebuild the image when
the source code of Airflow evolves. This means that if you already have the image locally downloaded and
@@ -129,8 +128,7 @@ parameter to Breeze:
.. code-block:: bash
- ./breeze build-image --python 3.7 --additional-extras=trino \
- --production-image --install-airflow-version=2.0.0
+ breeze build-prod-image --python 3.7 --additional-extras=trino --install-airflow-version=2.0.0
This will build the image using command similar to:
@@ -138,7 +136,7 @@ This will build the image using command similar to:
pip install \
apache-airflow[async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,microsoft.azure,mysql,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv]==2.0.0 \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.0/constraints-3.6.txt"
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.0/constraints-3.7.txt"
.. note::
@@ -160,15 +158,15 @@ HEAD of development for constraints):
.. code-block:: bash
pip install "https://github.com/apache/airflow/archive/.tar.gz#egg=apache-airflow" \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.6.txt"
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.7.txt"
You can also skip installing airflow and install it from locally provided files by using
-``--install-from-docker-context-files`` parameter and ``--disable-pypi-when-building`` to Breeze:
+``--install-packages-from-context`` parameter to Breeze:
.. code-block:: bash
- ./breeze build-image --python 3.7 --additional-extras=trino \
- --production-image --disable-pypi-when-building --install-from-docker-context-files
+ breeze build-prod-image --python 3.7 --additional-extras=trino \
+ --airflow-is-in-context-pypi --install-packages-from-context
In this case you airflow and all packages (.whl files) should be placed in ``docker-context-files`` folder.
@@ -190,31 +188,37 @@ Dockerfile image= and scripts further rebuilds with local build cache will be co
You can also disable build cache altogether. This is the strategy used by the scheduled builds in CI - they
will always rebuild all the images from scratch.
-You can change the strategy by providing one of the ``--build-cache-local``, ``--build-cache-pulled`` or
-even ``--build-cache-disabled`` flags when you run Breeze commands. For example:
+You can change the strategy by providing one of the ``--build-cache`` flags: ``registry`` (default), ``local``,
+or ``disabled`` flags when you run Breeze commands. For example:
.. code-block:: bash
- ./breeze build-image --python 3.7 --build-cache-local
+ breeze build-image --python 3.7 --docker-cache local
Will build the CI image using local build cache (note that it will take quite a long time the first
time you run it).
.. code-block:: bash
- ./breeze build-image --python 3.7 --production-image --build-cache-pulled
+ breeze build-prod-image --python 3.7 --docker-cache registry
-Will build the production image with pulled images as cache.
+Will build the production image with cache used from registry.
.. code-block:: bash
- ./breeze build-image --python 3.7 --production-image --build-cache-disabled
+ breeze build-prod-image --python 3.7 --docker-cache disabled
Will build the production image from the scratch.
-You can also turn local docker caching by setting ``DOCKER_CACHE`` variable to "local", "pulled",
-"disabled" and exporting it.
+You can also turn local docker caching by setting ``DOCKER_CACHE`` variable to ``local``, ``registry``,
+``disabled`` and exporting it.
+
+.. code-block:: bash
+
+ export DOCKER_CACHE="registry"
+
+or
.. code-block:: bash
@@ -229,7 +233,7 @@ or
Naming conventions
==================
-By default images we are using cache for images in Github Container registry. We are using GitHub
+By default images we are using cache for images in GitHub Container registry. We are using GitHub
Container Registry as development image cache and CI registry for build images.
The images are all in organization wide "apache/" namespace. We are adding "airflow-" as prefix for
the image names of all Airflow images. The images are linked to the repository
@@ -247,13 +251,7 @@ currently run build. They are built once per each build and pulled by each test
ghcr.io/apache/airflow//ci/python: - for CI images
ghcr.io/apache/airflow//prod/python: - for production images
-
-The cache images (pushed when main merge succeeds) are kept with ``cache`` tag:
-
-.. code-block:: bash
-
- ghcr.io/apache/airflow//ci/python:cache - for CI images
- ghcr.io/apache/airflow//prod/python:cache - for production images
+Thoe image contain inlined cache.
You can see all the current GitHub images at ``_
@@ -291,7 +289,7 @@ For example this command will run the same Python 3.8 image as was used in build
.. code-block:: bash
- ./breeze --github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e \
+ ./breeze-legacy --github-image-id 9a621eaa394c0a0a336f8e1b31b35eff4e4ee86e \
--python 3.8 --integration rabbitmq
You can see more details and examples in `Breeze `_
@@ -311,8 +309,9 @@ Here just a few examples are presented which should give you general understandi
This builds the production image in version 3.7 with additional airflow extras from 2.0.0 PyPI package and
additional apt dev and runtime dependencies.
-It is recommended to build images with ``DOCKER_BUILDKIT=1`` variable
-(Breeze sets ``DOCKER_BUILDKIT=1`` variable automatically).
+As of Airflow 2.3.0, it is required to build images with ``DOCKER_BUILDKIT=1`` variable
+(Breeze sets ``DOCKER_BUILDKIT=1`` variable automatically) or via ``docker buildx build`` command if
+you have ``buildx`` plugin installed.
.. code-block:: bash
@@ -330,9 +329,7 @@ the same image can be built using ``breeze`` (it supports auto-completion of the
.. code-block:: bash
- ./breeze build-image -f Dockerfile.ci \
- --production-image --python 3.7 \
- --additional-extras=jdbc --additional-python-deps="pandas" \
+ breeze build-prod-image --python 3.7 --additional-extras=jdbc --additional-python-deps="pandas" \
--additional-dev-apt-deps="gcc g++" --additional-runtime-apt-deps="default-jre-headless"
You can customize more aspects of the image - such as additional commands executed before apt dependencies
@@ -374,7 +371,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
+==========================================+==========================================+==========================================+
| ``PYTHON_BASE_IMAGE`` | ``python:3.7-slim-bullseye`` | Base Python image |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``PYTHON_MAJOR_MINOR_VERSION`` | ``3.6`` | major/minor version of Python (should |
+| ``PYTHON_MAJOR_MINOR_VERSION`` | ``3.7`` | major/minor version of Python (should |
| | | match base image) |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``DEPENDENCIES_EPOCH_NUMBER`` | ``2`` | increasing this number will reinstall |
@@ -427,9 +424,11 @@ The following build arguments (``--build-arg`` in docker build command) can be u
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_EXTRAS`` | ``all`` | extras to install |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``UPGRADE_TO_NEWER_DEPENDENCIES`` | ``false`` | If set to true, the dependencies are |
-| | | upgraded to newer versions matching |
-| | | setup.py before installation. |
+| ``UPGRADE_TO_NEWER_DEPENDENCIES`` | ``false`` | If set to a value different than "false" |
+| | | the dependencies are upgraded to newer |
+| | | versions. In CI it is set to build id |
+| | | to make sure subsequent builds are not |
+| | | reusing cached images with same value. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_PRE_CACHED_PIP_PACKAGES`` | ``true`` | Allows to pre-cache airflow PIP packages |
| | | from the GitHub of Apache Airflow |
@@ -476,7 +475,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
| ``ADDITIONAL_RUNTIME_APT_ENV`` | | Additional env variables defined |
| | | when installing runtime deps |
+------------------------------------------+------------------------------------------+------------------------------------------+
-| ``AIRFLOW_PIP_VERSION`` | ``22.0.4`` | PIP version used. |
+| ``AIRFLOW_PIP_VERSION`` | ``22.1.2`` | PIP version used. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``PIP_PROGRESS_BAR`` | ``on`` | Progress bar for PIP installation |
+------------------------------------------+------------------------------------------+------------------------------------------+
@@ -492,7 +491,7 @@ This builds the CI image in version 3.7 with default extras ("all").
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-bullseye" --tag my-image:0.0.1
-This builds the CI image in version 3.6 with "gcp" extra only.
+This builds the CI image in version 3.7 with "gcp" extra only.
.. code-block:: bash
@@ -502,7 +501,7 @@ This builds the CI image in version 3.6 with "gcp" extra only.
--build-arg AIRFLOW_EXTRAS=gcp --tag my-image:0.0.1
-This builds the CI image in version 3.6 with "apache-beam" extra added.
+This builds the CI image in version 3.7 with "apache-beam" extra added.
.. code-block:: bash
@@ -511,7 +510,7 @@ This builds the CI image in version 3.6 with "apache-beam" extra added.
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-bullseye" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="apache-beam" --tag my-image:0.0.1
-This builds the CI image in version 3.6 with "mssql" additional package added.
+This builds the CI image in version 3.7 with "mssql" additional package added.
.. code-block:: bash
@@ -520,7 +519,7 @@ This builds the CI image in version 3.6 with "mssql" additional package added.
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-bullseye" \
--build-arg ADDITIONAL_PYTHON_DEPS="mssql" --tag my-image:0.0.1
-This builds the CI image in version 3.6 with "gcc" and "g++" additional apt dev dependencies added.
+This builds the CI image in version 3.7 with "gcc" and "g++" additional apt dev dependencies added.
.. code-block::
@@ -529,7 +528,7 @@ This builds the CI image in version 3.6 with "gcc" and "g++" additional apt dev
--build-arg PYTHON_BASE_IMAGE="python:3.7-slim-bullseye" \
--build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++" --tag my-image:0.0.1
-This builds the CI image in version 3.6 with "jdbc" extra and "default-jre-headless" additional apt runtime dependencies added.
+This builds the CI image in version 3.7 with "jdbc" extra and "default-jre-headless" additional apt runtime dependencies added.
.. code-block::
@@ -543,7 +542,7 @@ Running the CI image
--------------------
The entrypoint in the CI image contains all the initialisation needed for tests to be immediately executed.
-It is copied from ``scripts/in_container/entrypoint_ci.sh``.
+It is copied from ``scripts/docker/entrypoint_ci.sh``.
The default behaviour is that you are dropped into bash shell. However if RUN_TESTS variable is
set to "true", then tests passed as arguments are executed
diff --git a/INSTALL b/INSTALL
index 4354d5e33f558..ba2f5500475ee 100644
--- a/INSTALL
+++ b/INSTALL
@@ -46,21 +46,21 @@ python setup.py install
# There are different constraint files for different python versions. For example"
pip install . \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.6.txt"
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.7.txt"
By default `pip install` in Airflow 2.0 installs only the provider packages that are needed by the extras and
install them as packages from PyPI rather than from local sources:
pip install .[google,amazon] \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.6.txt"
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.7.txt"
-You can upgrade just airflow, without paying attention to provider's dependencies by using 'no-providers'
+You can upgrade just airflow, without paying attention to provider's dependencies by using 'constraints-no-providers'
constraint files. This allows you to keep installed provider packages.
pip install . --upgrade \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.6.txt"
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.7.txt"
You can also install airflow in "editable mode" (with -e) flag and then provider packages are
@@ -75,7 +75,7 @@ and in ``CONTRIBUTING.rst`` for developing community maintained providers.
This is useful if you want to develop providers:
pip install -e . \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.6.txt"
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.7.txt"
You can also skip installing provider packages from PyPI by setting INSTALL_PROVIDERS_FROM_SOURCE to "true".
In this case Airflow will be installed in non-editable mode with all providers installed from the sources.
@@ -83,31 +83,29 @@ Additionally `provider.yaml` files will also be copied to providers folders whic
discoverable by Airflow even if they are not installed from packages in this case.
INSTALL_PROVIDERS_FROM_SOURCES="true" pip install . \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.6.txt"
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.7.txt"
Airflow can be installed with extras to install some additional features (for example 'async' or 'doc' or
to install automatically providers and all dependencies needed by that provider:
pip install .[async,google,amazon] \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.6.txt"
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.7.txt"
The list of available extras:
# START EXTRAS HERE
-
airbyte, alibaba, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra, apache.drill,
apache.druid, apache.hdfs, apache.hive, apache.kylin, apache.livy, apache.pig, apache.pinot,
-apache.spark, apache.sqoop, apache.webhdfs, asana, async, atlas, aws, azure, cassandra, celery,
-cgroups, cloudant, cncf.kubernetes, crypto, dask, databricks, datadog, dbt.cloud, deprecated_api,
-devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc, docker, druid, elasticsearch,
-exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google, google_auth, grpc,
-hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, jira, kerberos, kubernetes, ldap,
+apache.spark, apache.sqoop, apache.webhdfs, arangodb, asana, async, atlas, aws, azure, cassandra,
+celery, cgroups, cloudant, cncf.kubernetes, crypto, dask, databricks, datadog, dbt.cloud,
+deprecated_api, devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc, docker, druid,
+elasticsearch, exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google, google_auth,
+grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, jira, kerberos, kubernetes, ldap,
leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo, mssql, mysql,
neo4j, odbc, openfaas, opsgenie, oracle, pagerduty, pandas, papermill, password, pinot, plexus,
postgres, presto, qds, qubole, rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentry,
sftp, singularity, slack, snowflake, spark, sqlite, ssh, statsd, tableau, telegram, trino, vertica,
virtualenv, webhdfs, winrm, yandex, zendesk
-
# END EXTRAS HERE
# For installing Airflow in development environments - see CONTRIBUTING.rst
diff --git a/INTHEWILD.md b/INTHEWILD.md
index e316a648fa734..acf2a4f719564 100644
--- a/INTHEWILD.md
+++ b/INTHEWILD.md
@@ -224,7 +224,7 @@ Currently, **officially** using Airflow:
1. [Headout](https://headout.com/) [[@shivanshs9](https://github.com/shivanshs9/)]
1. [Healthjump](http://www.healthjump.com/) [[@miscbits](https://github.com/miscbits)]
1. [HelloFresh](https://www.hellofresh.com) [[@tammymendt](https://github.com/tammymendt) & [@davidsbatista](https://github.com/davidsbatista) & [@iuriinedostup](https://github.com/iuriinedostup)]
-1. [Hipages](https://www.hipages.com.au/) [[@arihantsurana](https://github.com/arihantsurana)]
+1. [Hipages](https://www.hipages.com.au/) [[@arihantsurana](https://github.com/arihantsurana) & [@koconder](https://github.com/koconder)]
1. [Holimetrix](http://holimetrix.com/) [[@thibault-ketterer](https://github.com/thibault-ketterer)]
1. [HomeToGo](https://www.hometogo.com/) [[@HomeToGo](https://github.com/hometogo), [@AurimasGr](https://github.com/AurimasGr)]
1. [Hootsuite](https://github.com/hootsuite)
@@ -349,6 +349,7 @@ Currently, **officially** using Airflow:
1. [Pronto Tools](http://www.prontotools.io/) [[@zkan](https://github.com/zkan) & [@mesodiar](https://github.com/mesodiar)]
1. [proton.ai](https://proton.ai/) [[@prmsolutions](https://github.com/prmsolutions)]
1. [PubNub](https://pubnub.com) [[@jzucker2](https://github.com/jzucker2)]
+1. [Pura Scents](https://www.trypura.com/) [[@mfjackson](https://github.com/mfjackson)]
1. [PXYData](https://www.pxydata.com) [[@patchus](https://github.com/patchus)]
1. [Qliro](https://www.qliro.com) [[@kvackkvackanka](https://github.com/kvackkvackanka)]
1. [Qoala](https://www.qoala.id) [[@gnomeria](https://github.com/gnomeria), [@qoala-engineering](https://github.com/qoala-engineering)]
@@ -411,7 +412,7 @@ Currently, **officially** using Airflow:
1. [Tesla](https://www.tesla.com/) [[@thoralf-gutierrez](https://github.com/thoralf-gutierrez)]
1. [TextNow](https://www.textnow.com/)
1. [The Climate Corporation](https://climate.com/) [[@jmelching](https://github.com/jmelching)]
-1. [The Dyrt](https://thedyrt.com/) [[@mfjackson](https://github.com/mfjackson)]
+1. [The Dyrt](https://thedyrt.com/)
1. [The Home Depot](https://www.homedepot.com/) [[@apekshithr](https://github.com/apekshithr)]
1. [THE ICONIC](https://www.theiconic.com.au/) [[@revathijay](https://github.com/revathijay), [@ilikedata](https://github.com/ilikedata)]
1. [theScore](https://www.thescore.com/) [[@kristenmalikk](https://github.com/kristenmalikk)]
@@ -464,7 +465,6 @@ Currently, **officially** using Airflow:
1. [WiseBanyan](https://wisebanyan.com/)
1. [Wise](https://wise.com) [[@koszti](https://github.com/koszti)]
1. [Wisr](https://wisr.com.au/) [[@fsodano](https://github.com/fsodano) & [@vincyf1](https://github.com/vincyf1)]
-1. [WixAnswers](https://www.wixanswers.com/) [[@eladkal](https://github.com/eladkal)]
1. [Wix](https://www.wix.com/) [[@eladkal](https://github.com/eladkal)]
1. [Wooga](https://www.wooga.com/)
1. [WorldRemit](https://www.worldremit.com/) [[@boittega](https://github.com/boittega)]
diff --git a/LICENSE b/LICENSE
index 34a85897c882f..81899d0fee17d 100644
--- a/LICENSE
+++ b/LICENSE
@@ -248,6 +248,7 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
(BSD 3 License) d3 v5.16.0 (https://d3js.org)
(BSD 3 License) d3-shape v2.1.0 (https://github.com/d3/d3-shape)
+ (BSD 3 License) cgroupspy 0.2.1 (https://github.com/cloudsigma/cgroupspy)
========================================================================
See licenses/LICENSES-ui.txt for packages used in `/airflow/www`
diff --git a/LOCAL_VIRTUALENV.rst b/LOCAL_VIRTUALENV.rst
index 925651abd64d1..1cbf5b6b12e81 100644
--- a/LOCAL_VIRTUALENV.rst
+++ b/LOCAL_VIRTUALENV.rst
@@ -51,7 +51,7 @@ Required Software Packages
Use system-level package managers like yum, apt-get for Linux, or
Homebrew for macOS to install required software packages:
-* Python (One of: 3.7, 3.8, 3.9)
+* Python (One of: 3.7, 3.8, 3.9, 3.10)
* MySQL 5.7+
* libxml
@@ -102,7 +102,7 @@ Creating a Local virtualenv
To use your IDE for Airflow development and testing, you need to configure a virtual
environment. Ideally you should set up virtualenv for all Python versions that Airflow
-supports (3.7, 3.8, 3.9).
+supports (3.7, 3.8, 3.9, 3.10).
To create and initialize the local virtualenv:
@@ -122,7 +122,7 @@ To create and initialize the local virtualenv:
.. code-block:: bash
- conda create -n airflow python=3.7 # or 3.8, or 3.9
+ conda create -n airflow python=3.7 # or 3.8, 3.9, 3.10
conda activate airflow
2. Install Python PIP requirements:
@@ -150,7 +150,7 @@ for different python versions). For development on current main source:
.. code-block:: bash
- # use the same version of python as you are working with, 3.7, 3.8, or 3.9
+ # use the same version of python as you are working with, 3.7, 3.8, 3.9, or 3.10
pip install -e ".[devel,]" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
@@ -163,7 +163,7 @@ You can also install Airflow in non-editable mode:
.. code-block:: bash
- # use the same version of python as you are working with, 3.7, 3.8, or 3.9
+ # use the same version of python as you are working with, 3.7, 3.8, 3.9, or 3.10
pip install ".[devel,]" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
@@ -173,7 +173,7 @@ sources, unless you set ``INSTALL_PROVIDERS_FROM_SOURCES`` environment variable
.. code-block:: bash
- # use the same version of python as you are working with, 3.7, 3.8, or 3.9
+ # use the same version of python as you are working with, 3.7, 3.8, 3.9, or 3.10
INSTALL_PROVIDERS_FROM_SOURCES="true" pip install ".[devel,]" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.7.txt"
@@ -205,14 +205,14 @@ Activate your virtualenv, e.g. by using ``workon``, and once you are in it, run:
.. code-block:: bash
- ./breeze initialize-local-virtualenv
+ ./breeze-legacy initialize-local-virtualenv
By default Breeze installs the ``devel`` extra only. You can optionally control which extras are installed by exporting ``VIRTUALENV_EXTRAS`` before calling Breeze:
.. code-block:: bash
export VIRTUALENV_EXTRAS="devel,google,postgres"
- ./breeze initialize-local-virtualenv
+ ./breeze-legacy initialize-local-virtualenv
5. (optionally) run yarn build if you plan to run the webserver
diff --git a/MANIFEST.in b/MANIFEST.in
index 8291ad54d432e..8f4b22b7ca01c 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -18,7 +18,7 @@
include NOTICE
include LICENSE
-include CHANGELOG.txt
+include RELEASE_NOTES.rst
include README.md
graft licenses
graft airflow/www
diff --git a/PULL_REQUEST_WORKFLOW.rst b/PULL_REQUEST_WORKFLOW.rst
index e5802b24c41d1..d7ca2f9b93eaa 100644
--- a/PULL_REQUEST_WORKFLOW.rst
+++ b/PULL_REQUEST_WORKFLOW.rst
@@ -57,12 +57,12 @@ We approached the problem by:
3) Even more optimisation came from limiting the scope of tests to only "default" matrix parameters. So far
in Airflow we always run all tests for all matrix combinations. The primary matrix components are:
- * Python versions (currently 3.6, 3.7, 3.8, 3.9)
+ * Python versions (currently 3.7, 3.8, 3.9, 3.10)
* Backend types (currently MySQL/Postgres)
* Backed version (currently MySQL 5.7, MySQL 8, Postgres 13
We've decided that instead of running all the combinations of parameters for all matrix component we will
- only run default values (Python 3.6, Mysql 5.7, Postgres 13) for all PRs which are not approved yet by
+ only run default values (Python 3.7, Mysql 5.7, Postgres 13) for all PRs which are not approved yet by
the committers. This has a nice effect, that full set of tests (though with limited combinations of
the matrix) are still run in the CI for every Pull Request that needs tests at all - allowing the
contributors to make sure that their PR is "good enough" to be reviewed.
diff --git a/README.md b/README.md
index e30361470ee24..3acb883dc4ba8 100644
--- a/README.md
+++ b/README.md
@@ -85,15 +85,15 @@ Airflow is not a streaming solution, but it is often used to process real-time d
Apache Airflow is tested with:
-| | Main version (dev) | Stable version (2.2.4) |
-|---------------------|---------------------|--------------------------|
-| Python | 3.7, 3.8, 3.9 | 3.6, 3.7, 3.8, 3.9 |
-| Platform | AMD64/ARM64(\*) | AMD64 |
-| Kubernetes | 1.20, 1.21 | 1.18, 1.19, 1.20 |
-| PostgreSQL | 10, 11, 12, 13 | 9.6, 10, 11, 12, 13 |
-| MySQL | 5.7, 8 | 5.7, 8 |
-| SQLite | 3.15.0+ | 3.15.0+ |
-| MSSQL | 2017(\*), 2019 (\*) | |
+| | Main version (dev) | Stable version (2.3.1) |
+|---------------------|------------------------------|------------------------------|
+| Python | 3.7, 3.8, 3.9, 3.10 | 3.7, 3.8, 3.9, 3.10 |
+| Platform | AMD64/ARM64(\*) | AMD64/ARM64(\*) |
+| Kubernetes | 1.20, 1.21, 1.22, 1.23, 1.24 | 1.20, 1.21, 1.22, 1.23, 1.24 |
+| PostgreSQL | 10, 11, 12, 13, 14 | 10, 11, 12, 13, 14 |
+| MySQL | 5.7, 8 | 5.7, 8 |
+| SQLite | 3.15.0+ | 3.15.0+ |
+| MSSQL | 2017(\*), 2019 (\*) | 2017(\*), 2019 (\*) |
\* Experimental
@@ -104,7 +104,8 @@ MariaDB is not tested/recommended.
**Note**: SQLite is used in Airflow tests. Do not use it in production. We recommend
using the latest stable version of SQLite for local development.
-**Note**: Python v3.10 is not supported yet. For details, see [#19059](https://github.com/apache/airflow/issues/19059).
+**Note**: Support for Python v3.10 will be available from Airflow 2.3.0. The `main` (development) branch
+already supports Python 3.10.
**Note**: Airflow currently can be run on POSIX-compliant Operating Systems. For development it is regularly
tested on fairly modern Linux Distros and recent versions of MacOS.
@@ -159,15 +160,15 @@ them to the appropriate format and workflow that your tool requires.
```bash
-pip install 'apache-airflow==2.2.4' \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.4/constraints-3.7.txt"
+pip install 'apache-airflow==2.3.1' \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.3.1/constraints-3.7.txt"
```
2. Installing with extras (i.e., postgres, google)
```bash
-pip install 'apache-airflow[postgres,google]==2.2.4' \
- --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.4/constraints-3.7.txt"
+pip install 'apache-airflow[postgres,google]==2.3.1' \
+ --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.3.1/constraints-3.7.txt"
```
For information on installing provider packages, check
@@ -214,9 +215,9 @@ following the ASF Policy.
-- **Tree**: Tree representation of a DAG that spans across time.
+- **Grid**: Grid representation of a DAG that spans across time.
- 
+ 
- **Graph**: Visualization of a DAG's dependencies and their current status for a specific run.
@@ -267,12 +268,12 @@ packages:
Apache Airflow version life cycle:
-
+
| Version | Current Patch/Minor | State | First Release | Limited Support | EOL/Terminated |
|-----------|-----------------------|-----------|-----------------|-------------------|------------------|
-| 2 | 2.2.4 | Supported | Dec 17, 2020 | TBD | TBD |
+| 2 | 2.3.1 | Supported | Dec 17, 2020 | TBD | TBD |
| 1.10 | 1.10.15 | EOL | Aug 27, 2018 | Dec 17, 2020 | June 17, 2021 |
| 1.9 | 1.9.0 | EOL | Jan 03, 2018 | Aug 27, 2018 | Aug 27, 2018 |
| 1.8 | 1.8.2 | EOL | Mar 19, 2017 | Jan 03, 2018 | Jan 03, 2018 |
@@ -292,20 +293,21 @@ They are based on the official release schedule of Python and Kubernetes, nicely
[Python Developer's Guide](https://devguide.python.org/#status-of-python-branches) and
[Kubernetes version skew policy](https://kubernetes.io/docs/setup/release/version-skew-policy/).
-1. We drop support for Python and Kubernetes versions when they reach EOL. We drop support for those
- EOL versions in main right after EOL date, and it is effectively removed when we release the
- first new MINOR (Or MAJOR if there is no new MINOR version) of Airflow
- For example, for Python 3.7 it means that we will drop support in main right after 27.06.2023, and
- the first MAJOR or MINOR version of Airflow released after will not have it.
+1. We drop support for Python and Kubernetes versions when they reach EOL. Except for kubernetes, a
+ version stay supported by Airflow if two major cloud provider still provide support for it. We drop
+ support for those EOL versions in main right after EOL date, and it is effectively removed when we release
+ the first new MINOR (Or MAJOR if there is no new MINOR version) of Airflow. For example, for Python 3.7 it
+ means that we will drop support in main right after 27.06.2023, and the first MAJOR or MINOR version of
+ Airflow released after will not have it.
2. The "oldest" supported version of Python/Kubernetes is the default one until we decide to switch to
later version. "Default" is only meaningful in terms of "smoke tests" in CI PRs, which are run using this
default version and the default reference image available. Currently `apache/airflow:latest`
- and `apache/airflow:2.2.4` images are Python 3.7 images. This means that default reference image will
+ and `apache/airflow:2.3.1` images are Python 3.7 images. This means that default reference image will
become the default at the time when we start preparing for dropping 3.7 support which is few months
before the end of life for Python 3.7.
-4. We support a new version of Python/Kubernetes in main after they are officially released, as soon as we
+3. We support a new version of Python/Kubernetes in main after they are officially released, as soon as we
make them work in our CI pipeline (which might not be immediate due to dependencies catching up with
new versions of Python mostly) we release new images/support in Airflow based on the working CI setup.
@@ -378,6 +380,14 @@ The important dependencies are:
are very likely to introduce breaking changes across those so limiting it to MAJOR version makes sense
* `werkzeug`: the library is known to cause problems in new versions. It is tightly coupled with Flask
libraries, and we should update them together
+* `celery`: Celery is crucial component of Airflow as it used for CeleryExecutor (and similar). Celery
+ [follows SemVer](https://docs.celeryq.dev/en/stable/contributing.html?highlight=semver#versions), so
+ we should upper-bound it to the next MAJOR version. Also when we bump the upper version of the library,
+ we should make sure Celery Provider minimum Airflow version is updated).
+* `kubernetes`: Kubernetes is a crucial component of Airflow as it is used for the KubernetesExecutor
+ (and similar). Kubernetes Python library [follows SemVer](https://github.com/kubernetes-client/python#compatibility),
+ so we should upper-bound it to the next MAJOR version. Also when we bump the upper version of the library,
+ we should make sure Kubernetes Provider minimum Airflow version is updated.
### Approach for dependencies in Airflow Providers and extras
diff --git a/CHANGELOG.txt b/RELEASE_NOTES.rst
similarity index 56%
rename from CHANGELOG.txt
rename to RELEASE_NOTES.rst
index c890f68e95084..87319b55aaebb 100644
--- a/CHANGELOG.txt
+++ b/RELEASE_NOTES.rst
@@ -1,8 +1,743 @@
-Airflow 2.2.4, 2022-02-22
--------------------------
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. contents:: Apache Airflow Releases
+ :local:
+ :depth: 1
+
+.. towncrier release notes start
+
+Airflow 2.3.1 (2022-05-25)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+No significant changes.
+
+Bug Fixes
+^^^^^^^^^
+
+- Automatically reschedule stalled queued tasks in ``CeleryExecutor`` (#23690)
+- Fix expand/collapse all buttons (#23590)
+- Grid view status filters (#23392)
+- Expand/collapse all groups (#23487)
+- Fix retrieval of deprecated non-config values (#23723)
+- Fix secrets rendered in UI when task is not executed. (#22754)
+- Fix provider import error matching (#23825)
+- Fix regression in ignoring symlinks (#23535)
+- Fix ``dag-processor`` fetch metadata database config (#23575)
+- Fix auto upstream dep when expanding non-templated field (#23771)
+- Fix task log is not captured (#23684)
+- Add ``reschedule`` to the serialized fields for the ``BaseSensorOperator`` (#23674)
+- Modify db clean to also catch the ProgrammingError exception (#23699)
+- Remove titles from link buttons (#23736)
+- Fix grid details header text overlap (#23728)
+- Ensure ``execution_timeout`` as timedelta (#23655)
+- Don't run pre-migration checks for downgrade (#23634)
+- Add index for event column in log table (#23625)
+- Implement ``send_callback`` method for ``CeleryKubernetesExecutor`` and ``LocalKubernetesExecutor`` (#23617)
+- Fix ``PythonVirtualenvOperator`` templated_fields (#23559)
+- Apply specific ID collation to ``root_dag_id`` too (#23536)
+- Prevent ``KubernetesJobWatcher`` getting stuck on resource too old (#23521)
+- Fix scheduler crash when expanding with mapped task that returned none (#23486)
+- Fix broken dagrun links when many runs start at the same time (#23462)
+- Fix: Exception when parsing log #20966 (#23301)
+- Handle invalid date parsing in webserver views. (#23161)
+- Pools with negative open slots should not block other pools (#23143)
+- Move around overflow, position and padding (#23044)
+- Change approach to finding bad rows to LEFT OUTER JOIN. (#23528)
+- Only count bad refs when ``moved`` table exists (#23491)
+- Visually distinguish task group summary (#23488)
+- Remove color change for highly nested groups (#23482)
+- Optimize 2.3.0 pre-upgrade check queries (#23458)
+- Add backward compatibility for ``core__sql_alchemy_conn__cmd`` (#23441)
+- Fix literal cross product expansion (#23434)
+- Fix broken task instance link in xcom list (#23367)
+- Fix connection test button (#23345)
+- fix cli ``airflow dags show`` for mapped operator (#23339)
+- Hide some task instance attributes (#23338)
+- Don't show grid actions if server would reject with permission denied (#23332)
+- Use run_id for ``ti.mark_success_url`` (#23330)
+- Fix update user auth stats (#23314)
+- Use ```` in Mapped Instance table (#23313)
+- Fix duplicated Kubernetes DeprecationWarnings (#23302)
+- Store grid view selection in url params (#23290)
+- Remove custom signal handling in Triggerer (#23274)
+- Override pool for TaskInstance when pool is passed from cli. (#23258)
+- Show warning if '/' is used in a DAG run ID (#23106)
+- Use kubernetes queue in kubernetes hybrid executors (#23048)
+- Add tags inside try block. (#21784)
+
+Doc only changes
+^^^^^^^^^^^^^^^^
+
+- Move ``dag_processing.processor_timeouts`` to counters section (#23393)
+- Clarify that bundle extras should not be used for PyPi installs (#23697)
+- Synchronize support for Postgres and K8S in docs (#23673)
+- Replace DummyOperator references in docs (#23502)
+- Add doc notes for keyword-only args for ``expand()`` and ``partial()`` (#23373)
+- Document fix for broken elasticsearch logs with 2.3.0+ upgrade (#23821)
+
+Misc/Internal
+^^^^^^^^^^^^^
+
+- Add typing for airflow/configuration.py (#23716)
+- Disable Flower by default from docker-compose (#23685)
+- Added postgres 14 to support versions(including breeze) (#23506)
+- add K8S 1.24 support (#23637)
+- Refactor code references from tree to grid (#23254)
+
+
+Airflow 2.3.0 (2022-04-30)
+--------------------------
+
+For production docker image related changes, see the `Docker Image Changelog `_.
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+Passing ``execution_date`` to ``XCom.set()``, ``XCom.clear()`` , ``XCom.get_one()`` , and ``XCom.get_many()`` is deprecated (#19825)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Continuing the effort to bind TaskInstance to a DagRun, XCom entries are now also tied to a DagRun. Use the ``run_id`` argument to specify the DagRun instead.
+
+Task log templates are now read from the metadata database instead of ``airflow.cfg`` (#20165)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Previously, a task’s log is dynamically rendered from the ``[core] log_filename_template`` and ``[elasticsearch] log_id_template`` config values at runtime. This resulted in unfortunate characteristics, e.g. it is impractical to modify the config value after an Airflow instance is running for a while, since all existing task logs have be saved under the previous format and cannot be found with the new config value.
+
+A new ``log_template`` table is introduced to solve this problem. This table is synchronized with the aforementioned config values every time Airflow starts, and a new field ``log_template_id`` is added to every DAG run to point to the format used by tasks (``NULL`` indicates the first ever entry for compatibility).
+
+Minimum kubernetes library version bumped from ``3.0.0`` to ``21.7.0`` (#20759)
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+.. note::
+
+ This is only about changing the ``kubernetes`` library, not the Kubernetes cluster. Airflow support for
+ Kubernetes version is described in `Installation prerequisites `_.
+
+No change in behavior is expected. This was necessary in order to take advantage of a `bugfix `_ concerning refreshing of Kubernetes API tokens with EKS, which enabled the removal of some `workaround code `_.
+
+XCom now defined by ``run_id`` instead of ``execution_date`` (#20975)
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+As a continuation to the TaskInstance-DagRun relation change started in Airflow 2.2, the ``execution_date`` columns on XCom has been removed from the database, and replaced by an `association proxy `_ field at the ORM level. If you access Airflow’s metadata database directly, you should rewrite the implementation to use the ``run_id`` column instead.
+
+Note that Airflow’s metadatabase definition on both the database and ORM levels are considered implementation detail without strict backward compatibility guarantees.
+
+Non-JSON-serializable params deprecated (#21135).
+"""""""""""""""""""""""""""""""""""""""""""""""""
+
+It was previously possible to use dag or task param defaults that were not JSON-serializable.
+
+For example this worked previously:
+
+.. code-block:: python
+
+ @dag.task(params={"a": {1, 2, 3}, "b": pendulum.now()})
+ def datetime_param(value):
+ print(value)
+
+
+ datetime_param("{{ params.a }} | {{ params.b }}")
+
+Note the use of ``set`` and ``datetime`` types, which are not JSON-serializable. This behavior is problematic because to override these values in a dag run conf, you must use JSON, which could make these params non-overridable. Another problem is that the support for param validation assumes JSON. Use of non-JSON-serializable params will be removed in Airflow 3.0 and until then, use of them will produce a warning at parse time.
+
+You must use ``postgresql://`` instead of ``postgres://`` in ``sql_alchemy_conn`` for SQLAlchemy 1.4.0+ (#21205)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+When you use SQLAlchemy 1.4.0+, you need to use ``postgresql://`` as the scheme in the ``sql_alchemy_conn``.
+In the previous versions of SQLAlchemy it was possible to use ``postgres://`` , but using it in
+SQLAlchemy 1.4.0+ results in:
+
+.. code-block::
+
+ > raise exc.NoSuchModuleError(
+ "Can't load plugin: %s:%s" % (self.group, name)
+ )
+ E sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgres
+
+If you cannot change the scheme of your URL immediately, Airflow continues to work with SQLAlchemy
+1.3 and you can downgrade SQLAlchemy, but we recommend updating the scheme.
+Details in the `SQLAlchemy Changelog `_.
+
+``auth_backends`` replaces ``auth_backend`` configuration setting (#21472)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Previously, only one backend was used to authorize use of the REST API. In 2.3 this was changed to support multiple backends, separated by whitespace. Each will be tried in turn until a successful response is returned.
+
+This setting is also used for the deprecated experimental API, which only uses the first option even if multiple are given.
+
+``airflow.models.base.Operator`` is removed (#21505)
+""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Previously, there was an empty class ``airflow.models.base.Operator`` for “type hinting”. This class was never really useful for anything (everything it did could be done better with ``airflow.models.baseoperator.BaseOperator``), and has been removed. If you are relying on the class’s existence, use ``BaseOperator`` (for concrete operators), ``airflow.models.abstractoperator.AbstractOperator`` (the base class of both ``BaseOperator`` and the AIP-42 ``MappedOperator``), or ``airflow.models.operator.Operator`` (a union type ``BaseOperator | MappedOperator`` for type annotation).
+
+Zip files in the DAGs folder can no longer have a ``.py`` extension (#21538)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+It was previously possible to have any extension for zip files in the DAGs folder. Now ``.py`` files are going to be loaded as modules without checking whether it is a zip file, as it leads to less IO. If a ``.py`` file in the DAGs folder is a zip compressed file, parsing it will fail with an exception.
+
+``auth_backends`` includes session (#21640)
+"""""""""""""""""""""""""""""""""""""""""""
+
+To allow the Airflow UI to use the API, the previous default authorization backend ``airflow.api.auth.backend.deny_all`` is changed to ``airflow.api.auth.backend.session``, and this is automatically added to the list of API authorization backends if a non-default value is set.
+
+Default templates for log filenames and elasticsearch log_id changed (#21734)
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+In order to support Dynamic Task Mapping the default templates for per-task instance logging has changed. If your config contains the old default values they will be upgraded-in-place.
+
+If you are happy with the new config values you should *remove* the setting in ``airflow.cfg`` and let the default value be used. Old default values were:
+
+
+* ``[core] log_filename_template``: ``{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log``
+* ``[elasticsearch] log_id_template``: ``{dag_id}-{task_id}-{execution_date}-{try_number}``
+
+``[core] log_filename_template`` now uses "hive partition style" of ``dag_id=/run_id=`` by default, which may cause problems on some older FAT filesystems. If this affects you then you will have to change the log template.
+
+If you have customized the templates you should ensure that they contain ``{{ ti.map_index }}`` if you want to use dynamically mapped tasks.
+
+If after upgrading you find your task logs are no longer accessible, try adding a row in the ``log_template`` table with ``id=0``
+containing your previous ``log_id_template`` and ``log_filename_template``. For example, if you used the defaults in 2.2.5:
+
+.. code-block:: sql
+
+ INSERT INTO log_template (id, filename, elasticsearch_id, created_at) VALUES (0, '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log', '{dag_id}_{task_id}_{run_id}_{try_number}', NOW());
+
+BaseOperatorLink's ``get_link`` method changed to take a ``ti_key`` keyword argument (#21798)
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+In v2.2 we "deprecated" passing an execution date to XCom.get methods, but there was no other option for operator links as they were only passed an execution_date.
+
+Now in 2.3 as part of Dynamic Task Mapping (AIP-42) we will need to add map_index to the XCom row to support the "reduce" part of the API.
+
+In order to support that cleanly we have changed the interface for BaseOperatorLink to take an TaskInstanceKey as the ``ti_key`` keyword argument (as execution_date + task is no longer unique for mapped operators).
+
+The existing signature will be detected (by the absence of the ``ti_key`` argument) and continue to work.
+
+``ReadyToRescheduleDep`` now only runs when ``reschedule`` is *True* (#21815)
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+When a ``ReadyToRescheduleDep`` is run, it now checks whether the ``reschedule`` attribute on the operator, and always reports itself as *passed* unless it is set to *True*. If you use this dep class on your custom operator, you will need to add this attribute to the operator class. Built-in operator classes that use this dep class (including sensors and all subclasses) already have this attribute and are not affected.
+
+The ``deps`` attribute on an operator class should be a class level attribute (#21815)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+To support operator-mapping (AIP 42), the ``deps`` attribute on operator class must be a set at the class level. This means that if a custom operator implements this as an instance-level variable, it will not be able to be used for operator-mapping. This does not affect existing code, but we highly recommend you to restructure the operator's dep logic in order to support the new feature.
+
+Deprecation: ``Connection.extra`` must be JSON-encoded dict (#21816)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+TLDR
+~~~~
+
+From Airflow 3.0, the ``extra`` field in airflow connections must be a JSON-encoded Python dict.
+
+What, why, and when?
+~~~~~~~~~~~~~~~~~~~~
+
+Airflow's Connection is used for storing credentials. For storage of information that does not
+fit into user / password / host / schema / port, we have the ``extra`` string field. Its intention
+was always to provide for storage of arbitrary key-value pairs, like ``no_host_key_check`` in the SSH
+hook, or ``keyfile_dict`` in GCP.
+
+But since the field is string, it's technically been permissible to store any string value. For example
+one could have stored the string value ``'my-website.com'`` and used this in the hook. But this is a very
+bad practice. One reason is intelligibility: when you look at the value for ``extra`` , you don't have any idea
+what its purpose is. Better would be to store ``{"api_host": "my-website.com"}`` which at least tells you
+*something* about the value. Another reason is extensibility: if you store the API host as a simple string
+value, what happens if you need to add more information, such as the API endpoint, or credentials? Then
+you would need to convert the string to a dict, and this would be a breaking change.
+
+For these reason, starting in Airflow 3.0 we will require that the ``Connection.extra`` field store
+a JSON-encoded Python dict.
+
+How will I be affected?
+~~~~~~~~~~~~~~~~~~~~~~~
+
+For users of providers that are included in the Airflow codebase, you should not have to make any changes
+because in the Airflow codebase we should not allow hooks to misuse the ``Connection.extra`` field in this way.
+
+However, if you have any custom hooks that store something other than JSON dict, you will have to update it.
+If you do, you should see a warning any time that this connection is retrieved or instantiated (e.g. it should show up in
+task logs).
+
+To see if you have any connections that will need to be updated, you can run this command:
+
+.. code-block:: shell
+
+ airflow connections export - 2>&1 >/dev/null | grep 'non-JSON'
+
+This will catch any warnings about connections that are storing something other than JSON-encoded Python dict in the ``extra`` field.
+
+The ``tree`` default view setting has been renamed to ``grid`` (#22167)
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+If you set the ``dag_default_view`` config option or the ``default_view`` argument to ``DAG()`` to ``tree`` you will need to update your deployment. The old name will continue to work but will issue warnings.
+
+Database configuration moved to new section (#22284)
+""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The following configurations have been moved from ``[core]`` to the new ``[database]`` section. However when reading the new option, the old option will be checked to see if it exists. If it does a DeprecationWarning will be issued and the old option will be used instead.
+
+* sql_alchemy_conn
+* sql_engine_encoding
+* sql_engine_collation_for_ids
+* sql_alchemy_pool_enabled
+* sql_alchemy_pool_size
+* sql_alchemy_max_overflow
+* sql_alchemy_pool_recycle
+* sql_alchemy_pool_pre_ping
+* sql_alchemy_schema
+* sql_alchemy_connect_args
+* load_default_connections
+* max_db_retries
+
+Remove requirement that custom connection UI fields be prefixed (#22607)
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Hooks can define custom connection fields for their connection type by implementing method ``get_connection_form_widgets``. These custom fields appear in the web UI as additional connection attributes, but internally they are stored in the connection ``extra`` dict field. For technical reasons, previously, when stored in the ``extra`` dict, the custom field's dict key had to take the form ``extra____``. This had the consequence of making it more cumbersome to define connections outside of the UI, since the prefix ``extra____`` makes it tougher to read and work with. With #22607, we make it so that you can now define custom fields such that they can be read from and stored in ``extra`` without the prefix.
+
+To enable this, update the dict returned by the ``get_connection_form_widgets`` method to remove the prefix from the keys. Internally, the providers manager will still use a prefix to ensure each custom field is globally unique, but the absence of a prefix in the returned widget dict will signal to the Web UI to read and store custom fields without the prefix. Note that this is only a change to the Web UI behavior; when updating your hook in this way, you must make sure that when your *hook* reads the ``extra`` field, it will also check for the prefixed value for backward compatibility.
+
+The webserver.X_FRAME_ENABLED configuration works according to description now (#23222).
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+In Airflow 2.0.0 - 2.2.4 the webserver.X_FRAME_ENABLED parameter worked the opposite of its description,
+setting the value to "true" caused "X-Frame-Options" header to "DENY" (not allowing Airflow to be used
+in an iframe). When you set it to "false", the header was not added, so Airflow could be embedded in an
+iframe. By default Airflow could not be embedded in an iframe.
+
+In Airflow 2.2.5 there was a bug introduced that made it impossible to disable Airflow to
+work in iframe. No matter what the configuration was set, it was possible to embed Airflow in an iframe.
+
+Airflow 2.3.0 restores the original meaning to the parameter. If you set it to "true" (default) Airflow
+can be embedded in an iframe (no header is added), but when you set it to "false" the header is added
+and Airflow cannot be embedded in an iframe.
+
+
+New Features
+^^^^^^^^^^^^
+
+- Add dynamic task mapping (`AIP-42 `_)
+- New Grid View replaces Tree View (#18675)
+- Templated ``requirements.txt`` in Python Operators (#17349)
+- Allow reuse of decorated tasks (#22941)
+- Move the database configuration to a new section (#22284)
+- Add ``SmoothOperator`` (#22813)
+- Make operator's ``execution_timeout`` configurable (#22389)
+- Events Timetable (#22332)
+- Support dag serialization with custom ``ti_deps`` rules (#22698)
+- Support log download in task log view (#22804)
+- support for continue backfill on failures (#22697)
+- Add ``dag-processor`` cli command (#22305)
+- Add possibility to create users in LDAP mode (#22619)
+- Add ``ignore_first_depends_on_past`` for scheduled jobs (#22491)
+- Update base sensor operator to support XCOM return value (#20656)
+- Add an option for run id in the ui trigger screen (#21851)
+- Enable JSON serialization for connections (#19857)
+- Add REST API endpoint for bulk update of DAGs (#19758)
+- Add queue button to click-on-DagRun interface. (#21555)
+- Add ``list-import-errors`` to ``airflow dags`` command (#22084)
+- Store callbacks in database if ``standalone_dag_processor`` config is True. (#21731)
+- Add LocalKubernetesExecutor (#19729)
+- Add ``celery.task_timeout_error`` metric (#21602)
+- Airflow ``db downgrade`` cli command (#21596)
+- Add ``ALL_SKIPPED`` trigger rule (#21662)
+- Add ``db clean`` CLI command for purging old data (#20838)
+- Add ``celery_logging_level`` (#21506)
+- Support different timeout value for dag file parsing (#21501)
+- Support generating SQL script for upgrades (#20962)
+- Add option to compress Serialized dag data (#21332)
+- Branch python operator decorator (#20860)
+- Add Audit Log View to Dag View (#20733)
+- Add missing StatsD metric for failing SLA Callback notification (#20924)
+- Add ``ShortCircuitOperator`` configurability for respecting downstream trigger rules (#20044)
+- Allow using Markup in page title in Webserver (#20888)
+- Add Listener Plugin API that tracks TaskInstance state changes (#20443)
+- Add context var hook to inject more env vars (#20361)
+- Add a button to set all tasks to skipped (#20455)
+- Cleanup pending pods (#20438)
+- Add config to warn public deployment exposure in UI (#18557)
+- Log filename template records (#20165)
+- Added windows extensions (#16110)
+- Showing approximate time until next dag_run in Airflow (#20273)
+- Extend config window on UI (#20052)
+- Add show dag dependencies feature to CLI (#19985)
+- Add cli command for 'airflow dags reserialize` (#19471)
+- Add missing description field to Pool schema(REST API) (#19841)
+- Introduce DagRun action to change state to queued. (#19353)
+- Add DAG run details page (#19705)
+- Add role export/import to cli tools (#18916)
+- Adding ``dag_id_pattern`` parameter to the ``/dags`` endpoint (#18924)
+
+
+Improvements
+^^^^^^^^^^^^
+
+- Show schedule_interval/timetable description in UI (#16931)
+- Added column duration to DAG runs view (#19482)
+- Enable use of custom conn extra fields without prefix (#22607)
+- Initialize finished counter at zero (#23080)
+- Improve logging of optional provider features messages (#23037)
+- Meaningful error message in resolve_template_files (#23027)
+- Update ImportError items instead of deleting and recreating them (#22928)
+- Add option ``--skip-init`` to db reset command (#22989)
+- Support importing connections from files with ".yml" extension (#22872)
+- Support glob syntax in ``.airflowignore`` files (#21392) (#22051)
+- Hide pagination when data is a single page (#22963)
+- Support for sorting DAGs in the web UI (#22671)
+- Speed up ``has_access`` decorator by ~200ms (#22858)
+- Add XComArg to lazy-imported list of Airflow module (#22862)
+- Add more fields to REST API dags/dag_id/details endpoint (#22756)
+- Don't show irrelevant/duplicated/"internal" Task attrs in UI (#22812)
+- No need to load whole ti in current_state (#22764)
+- Pickle dag exception string fix (#22760)
+- Better verification of Localexecutor's parallelism option (#22711)
+- log backfill exceptions to sentry (#22704)
+- retry commit on MySQL deadlocks during backfill (#22696)
+- Add more fields to REST API get DAG(dags/dag_id) endpoint (#22637)
+- Use timetable to generate planned days for current year (#22055)
+- Disable connection pool for celery worker (#22493)
+- Make date picker label visible in trigger dag view (#22379)
+- Expose ``try_number`` in airflow vars (#22297)
+- Add generic connection type (#22310)
+- Add a few more fields to the taskinstance finished log message (#22262)
+- Pause auto-refresh if scheduler isn't running (#22151)
+- Show DagModel details. (#21868)
+- Add pip_install_options to PythonVirtualenvOperator (#22158)
+- Show import error for ``airflow dags list`` CLI command (#21991)
+- Pause auto-refresh when page is hidden (#21904)
+- Default args type check (#21809)
+- Enhance magic methods on XComArg for UX (#21882)
+- py files don't have to be checked ``is_zipfiles`` in refresh_dag (#21926)
+- Fix TaskDecorator type hints (#21881)
+- Add 'Show record' option for variables (#21342)
+- Use DB where possible for quicker ``airflow dag`` subcommands (#21793)
+- REST API: add rendered fields in task instance. (#21741)
+- Change the default auth backend to session (#21640)
+- Don't check if ``py`` DAG files are zipped during parsing (#21538)
+- Switch XCom implementation to use ``run_id`` (#20975)
+- Action log on Browse Views (#21569)
+- Implement multiple API auth backends (#21472)
+- Change logging level details of connection info in ``get_connection()`` (#21162)
+- Support mssql in airflow db shell (#21511)
+- Support config ``worker_enable_remote_control`` for celery (#21507)
+- Log memory usage in ``CgroupTaskRunner`` (#21481)
+- Modernize DAG-related URL routes and rename "tree" to "grid" (#20730)
+- Move Zombie detection to ``SchedulerJob`` (#21181)
+- Improve speed to run ``airflow`` by 6x (#21438)
+- Add more SQL template fields renderers (#21237)
+- Simplify fab has access lookup (#19294)
+- Log context only for default method (#21244)
+- Log trigger status only if at least one is running (#21191)
+- Add optional features in providers. (#21074)
+- Better multiple_outputs inferral for @task.python (#20800)
+- Improve handling of string type and non-attribute ``template_fields`` (#21054)
+- Remove un-needed deps/version requirements (#20979)
+- Correctly specify overloads for TaskFlow API for type-hinting (#20933)
+- Introduce notification_sent to SlaMiss view (#20923)
+- Rewrite the task decorator as a composition (#20868)
+- Add "Greater/Smaller than or Equal" to filters in the browse views (#20602) (#20798)
+- Rewrite DAG run retrieval in task command (#20737)
+- Speed up creation of DagRun for large DAGs (5k+ tasks) by 25-130% (#20722)
+- Make native environment Airflow-flavored like sandbox (#20704)
+- Better error when param value has unexpected type (#20648)
+- Add filter by state in DagRun REST API (List Dag Runs) (#20485)
+- Prevent exponential memory growth in Tasks with custom logging handler (#20541)
+- Set default logger in logging Mixin (#20355)
+- Reduce deprecation warnings from www (#20378)
+- Add hour and minute to time format on x-axis of all charts using nvd3.lineChart (#20002)
+- Add specific warning when Task asks for more slots than pool defined with (#20178)
+- UI: Update duration column for better human readability (#20112)
+- Use Viewer role as example public role (#19215)
+- Properly implement DAG param dict copying (#20216)
+- ``ShortCircuitOperator`` push XCom by returning python_callable result (#20071)
+- Add clear logging to tasks killed due to a Dagrun timeout (#19950)
+- Change log level for Zombie detection messages (#20204)
+- Better confirmation prompts (#20183)
+- Only execute TIs of running DagRuns (#20182)
+- Check and run migration in commands if necessary (#18439)
+- Log only when Zombies exists (#20118)
+- Increase length of the email and username (#19932)
+- Add more filtering options for TI's in the UI (#19910)
+- Dynamically enable "Test Connection" button by connection type (#19792)
+- Avoid littering postgres server logs with "could not obtain lock" with HA schedulers (#19842)
+- Renamed ``Connection.get_hook`` parameter to make it the same as in ``SqlSensor`` and ``SqlOperator``. (#19849)
+- Add hook_params in SqlSensor using the latest changes from PR #18718. (#18431)
+- Speed up webserver boot time by delaying provider initialization (#19709)
+- Configurable logging of ``XCOM`` value in PythonOperator (#19378)
+- Minimize production js files (#19658)
+- Add ``hook_params`` in ``BaseSqlOperator`` (#18718)
+- Add missing "end_date" to hash components (#19281)
+- More friendly output of the airflow plugins command + add timetables (#19298)
+- Add sensor default timeout config (#19119)
+- Update ``taskinstance`` REST API schema to include dag_run_id field (#19105)
+- Adding feature in bash operator to append the user defined env variable to system env variable (#18944)
+- Duplicate Connection: Added logic to query if a connection id exists before creating one (#18161)
+
+
+Bug Fixes
+^^^^^^^^^
+
+- Use inherited 'trigger_tasks' method (#23016)
+- In DAG dependency detector, use class type instead of class name (#21706)
+- Fix tasks being wrongly skipped by schedule_after_task_execution (#23181)
+- Fix X-Frame enabled behaviour (#23222)
+- Allow ``extra`` to be nullable in connection payload as per schema(REST API). (#23183)
+- Fix ``dag_id`` extraction for dag level access checks in web ui (#23015)
+- Fix timezone display for logs on UI (#23075)
+- Include message in graph errors (#23021)
+- Change trigger dropdown left position (#23013)
+- Don't add planned tasks for legacy DAG runs (#23007)
+- Add dangling rows check for TaskInstance references (#22924)
+- Validate the input params in connection ``CLI`` command (#22688)
+- Fix trigger event payload is not persisted in db (#22944)
+- Drop "airflow moved" tables in command ``db reset`` (#22990)
+- Add max width to task group tooltips (#22978)
+- Add template support for ``external_task_ids``. (#22809)
+- Allow ``DagParam`` to hold falsy values (#22964)
+- Fix regression in pool metrics (#22939)
+- Priority order tasks even when using pools (#22483)
+- Do not clear XCom when resuming from deferral (#22932)
+- Handle invalid JSON metadata in ``get_logs_with_metadata endpoint``. (#22898)
+- Fix pre-upgrade check for rows dangling w.r.t. dag_run (#22850)
+- Fixed backfill interference with scheduler (#22701)
+- Support conf param override for backfill runs (#22837)
+- Correctly interpolate pool name in ``PoolSlotsAvailableDep`` statues (#22807)
+- Fix ``email_on_failure`` with ``render_template_as_native_obj`` (#22770)
+- Fix processor cleanup on ``DagFileProcessorManager`` (#22685)
+- Prevent meta name clash for task instances (#22783)
+- remove json parse for gantt chart (#22780)
+- Check for missing dagrun should know version (#22752)
+- Fixes ``ScheduleInterval`` spec (#22635)
+- Fixing task status for non-running and non-committed tasks (#22410)
+- Do not log the hook connection details even at DEBUG level (#22627)
+- Stop crashing when empty logs are received from kubernetes client (#22566)
+- Fix bugs about timezone change (#22525)
+- Fix entire DAG stops when one task has end_date (#20920)
+- Use logger to print message during task execution. (#22488)
+- Make sure finalizers are not skipped during exception handling (#22475)
+- update smart sensor docs and minor fix on ``is_smart_sensor_compatible()`` (#22386)
+- Fix ``run_id`` k8s and elasticsearch compatibility with Airflow 2.1 (#22385)
+- Allow to ``except_skip`` None on ``BranchPythonOperator`` (#20411)
+- Fix incorrect datetime details (DagRun views) (#21357)
+- Remove incorrect deprecation warning in secrets backend (#22326)
+- Remove ``RefreshConfiguration`` workaround for K8s token refreshing (#20759)
+- Masking extras in GET ``/connections/`` endpoint (#22227)
+- Set ``queued_dttm`` when submitting task to directly to executor (#22259)
+- Addressed some issues in the tutorial mentioned in discussion #22233 (#22236)
+- Change default python executable to python3 for docker decorator (#21973)
+- Don't validate that Params are JSON when NOTSET (#22000)
+- Add per-DAG delete permissions (#21938)
+- Fix handling some None parameters in kubernetes 23 libs. (#21905)
+- Fix handling of empty (None) tags in ``bulk_write_to_db`` (#21757)
+- Fix DAG date range bug (#20507)
+- Removed ``request.referrer`` from views.py (#21751)
+- Make ``DbApiHook`` use ``get_uri`` from Connection (#21764)
+- Fix some migrations (#21670)
+- [de]serialize resources on task correctly (#21445)
+- Add params ``dag_id``, ``task_id`` etc to ``XCom.serialize_value`` (#19505)
+- Update test connection functionality to use custom form fields (#21330)
+- fix all "high" npm vulnerabilities (#21526)
+- Fix bug incorrectly removing action from role, rather than permission. (#21483)
+- Fix relationship join bug in FAB/SecurityManager with SQLA 1.4 (#21296)
+- Use Identity instead of Sequence in SQLAlchemy 1.4 for MSSQL (#21238)
+- Ensure ``on_task_instance_running`` listener can get at task (#21157)
+- Return to the same place when triggering a DAG (#20955)
+- Fix task ID deduplication in ``@task_group`` (#20870)
+- Add downgrade to some FAB migrations (#20874)
+- Only validate Params when DAG is triggered (#20802)
+- Fix ``airflow trigger`` cli (#20781)
+- Fix task instances iteration in a pool to prevent blocking (#20816)
+- Allow depending to a ``@task_group`` as a whole (#20671)
+- Use original task's ``start_date`` if a task continues after deferral (#20062)
+- Disabled edit button in task instances list view page (#20659)
+- Fix a package name import error (#20519) (#20519)
+- Remove ``execution_date`` label when get cleanup pods list (#20417)
+- Remove unneeded FAB REST API endpoints (#20487)
+- Fix parsing of Cloudwatch log group arn containing slashes (#14667) (#19700)
+- Sanity check for MySQL's TIMESTAMP column (#19821)
+- Allow using default celery command group with executors subclassed from Celery-based executors. (#18189)
+- Move ``class_permission_name`` to mixin so it applies to all classes (#18749)
+- Adjust trimmed_pod_id and replace '.' with '-' (#19036)
+- Pass custom_headers to send_email and send_email_smtp (#19009)
+- Ensure ``catchup=False`` is used in example dags (#19396)
+- Edit permalinks in OpenApi description file (#19244)
+- Navigate directly to DAG when selecting from search typeahead list (#18991)
+- [Minor] Fix padding on home page (#19025)
+
+
+Doc only changes
+^^^^^^^^^^^^^^^^
+
+- Update doc for DAG file processing (#23209)
+- Replace changelog/updating with release notes and ``towncrier`` now (#22003)
+- Fix wrong reference in tracking-user-activity.rst (#22745)
+- Remove references to ``rbac = True`` from docs (#22725)
+- Doc: Update description for executor-bound dependencies (#22601)
+- Update check-health.rst (#22372)
+- Stronger language about Docker Compose customizability (#22304)
+- Update logging-tasks.rst (#22116)
+- Add example config of ``sql_alchemy_connect_args`` (#22045)
+- Update best-practices.rst (#22053)
+- Add information on DAG pausing/deactivation/deletion (#22025)
+- Add brief examples of integration test dags you might want (#22009)
+- Run inclusive language check on CHANGELOG (#21980)
+- Add detailed email docs for Sendgrid (#21958)
+- Add docs for ``db upgrade`` / ``db downgrade`` (#21879)
+- Update modules_management.rst (#21889)
+- Fix UPDATING section on SqlAlchemy 1.4 scheme changes (#21887)
+- Update TaskFlow tutorial doc to show how to pass "operator-level" args. (#21446)
+- Fix doc - replace decreasing by increasing (#21805)
+- Add another way to dynamically generate DAGs to docs (#21297)
+- Add extra information about time synchronization needed (#21685)
+- Update debug.rst docs (#21246)
+- Replaces the usage of ``postgres://`` with ``postgresql://`` (#21205)
+- Fix task execution process in ``CeleryExecutor`` docs (#20783)
+
+
+Misc/Internal
+^^^^^^^^^^^^^
+
+- Bring back deprecated security manager functions (#23243)
+- Replace usage of ``DummyOperator`` with ``EmptyOperator`` (#22974)
+- Deprecate ``DummyOperator`` in favor of ``EmptyOperator`` (#22832)
+- Remove unnecessary python 3.6 conditionals (#20549)
+- Bump ``moment`` from 2.29.1 to 2.29.2 in /airflow/www (#22873)
+- Bump ``prismjs`` from 1.26.0 to 1.27.0 in /airflow/www (#22823)
+- Bump ``nanoid`` from 3.1.23 to 3.3.2 in /airflow/www (#22803)
+- Bump ``minimist`` from 1.2.5 to 1.2.6 in /airflow/www (#22798)
+- Remove dag parsing from db init command (#22531)
+- Update our approach for executor-bound dependencies (#22573)
+- Use ``Airflow.Base.metadata`` in FAB models (#22353)
+- Limit docutils to make our documentation pretty again (#22420)
+- Add Python 3.10 support (#22050)
+- [FEATURE] add 1.22 1.23 K8S support (#21902)
+- Remove pandas upper limit now that SQLA is 1.4+ (#22162)
+- Patch ``sql_alchemy_conn`` if old postgres scheme used (#22333)
+- Protect against accidental misuse of XCom.get_value() (#22244)
+- Order filenames for migrations (#22168)
+- Don't try to auto generate migrations for Celery tables (#22120)
+- Require SQLAlchemy 1.4 (#22114)
+- bump sphinx-jinja (#22101)
+- Add compat shim for SQLAlchemy to avoid warnings (#21959)
+- Rename ``xcom.dagrun_id`` to ``xcom.dag_run_id`` (#21806)
+- Deprecate non-JSON ``conn.extra`` (#21816)
+- Bump upper bound version of ``jsonschema`` to 5.0 (#21712)
+- Deprecate helper utility ``days_ago`` (#21653)
+- Remove ```:type``` lines now ``sphinx-autoapi`` supports type hints (#20951)
+- Silence deprecation warning in tests (#20900)
+- Use ``DagRun.run_id`` instead of ``execution_date`` when updating state of TIs (UI & REST API) (#18724)
+- Add Context stub to Airflow packages (#20817)
+- Update Kubernetes library version (#18797)
+- Rename ``PodLauncher`` to ``PodManager`` (#20576)
+- Removes Python 3.6 support (#20467)
+- Add deprecation warning for non-json-serializable params (#20174)
+- Rename TaskMixin to DependencyMixin (#20297)
+- Deprecate passing execution_date to XCom methods (#19825)
+- Remove ``get_readable_dags`` and ``get_editable_dags``, and ``get_accessible_dags``. (#19961)
+- Remove postgres 9.6 support (#19987)
+- Removed hardcoded connection types. Check if hook is instance of DbApiHook. (#19639)
+- add kubernetes 1.21 support (#19557)
+- Add FAB base class and set import_name explicitly. (#19667)
+- Removes unused state transitions to handle auto-changing view permissions. (#19153)
+- Chore: Use enum for ``__var`` and ``__type`` members (#19303)
+- Use fab models (#19121)
+- Consolidate method names between Airflow Security Manager and FAB default (#18726)
+- Remove distutils usages for Python 3.10 (#19064)
+- Removing redundant ``max_tis_per_query`` initialisation on SchedulerJob (#19020)
+- Remove deprecated usage of ``init_role()`` from API (#18820)
+- Remove duplicate code on dbapi hook (#18821)
+
+
+Airflow 2.2.5, (2022-04-04)
+---------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+No significant changes.
Bug Fixes
"""""""""
+- Check and disallow a relative path for sqlite (#22530)
+- Fixed dask executor and tests (#22027)
+- Fix broken links to celery documentation (#22364)
+- Fix incorrect data provided to tries & landing times charts (#21928)
+- Fix assignment of unassigned triggers (#21770)
+- Fix triggerer ``--capacity`` parameter (#21753)
+- Fix graph auto-refresh on page load (#21736)
+- Fix filesystem sensor for directories (#21729)
+- Fix stray ``order_by(TaskInstance.execution_date)`` (#21705)
+- Correctly handle multiple '=' in LocalFileSystem secrets. (#21694)
+- Log exception in local executor (#21667)
+- Disable ``default_pool`` delete on web ui (#21658)
+- Extends ``typing-extensions`` to be installed with python 3.8+ #21566 (#21567)
+- Dispose unused connection pool (#21565)
+- Fix logging JDBC SQL error when task fails (#21540)
+- Filter out default configs when overrides exist. (#21539)
+- Fix Resources ``__eq__`` check (#21442)
+- Fix ``max_active_runs=1`` not scheduling runs when ``min_file_process_interval`` is high (#21413)
+- Reduce DB load incurred by Stale DAG deactivation (#21399)
+- Fix race condition between triggerer and scheduler (#21316)
+- Fix trigger dag redirect from task instance log view (#21239)
+- Log traceback in trigger exceptions (#21213)
+- A trigger might use a connection; make sure we mask passwords (#21207)
+- Update ``ExternalTaskSensorLink`` to handle templated ``external_dag_id`` (#21192)
+- Ensure ``clear_task_instances`` sets valid run state (#21116)
+- Fix: Update custom connection field processing (#20883)
+- Truncate stack trace to DAG user code for exceptions raised during execution (#20731)
+- Fix duplicate trigger creation race condition (#20699)
+- Fix Tasks getting stuck in scheduled state (#19747)
+- Fix: Do not render undefined graph edges (#19684)
+- Set ``X-Frame-Options`` header to DENY only if ``X_FRAME_ENABLED`` is set to true. (#19491)
+
+Doc only changes
+""""""""""""""""
+- adding ``on_execute_callback`` to callbacks docs (#22362)
+- Add documentation on specifying a DB schema. (#22347)
+- Fix postgres part of pipeline example of tutorial (#21586)
+- Extend documentation for states of DAGs & tasks and update trigger rules docs (#21382)
+- DB upgrade is required when updating Airflow (#22061)
+- Remove misleading MSSQL information from the docs (#21998)
+
+Misc
+""""
+- Add the new Airflow Trove Classifier to setup.cfg (#22241)
+- Rename ``to_delete`` to ``to_cancel`` in TriggerRunner (#20658)
+- Update Flask-AppBuilder to ``3.4.5`` (#22596)
+
+Airflow 2.2.4, (2022-02-22)
+---------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+Smart sensors deprecated
+""""""""""""""""""""""""
+
+Smart sensors, an "early access" feature added in Airflow 2, are now deprecated and will be removed in Airflow 2.4.0. They have been superseded by Deferrable Operators, added in Airflow 2.2.0.
+
+See `Migrating to Deferrable Operators `_ for details on how to migrate.
+
+Bug Fixes
+^^^^^^^^^
- Adding missing login provider related methods from Flask-Appbuilder (#21294)
- Fix slow DAG deletion due to missing ``dag_id`` index for job table (#20282)
@@ -42,7 +777,7 @@ Bug Fixes
- Pin MarkupSafe until we are able to upgrade Flask/Jinja (#21664)
Doc only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
- Added explaining concept of logical date in DAG run docs (#21433)
- Add note about Variable precedence with env vars (#21568)
@@ -65,16 +800,21 @@ Doc only changes
- Update upgrading.rst with detailed code example of how to resolve post-upgrade warning (#19993)
Misc
-""""
+^^^^
- Deprecate some functions in the experimental API (#19931)
- Deprecate smart sensors (#20151)
-Airflow 2.2.3, 2021-12-21
--------------------------
+Airflow 2.2.3, (2021-12-21)
+---------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+No significant changes.
Bug Fixes
-"""""""""
+^^^^^^^^^
- Lazy Jinja2 context (#20217)
- Exclude ``snowflake-sqlalchemy`` v1.2.5 (#20245)
@@ -113,7 +853,7 @@ Bug Fixes
- Fix moved data migration check for MySQL when replication is used (#19999)
Doc only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
- Warn without tracebacks when example_dags are missing deps (#20295)
- Deferrable operators doc clarification (#20150)
@@ -136,11 +876,16 @@ Doc only changes
- Add description on how you can customize image entrypoint (#18915)
- Dags-in-image pod template example should not have dag mounts (#19337)
-Airflow 2.2.2, 2021-11-15
--------------------------
+Airflow 2.2.2 (2021-11-15)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+No significant changes.
Bug Fixes
-"""""""""
+^^^^^^^^^
- Fix bug when checking for existence of a Variable (#19395)
- Fix Serialization when ``relativedelta`` is passed as ``schedule_interval`` (#19418)
@@ -165,17 +910,41 @@ Bug Fixes
- Fix missing dagruns when ``catchup=True`` (#19528)
Doc only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
- Add missing parameter documentation for "timetable" (#19282)
- Improve Kubernetes Executor docs (#19339)
- Update image tag used in docker docs
-Airflow 2.2.1, 2021-10-29
--------------------------
+Airflow 2.2.1 (2021-10-29)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+``Param``'s default value for ``default`` removed
+"""""""""""""""""""""""""""""""""""""""""""""""""
+
+``Param``, introduced in Airflow 2.2.0, accidentally set the default value to ``None``. This default has been removed. If you want ``None`` as your default, explicitly set it as such. For example:
+
+.. code-block:: python
+
+ Param(None, type=["null", "string"])
+
+Now if you resolve a ``Param`` without a default and don't pass a value, you will get an ``TypeError``. For Example:
+
+.. code-block:: python
+
+ Param().resolve() # raises TypeError
+
+``max_queued_runs_per_dag`` configuration has been removed
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The ``max_queued_runs_per_dag`` configuration option in ``[core]`` section has been removed. Previously, this controlled the number of queued dagrun
+the scheduler can create in a dag. Now, the maximum number is controlled internally by the DAG's ``max_active_runs``
Bug Fixes
-"""""""""
+^^^^^^^^^
- Fix Unexpected commit error in SchedulerJob (#19213)
- Add DagRun.logical_date as a property (#19198)
@@ -208,18 +977,236 @@ Bug Fixes
- Check python version before starting triggerer (#18926)
Doc only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
- Update access control documentation for TaskInstances and DagRuns (#18644)
- Add information about keepalives for managed Postgres (#18850)
- Doc: Add Callbacks Section to Logging & Monitoring (#18842)
- Group PATCH DAGrun together with other DAGRun endpoints (#18885)
-Airflow 2.2.0, 2021-10-11
--------------------------
+Airflow 2.2.0 (2021-10-11)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+Note: Upgrading the database to ``2.2.0`` or later can take some time to complete, particularly if you have a large ``task_instance`` table.
+
+``worker_log_server_port`` configuration has been moved to the ``logging`` section.
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The ``worker_log_server_port`` configuration option has been moved from ``[celery]`` section to ``[logging]`` section to allow for re-use between different executors.
+
+``pandas`` is now an optional dependency
+""""""""""""""""""""""""""""""""""""""""""""
+
+Previously ``pandas`` was a core requirement so when you run ``pip install apache-airflow`` it looked for ``pandas``
+library and installed it if it does not exist.
+
+If you want to install ``pandas`` compatible with Airflow, you can use ``[pandas]`` extra while
+installing Airflow, example for Python 3.8 and Airflow 2.1.2:
+
+.. code-block:: shell
+
+ pip install -U "apache-airflow[pandas]==2.1.2" \
+ --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-3.8.txt"
+
+``none_failed_or_skipped`` trigger rule has been deprecated
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+``TriggerRule.NONE_FAILED_OR_SKIPPED`` is replaced by ``TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS``.
+This is only name change, no functionality changes made.
+This change is backward compatible however ``TriggerRule.NONE_FAILED_OR_SKIPPED`` will be removed in next major release.
+
+Dummy trigger rule has been deprecated
+""""""""""""""""""""""""""""""""""""""
+
+``TriggerRule.DUMMY`` is replaced by ``TriggerRule.ALWAYS``.
+This is only name change, no functionality changes made.
+This change is backward compatible however ``TriggerRule.DUMMY`` will be removed in next major release.
+
+DAG concurrency settings have been renamed
+""""""""""""""""""""""""""""""""""""""""""
+
+``[core] dag_concurrency`` setting in ``airflow.cfg`` has been renamed to ``[core] max_active_tasks_per_dag``
+for better understanding.
+
+It is the maximum number of task instances allowed to run concurrently in each DAG. To calculate
+the number of tasks that is running concurrently for a DAG, add up the number of running
+tasks for all DAG runs of the DAG.
+
+This is configurable at the DAG level with ``max_active_tasks`` and a default can be set in ``airflow.cfg`` as
+``[core] max_active_tasks_per_dag``.
+
+**Before**\ :
+
+.. code-block:: ini
+
+ [core]
+ dag_concurrency = 16
+
+**Now**\ :
+
+.. code-block:: ini
+
+ [core]
+ max_active_tasks_per_dag = 16
+
+Similarly, ``DAG.concurrency`` has been renamed to ``DAG.max_active_tasks``.
+
+**Before**\ :
+
+.. code-block:: python
+
+ dag = DAG(
+ dag_id="example_dag",
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ concurrency=3,
+ )
+
+**Now**\ :
+
+.. code-block:: python
+
+ dag = DAG(
+ dag_id="example_dag",
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ max_active_tasks=3,
+ )
+
+If you are using DAGs Details API endpoint, use ``max_active_tasks`` instead of ``concurrency``.
+
+Task concurrency parameter has been renamed
+"""""""""""""""""""""""""""""""""""""""""""
+
+``BaseOperator.task_concurrency`` has been deprecated and renamed to ``max_active_tis_per_dag`` for
+better understanding.
+
+This parameter controls the number of concurrent running task instances across ``dag_runs``
+per task.
+
+**Before**\ :
+
+.. code-block:: python
+
+ with DAG(dag_id="task_concurrency_example"):
+ BashOperator(task_id="t1", task_concurrency=2, bash_command="echo Hi")
+
+**After**\ :
+
+.. code-block:: python
+
+ with DAG(dag_id="task_concurrency_example"):
+ BashOperator(task_id="t1", max_active_tis_per_dag=2, bash_command="echo Hi")
+
+``processor_poll_interval`` config have been renamed to ``scheduler_idle_sleep_time``
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+``[scheduler] processor_poll_interval`` setting in ``airflow.cfg`` has been renamed to ``[scheduler] scheduler_idle_sleep_time``
+for better understanding.
+
+It controls the 'time to sleep' at the end of the Scheduler loop if nothing was scheduled inside ``SchedulerJob``.
+
+**Before**\ :
+
+.. code-block:: ini
+
+ [scheduler]
+ processor_poll_interval = 16
+
+**Now**\ :
+
+.. code-block:: ini
+
+ [scheduler]
+ scheduler_idle_sleep_time = 16
+
+Marking success/failed automatically clears failed downstream tasks
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+When marking a task success/failed in Graph View, its downstream tasks that are in failed/upstream_failed state are automatically cleared.
+
+``[core] store_dag_code`` has been removed
+""""""""""""""""""""""""""""""""""""""""""""""
+
+While DAG Serialization is a strict requirements since Airflow 2, we allowed users to control
+where the Webserver looked for when showing the **Code View**.
+
+If ``[core] store_dag_code`` was set to ``True``\ , the Scheduler stored the code in the DAG file in the
+DB (in ``dag_code`` table) as a plain string, and the webserver just read it from the same table.
+If the value was set to ``False``\ , the webserver read it from the DAG file.
+
+While this setting made sense for Airflow < 2, it caused some confusion to some users where they thought
+this setting controlled DAG Serialization.
+
+From Airflow 2.2, Airflow will only look for DB when a user clicks on **Code View** for a DAG.
+
+Clearing a running task sets its state to ``RESTARTING``
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Previously, clearing a running task sets its state to ``SHUTDOWN``. The task gets killed and goes into ``FAILED`` state. After `#16681 `_\ , clearing a running task sets its state to ``RESTARTING``. The task is eligible for retry without going into ``FAILED`` state.
+
+Remove ``TaskInstance.log_filepath`` attribute
+""""""""""""""""""""""""""""""""""""""""""""""""""
+
+This method returned incorrect values for a long time, because it did not take into account the different
+logger configuration and task retries. We have also started supporting more advanced tools that don't use
+files, so it is impossible to determine the correct file path in every case e.g. Stackdriver doesn't use files
+but identifies logs based on labels. For this reason, we decided to delete this attribute.
+
+If you need to read logs, you can use ``airflow.utils.log.log_reader.TaskLogReader`` class, which does not have
+the above restrictions.
+
+If a sensor times out, it will not retry
+""""""""""""""""""""""""""""""""""""""""
+
+Previously, a sensor is retried when it times out until the number of ``retries`` are exhausted. So the effective timeout of a sensor is ``timeout * (retries + 1)``. This behaviour is now changed. A sensor will immediately fail without retrying if ``timeout`` is reached. If it's desirable to let the sensor continue running for longer time, set a larger ``timeout`` instead.
+
+Default Task Pools Slots can be set using ``[core] default_pool_task_slot_count``
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+By default tasks are running in ``default_pool``. ``default_pool`` is initialized with ``128`` slots and user can change the
+number of slots through UI/CLI/API for an existing deployment.
+
+For new deployments, you can use ``default_pool_task_slot_count`` setting in ``[core]`` section. This setting would
+not have any effect in an existing deployment where the ``default_pool`` already exists.
+
+Previously this was controlled by ``non_pooled_task_slot_count`` in ``[core]`` section, which was not documented.
+
+Webserver DAG refresh buttons removed
+"""""""""""""""""""""""""""""""""""""
+
+Now that the DAG parser syncs DAG permissions there is no longer a need for manually refreshing DAGs. As such, the buttons to refresh a DAG have been removed from the UI.
+
+In addition, the ``/refresh`` and ``/refresh_all`` webserver endpoints have also been removed.
+
+TaskInstances now *require* a DagRun
+""""""""""""""""""""""""""""""""""""""""
+
+Under normal operation every TaskInstance row in the database would have DagRun row too, but it was possible to manually delete the DagRun and Airflow would still schedule the TaskInstances.
+
+In Airflow 2.2 we have changed this and now there is a database-level foreign key constraint ensuring that every TaskInstance has a DagRun row.
+
+Before updating to this 2.2 release you will have to manually resolve any inconsistencies (add back DagRun rows, or delete TaskInstances) if you have any "dangling" TaskInstance" rows.
+
+As part of this change the ``clean_tis_without_dagrun_interval`` config option under ``[scheduler]`` section has been removed and has no effect.
+
+TaskInstance and TaskReschedule now define ``run_id`` instead of ``execution_date``
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+As a part of the TaskInstance-DagRun relation change, the ``execution_date`` columns on TaskInstance and TaskReschedule have been removed from the database, and replaced by `association proxy `_ fields at the ORM level. If you access Airflow’s metadatabase directly, you should rewrite the implementation to use the ``run_id`` columns instead.
+
+Note that Airflow’s metadatabase definition on both the database and ORM levels are considered implementation detail without strict backward compatibility guarantees.
+
+DaskExecutor - Dask Worker Resources and queues
+"""""""""""""""""""""""""""""""""""""""""""""""
+
+If dask workers are not started with complementary resources to match the specified queues, it will now result in an ``AirflowException``\ , whereas before it would have just ignored the ``queue`` argument.
New Features
-""""""""""""
+^^^^^^^^^^^^
- AIP-39: Add (customizable) Timetable class to Airflow for richer scheduling behaviour (#15397, #16030,
#16352, #17030, #17122, #17414, #17552, #17755, #17989, #18084, #18088, #18244, #18266, #18420, #18434,
@@ -256,7 +1243,8 @@ New Features
- Make auto refresh interval configurable (#18107)
Improvements
-""""""""""""
+^^^^^^^^^^^^
+
- Small improvements for Airflow UI (#18715, #18795)
- Rename ``processor_poll_interval`` to ``scheduler_idle_sleep_time`` (#18704)
- Check the allowed values for the logging level (#18651)
@@ -339,7 +1327,8 @@ Improvements
- Accept custom run ID in TriggerDagRunOperator (#18788)
Bug Fixes
-"""""""""
+^^^^^^^^^
+
- Make REST API patch user endpoint work the same way as the UI (#18757)
- Properly set ``start_date`` for cleared tasks (#18708)
- Ensure task_instance exists before running update on its state(REST API) (#18642)
@@ -416,7 +1405,8 @@ Bug Fixes
- Fix bug that Backfill job fail to run when there are tasks run into ``reschedule`` state (#17305, #18806)
Doc only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
+
- Update ``dagbag_size`` documentation (#18824)
- Update documentation about bundle extras (#18828)
- Fix wrong Postgres ``search_path`` set up instructions (#17600)
@@ -481,7 +1471,8 @@ Doc only changes
- Docs: Fix API verb from ``POST`` to ``PATCH`` (#16511)
Misc/Internal
-"""""""""""""
+^^^^^^^^^^^^^
+
- Renaming variables to be consistent with code logic (#18685)
- Simplify strings previously split across lines (#18679)
- fix exception string of ``BranchPythonOperator`` (#18623)
@@ -527,11 +1518,16 @@ Misc/Internal
- Fix deprecated default for ``fab_logging_level`` to ``WARNING`` (#18783)
- Allow running tasks from UI when using ``CeleryKubernetesExecutor`` (#18441)
-Airflow 2.1.4, 2021-09-18
--------------------------
+Airflow 2.1.4 (2021-09-18)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+No significant changes.
Bug Fixes
-"""""""""
+^^^^^^^^^
- Fix deprecation error message rather than silencing it (#18126)
- Limit the number of queued dagruns created by the Scheduler (#18065)
@@ -560,7 +1556,7 @@ Bug Fixes
- Sort adopted tasks in ``_check_for_stalled_adopted_tasks`` method (#18208)
Doc only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
- Update version added fields in airflow/config_templates/config.yml (#18128)
- Improve the description of how to handle dynamic task generation (#17963)
@@ -574,11 +1570,16 @@ Doc only changes
- Doc: Improve installing from sources (#18194)
- Improves installing from sources pages for all components (#18251)
-Airflow 2.1.3, 2021-08-23
--------------------------
+Airflow 2.1.3 (2021-08-23)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+No significant changes.
Bug Fixes
-"""""""""
+^^^^^^^^^
- Fix task retries when they receive ``sigkill`` and have retries and properly handle ``sigterm`` (#16301)
- Fix redacting secrets in context exceptions. (#17618)
@@ -612,7 +1613,7 @@ Bug Fixes
- Stop checking ``execution_date`` in ``task_instance.refresh_from_db`` (#16809)
Improvements
-""""""""""""
+^^^^^^^^^^^^
- Run mini scheduler in ``LocalTaskJob`` during task exit (#16289)
- Remove ``SQLAlchemy<1.4`` constraint (#16630)
@@ -624,21 +1625,26 @@ Improvements
- Move ``DagFileProcessor`` and ``DagFileProcessorProcess`` out of ``scheduler_job.py`` (#16581)
Doc only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
- Fix inconsistencies in configuration docs (#17317)
- Fix docs link for using SQLite as Metadata DB (#17308)
Misc
-""""
+^^^^
- Switch back http provider after requests removes LGPL dependency (#16974)
-Airflow 2.1.2, 2021-07-14
--------------------------
+Airflow 2.1.2 (2021-07-14)
+--------------------------
-Bug Fixes
-"""""""""
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+No significant changes.
+
+Bug Fixes
+^^^^^^^^^
- Only allow webserver to request from the worker log server (#16754)
- Fix "Invalid JSON configuration, must be a dict" bug (#16648)
@@ -648,16 +1654,29 @@ Bug Fixes
- Resolve all npm vulnerabilities including bumping ``jQuery`` to ``3.5`` (#16440)
Misc
-""""
+^^^^
- Add Python 3.9 support (#15515)
-Airflow 2.1.1, 2021-07-02
--------------------------
+Airflow 2.1.1 (2021-07-02)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+``activate_dag_runs`` argument of the function ``clear_task_instances`` is replaced with ``dag_run_state``
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+To achieve the previous default behaviour of ``clear_task_instances`` with ``activate_dag_runs=True``\ , no change is needed. To achieve the previous behaviour of ``activate_dag_runs=False``\ , pass ``dag_run_state=False`` instead. (The previous parameter is still accepted, but is deprecated)
+
+``dag.set_dag_runs_state`` is deprecated
+""""""""""""""""""""""""""""""""""""""""""""
+
+The method ``set_dag_runs_state`` is no longer needed after a bug fix in PR: `#15382 `_. This method is now deprecated and will be removed in a future version.
Bug Fixes
-"""""""""
+^^^^^^^^^
- Don't crash attempting to mask secrets in dict with non-string keys (#16601)
- Always install sphinx_airflow_theme from ``PyPI`` (#16594)
@@ -703,11 +1722,75 @@ Bug Fixes
- Fix Celery executor getting stuck randomly because of reset_signals in multiprocessing (#15989)
-Airflow 2.1.0, 2021-05-21
--------------------------
+Airflow 2.1.0 (2021-05-21)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+New "deprecated_api" extra
+""""""""""""""""""""""""""
+
+We have a new '[deprecated_api]' extra that should be used when installing airflow when the deprecated API
+is going to be used. This is now an optional feature of Airflow now because it pulls in ``requests`` which
+(as of 14 May 2021) pulls LGPL ``chardet`` dependency.
+
+The ``http`` provider is not installed by default
+"""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The ``http`` provider is now optional and not installed by default, until ``chardet`` becomes an optional
+dependency of ``requests``.
+See `PR to replace chardet with charset-normalizer `_
+
+``@apply_default`` decorator isn't longer necessary
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+This decorator is now automatically added to all operators via the metaclass on BaseOperator
+
+Change the configuration options for field masking
+""""""""""""""""""""""""""""""""""""""""""""""""""
+
+We've improved masking for sensitive data in Web UI and logs. As part of it, the following configurations have been changed:
+
+
+* ``hide_sensitive_variable_fields`` option in ``admin`` section has been replaced by ``hide_sensitive_var_conn_fields`` section in ``core`` section,
+* ``sensitive_variable_fields`` option in ``admin`` section has been replaced by ``sensitive_var_conn_names`` section in ``core`` section.
+
+Deprecated PodDefaults and add_xcom_sidecar in airflow.kubernetes.pod_generator
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+We have moved PodDefaults from ``airflow.kubernetes.pod_generator.PodDefaults`` to
+``airflow.providers.cncf.kubernetes.utils.xcom_sidecar.PodDefaults`` and moved add_xcom_sidecar
+from ``airflow.kubernetes.pod_generator.PodGenerator.add_xcom_sidecar``\ to
+``airflow.providers.cncf.kubernetes.utils.xcom_sidecar.add_xcom_sidecar``.
+This change will allow us to modify the KubernetesPodOperator XCom functionality without requiring airflow upgrades.
+
+Removed pod_launcher from core airflow
+""""""""""""""""""""""""""""""""""""""
+
+Moved the pod launcher from ``airflow.kubernetes.pod_launcher`` to ``airflow.providers.cncf.kubernetes.utils.pod_launcher``
+
+This will allow users to update the pod_launcher for the KubernetesPodOperator without requiring an airflow upgrade
+
+Default ``[webserver] worker_refresh_interval`` is changed to ``6000`` seconds
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The default value for ``[webserver] worker_refresh_interval`` was ``30`` seconds for
+Airflow <=2.0.1. However, since Airflow 2.0 DAG Serialization is a hard requirement
+and the Webserver used the serialized DAGs, there is no need to kill an existing
+worker and create a new one as frequently as ``30`` seconds.
+
+This setting can be raised to an even higher value, currently it is
+set to ``6000`` seconds (100 minutes) to
+serve as a DagBag cache burst time.
+
+``default_queue`` configuration has been moved to the ``operators`` section.
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The ``default_queue`` configuration option has been moved from ``[celery]`` section to ``[operators]`` section to allow for re-use between different executors.
New Features
-""""""""""""
+^^^^^^^^^^^^
- Add ``PythonVirtualenvDecorator`` to Taskflow API (#14761)
- Add ``Taskgroup`` decorator (#15034)
@@ -746,7 +1829,7 @@ New Features
- Rename ``DateTimeBranchOperator`` to ``BranchDateTimeOperator`` (#14720)
Improvements
-""""""""""""
+^^^^^^^^^^^^
- Add optional result handler callback to ``DbApiHook`` (#15581)
- Update Flask App Builder limit to recently released 3.3 (#15792)
@@ -786,7 +1869,7 @@ Improvements
- Move ``[celery] default_queue`` config to ``[operators] default_queue`` to re-use between executors (#14699)
Bug Fixes
-"""""""""
+^^^^^^^^^
- Fix 500 error from ``updateTaskInstancesState`` API endpoint when ``dry_run`` not passed (#15889)
- Ensure that task preceding a PythonVirtualenvOperator doesn't fail (#15822)
@@ -839,7 +1922,7 @@ Bug Fixes
- Fix ``task_instance_mutation_hook`` when importing airflow.models.dagrun (#15851)
Doc only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
- Fix docstring of SqlSensor (#15466)
- Small changes on "DAGs and Tasks documentation" (#14853)
@@ -869,7 +1952,7 @@ Doc only changes
- Add note in Updating.md about FAB data model change (#14478)
Misc/Internal
-"""""""""""""
+^^^^^^^^^^^^^
- Fix ``logging.exception`` redundancy (#14823)
- Bump ``stylelint`` to remove vulnerable sub-dependency (#15784)
@@ -911,11 +1994,26 @@ Misc/Internal
- Disable suppress_logs_and_warning in cli when debugging (#13180)
-Airflow 2.0.2, 2021-04-19
--------------------------
+Airflow 2.0.2 (2021-04-19)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+Default ``[kubernetes] enable_tcp_keepalive`` is changed to ``True``
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+This allows Airflow to work more reliably with some environments (like Azure) by default.
+
+``sync-perm`` CLI no longer syncs DAG specific permissions by default
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The ``sync-perm`` CLI command will no longer sync DAG specific permissions by default as they are now being handled during
+DAG parsing. If you need or want the old behavior, you can pass ``--include-dags`` to have ``sync-perm`` also sync DAG
+specific permissions.
Bug Fixes
-"""""""""
+^^^^^^^^^
* Bugfix: ``TypeError`` when Serializing & sorting iterable properties of DAGs (#15395)
* Fix missing ``on_load`` trigger for folder-based plugins (#15208)
@@ -949,7 +2047,7 @@ Bug Fixes
* BugFix: TypeError in airflow.kubernetes.pod_launcher's monitor_pod (#14513)
* Bugfix: Fix wrong output of tags and owners in dag detail API endpoint (#14490)
* Fix logging error with task error when JSON logging is enabled (#14456)
-* Fix statsd metrics not sending when using daemon mode (#14454)
+* Fix StatsD metrics not sending when using daemon mode (#14454)
* Gracefully handle missing start_date and end_date for DagRun (#14452)
* BugFix: Serialize max_retry_delay as a timedelta (#14436)
* Fix crash when user clicks on "Task Instance Details" caused by start_date being None (#14416)
@@ -962,7 +2060,7 @@ Bug Fixes
* Fix running child tasks in a subdag after clearing a successful subdag (#14776)
Improvements
-""""""""""""
+^^^^^^^^^^^^
* Remove unused JS packages causing false security alerts (#15383)
* Change default of ``[kubernetes] enable_tcp_keepalive`` for new installs to ``True`` (#15338)
@@ -988,7 +2086,7 @@ Improvements
* Add better description and guidance in case of sqlite version mismatch (#14209)
Doc only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
* Add documentation create/update community providers (#15061)
* Fix mistake and typos in airflow.utils.timezone docstrings (#15180)
@@ -999,18 +2097,51 @@ Doc only changes
* Replace Graph View Screenshot to show Auto-refresh (#14571)
Misc/Internal
-"""""""""""""
+^^^^^^^^^^^^^
* Import Connection lazily in hooks to avoid cycles (#15361)
* Rename last_scheduler_run into last_parsed_time, and ensure it's updated in DB (#14581)
* Make TaskInstance.pool_slots not nullable with a default of 1 (#14406)
* Log migrations info in consistent way (#14158)
-Airflow 2.0.1, 2021-02-08
--------------------------
+Airflow 2.0.1 (2021-02-08)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+Permission to view Airflow Configurations has been removed from ``User`` and ``Viewer`` role
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Previously, Users with ``User`` or ``Viewer`` role were able to get/view configurations using
+the REST API or in the Webserver. From Airflow 2.0.1, only users with ``Admin`` or ``Op`` role would be able
+to get/view Configurations.
+
+To allow users with other roles to view configuration, add ``can read on Configurations`` permissions to that role.
+
+Note that if ``[webserver] expose_config`` is set to ``False``\ , the API will throw a ``403`` response even if
+the user has role with ``can read on Configurations`` permission.
+
+Default ``[celery] worker_concurrency`` is changed to ``16``
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The default value for ``[celery] worker_concurrency`` was ``16`` for Airflow <2.0.0.
+However, it was unintentionally changed to ``8`` in 2.0.0.
+
+From Airflow 2.0.1, we revert to the old default of ``16``.
+
+Default ``[scheduler] min_file_process_interval`` is changed to ``30``
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The default value for ``[scheduler] min_file_process_interval`` was ``0``\ ,
+due to which the CPU Usage mostly stayed around 100% as the DAG files are parsed
+constantly.
+
+From Airflow 2.0.0, the scheduling decisions have been moved from
+DagFileProcessor to Scheduler, so we can keep the default a bit higher: ``30``.
Bug Fixes
-"""""""""
+^^^^^^^^^
- Bugfix: Return XCom Value in the XCom Endpoint API (#13684)
- Bugfix: Import error when using custom backend and ``sql_alchemy_conn_secret`` (#13260)
@@ -1064,7 +2195,7 @@ Bug Fixes
- Fix DB Migration from 2.0.1rc1
Improvements
-""""""""""""
+^^^^^^^^^^^^
- Increase the default ``min_file_process_interval`` to decrease CPU Usage (#13664)
- Dispose connections when running tasks with ``os.fork`` & ``CeleryExecutor`` (#13265)
@@ -1107,9 +2238,9 @@ Improvements
- Retry critical methods in Scheduler loop in case of ``OperationalError`` (#14032)
Doc only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
-- Add Missing Statsd Metrics in Docs (#13708)
+- Add Missing StatsD Metrics in Docs (#13708)
- Add Missing Email configs in Configuration doc (#13709)
- Add quick start for Airflow on Docker (#13660)
- Describe which Python versions are supported (#13259)
@@ -1151,90 +2282,2522 @@ Doc only changes
- Add deprecated config options to docs (#13883)
- Added a FAQ section to the Upgrading to 2 doc (#13979)
-Airflow 2.0.0, 2020-12-18
-----------------------------
+Airflow 2.0.0 (2020-12-18)
+--------------------------
The full changelog is about 3,000 lines long (already excluding everything backported to 1.10)
so please check `Airflow 2.0.0 Highligths Blog Post `_.
-Improvements
-""""""""""""
+Major changes
+"""""""""""""
-- Adds airflow as viable docker command in official image (#12878)
-- ``StreamLogWriter``: Provide (no-op) close method (#10885)
-- Add 'airflow variables list' command for 1.10.x transition version (#14462)
+This section describes the major changes that have been made in this release.
-Doc only changes
-""""""""""""""""
+The experimental REST API is disabled by default
+""""""""""""""""""""""""""""""""""""""""""""""""
-- Update URL for Airflow docs (#13561)
-- Clarifies version args for installing 1.10 in Docker (#12875)
+The experimental REST API is disabled by default. To restore these APIs while migrating to
+the stable REST API, set ``enable_experimental_api`` option in ``[api]`` section to ``True``.
-Airflow 1.10.14, 2020-12-10
-----------------------------
+Please note that the experimental REST API do not have access control.
+The authenticated user has full access.
-Bug Fixes
-"""""""""
+SparkJDBCHook default connection
+""""""""""""""""""""""""""""""""
-- BugFix: Tasks with ``depends_on_past`` or ``task_concurrency`` are stuck (#12663)
-- Fix issue with empty Resources in executor_config (#12633)
-- Fix: Deprecated config ``force_log_out_after`` was not used (#12661)
-- Fix empty asctime field in JSON formatted logs (#10515)
-- [AIRFLOW-2809] Fix security issue regarding Flask SECRET_KEY (#3651)
-- [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac (#3729)
-- [AIRFLOW-2886] Generate random Flask SECRET_KEY in default config (#3738)
-- Add missing comma in setup.py (#12790)
-- Bugfix: Unable to import Airflow plugins on Python 3.8 (#12859)
-- Fix setup.py missing comma in ``setup_requires`` (#12880)
-- Don't emit first_task_scheduling_delay metric for only-once dags (#12835)
+For SparkJDBCHook default connection was ``spark-default``\ , and for SparkSubmitHook it was
+``spark_default``. Both hooks now use the ``spark_default`` which is a common pattern for the connection
+names used across all providers.
-Improvements
-""""""""""""
+Changes to output argument in commands
+""""""""""""""""""""""""""""""""""""""
-- Update setup.py to get non-conflicting set of dependencies (#12636)
-- Rename ``[scheduler] max_threads`` to ``[scheduler] parsing_processes`` (#12605)
-- Add metric for scheduling delay between first run task & expected start time (#9544)
-- Add new-style 2.0 command names for Airflow 1.10.x (#12725)
-- Add Kubernetes cleanup-pods CLI command for Helm Chart (#11802)
-- Don't let webserver run with dangerous config (#12747)
-- Replace pkg_resources with ``importlib.metadata`` to avoid VersionConflict errors (#12694)
+From Airflow 2.0, We are replacing `tabulate `_ to render commands output. Due to this change, the ``--output`` argument
+will no longer accept formats of tabulate tables. Instead, it now accepts:
-Doc only changes
-""""""""""""""""
-- Clarified information about supported Databases
+* ``table`` - will render the output in predefined table
+* ``json`` - will render the output as a json
+* ``yaml`` - will render the output as yaml
-Airflow 1.10.13, 2020-11-25
-----------------------------
+By doing this we increased consistency and gave users possibility to manipulate the
+output programmatically (when using json or yaml).
-New Features
-""""""""""""
+Affected commands:
+
+
+* ``airflow dags list``
+* ``airflow dags report``
+* ``airflow dags list-runs``
+* ``airflow dags list-jobs``
+* ``airflow connections list``
+* ``airflow connections get``
+* ``airflow pools list``
+* ``airflow pools get``
+* ``airflow pools set``
+* ``airflow pools delete``
+* ``airflow pools import``
+* ``airflow pools export``
+* ``airflow role list``
+* ``airflow providers list``
+* ``airflow providers get``
+* ``airflow providers hooks``
+* ``airflow tasks states-for-dag-run``
+* ``airflow users list``
+* ``airflow variables list``
+
+Azure Wasb Hook does not work together with Snowflake hook
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The WasbHook in Apache Airflow use a legacy version of Azure library. While the conflict is not
+significant for most of the Azure hooks, it is a problem for Wasb Hook because the ``blob`` folders
+for both libraries overlap. Installing both Snowflake and Azure extra will result in non-importable
+WasbHook.
+
+Rename ``all`` to ``devel_all`` extra
+"""""""""""""""""""""""""""""""""""""""""""""
+
+The ``all`` extras were reduced to include only user-facing dependencies. This means
+that this extra does not contain development dependencies. If you were relying on
+``all`` extra then you should use now ``devel_all`` or figure out if you need development
+extras at all.
+
+Context variables ``prev_execution_date_success`` and ``prev_execution_date_success`` are now ``pendulum.DateTime``
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Rename policy to task_policy
+""""""""""""""""""""""""""""
+
+Because Airflow introduced DAG level policy (\ ``dag_policy``\ ) we decided to rename existing ``policy``
+function to ``task_policy`` to make the distinction more profound and avoid any confusion.
+
+Users using cluster policy need to rename their ``policy`` functions in ``airflow_local_settings.py``
+to ``task_policy``.
+
+Default value for ``[celery] operation_timeout`` has changed to ``1.0``
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+From Airflow 2, by default Airflow will retry 3 times to publish task to Celery broker. This is controlled by
+``[celery] task_publish_max_retries``. Because of this we can now have a lower Operation timeout that raises
+``AirflowTaskTimeout``. This generally occurs during network blips or intermittent DNS issues.
+
+Adding Operators and Sensors via plugins is no longer supported
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Operators and Sensors should no longer be registered or imported via Airflow's plugin mechanism -- these types of classes are just treated as plain python classes by Airflow, so there is no need to register them with Airflow.
+
+If you previously had a ``plugins/my_plugin.py`` and you used it like this in a DAG:
+
+.. code-block::
+
+ from airflow.operators.my_plugin import MyOperator
+
+You should instead import it as:
+
+.. code-block::
+
+ from my_plugin import MyOperator
+
+The name under ``airflow.operators.`` was the plugin name, where as in the second example it is the python module name where the operator is defined.
+
+See https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html for more info.
+
+Importing Hooks via plugins is no longer supported
+""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Importing hooks added in plugins via ``airflow.hooks.`` is no longer supported, and hooks should just be imported as regular python modules.
+
+.. code-block::
+
+ from airflow.hooks.my_plugin import MyHook
+
+You should instead import it as:
+
+.. code-block::
+
+ from my_plugin import MyHook
+
+It is still possible (but not required) to "register" hooks in plugins. This is to allow future support for dynamically populating the Connections form in the UI.
+
+See https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html for more info.
+
+The default value for ``[core] enable_xcom_pickling`` has been changed to ``False``
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The pickle type for XCom messages has been replaced to JSON by default to prevent RCE attacks.
+Note that JSON serialization is stricter than pickling, so for example if you want to pass
+raw bytes through XCom you must encode them using an encoding like ``base64``.
+If you understand the risk and still want to use `pickling `_\ ,
+set ``enable_xcom_pickling = True`` in your Airflow config's ``core`` section.
+
+Airflowignore of base path
+""""""""""""""""""""""""""
+
+There was a bug fixed in https://github.com/apache/airflow/pull/11993 that the "airflowignore" checked
+the base path of the dag folder for forbidden dags, not only the relative part. This had the effect
+that if the base path contained the excluded word the whole dag folder could have been excluded. For
+example if the airflowignore file contained x, and the dags folder was '/var/x/dags', then all dags in
+the folder would be excluded. The fix only matches the relative path only now which means that if you
+previously used full path as ignored, you should change it to relative one. For example if your dag
+folder was '/var/dags/' and your airflowignore contained '/var/dag/excluded/', you should change it
+to 'excluded/'.
+
+``ExternalTaskSensor`` provides all task context variables to ``execution_date_fn`` as keyword arguments
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The old syntax of passing ``context`` as a dictionary will continue to work with the caveat that the argument must be named ``context``. The following will break. To fix it, change ``ctx`` to ``context``.
+
+.. code-block:: python
+
+ def execution_date_fn(execution_date, ctx):
+ ...
+
+``execution_date_fn`` can take in any number of keyword arguments available in the task context dictionary. The following forms of ``execution_date_fn`` are all supported:
+
+.. code-block:: python
+
+ def execution_date_fn(dt):
+ ...
+
+
+ def execution_date_fn(execution_date):
+ ...
+
+
+ def execution_date_fn(execution_date, ds_nodash):
+ ...
+
+
+ def execution_date_fn(execution_date, ds_nodash, dag):
+ ...
+
+The default value for ``[webserver] cookie_samesite`` has been changed to ``Lax``
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+As `recommended `_ by Flask, the
+``[webserver] cookie_samesite`` has been changed to ``Lax`` from ``''`` (empty string) .
+
+Changes to import paths
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Formerly the core code was maintained by the original creators - Airbnb. The code that was in the contrib
+package was supported by the community. The project was passed to the Apache community and currently the
+entire code is maintained by the community, so now the division has no justification, and it is only due
+to historical reasons. In Airflow 2.0, we want to organize packages and move integrations
+with third party services to the ``airflow.providers`` package.
+
+All changes made are backward compatible, but if you use the old import paths you will
+see a deprecation warning. The old import paths can be abandoned in the future.
+
+According to `AIP-21 `_
+``_operator`` suffix has been removed from operators. A deprecation warning has also been raised for paths
+importing with the suffix.
+
+The following table shows changes in import paths.
+
+.. list-table::
+ :header-rows: 1
+
+ * - Old path
+ - New path
+ * - ``airflow.hooks.base_hook.BaseHook``
+ - ``airflow.hooks.base.BaseHook``
+ * - ``airflow.hooks.dbapi_hook.DbApiHook``
+ - ``airflow.hooks.dbapi.DbApiHook``
+ * - ``airflow.operators.dummy_operator.DummyOperator``
+ - ``airflow.operators.dummy.DummyOperator``
+ * - ``airflow.operators.dagrun_operator.TriggerDagRunOperator``
+ - ``airflow.operators.trigger_dagrun.TriggerDagRunOperator``
+ * - ``airflow.operators.branch_operator.BaseBranchOperator``
+ - ``airflow.operators.branch.BaseBranchOperator``
+ * - ``airflow.operators.subdag_operator.SubDagOperator``
+ - ``airflow.operators.subdag.SubDagOperator``
+ * - ``airflow.sensors.base_sensor_operator.BaseSensorOperator``
+ - ``airflow.sensors.base.BaseSensorOperator``
+ * - ``airflow.sensors.date_time_sensor.DateTimeSensor``
+ - ``airflow.sensors.date_time.DateTimeSensor``
+ * - ``airflow.sensors.external_task_sensor.ExternalTaskMarker``
+ - ``airflow.sensors.external_task.ExternalTaskMarker``
+ * - ``airflow.sensors.external_task_sensor.ExternalTaskSensor``
+ - ``airflow.sensors.external_task.ExternalTaskSensor``
+ * - ``airflow.sensors.sql_sensor.SqlSensor``
+ - ``airflow.sensors.sql.SqlSensor``
+ * - ``airflow.sensors.time_delta_sensor.TimeDeltaSensor``
+ - ``airflow.sensors.time_delta.TimeDeltaSensor``
+ * - ``airflow.contrib.sensors.weekday_sensor.DayOfWeekSensor``
+ - ``airflow.sensors.weekday.DayOfWeekSensor``
+
+
+Database schema changes
+"""""""""""""""""""""""
+
+In order to migrate the database, you should use the command ``airflow db upgrade``\ , but in
+some cases manual steps are required.
+
+Unique conn_id in connection table
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Previously, Airflow allowed users to add more than one connection with the same ``conn_id`` and on access it would choose one connection randomly. This acted as a basic load balancing and fault tolerance technique, when used in conjunction with retries.
+
+This behavior caused some confusion for users, and there was no clear evidence if it actually worked well or not.
+
+Now the ``conn_id`` will be unique. If you already have duplicates in your metadata database, you will have to manage those duplicate connections before upgrading the database.
+
+Not-nullable conn_type column in connection table
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``conn_type`` column in the ``connection`` table must contain content. Previously, this rule was enforced
+by application logic, but was not enforced by the database schema.
+
+If you made any modifications to the table directly, make sure you don't have
+null in the ``conn_type`` column.
+
+Configuration changes
+"""""""""""""""""""""
+
+This release contains many changes that require a change in the configuration of this application or
+other application that integrate with it.
+
+This section describes the changes that have been made, and what you need to do to.
+
+airflow.contrib.utils.log has been moved
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Formerly the core code was maintained by the original creators - Airbnb. The code that was in the contrib
+package was supported by the community. The project was passed to the Apache community and currently the
+entire code is maintained by the community, so now the division has no justification, and it is only due
+to historical reasons. In Airflow 2.0, we want to organize packages and move integrations
+with third party services to the ``airflow.providers`` package.
+
+To clean up, the following packages were moved:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Old package
+ - New package
+ * - ``airflow.contrib.utils.log``
+ - ``airflow.utils.log``
+ * - ``airflow.utils.log.gcs_task_handler``
+ - ``airflow.providers.google.cloud.log.gcs_task_handler``
+ * - ``airflow.utils.log.wasb_task_handler``
+ - ``airflow.providers.microsoft.azure.log.wasb_task_handler``
+ * - ``airflow.utils.log.stackdriver_task_handler``
+ - ``airflow.providers.google.cloud.log.stackdriver_task_handler``
+ * - ``airflow.utils.log.s3_task_handler``
+ - ``airflow.providers.amazon.aws.log.s3_task_handler``
+ * - ``airflow.utils.log.es_task_handler``
+ - ``airflow.providers.elasticsearch.log.es_task_handler``
+ * - ``airflow.utils.log.cloudwatch_task_handler``
+ - ``airflow.providers.amazon.aws.log.cloudwatch_task_handler``
+
+You should update the import paths if you are setting log configurations with the ``logging_config_class`` option.
+The old import paths still works but can be abandoned.
+
+SendGrid emailer has been moved
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Formerly the core code was maintained by the original creators - Airbnb. The code that was in the contrib
+package was supported by the community. The project was passed to the Apache community and currently the
+entire code is maintained by the community, so now the division has no justification, and it is only due
+to historical reasons.
+
+To clean up, the ``send_mail`` function from the ``airflow.contrib.utils.sendgrid`` module has been moved.
+
+If your configuration file looks like this:
+
+.. code-block:: ini
+
+ [email]
+ email_backend = airflow.contrib.utils.sendgrid.send_email
+
+It should look like this now:
+
+.. code-block:: ini
+
+ [email]
+ email_backend = airflow.providers.sendgrid.utils.emailer.send_email
+
+The old configuration still works but can be abandoned.
+
+Unify ``hostname_callable`` option in ``core`` section
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The previous option used a colon(\ ``:``\ ) to split the module from function. Now the dot(\ ``.``\ ) is used.
+
+The change aims to unify the format of all options that refer to objects in the ``airflow.cfg`` file.
+
+Custom executors is loaded using full import path
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In previous versions of Airflow it was possible to use plugins to load custom executors. It is still
+possible, but the configuration has changed. Now you don't have to create a plugin to configure a
+custom executor, but you need to provide the full path to the module in the ``executor`` option
+in the ``core`` section. The purpose of this change is to simplify the plugin mechanism and make
+it easier to configure executor.
+
+If your module was in the path ``my_acme_company.executors.MyCustomExecutor`` and the plugin was
+called ``my_plugin`` then your configuration looks like this
+
+.. code-block:: ini
+
+ [core]
+ executor = my_plugin.MyCustomExecutor
+
+And now it should look like this:
+
+.. code-block:: ini
+
+ [core]
+ executor = my_acme_company.executors.MyCustomExecutor
+
+The old configuration is still works but can be abandoned at any time.
+
+Use ``CustomSQLAInterface`` instead of ``SQLAInterface`` for custom data models.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+From Airflow 2.0, if you want to define your own Flask App Builder data models you need to use CustomSQLAInterface
+instead of SQLAInterface.
+
+For Non-RBAC replace:
+
+.. code-block:: python
+
+ from flask_appbuilder.models.sqla.interface import SQLAInterface
+
+ datamodel = SQLAInterface(your_data_model)
+
+with RBAC (in 1.10):
+
+.. code-block:: python
+
+ from airflow.www_rbac.utils import CustomSQLAInterface
+
+ datamodel = CustomSQLAInterface(your_data_model)
+
+and in 2.0:
+
+.. code-block:: python
+
+ from airflow.www.utils import CustomSQLAInterface
+
+ datamodel = CustomSQLAInterface(your_data_model)
+
+Drop plugin support for stat_name_handler
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In previous version, you could use plugins mechanism to configure ``stat_name_handler``. You should now use the ``stat_name_handler``
+option in ``[scheduler]`` section to achieve the same effect.
+
+If your plugin looked like this and was available through the ``test_plugin`` path:
+
+.. code-block:: python
+
+ def my_stat_name_handler(stat):
+ return stat
+
+
+ class AirflowTestPlugin(AirflowPlugin):
+ name = "test_plugin"
+ stat_name_handler = my_stat_name_handler
+
+then your ``airflow.cfg`` file should look like this:
+
+.. code-block:: ini
+
+ [scheduler]
+ stat_name_handler=test_plugin.my_stat_name_handler
+
+This change is intended to simplify the statsd configuration.
+
+Logging configuration has been moved to new section
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following configurations have been moved from ``[core]`` to the new ``[logging]`` section.
+
+
+* ``base_log_folder``
+* ``remote_logging``
+* ``remote_log_conn_id``
+* ``remote_base_log_folder``
+* ``encrypt_s3_logs``
+* ``logging_level``
+* ``fab_logging_level``
+* ``logging_config_class``
+* ``colored_console_log``
+* ``colored_log_format``
+* ``colored_formatter_class``
+* ``log_format``
+* ``simple_log_format``
+* ``task_log_prefix_template``
+* ``log_filename_template``
+* ``log_processor_filename_template``
+* ``dag_processor_manager_log_location``
+* ``task_log_reader``
+
+Metrics configuration has been moved to new section
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following configurations have been moved from ``[scheduler]`` to the new ``[metrics]`` section.
+
+
+* ``statsd_on``
+* ``statsd_host``
+* ``statsd_port``
+* ``statsd_prefix``
+* ``statsd_allow_list``
+* ``stat_name_handler``
+* ``statsd_datadog_enabled``
+* ``statsd_datadog_tags``
+* ``statsd_custom_client_path``
+
+Changes to Elasticsearch logging provider
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When JSON output to stdout is enabled, log lines will now contain the ``log_id`` & ``offset`` fields, this should make reading task logs from elasticsearch on the webserver work out of the box. Example configuration:
+
+.. code-block:: ini
+
+ [logging]
+ remote_logging = True
+ [elasticsearch]
+ host = http://es-host:9200
+ write_stdout = True
+ json_format = True
+
+Note that the webserver expects the log line data itself to be present in the ``message`` field of the document.
+
+Remove gcp_service_account_keys option in airflow.cfg file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This option has been removed because it is no longer supported by the Google Kubernetes Engine. The new
+recommended service account keys for the Google Cloud management method is
+`Workload Identity `_.
+
+Fernet is enabled by default
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The fernet mechanism is enabled by default to increase the security of the default installation. In order to
+restore the previous behavior, the user must consciously set an empty key in the ``fernet_key`` option of
+section ``[core]`` in the ``airflow.cfg`` file.
+
+At the same time, this means that the ``apache-airflow[crypto]`` extra-packages are always installed.
+However, this requires that your operating system has ``libffi-dev`` installed.
+
+Changes to propagating Kubernetes worker annotations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``kubernetes_annotations`` configuration section has been removed.
+A new key ``worker_annotations`` has been added to existing ``kubernetes`` section instead.
+That is to remove restriction on the character set for k8s annotation keys.
+All key/value pairs from ``kubernetes_annotations`` should now go to ``worker_annotations`` as a json. I.e. instead of e.g.
+
+.. code-block::
+
+ [kubernetes_annotations]
+ annotation_key = annotation_value
+ annotation_key2 = annotation_value2
+
+it should be rewritten to
+
+.. code-block::
+
+ [kubernetes]
+ worker_annotations = { "annotation_key" : "annotation_value", "annotation_key2" : "annotation_value2" }
+
+Remove run_duration
+~~~~~~~~~~~~~~~~~~~
+
+We should not use the ``run_duration`` option anymore. This used to be for restarting the scheduler from time to time, but right now the scheduler is getting more stable and therefore using this setting is considered bad and might cause an inconsistent state.
+
+Rename pool statsd metrics
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Used slot has been renamed to running slot to make the name self-explanatory
+and the code more maintainable.
+
+This means ``pool.used_slots.`` metric has been renamed to
+``pool.running_slots.``. The ``Used Slots`` column in Pools Web UI view
+has also been changed to ``Running Slots``.
+
+Removal of Mesos Executor
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The Mesos Executor is removed from the code base as it was not widely used and not maintained. `Mailing List Discussion on deleting it `_.
+
+Change dag loading duration metric name
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Change DAG file loading duration metric from
+``dag.loading-duration.`` to ``dag.loading-duration.``. This is to
+better handle the case when a DAG file has multiple DAGs.
+
+Sentry is disabled by default
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Sentry is disabled by default. To enable these integrations, you need set ``sentry_on`` option
+in ``[sentry]`` section to ``"True"``.
+
+Simplified GCSTaskHandler configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In previous versions, in order to configure the service account key file, you had to create a connection entry.
+In the current version, you can configure ``google_key_path`` option in ``[logging]`` section to set
+the key file path.
+
+Users using Application Default Credentials (ADC) need not take any action.
+
+The change aims to simplify the configuration of logging, to prevent corruption of
+the instance configuration by changing the value controlled by the user - connection entry. If you
+configure a backend secret, it also means the webserver doesn't need to connect to it. This
+simplifies setups with multiple GCP projects, because only one project will require the Secret Manager API
+to be enabled.
+
+Changes to the core operators/hooks
+"""""""""""""""""""""""""""""""""""
+
+We strive to ensure that there are no changes that may affect the end user and your files, but this
+release may contain changes that will require changes to your DAG files.
+
+This section describes the changes that have been made, and what you need to do to update your DAG File,
+if you use core operators or any other.
+
+BaseSensorOperator now respects the trigger_rule of downstream tasks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Previously, BaseSensorOperator with setting ``soft_fail=True`` skips itself
+and skips all its downstream tasks unconditionally, when it fails i.e the trigger_rule of downstream tasks is not
+respected.
+
+In the new behavior, the trigger_rule of downstream tasks is respected.
+User can preserve/achieve the original behaviour by setting the trigger_rule of each downstream task to ``all_success``.
+
+BaseOperator uses metaclass
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``BaseOperator`` class uses a ``BaseOperatorMeta`` as a metaclass. This meta class is based on
+``abc.ABCMeta``. If your custom operator uses different metaclass then you will have to adjust it.
+
+Remove SQL support in BaseHook
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Remove ``get_records`` and ``get_pandas_df`` and ``run`` from BaseHook, which only apply for SQL-like hook,
+If want to use them, or your custom hook inherit them, please use ``airflow.hooks.dbapi.DbApiHook``
+
+Assigning task to a DAG using bitwise shift (bit-shift) operators are no longer supported
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Previously, you could assign a task to a DAG as follows:
+
+.. code-block:: python
+
+ dag = DAG("my_dag")
+ dummy = DummyOperator(task_id="dummy")
+
+ dag >> dummy
+
+This is no longer supported. Instead, we recommend using the DAG as context manager:
+
+.. code-block:: python
+
+ with DAG("my_dag") as dag:
+ dummy = DummyOperator(task_id="dummy")
+
+Removed deprecated import mechanism
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The deprecated import mechanism has been removed so the import of modules becomes more consistent and explicit.
+
+For example: ``from airflow.operators import BashOperator``
+becomes ``from airflow.operators.bash_operator import BashOperator``
+
+Changes to sensor imports
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Sensors are now accessible via ``airflow.sensors`` and no longer via ``airflow.operators.sensors``.
+
+For example: ``from airflow.operators.sensors import BaseSensorOperator``
+becomes ``from airflow.sensors.base import BaseSensorOperator``
+
+Skipped tasks can satisfy wait_for_downstream
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Previously, a task instance with ``wait_for_downstream=True`` will only run if the downstream task of
+the previous task instance is successful. Meanwhile, a task instance with ``depends_on_past=True``
+will run if the previous task instance is either successful or skipped. These two flags are close siblings
+yet they have different behavior. This inconsistency in behavior made the API less intuitive to users.
+To maintain consistent behavior, both successful or skipped downstream task can now satisfy the
+``wait_for_downstream=True`` flag.
+
+``airflow.utils.helpers.cross_downstream``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.utils.helpers.chain``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``chain`` and ``cross_downstream`` methods are now moved to airflow.models.baseoperator module from
+``airflow.utils.helpers`` module.
+
+The ``baseoperator`` module seems to be a better choice to keep
+closely coupled methods together. Helpers module is supposed to contain standalone helper methods
+that can be imported by all classes.
+
+The ``chain`` method and ``cross_downstream`` method both use BaseOperator. If any other package imports
+any classes or functions from helpers module, then it automatically has an
+implicit dependency to BaseOperator. That can often lead to cyclic dependencies.
+
+More information in `AIRFLOW-6392 `_
+
+In Airflow < 2.0 you imported those two methods like this:
+
+.. code-block:: python
+
+ from airflow.utils.helpers import chain
+ from airflow.utils.helpers import cross_downstream
+
+In Airflow 2.0 it should be changed to:
+
+.. code-block:: python
+
+ from airflow.models.baseoperator import chain
+ from airflow.models.baseoperator import cross_downstream
+
+``airflow.operators.python.BranchPythonOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``BranchPythonOperator`` will now return a value equal to the ``task_id`` of the chosen branch,
+where previously it returned None. Since it inherits from BaseOperator it will do an
+``xcom_push`` of this value if ``do_xcom_push=True``. This is useful for downstream decision-making.
+
+``airflow.sensors.sql_sensor.SqlSensor``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+SQLSensor now consistent with python ``bool()`` function and the ``allow_null`` parameter has been removed.
+
+It will resolve after receiving any value that is casted to ``True`` with python ``bool(value)``. That
+changes the previous response receiving ``NULL`` or ``'0'``. Earlier ``'0'`` has been treated as success
+criteria. ``NULL`` has been treated depending on value of ``allow_null``\ parameter. But all the previous
+behaviour is still achievable setting param ``success`` to ``lambda x: x is None or str(x) not in ('0', '')``.
+
+``airflow.operators.trigger_dagrun.TriggerDagRunOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The TriggerDagRunOperator now takes a ``conf`` argument to which a dict can be provided as conf for the DagRun.
+As a result, the ``python_callable`` argument was removed. PR: https://github.com/apache/airflow/pull/6317.
+
+``airflow.operators.python.PythonOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``provide_context`` argument on the PythonOperator was removed. The signature of the callable passed to the PythonOperator is now inferred and argument values are always automatically provided. There is no need to explicitly provide or not provide the context anymore. For example:
+
+.. code-block:: python
+
+ def myfunc(execution_date):
+ print(execution_date)
+
+
+ python_operator = PythonOperator(task_id="mytask", python_callable=myfunc, dag=dag)
+
+Notice you don't have to set provide_context=True, variables from the task context are now automatically detected and provided.
+
+All context variables can still be provided with a double-asterisk argument:
+
+.. code-block:: python
+
+ def myfunc(**context):
+ print(context) # all variables will be provided to context
+
+
+ python_operator = PythonOperator(task_id="mytask", python_callable=myfunc)
+
+The task context variable names are reserved names in the callable function, hence a clash with ``op_args`` and ``op_kwargs`` results in an exception:
+
+.. code-block:: python
+
+ def myfunc(dag):
+ # raises a ValueError because "dag" is a reserved name
+ # valid signature example: myfunc(mydag)
+ print("output")
+
+
+ python_operator = PythonOperator(
+ task_id="mytask",
+ op_args=[1],
+ python_callable=myfunc,
+ )
+
+The change is backwards compatible, setting ``provide_context`` will add the ``provide_context`` variable to the ``kwargs`` (but won't do anything).
+
+PR: `#5990 `_
+
+``airflow.sensors.filesystem.FileSensor``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+FileSensor is now takes a glob pattern, not just a filename. If the filename you are looking for has ``*``\ , ``?``\ , or ``[`` in it then you should replace these with ``[*]``\ , ``[?]``\ , and ``[[]``.
+
+``airflow.operators.subdag_operator.SubDagOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``SubDagOperator`` is changed to use Airflow scheduler instead of backfill
+to schedule tasks in the subdag. User no longer need to specify the executor
+in ``SubDagOperator``.
+
+``airflow.providers.google.cloud.operators.datastore.CloudDatastoreExportEntitiesOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.datastore.CloudDatastoreImportEntitiesOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.ssh.operators.ssh.SSHOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.microsoft.winrm.operators.winrm.WinRMOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.operators.bash.BashOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.docker.operators.docker.DockerOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.http.operators.http.SimpleHttpOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``do_xcom_push`` flag (a switch to push the result of an operator to xcom or not) was appearing in different incarnations in different operators. It's function has been unified under a common name (\ ``do_xcom_push``\ ) on ``BaseOperator``. This way it is also easy to globally disable pushing results to xcom.
+
+The following operators were affected:
+
+
+* DatastoreExportOperator (Backwards compatible)
+* DatastoreImportOperator (Backwards compatible)
+* KubernetesPodOperator (Not backwards compatible)
+* SSHOperator (Not backwards compatible)
+* WinRMOperator (Not backwards compatible)
+* BashOperator (Not backwards compatible)
+* DockerOperator (Not backwards compatible)
+* SimpleHttpOperator (Not backwards compatible)
+
+See `AIRFLOW-3249 `_ for details
+
+``airflow.operators.latest_only_operator.LatestOnlyOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In previous versions, the ``LatestOnlyOperator`` forcefully skipped all (direct and indirect) downstream tasks on its own. From this version on the operator will **only skip direct downstream** tasks and the scheduler will handle skipping any further downstream dependencies.
+
+No change is needed if only the default trigger rule ``all_success`` is being used.
+
+If the DAG relies on tasks with other trigger rules (i.e. ``all_done``\ ) being skipped by the ``LatestOnlyOperator``\ , adjustments to the DAG need to be made to accommodate the change in behaviour, i.e. with additional edges from the ``LatestOnlyOperator``.
+
+The goal of this change is to achieve a more consistent and configurable cascading behaviour based on the ``BaseBranchOperator`` (see `AIRFLOW-2923 `_ and `AIRFLOW-1784 `_\ ).
+
+Changes to the core Python API
+""""""""""""""""""""""""""""""
+
+We strive to ensure that there are no changes that may affect the end user, and your Python files, but this
+release may contain changes that will require changes to your plugins, DAG File or other integration.
+
+Only changes unique to this provider are described here. You should still pay attention to the changes that
+have been made to the core (including core operators) as they can affect the integration behavior
+of this provider.
+
+This section describes the changes that have been made, and what you need to do to update your Python files.
+
+Removed sub-package imports from ``airflow/__init__.py``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The imports ``LoggingMixin``\ , ``conf``\ , and ``AirflowException`` have been removed from ``airflow/__init__.py``.
+All implicit references of these objects will no longer be valid. To migrate, all usages of each old path must be
+replaced with its corresponding new path.
+
+.. list-table::
+ :header-rows: 1
+
+ * - Old Path (Implicit Import)
+ - New Path (Explicit Import)
+ * - ``airflow.LoggingMixin``
+ - ``airflow.utils.log.logging_mixin.LoggingMixin``
+ * - ``airflow.conf``
+ - ``airflow.configuration.conf``
+ * - ``airflow.AirflowException``
+ - ``airflow.exceptions.AirflowException``
+
+
+Variables removed from the task instance context
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following variables were removed from the task instance context:
+
+
+* end_date
+* latest_date
+* tables
+
+``airflow.contrib.utils.Weekday``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Formerly the core code was maintained by the original creators - Airbnb. The code that was in the contrib
+package was supported by the community. The project was passed to the Apache community and currently the
+entire code is maintained by the community, so now the division has no justification, and it is only due
+to historical reasons.
+
+To clean up, ``Weekday`` enum has been moved from ``airflow.contrib.utils`` into ``airflow.utils`` module.
+
+``airflow.models.connection.Connection``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The connection module has new deprecated methods:
+
+
+* ``Connection.parse_from_uri``
+* ``Connection.log_info``
+* ``Connection.debug_info``
+
+and one deprecated function:
+
+
+* ``parse_netloc_to_hostname``
+
+Previously, users could create a connection object in two ways
+
+.. code-block::
+
+ conn_1 = Connection(conn_id="conn_a", uri="mysql://AAA/")
+ # or
+ conn_2 = Connection(conn_id="conn_a")
+ conn_2.parse_uri(uri="mysql://AAA/")
+
+Now the second way is not supported.
+
+``Connection.log_info`` and ``Connection.debug_info`` method have been deprecated. Read each Connection field individually or use the
+default representation (\ ``__repr__``\ ).
+
+The old method is still works but can be abandoned at any time. The changes are intended to delete method
+that are rarely used.
+
+``airflow.models.dag.DAG.create_dagrun``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+DAG.create_dagrun accepts run_type and does not require run_id
+This change is caused by adding ``run_type`` column to ``DagRun``.
+
+Previous signature:
+
+.. code-block:: python
+
+ def create_dagrun(
+ self,
+ run_id,
+ state,
+ execution_date=None,
+ start_date=None,
+ external_trigger=False,
+ conf=None,
+ session=None,
+ ):
+ ...
+
+current:
+
+.. code-block:: python
+
+ def create_dagrun(
+ self,
+ state,
+ execution_date=None,
+ run_id=None,
+ start_date=None,
+ external_trigger=False,
+ conf=None,
+ run_type=None,
+ session=None,
+ ):
+ ...
+
+If user provides ``run_id`` then the ``run_type`` will be derived from it by checking prefix, allowed types
+: ``manual``\ , ``scheduled``\ , ``backfill`` (defined by ``airflow.utils.types.DagRunType``\ ).
+
+If user provides ``run_type`` and ``execution_date`` then ``run_id`` is constructed as
+``{run_type}__{execution_data.isoformat()}``.
+
+Airflow should construct dagruns using ``run_type`` and ``execution_date``\ , creation using
+``run_id`` is preserved for user actions.
+
+``airflow.models.dagrun.DagRun``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Use DagRunType.SCHEDULED.value instead of DagRun.ID_PREFIX
+
+All the run_id prefixes for different kind of DagRuns have been grouped into a single
+enum in ``airflow.utils.types.DagRunType``.
+
+Previously, there were defined in various places, example as ``ID_PREFIX`` class variables for
+``DagRun``\ , ``BackfillJob`` and in ``_trigger_dag`` function.
+
+Was:
+
+.. code-block:: pycon
+
+ >> from airflow.models.dagrun import DagRun
+ >> DagRun.ID_PREFIX
+ scheduled__
+
+Replaced by:
+
+.. code-block:: pycon
+
+ >> from airflow.utils.types import DagRunType
+ >> DagRunType.SCHEDULED.value
+ scheduled
+
+``airflow.utils.file.TemporaryDirectory``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We remove ``airflow.utils.file.TemporaryDirectory``
+Since Airflow dropped support for Python < 3.5 there's no need to have this custom
+implementation of ``TemporaryDirectory`` because the same functionality is provided by
+``tempfile.TemporaryDirectory``.
+
+Now users instead of ``import from airflow.utils.files import TemporaryDirectory`` should
+do ``from tempfile import TemporaryDirectory``. Both context managers provide the same
+interface, thus no additional changes should be required.
+
+``airflow.AirflowMacroPlugin``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We removed ``airflow.AirflowMacroPlugin`` class. The class was there in airflow package but it has not been used (apparently since 2015).
+It has been removed.
+
+``airflow.settings.CONTEXT_MANAGER_DAG``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+CONTEXT_MANAGER_DAG was removed from settings. It's role has been taken by ``DagContext`` in
+'airflow.models.dag'. One of the reasons was that settings should be rather static than store
+dynamic context from the DAG, but the main one is that moving the context out of settings allowed to
+untangle cyclic imports between DAG, BaseOperator, SerializedDAG, SerializedBaseOperator which was
+part of AIRFLOW-6010.
+
+``airflow.utils.log.logging_mixin.redirect_stderr``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.utils.log.logging_mixin.redirect_stdout``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Function ``redirect_stderr`` and ``redirect_stdout`` from ``airflow.utils.log.logging_mixin`` module has
+been deleted because it can be easily replaced by the standard library.
+The functions of the standard library are more flexible and can be used in larger cases.
+
+The code below
+
+.. code-block:: python
+
+ import logging
+
+ from airflow.utils.log.logging_mixin import redirect_stderr, redirect_stdout
+
+ logger = logging.getLogger("custom-logger")
+ with redirect_stdout(logger, logging.INFO), redirect_stderr(logger, logging.WARN):
+ print("I love Airflow")
+
+can be replaced by the following code:
+
+.. code-block:: python
+
+ from contextlib import redirect_stdout, redirect_stderr
+ import logging
+
+ from airflow.utils.log.logging_mixin import StreamLogWriter
+
+ logger = logging.getLogger("custom-logger")
+
+ with redirect_stdout(StreamLogWriter(logger, logging.INFO)), redirect_stderr(
+ StreamLogWriter(logger, logging.WARN)
+ ):
+ print("I Love Airflow")
+
+``airflow.models.baseoperator.BaseOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now, additional arguments passed to BaseOperator cause an exception. Previous versions of Airflow took additional arguments and displayed a message on the console. When the
+message was not noticed by users, it caused very difficult to detect errors.
+
+In order to restore the previous behavior, you must set an ``True`` in the ``allow_illegal_arguments``
+option of section ``[operators]`` in the ``airflow.cfg`` file. In the future it is possible to completely
+delete this option.
+
+``airflow.models.dagbag.DagBag``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Passing ``store_serialized_dags`` argument to DagBag.\ **init** and accessing ``DagBag.store_serialized_dags`` property
+are deprecated and will be removed in future versions.
+
+**Previous signature**\ :
+
+.. code-block:: python
+
+ def __init__(
+ dag_folder=None,
+ include_examples=conf.getboolean("core", "LOAD_EXAMPLES"),
+ safe_mode=conf.getboolean("core", "DAG_DISCOVERY_SAFE_MODE"),
+ store_serialized_dags=False,
+ ):
+ ...
+
+**current**\ :
+
+.. code-block:: python
+
+ def __init__(
+ dag_folder=None,
+ include_examples=conf.getboolean("core", "LOAD_EXAMPLES"),
+ safe_mode=conf.getboolean("core", "DAG_DISCOVERY_SAFE_MODE"),
+ read_dags_from_db=False,
+ ):
+ ...
+
+If you were using positional arguments, it requires no change but if you were using keyword
+arguments, please change ``store_serialized_dags`` to ``read_dags_from_db``.
+
+Similarly, if you were using ``DagBag().store_serialized_dags`` property, change it to
+``DagBag().read_dags_from_db``.
+
+Changes in ``google`` provider package
+""""""""""""""""""""""""""""""""""""""""""
+
+We strive to ensure that there are no changes that may affect the end user and your Python files, but this
+release may contain changes that will require changes to your configuration, DAG Files or other integration
+e.g. custom operators.
+
+Only changes unique to this provider are described here. You should still pay attention to the changes that
+have been made to the core (including core operators) as they can affect the integration behavior
+of this provider.
+
+This section describes the changes that have been made, and what you need to do to update your if
+you use operators or hooks which integrate with Google services (including Google Cloud - GCP).
+
+Direct impersonation added to operators communicating with Google services
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`Directly impersonating a service account `_
+has been made possible for operators communicating with Google services via new argument called ``impersonation_chain``
+(\ ``google_impersonation_chain`` in case of operators that also communicate with services of other cloud providers).
+As a result, GCSToS3Operator no longer derivatives from GCSListObjectsOperator.
+
+Normalize gcp_conn_id for Google Cloud
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Previously not all hooks and operators related to Google Cloud use
+``gcp_conn_id`` as parameter for GCP connection. There is currently one parameter
+which apply to most services. Parameters like ``datastore_conn_id``\ , ``bigquery_conn_id``\ ,
+``google_cloud_storage_conn_id`` and similar have been deprecated. Operators that require two connections are not changed.
+
+Following components were affected by normalization:
+
+
+* ``airflow.providers.google.cloud.hooks.datastore.DatastoreHook``
+* ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook``
+* ``airflow.providers.google.cloud.hooks.gcs.GoogleCloudStorageHook``
+* ``airflow.providers.google.cloud.operators.bigquery.BigQueryCheckOperator``
+* ``airflow.providers.google.cloud.operators.bigquery.BigQueryValueCheckOperator``
+* ``airflow.providers.google.cloud.operators.bigquery.BigQueryIntervalCheckOperator``
+* ``airflow.providers.google.cloud.operators.bigquery.BigQueryGetDataOperator``
+* ``airflow.providers.google.cloud.operators.bigquery.BigQueryOperator``
+* ``airflow.providers.google.cloud.operators.bigquery.BigQueryDeleteDatasetOperator``
+* ``airflow.providers.google.cloud.operators.bigquery.BigQueryCreateEmptyDatasetOperator``
+* ``airflow.providers.google.cloud.operators.bigquery.BigQueryTableDeleteOperator``
+* ``airflow.providers.google.cloud.operators.gcs.GoogleCloudStorageCreateBucketOperator``
+* ``airflow.providers.google.cloud.operators.gcs.GoogleCloudStorageListOperator``
+* ``airflow.providers.google.cloud.operators.gcs.GoogleCloudStorageDownloadOperator``
+* ``airflow.providers.google.cloud.operators.gcs.GoogleCloudStorageDeleteOperator``
+* ``airflow.providers.google.cloud.operators.gcs.GoogleCloudStorageBucketCreateAclEntryOperator``
+* ``airflow.providers.google.cloud.operators.gcs.GoogleCloudStorageObjectCreateAclEntryOperator``
+* ``airflow.operators.sql_to_gcs.BaseSQLToGoogleCloudStorageOperator``
+* ``airflow.operators.adls_to_gcs.AdlsToGoogleCloudStorageOperator``
+* ``airflow.operators.gcs_to_s3.GoogleCloudStorageToS3Operator``
+* ``airflow.operators.gcs_to_gcs.GoogleCloudStorageToGoogleCloudStorageOperator``
+* ``airflow.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator``
+* ``airflow.operators.local_to_gcs.FileToGoogleCloudStorageOperator``
+* ``airflow.operators.cassandra_to_gcs.CassandraToGoogleCloudStorageOperator``
+* ``airflow.operators.bigquery_to_bigquery.BigQueryToBigQueryOperator``
+
+Changes to import paths and names of GCP operators and hooks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+According to `AIP-21 `_
+operators related to Google Cloud has been moved from contrib to core.
+The following table shows changes in import paths.
+
+.. list-table::
+ :header-rows: 1
+
+ * - Old path
+ - New path
+ * - ``airflow.contrib.hooks.bigquery_hook.BigQueryHook``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook``
+ * - ``airflow.contrib.hooks.datastore_hook.DatastoreHook``
+ - ``airflow.providers.google.cloud.hooks.datastore.DatastoreHook``
+ * - ``airflow.contrib.hooks.gcp_bigtable_hook.BigtableHook``
+ - ``airflow.providers.google.cloud.hooks.bigtable.BigtableHook``
+ * - ``airflow.contrib.hooks.gcp_cloud_build_hook.CloudBuildHook``
+ - ``airflow.providers.google.cloud.hooks.cloud_build.CloudBuildHook``
+ * - ``airflow.contrib.hooks.gcp_container_hook.GKEClusterHook``
+ - ``airflow.providers.google.cloud.hooks.kubernetes_engine.GKEHook``
+ * - ``airflow.contrib.hooks.gcp_compute_hook.GceHook``
+ - ``airflow.providers.google.cloud.hooks.compute.ComputeEngineHook``
+ * - ``airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook``
+ - ``airflow.providers.google.cloud.hooks.dataflow.DataflowHook``
+ * - ``airflow.contrib.hooks.gcp_dataproc_hook.DataProcHook``
+ - ``airflow.providers.google.cloud.hooks.dataproc.DataprocHook``
+ * - ``airflow.contrib.hooks.gcp_dlp_hook.CloudDLPHook``
+ - ``airflow.providers.google.cloud.hooks.dlp.CloudDLPHook``
+ * - ``airflow.contrib.hooks.gcp_function_hook.GcfHook``
+ - ``airflow.providers.google.cloud.hooks.functions.CloudFunctionsHook``
+ * - ``airflow.contrib.hooks.gcp_kms_hook.GoogleCloudKMSHook``
+ - ``airflow.providers.google.cloud.hooks.kms.CloudKMSHook``
+ * - ``airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook``
+ - ``airflow.providers.google.cloud.hooks.mlengine.MLEngineHook``
+ * - ``airflow.contrib.hooks.gcp_natural_language_hook.CloudNaturalLanguageHook``
+ - ``airflow.providers.google.cloud.hooks.natural_language.CloudNaturalLanguageHook``
+ * - ``airflow.contrib.hooks.gcp_pubsub_hook.PubSubHook``
+ - ``airflow.providers.google.cloud.hooks.pubsub.PubSubHook``
+ * - ``airflow.contrib.hooks.gcp_speech_to_text_hook.GCPSpeechToTextHook``
+ - ``airflow.providers.google.cloud.hooks.speech_to_text.CloudSpeechToTextHook``
+ * - ``airflow.contrib.hooks.gcp_spanner_hook.CloudSpannerHook``
+ - ``airflow.providers.google.cloud.hooks.spanner.SpannerHook``
+ * - ``airflow.contrib.hooks.gcp_sql_hook.CloudSqlDatabaseHook``
+ - ``airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLDatabaseHook``
+ * - ``airflow.contrib.hooks.gcp_sql_hook.CloudSqlHook``
+ - ``airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLHook``
+ * - ``airflow.contrib.hooks.gcp_tasks_hook.CloudTasksHook``
+ - ``airflow.providers.google.cloud.hooks.tasks.CloudTasksHook``
+ * - ``airflow.contrib.hooks.gcp_text_to_speech_hook.GCPTextToSpeechHook``
+ - ``airflow.providers.google.cloud.hooks.text_to_speech.CloudTextToSpeechHook``
+ * - ``airflow.contrib.hooks.gcp_transfer_hook.GCPTransferServiceHook``
+ - ``airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.CloudDataTransferServiceHook``
+ * - ``airflow.contrib.hooks.gcp_translate_hook.CloudTranslateHook``
+ - ``airflow.providers.google.cloud.hooks.translate.CloudTranslateHook``
+ * - ``airflow.contrib.hooks.gcp_video_intelligence_hook.CloudVideoIntelligenceHook``
+ - ``airflow.providers.google.cloud.hooks.video_intelligence.CloudVideoIntelligenceHook``
+ * - ``airflow.contrib.hooks.gcp_vision_hook.CloudVisionHook``
+ - ``airflow.providers.google.cloud.hooks.vision.CloudVisionHook``
+ * - ``airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook``
+ - ``airflow.providers.google.cloud.hooks.gcs.GCSHook``
+ * - ``airflow.contrib.operators.adls_to_gcs.AdlsToGoogleCloudStorageOperator``
+ - ``airflow.operators.adls_to_gcs.AdlsToGoogleCloudStorageOperator``
+ * - ``airflow.contrib.operators.bigquery_check_operator.BigQueryCheckOperator``
+ - ``airflow.providers.google.cloud.operators.bigquery.BigQueryCheckOperator``
+ * - ``airflow.contrib.operators.bigquery_check_operator.BigQueryIntervalCheckOperator``
+ - ``airflow.providers.google.cloud.operators.bigquery.BigQueryIntervalCheckOperator``
+ * - ``airflow.contrib.operators.bigquery_check_operator.BigQueryValueCheckOperator``
+ - ``airflow.providers.google.cloud.operators.bigquery.BigQueryValueCheckOperator``
+ * - ``airflow.contrib.operators.bigquery_get_data.BigQueryGetDataOperator``
+ - ``airflow.providers.google.cloud.operators.bigquery.BigQueryGetDataOperator``
+ * - ``airflow.contrib.operators.bigquery_operator.BigQueryCreateEmptyDatasetOperator``
+ - ``airflow.providers.google.cloud.operators.bigquery.BigQueryCreateEmptyDatasetOperator``
+ * - ``airflow.contrib.operators.bigquery_operator.BigQueryCreateEmptyTableOperator``
+ - ``airflow.providers.google.cloud.operators.bigquery.BigQueryCreateEmptyTableOperator``
+ * - ``airflow.contrib.operators.bigquery_operator.BigQueryCreateExternalTableOperator``
+ - ``airflow.providers.google.cloud.operators.bigquery.BigQueryCreateExternalTableOperator``
+ * - ``airflow.contrib.operators.bigquery_operator.BigQueryDeleteDatasetOperator``
+ - ``airflow.providers.google.cloud.operators.bigquery.BigQueryDeleteDatasetOperator``
+ * - ``airflow.contrib.operators.bigquery_operator.BigQueryOperator``
+ - ``airflow.providers.google.cloud.operators.bigquery.BigQueryExecuteQueryOperator``
+ * - ``airflow.contrib.operators.bigquery_table_delete_operator.BigQueryTableDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.bigquery.BigQueryDeleteTableOperator``
+ * - ``airflow.contrib.operators.bigquery_to_bigquery.BigQueryToBigQueryOperator``
+ - ``airflow.operators.bigquery_to_bigquery.BigQueryToBigQueryOperator``
+ * - ``airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator``
+ - ``airflow.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator``
+ * - ``airflow.contrib.operators.bigquery_to_mysql_operator.BigQueryToMySqlOperator``
+ - ``airflow.operators.bigquery_to_mysql.BigQueryToMySqlOperator``
+ * - ``airflow.contrib.operators.dataflow_operator.DataFlowJavaOperator``
+ - ``airflow.providers.google.cloud.operators.dataflow.DataFlowJavaOperator``
+ * - ``airflow.contrib.operators.dataflow_operator.DataFlowPythonOperator``
+ - ``airflow.providers.google.cloud.operators.dataflow.DataFlowPythonOperator``
+ * - ``airflow.contrib.operators.dataflow_operator.DataflowTemplateOperator``
+ - ``airflow.providers.google.cloud.operators.dataflow.DataflowTemplateOperator``
+ * - ``airflow.contrib.operators.dataproc_operator.DataProcHadoopOperator``
+ - ``airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHadoopJobOperator``
+ * - ``airflow.contrib.operators.dataproc_operator.DataProcHiveOperator``
+ - ``airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHiveJobOperator``
+ * - ``airflow.contrib.operators.dataproc_operator.DataProcJobBaseOperator``
+ - ``airflow.providers.google.cloud.operators.dataproc.DataprocJobBaseOperator``
+ * - ``airflow.contrib.operators.dataproc_operator.DataProcPigOperator``
+ - ``airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPigJobOperator``
+ * - ``airflow.contrib.operators.dataproc_operator.DataProcPySparkOperator``
+ - ``airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPySparkJobOperator``
+ * - ``airflow.contrib.operators.dataproc_operator.DataProcSparkOperator``
+ - ``airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkJobOperator``
+ * - ``airflow.contrib.operators.dataproc_operator.DataProcSparkSqlOperator``
+ - ``airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkSqlJobOperator``
+ * - ``airflow.contrib.operators.dataproc_operator.DataprocClusterCreateOperator``
+ - ``airflow.providers.google.cloud.operators.dataproc.DataprocCreateClusterOperator``
+ * - ``airflow.contrib.operators.dataproc_operator.DataprocClusterDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.dataproc.DataprocDeleteClusterOperator``
+ * - ``airflow.contrib.operators.dataproc_operator.DataprocClusterScaleOperator``
+ - ``airflow.providers.google.cloud.operators.dataproc.DataprocScaleClusterOperator``
+ * - ``airflow.contrib.operators.dataproc_operator.DataprocOperationBaseOperator``
+ - ``airflow.providers.google.cloud.operators.dataproc.DataprocOperationBaseOperator``
+ * - ``airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateInstantiateInlineOperator``
+ - ``airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateInlineWorkflowTemplateOperator``
+ * - ``airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateInstantiateOperator``
+ - ``airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateWorkflowTemplateOperator``
+ * - ``airflow.contrib.operators.datastore_export_operator.DatastoreExportOperator``
+ - ``airflow.providers.google.cloud.operators.datastore.DatastoreExportOperator``
+ * - ``airflow.contrib.operators.datastore_import_operator.DatastoreImportOperator``
+ - ``airflow.providers.google.cloud.operators.datastore.DatastoreImportOperator``
+ * - ``airflow.contrib.operators.file_to_gcs.FileToGoogleCloudStorageOperator``
+ - ``airflow.providers.google.cloud.transfers.local_to_gcs.FileToGoogleCloudStorageOperator``
+ * - ``airflow.contrib.operators.gcp_bigtable_operator.BigtableClusterUpdateOperator``
+ - ``airflow.providers.google.cloud.operators.bigtable.BigtableUpdateClusterOperator``
+ * - ``airflow.contrib.operators.gcp_bigtable_operator.BigtableInstanceCreateOperator``
+ - ``airflow.providers.google.cloud.operators.bigtable.BigtableCreateInstanceOperator``
+ * - ``airflow.contrib.operators.gcp_bigtable_operator.BigtableInstanceDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.bigtable.BigtableDeleteInstanceOperator``
+ * - ``airflow.contrib.operators.gcp_bigtable_operator.BigtableTableCreateOperator``
+ - ``airflow.providers.google.cloud.operators.bigtable.BigtableCreateTableOperator``
+ * - ``airflow.contrib.operators.gcp_bigtable_operator.BigtableTableDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.bigtable.BigtableDeleteTableOperator``
+ * - ``airflow.contrib.operators.gcp_bigtable_operator.BigtableTableWaitForReplicationSensor``
+ - ``airflow.providers.google.cloud.sensors.bigtable.BigtableTableReplicationCompletedSensor``
+ * - ``airflow.contrib.operators.gcp_cloud_build_operator.CloudBuildCreateBuildOperator``
+ - ``airflow.providers.google.cloud.operators.cloud_build.CloudBuildCreateBuildOperator``
+ * - ``airflow.contrib.operators.gcp_compute_operator.GceBaseOperator``
+ - ``airflow.providers.google.cloud.operators.compute.GceBaseOperator``
+ * - ``airflow.contrib.operators.gcp_compute_operator.GceInstanceGroupManagerUpdateTemplateOperator``
+ - ``airflow.providers.google.cloud.operators.compute.GceInstanceGroupManagerUpdateTemplateOperator``
+ * - ``airflow.contrib.operators.gcp_compute_operator.GceInstanceStartOperator``
+ - ``airflow.providers.google.cloud.operators.compute.GceInstanceStartOperator``
+ * - ``airflow.contrib.operators.gcp_compute_operator.GceInstanceStopOperator``
+ - ``airflow.providers.google.cloud.operators.compute.GceInstanceStopOperator``
+ * - ``airflow.contrib.operators.gcp_compute_operator.GceInstanceTemplateCopyOperator``
+ - ``airflow.providers.google.cloud.operators.compute.GceInstanceTemplateCopyOperator``
+ * - ``airflow.contrib.operators.gcp_compute_operator.GceSetMachineTypeOperator``
+ - ``airflow.providers.google.cloud.operators.compute.GceSetMachineTypeOperator``
+ * - ``airflow.contrib.operators.gcp_container_operator.GKEClusterCreateOperator``
+ - ``airflow.providers.google.cloud.operators.kubernetes_engine.GKECreateClusterOperator``
+ * - ``airflow.contrib.operators.gcp_container_operator.GKEClusterDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.kubernetes_engine.GKEDeleteClusterOperator``
+ * - ``airflow.contrib.operators.gcp_container_operator.GKEPodOperator``
+ - ``airflow.providers.google.cloud.operators.kubernetes_engine.GKEStartPodOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPCancelDLPJobOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPCancelDLPJobOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPCreateDLPJobOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPCreateDLPJobOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPCreateDeidentifyTemplateOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPCreateDeidentifyTemplateOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPCreateInspectTemplateOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPCreateInspectTemplateOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPCreateJobTriggerOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPCreateJobTriggerOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPCreateStoredInfoTypeOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPCreateStoredInfoTypeOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPDeidentifyContentOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPDeidentifyContentOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPDeleteDeidentifyTemplateOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPDeleteDeidentifyTemplateOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPDeleteDlpJobOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPDeleteDLPJobOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPDeleteInspectTemplateOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPDeleteInspectTemplateOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPDeleteJobTriggerOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPDeleteJobTriggerOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPDeleteStoredInfoTypeOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPDeleteStoredInfoTypeOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPGetDeidentifyTemplateOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPGetDeidentifyTemplateOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPGetDlpJobOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPGetDLPJobOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPGetInspectTemplateOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPGetInspectTemplateOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPGetJobTripperOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPGetJobTriggerOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPGetStoredInfoTypeOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPGetStoredInfoTypeOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPInspectContentOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPInspectContentOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPListDeidentifyTemplatesOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPListDeidentifyTemplatesOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPListDlpJobsOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPListDLPJobsOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPListInfoTypesOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPListInfoTypesOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPListInspectTemplatesOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPListInspectTemplatesOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPListJobTriggersOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPListJobTriggersOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPListStoredInfoTypesOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPListStoredInfoTypesOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPRedactImageOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPRedactImageOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPReidentifyContentOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPReidentifyContentOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPUpdateDeidentifyTemplateOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPUpdateDeidentifyTemplateOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPUpdateInspectTemplateOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPUpdateInspectTemplateOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPUpdateJobTriggerOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPUpdateJobTriggerOperator``
+ * - ``airflow.contrib.operators.gcp_dlp_operator.CloudDLPUpdateStoredInfoTypeOperator``
+ - ``airflow.providers.google.cloud.operators.dlp.CloudDLPUpdateStoredInfoTypeOperator``
+ * - ``airflow.contrib.operators.gcp_function_operator.GcfFunctionDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.functions.GcfFunctionDeleteOperator``
+ * - ``airflow.contrib.operators.gcp_function_operator.GcfFunctionDeployOperator``
+ - ``airflow.providers.google.cloud.operators.functions.GcfFunctionDeployOperator``
+ * - ``airflow.contrib.operators.gcp_natural_language_operator.CloudNaturalLanguageAnalyzeEntitiesOperator``
+ - ``airflow.providers.google.cloud.operators.natural_language.CloudNaturalLanguageAnalyzeEntitiesOperator``
+ * - ``airflow.contrib.operators.gcp_natural_language_operator.CloudNaturalLanguageAnalyzeEntitySentimentOperator``
+ - ``airflow.providers.google.cloud.operators.natural_language.CloudNaturalLanguageAnalyzeEntitySentimentOperator``
+ * - ``airflow.contrib.operators.gcp_natural_language_operator.CloudNaturalLanguageAnalyzeSentimentOperator``
+ - ``airflow.providers.google.cloud.operators.natural_language.CloudNaturalLanguageAnalyzeSentimentOperator``
+ * - ``airflow.contrib.operators.gcp_natural_language_operator.CloudNaturalLanguageClassifyTextOperator``
+ - ``airflow.providers.google.cloud.operators.natural_language.CloudNaturalLanguageClassifyTextOperator``
+ * - ``airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.spanner.SpannerDeleteDatabaseInstanceOperator``
+ * - ``airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseDeployOperator``
+ - ``airflow.providers.google.cloud.operators.spanner.SpannerDeployDatabaseInstanceOperator``
+ * - ``airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseQueryOperator``
+ - ``airflow.providers.google.cloud.operators.spanner.SpannerQueryDatabaseInstanceOperator``
+ * - ``airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseUpdateOperator``
+ - ``airflow.providers.google.cloud.operators.spanner.SpannerUpdateDatabaseInstanceOperator``
+ * - ``airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.spanner.SpannerDeleteInstanceOperator``
+ * - ``airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDeployOperator``
+ - ``airflow.providers.google.cloud.operators.spanner.SpannerDeployInstanceOperator``
+ * - ``airflow.contrib.operators.gcp_speech_to_text_operator.GcpSpeechToTextRecognizeSpeechOperator``
+ - ``airflow.providers.google.cloud.operators.speech_to_text.CloudSpeechToTextRecognizeSpeechOperator``
+ * - ``airflow.contrib.operators.gcp_text_to_speech_operator.GcpTextToSpeechSynthesizeOperator``
+ - ``airflow.providers.google.cloud.operators.text_to_speech.CloudTextToSpeechSynthesizeOperator``
+ * - ``airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceJobCreateOperator``
+ - ``airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceCreateJobOperator``
+ * - ``airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceJobDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceDeleteJobOperator``
+ * - ``airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceJobUpdateOperator``
+ - ``airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceUpdateJobOperator``
+ * - ``airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceOperationCancelOperator``
+ - ``airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceCancelOperationOperator``
+ * - ``airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceOperationGetOperator``
+ - ``airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceGetOperationOperator``
+ * - ``airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceOperationPauseOperator``
+ - ``airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServicePauseOperationOperator``
+ * - ``airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceOperationResumeOperator``
+ - ``airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceResumeOperationOperator``
+ * - ``airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceOperationsListOperator``
+ - ``airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceListOperationsOperator``
+ * - ``airflow.contrib.operators.gcp_transfer_operator.GoogleCloudStorageToGoogleCloudStorageTransferOperator``
+ - ``airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceGCSToGCSOperator``
+ * - ``airflow.contrib.operators.gcp_translate_operator.CloudTranslateTextOperator``
+ - ``airflow.providers.google.cloud.operators.translate.CloudTranslateTextOperator``
+ * - ``airflow.contrib.operators.gcp_translate_speech_operator.GcpTranslateSpeechOperator``
+ - ``airflow.providers.google.cloud.operators.translate_speech.GcpTranslateSpeechOperator``
+ * - ``airflow.contrib.operators.gcp_video_intelligence_operator.CloudVideoIntelligenceDetectVideoExplicitContentOperator``
+ - ``airflow.providers.google.cloud.operators.video_intelligence.CloudVideoIntelligenceDetectVideoExplicitContentOperator``
+ * - ``airflow.contrib.operators.gcp_video_intelligence_operator.CloudVideoIntelligenceDetectVideoLabelsOperator``
+ - ``airflow.providers.google.cloud.operators.video_intelligence.CloudVideoIntelligenceDetectVideoLabelsOperator``
+ * - ``airflow.contrib.operators.gcp_video_intelligence_operator.CloudVideoIntelligenceDetectVideoShotsOperator``
+ - ``airflow.providers.google.cloud.operators.video_intelligence.CloudVideoIntelligenceDetectVideoShotsOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionAddProductToProductSetOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionAddProductToProductSetOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionAnnotateImageOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionImageAnnotateOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionDetectDocumentTextOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionTextDetectOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionDetectImageLabelsOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionDetectImageLabelsOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionDetectImageSafeSearchOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionDetectImageSafeSearchOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionDetectTextOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionDetectTextOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionProductCreateOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionCreateProductOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionProductDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionDeleteProductOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionProductGetOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionGetProductOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionProductSetCreateOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionCreateProductSetOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionProductSetDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionDeleteProductSetOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionProductSetGetOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionGetProductSetOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionProductSetUpdateOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionUpdateProductSetOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionProductUpdateOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionUpdateProductOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionReferenceImageCreateOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionCreateReferenceImageOperator``
+ * - ``airflow.contrib.operators.gcp_vision_operator.CloudVisionRemoveProductFromProductSetOperator``
+ - ``airflow.providers.google.cloud.operators.vision.CloudVisionRemoveProductFromProductSetOperator``
+ * - ``airflow.contrib.operators.gcs_acl_operator.GoogleCloudStorageBucketCreateAclEntryOperator``
+ - ``airflow.providers.google.cloud.operators.gcs.GCSBucketCreateAclEntryOperator``
+ * - ``airflow.contrib.operators.gcs_acl_operator.GoogleCloudStorageObjectCreateAclEntryOperator``
+ - ``airflow.providers.google.cloud.operators.gcs.GCSObjectCreateAclEntryOperator``
+ * - ``airflow.contrib.operators.gcs_delete_operator.GoogleCloudStorageDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.gcs.GCSDeleteObjectsOperator``
+ * - ``airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator``
+ - ``airflow.providers.google.cloud.operators.gcs.GCSToLocalFilesystemOperator``
+ * - ``airflow.contrib.operators.gcs_list_operator.GoogleCloudStorageListOperator``
+ - ``airflow.providers.google.cloud.operators.gcs.GCSListObjectsOperator``
+ * - ``airflow.contrib.operators.gcs_operator.GoogleCloudStorageCreateBucketOperator``
+ - ``airflow.providers.google.cloud.operators.gcs.GCSCreateBucketOperator``
+ * - ``airflow.contrib.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator``
+ - ``airflow.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator``
+ * - ``airflow.contrib.operators.gcs_to_gcs.GoogleCloudStorageToGoogleCloudStorageOperator``
+ - ``airflow.operators.gcs_to_gcs.GoogleCloudStorageToGoogleCloudStorageOperator``
+ * - ``airflow.contrib.operators.gcs_to_s3.GoogleCloudStorageToS3Operator``
+ - ``airflow.operators.gcs_to_s3.GCSToS3Operator``
+ * - ``airflow.contrib.operators.mlengine_operator.MLEngineBatchPredictionOperator``
+ - ``airflow.providers.google.cloud.operators.mlengine.MLEngineStartBatchPredictionJobOperator``
+ * - ``airflow.contrib.operators.mlengine_operator.MLEngineModelOperator``
+ - ``airflow.providers.google.cloud.operators.mlengine.MLEngineManageModelOperator``
+ * - ``airflow.contrib.operators.mlengine_operator.MLEngineTrainingOperator``
+ - ``airflow.providers.google.cloud.operators.mlengine.MLEngineStartTrainingJobOperator``
+ * - ``airflow.contrib.operators.mlengine_operator.MLEngineVersionOperator``
+ - ``airflow.providers.google.cloud.operators.mlengine.MLEngineManageVersionOperator``
+ * - ``airflow.contrib.operators.mssql_to_gcs.MsSqlToGoogleCloudStorageOperator``
+ - ``airflow.operators.mssql_to_gcs.MsSqlToGoogleCloudStorageOperator``
+ * - ``airflow.contrib.operators.mysql_to_gcs.MySqlToGoogleCloudStorageOperator``
+ - ``airflow.operators.mysql_to_gcs.MySqlToGoogleCloudStorageOperator``
+ * - ``airflow.contrib.operators.postgres_to_gcs_operator.PostgresToGoogleCloudStorageOperator``
+ - ``airflow.operators.postgres_to_gcs.PostgresToGoogleCloudStorageOperator``
+ * - ``airflow.contrib.operators.pubsub_operator.PubSubPublishOperator``
+ - ``airflow.providers.google.cloud.operators.pubsub.PubSubPublishMessageOperator``
+ * - ``airflow.contrib.operators.pubsub_operator.PubSubSubscriptionCreateOperator``
+ - ``airflow.providers.google.cloud.operators.pubsub.PubSubCreateSubscriptionOperator``
+ * - ``airflow.contrib.operators.pubsub_operator.PubSubSubscriptionDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.pubsub.PubSubDeleteSubscriptionOperator``
+ * - ``airflow.contrib.operators.pubsub_operator.PubSubTopicCreateOperator``
+ - ``airflow.providers.google.cloud.operators.pubsub.PubSubCreateTopicOperator``
+ * - ``airflow.contrib.operators.pubsub_operator.PubSubTopicDeleteOperator``
+ - ``airflow.providers.google.cloud.operators.pubsub.PubSubDeleteTopicOperator``
+ * - ``airflow.contrib.operators.sql_to_gcs.BaseSQLToGoogleCloudStorageOperator``
+ - ``airflow.operators.sql_to_gcs.BaseSQLToGoogleCloudStorageOperator``
+ * - ``airflow.contrib.sensors.bigquery_sensor.BigQueryTableSensor``
+ - ``airflow.providers.google.cloud.sensors.bigquery.BigQueryTableExistenceSensor``
+ * - ``airflow.contrib.sensors.gcp_transfer_sensor.GCPTransferServiceWaitForJobStatusSensor``
+ - ``airflow.providers.google.cloud.sensors.cloud_storage_transfer_service.DataTransferServiceJobStatusSensor``
+ * - ``airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectSensor``
+ - ``airflow.providers.google.cloud.sensors.gcs.GCSObjectExistenceSensor``
+ * - ``airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectUpdatedSensor``
+ - ``airflow.providers.google.cloud.sensors.gcs.GCSObjectUpdateSensor``
+ * - ``airflow.contrib.sensors.gcs_sensor.GoogleCloudStoragePrefixSensor``
+ - ``airflow.providers.google.cloud.sensors.gcs.GCSObjectsWithPrefixExistenceSensor``
+ * - ``airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageUploadSessionCompleteSensor``
+ - ``airflow.providers.google.cloud.sensors.gcs.GCSUploadSessionCompleteSensor``
+ * - ``airflow.contrib.sensors.pubsub_sensor.PubSubPullSensor``
+ - ``airflow.providers.google.cloud.sensors.pubsub.PubSubPullSensor``
+
+
+Unify default conn_id for Google Cloud
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Previously not all hooks and operators related to Google Cloud use
+``google_cloud_default`` as a default conn_id. There is currently one default
+variant. Values like ``google_cloud_storage_default``\ , ``bigquery_default``\ ,
+``google_cloud_datastore_default`` have been deprecated. The configuration of
+existing relevant connections in the database have been preserved. To use those
+deprecated GCP conn_id, you need to explicitly pass their conn_id into
+operators/hooks. Otherwise, ``google_cloud_default`` will be used as GCP's conn_id
+by default.
+
+``airflow.providers.google.cloud.hooks.dataflow.DataflowHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.dataflow.DataflowCreateJavaJobOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.dataflow.DataflowTemplatedJobStartOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.dataflow.DataflowCreatePythonJobOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To use project_id argument consistently across GCP hooks and operators, we did the following changes:
+
+
+* Changed order of arguments in DataflowHook.start_python_dataflow. Uses
+ with positional arguments may break.
+* Changed order of arguments in DataflowHook.is_job_dataflow_running. Uses
+ with positional arguments may break.
+* Changed order of arguments in DataflowHook.cancel_job. Uses
+ with positional arguments may break.
+* Added optional project_id argument to DataflowCreateJavaJobOperator
+ constructor.
+* Added optional project_id argument to DataflowTemplatedJobStartOperator
+ constructor.
+* Added optional project_id argument to DataflowCreatePythonJobOperator
+ constructor.
+
+``airflow.providers.google.cloud.sensors.gcs.GCSUploadSessionCompleteSensor``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To provide more precise control in handling of changes to objects in
+underlying GCS Bucket the constructor of this sensor now has changed.
+
+
+* Old Behavior: This constructor used to optionally take ``previous_num_objects: int``.
+* New replacement constructor kwarg: ``previous_objects: Optional[Set[str]]``.
+
+Most users would not specify this argument because the bucket begins empty
+and the user wants to treat any files as new.
+
+Example of Updating usage of this sensor:
+Users who used to call:
+
+``GCSUploadSessionCompleteSensor(bucket='my_bucket', prefix='my_prefix', previous_num_objects=1)``
+
+Will now call:
+
+``GCSUploadSessionCompleteSensor(bucket='my_bucket', prefix='my_prefix', previous_num_objects={'.keep'})``
+
+Where '.keep' is a single file at your prefix that the sensor should not consider new.
+
+``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To simplify BigQuery operators (no need of ``Cursor``\ ) and standardize usage of hooks within all GCP integration methods from ``BiqQueryBaseCursor``
+were moved to ``BigQueryHook``. Using them by from ``Cursor`` object is still possible due to preserved backward compatibility but they will raise ``DeprecationWarning``.
+The following methods were moved:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Old path
+ - New path
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.cancel_query``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.cancel_query``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.create_empty_dataset``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.create_empty_dataset``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.create_empty_table``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.create_empty_table``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.create_external_table``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.create_external_table``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.delete_dataset``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.delete_dataset``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_dataset``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_dataset``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_dataset_tables``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_dataset_tables``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_dataset_tables_list``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_dataset_tables_list``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_datasets_list``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_datasets_list``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_schema``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_schema``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_tabledata``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_tabledata``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.insert_all``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.insert_all``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.patch_dataset``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.patch_dataset``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.patch_table``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.patch_table``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.poll_job_complete``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.poll_job_complete``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_copy``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_copy``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_extract``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_extract``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_grant_dataset_view_access``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_grant_dataset_view_access``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_load``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_load``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_query``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_query``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_table_delete``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_table_delete``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_table_upsert``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_table_upsert``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_with_configuration``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_with_configuration``
+ * - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.update_dataset``
+ - ``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.update_dataset``
+
+
+``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Since BigQuery is the part of the GCP it was possible to simplify the code by handling the exceptions
+by usage of the ``airflow.providers.google.common.hooks.base.GoogleBaseHook.catch_http_exception`` decorator however it changes
+exceptions raised by the following methods:
+
+
+* ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_table_delete`` raises ``AirflowException`` instead of ``Exception``.
+* ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.create_empty_dataset`` raises ``AirflowException`` instead of ``ValueError``.
+* ``airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_dataset`` raises ``AirflowException`` instead of ``ValueError``.
+
+``airflow.providers.google.cloud.operators.bigquery.BigQueryCreateEmptyTableOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.bigquery.BigQueryCreateEmptyDatasetOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Idempotency was added to ``BigQueryCreateEmptyTableOperator`` and ``BigQueryCreateEmptyDatasetOperator``.
+But to achieve that try / except clause was removed from ``create_empty_dataset`` and ``create_empty_table``
+methods of ``BigQueryHook``.
+
+``airflow.providers.google.cloud.hooks.dataflow.DataflowHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.hooks.mlengine.MLEngineHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.hooks.pubsub.PubSubHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The change in GCP operators implies that GCP Hooks for those operators require now keyword parameters rather
+than positional ones in all methods where ``project_id`` is used. The methods throw an explanatory exception
+in case they are called using positional parameters.
+
+Other GCP hooks are unaffected.
+
+``airflow.providers.google.cloud.hooks.pubsub.PubSubHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.pubsub.PubSubTopicCreateOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.pubsub.PubSubSubscriptionCreateOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.pubsub.PubSubTopicDeleteOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.pubsub.PubSubSubscriptionDeleteOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.pubsub.PubSubPublishOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.sensors.pubsub.PubSubPullSensor``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the ``PubSubPublishOperator`` and ``PubSubHook.publsh`` method the data field in a message should be bytestring (utf-8 encoded) rather than base64 encoded string.
+
+Due to the normalization of the parameters within GCP operators and hooks a parameters like ``project`` or ``topic_project``
+are deprecated and will be substituted by parameter ``project_id``.
+In ``PubSubHook.create_subscription`` hook method in the parameter ``subscription_project`` is replaced by ``subscription_project_id``.
+Template fields are updated accordingly and old ones may not work.
+
+It is required now to pass key-word only arguments to ``PubSub`` hook.
+
+These changes are not backward compatible.
+
+``airflow.providers.google.cloud.operators.kubernetes_engine.GKEStartPodOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The gcp_conn_id parameter in GKEPodOperator is required. In previous versions, it was possible to pass
+the ``None`` value to the ``gcp_conn_id`` in the GKEStartPodOperator
+operator, which resulted in credentials being determined according to the
+`Application Default Credentials `_ strategy.
+
+Now this parameter requires a value. To restore the previous behavior, configure the connection without
+specifying the service account.
+
+Detailed information about connection management is available:
+`Google Cloud Connection `_.
+
+``airflow.providers.google.cloud.hooks.gcs.GCSHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+*
+ The following parameters have been replaced in all the methods in GCSHook:
+
+
+ * ``bucket`` is changed to ``bucket_name``
+ * ``object`` is changed to ``object_name``
+
+*
+ The ``maxResults`` parameter in ``GoogleCloudStorageHook.list`` has been renamed to ``max_results`` for consistency.
+
+``airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPigJobOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHiveJobOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkSqlJobOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkJobOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHadoopJobOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPySparkJobOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The 'properties' and 'jars' properties for the Dataproc related operators (\ ``DataprocXXXOperator``\ ) have been renamed from
+``dataproc_xxxx_properties`` and ``dataproc_xxx_jars`` to ``dataproc_properties``
+and ``dataproc_jars``\ respectively.
+Arguments for dataproc_properties dataproc_jars
+
+``airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceCreateJobOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To obtain pylint compatibility the ``filter`` argument in ``CloudDataTransferServiceCreateJobOperator``
+has been renamed to ``request_filter``.
+
+``airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.CloudDataTransferServiceHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ To obtain pylint compatibility the ``filter`` argument in ``CloudDataTransferServiceHook.list_transfer_job`` and
+ ``CloudDataTransferServiceHook.list_transfer_operations`` has been renamed to ``request_filter``.
+
+``airflow.providers.google.cloud.hooks.bigquery.BigQueryHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In general all hook methods are decorated with ``@GoogleBaseHook.fallback_to_default_project_id`` thus
+parameters to hook can only be passed via keyword arguments.
+
+
+* ``create_empty_table`` method accepts now ``table_resource`` parameter. If provided all
+ other parameters are ignored.
+* ``create_empty_dataset`` will now use values from ``dataset_reference`` instead of raising error
+ if parameters were passed in ``dataset_reference`` and as arguments to method. Additionally validation
+ of ``dataset_reference`` is done using ``Dataset.from_api_repr``. Exception and log messages has been
+ changed.
+* ``update_dataset`` requires now new ``fields`` argument (breaking change)
+* ``delete_dataset`` has new signature (dataset_id, project_id, ...)
+ previous one was (project_id, dataset_id, ...) (breaking change)
+* ``get_tabledata`` returns list of rows instead of API response in dict format. This method is deprecated in
+ favor of ``list_rows``. (breaking change)
+
+``airflow.providers.google.cloud.hooks.cloud_build.CloudBuildHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.cloud_build.CloudBuildCreateBuildOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``api_version`` has been removed and will not be used since we migrate ``CloudBuildHook`` from using
+ Discovery API to native google-cloud-build python library.
+
+The ``body`` parameter in ``CloudBuildCreateBuildOperator`` has been deprecated.
+ Instead, you should pass body using the ``build`` parameter.
+
+``airflow.providers.google.cloud.hooks.dataflow.DataflowHook.start_python_dataflow``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.hooks.dataflow.DataflowHook.start_python_dataflow``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.google.cloud.operators.dataflow.DataflowCreatePythonJobOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Change python3 as Dataflow Hooks/Operators default interpreter
+
+Now the ``py_interpreter`` argument for DataFlow Hooks/Operators has been changed from python2 to python3.
+
+``airflow.providers.google.common.hooks.base_google.GoogleBaseHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To simplify the code, the decorator provide_gcp_credential_file has been moved from the inner-class.
+
+Instead of ``@GoogleBaseHook._Decorators.provide_gcp_credential_file``\ ,
+you should write ``@GoogleBaseHook.provide_gcp_credential_file``
+
+``airflow.providers.google.cloud.operators.dataproc.DataprocCreateClusterOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It is highly recommended to have 1TB+ disk size for Dataproc to have sufficient throughput:
+https://cloud.google.com/compute/docs/disks/performance
+
+Hence, the default value for ``master_disk_size`` in ``DataprocCreateClusterOperator`` has been changed from 500GB to 1TB.
+
+Generating Cluster Config
+"""""""""""""""""""""""""
+
+If you are upgrading from Airflow 1.10.x and are not using **CLUSTER_CONFIG**\ ,
+You can easily generate config using **make()** of ``airflow.providers.google.cloud.operators.dataproc.ClusterGenerator``
+
+This has been proved specially useful if you are using **metadata** argument from older API, refer `AIRFLOW-16911 `_ for details.
+
+eg. your cluster creation may look like this in **v1.10.x**
+
+.. code-block:: python
+
+ path = f"gs://goog-dataproc-initialization-actions-us-central1/python/pip-install.sh"
+
+ create_cluster = DataprocClusterCreateOperator(
+ task_id="create_dataproc_cluster",
+ cluster_name="test",
+ project_id="test",
+ zone="us-central1-a",
+ region="us-central1",
+ master_machine_type="n1-standard-4",
+ worker_machine_type="n1-standard-4",
+ num_workers=2,
+ storage_bucket="test_bucket",
+ init_actions_uris=[path],
+ metadata={"PIP_PACKAGES": "pyyaml requests pandas openpyxl"},
+ )
+
+After upgrading to **v2.x.x** and using **CLUSTER_CONFIG**\ , it will look like followed:
+
+.. code-block:: python
+
+ path = f"gs://goog-dataproc-initialization-actions-us-central1/python/pip-install.sh"
+
+ CLUSTER_CONFIG = ClusterGenerator(
+ project_id="test",
+ zone="us-central1-a",
+ master_machine_type="n1-standard-4",
+ worker_machine_type="n1-standard-4",
+ num_workers=2,
+ storage_bucket="test",
+ init_actions_uris=[path],
+ metadata={"PIP_PACKAGES": "pyyaml requests pandas openpyxl"},
+ ).make()
+
+ create_cluster_operator = DataprocClusterCreateOperator(
+ task_id="create_dataproc_cluster",
+ cluster_name="test",
+ project_id="test",
+ region="us-central1",
+ cluster_config=CLUSTER_CONFIG,
+ )
+
+``airflow.providers.google.cloud.operators.bigquery.BigQueryGetDatasetTablesOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We changed signature of ``BigQueryGetDatasetTablesOperator``.
+
+Before:
+
+.. code-block:: python
+
+ def __init__(
+ dataset_id: str,
+ dataset_resource: dict,
+ # ...
+ ):
+ ...
+
+After:
+
+.. code-block:: python
+
+ def __init__(
+ dataset_resource: dict,
+ dataset_id: Optional[str] = None,
+ # ...
+ ):
+ ...
+
+Changes in ``amazon`` provider package
+""""""""""""""""""""""""""""""""""""""""""
+
+We strive to ensure that there are no changes that may affect the end user, and your Python files, but this
+release may contain changes that will require changes to your configuration, DAG Files or other integration
+e.g. custom operators.
+
+Only changes unique to this provider are described here. You should still pay attention to the changes that
+have been made to the core (including core operators) as they can affect the integration behavior
+of this provider.
+
+This section describes the changes that have been made, and what you need to do to update your if
+you use operators or hooks which integrate with Amazon services (including Amazon Web Service - AWS).
+
+Migration of AWS components
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+All AWS components (hooks, operators, sensors, example DAGs) will be grouped together as decided in
+`AIP-21 `_. Migrated
+components remain backwards compatible but raise a ``DeprecationWarning`` when imported from the old module.
+Migrated are:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Old path
+ - New path
+ * - ``airflow.hooks.S3_hook.S3Hook``
+ - ``airflow.providers.amazon.aws.hooks.s3.S3Hook``
+ * - ``airflow.contrib.hooks.aws_athena_hook.AWSAthenaHook``
+ - ``airflow.providers.amazon.aws.hooks.athena.AWSAthenaHook``
+ * - ``airflow.contrib.hooks.aws_lambda_hook.AwsLambdaHook``
+ - ``airflow.providers.amazon.aws.hooks.lambda_function.AwsLambdaHook``
+ * - ``airflow.contrib.hooks.aws_sqs_hook.SQSHook``
+ - ``airflow.providers.amazon.aws.hooks.sqs.SQSHook``
+ * - ``airflow.contrib.hooks.aws_sns_hook.AwsSnsHook``
+ - ``airflow.providers.amazon.aws.hooks.sns.AwsSnsHook``
+ * - ``airflow.contrib.operators.aws_athena_operator.AWSAthenaOperator``
+ - ``airflow.providers.amazon.aws.operators.athena.AWSAthenaOperator``
+ * - ``airflow.contrib.operators.awsbatch.AWSBatchOperator``
+ - ``airflow.providers.amazon.aws.operators.batch.AwsBatchOperator``
+ * - ``airflow.contrib.operators.awsbatch.BatchProtocol``
+ - ``airflow.providers.amazon.aws.hooks.batch_client.AwsBatchProtocol``
+ * - private attrs and methods on ``AWSBatchOperator``
+ - ``airflow.providers.amazon.aws.hooks.batch_client.AwsBatchClient``
+ * - n/a
+ - ``airflow.providers.amazon.aws.hooks.batch_waiters.AwsBatchWaiters``
+ * - ``airflow.contrib.operators.aws_sqs_publish_operator.SQSPublishOperator``
+ - ``airflow.providers.amazon.aws.operators.sqs.SQSPublishOperator``
+ * - ``airflow.contrib.operators.aws_sns_publish_operator.SnsPublishOperator``
+ - ``airflow.providers.amazon.aws.operators.sns.SnsPublishOperator``
+ * - ``airflow.contrib.sensors.aws_athena_sensor.AthenaSensor``
+ - ``airflow.providers.amazon.aws.sensors.athena.AthenaSensor``
+ * - ``airflow.contrib.sensors.aws_sqs_sensor.SQSSensor``
+ - ``airflow.providers.amazon.aws.sensors.sqs.SQSSensor``
+
+
+``airflow.providers.amazon.aws.hooks.emr.EmrHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.amazon.aws.operators.emr_add_steps.EmrAddStepsOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.amazon.aws.operators.emr_create_job_flow.EmrCreateJobFlowOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.amazon.aws.operators.emr_terminate_job_flow.EmrTerminateJobFlowOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The default value for the `aws_conn_id `_ was accidentally set to 's3_default' instead of 'aws_default' in some of the emr operators in previous
+versions. This was leading to EmrStepSensor not being able to find their corresponding emr cluster. With the new
+changes in the EmrAddStepsOperator, EmrTerminateJobFlowOperator and EmrCreateJobFlowOperator this issue is
+solved.
+
+``airflow.providers.amazon.aws.operators.batch.AwsBatchOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``AwsBatchOperator`` was refactored to extract an ``AwsBatchClient`` (and inherit from it). The
+changes are mostly backwards compatible and clarify the public API for these classes; some
+private methods on ``AwsBatchOperator`` for polling a job status were relocated and renamed
+to surface new public methods on ``AwsBatchClient`` (and via inheritance on ``AwsBatchOperator``\ ). A
+couple of job attributes are renamed on an instance of ``AwsBatchOperator``\ ; these were mostly
+used like private attributes but they were surfaced in the public API, so any use of them needs
+to be updated as follows:
+
+
+* ``AwsBatchOperator().jobId`` -> ``AwsBatchOperator().job_id``
+* ``AwsBatchOperator().jobName`` -> ``AwsBatchOperator().job_name``
+
+The ``AwsBatchOperator`` gets a new option to define a custom model for waiting on job status changes.
+The ``AwsBatchOperator`` can use a new ``waiters`` parameter, an instance of ``AwsBatchWaiters``\ , to
+specify that custom job waiters will be used to monitor a batch job. See the latest API
+documentation for details.
+
+``airflow.providers.amazon.aws.sensors.athena.AthenaSensor``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Replace parameter ``max_retires`` with ``max_retries`` to fix typo.
+
+``airflow.providers.amazon.aws.hooks.s3.S3Hook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Note: The order of arguments has changed for ``check_for_prefix``.
+The ``bucket_name`` is now optional. It falls back to the ``connection schema`` attribute.
+The ``delete_objects`` now returns ``None`` instead of a response, since the method now makes multiple api requests when the keys list length is > 1000.
+
+Changes in other provider packages
+""""""""""""""""""""""""""""""""""
+
+We strive to ensure that there are no changes that may affect the end user and your Python files, but this
+release may contain changes that will require changes to your configuration, DAG Files or other integration
+e.g. custom operators.
+
+Only changes unique to providers are described here. You should still pay attention to the changes that
+have been made to the core (including core operators) as they can affect the integration behavior
+of this provider.
+
+This section describes the changes that have been made, and what you need to do to update your if
+you use any code located in ``airflow.providers`` package.
+
+Changed return type of ``list_prefixes`` and ``list_keys`` methods in ``S3Hook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Previously, the ``list_prefixes`` and ``list_keys`` methods returned ``None`` when there were no
+results. The behavior has been changed to return an empty list instead of ``None`` in this
+case.
+
+Removed HipChat integration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+HipChat has reached end of life and is no longer available.
+
+For more information please see
+https://community.atlassian.com/t5/Stride-articles/Stride-and-Hipchat-Cloud-have-reached-End-of-Life-updated/ba-p/940248
+
+``airflow.providers.salesforce.hooks.salesforce.SalesforceHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Replace parameter ``sandbox`` with ``domain``. According to change in simple-salesforce package.
+
+Rename ``sign_in`` function to ``get_conn``.
+
+``airflow.providers.apache.pinot.hooks.pinot.PinotAdminHook.create_segment``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Rename parameter name from ``format`` to ``segment_format`` in PinotAdminHook function create_segment for pylint compatible
+
+``airflow.providers.apache.hive.hooks.hive.HiveMetastoreHook.get_partitions``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Rename parameter name from ``filter`` to ``partition_filter`` in HiveMetastoreHook function get_partitions for pylint compatible
+
+``airflow.providers.ftp.hooks.ftp.FTPHook.list_directory``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Remove unnecessary parameter ``nlst`` in FTPHook function ``list_directory`` for pylint compatible
+
+``airflow.providers.postgres.hooks.postgres.PostgresHook.copy_expert``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Remove unnecessary parameter ``open`` in PostgresHook function ``copy_expert`` for pylint compatible
+
+``airflow.providers.opsgenie.operators.opsgenie_alert.OpsgenieAlertOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Change parameter name from ``visibleTo`` to ``visible_to`` in OpsgenieAlertOperator for pylint compatible
+
+``airflow.providers.imap.hooks.imap.ImapHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``airflow.providers.imap.sensors.imap_attachment.ImapAttachmentSensor``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ImapHook:
+
+
+* The order of arguments has changed for ``has_mail_attachment``\ ,
+ ``retrieve_mail_attachments`` and ``download_mail_attachments``.
+* A new ``mail_filter`` argument has been added to each of those.
+
+``airflow.providers.http.hooks.http.HttpHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The HTTPHook is now secured by default: ``verify=True`` (before: ``verify=False``\ )
+This can be overwritten by using the extra_options param as ``{'verify': False}``.
+
+``airflow.providers.cloudant.hooks.cloudant.CloudantHook``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+* upgraded cloudant version from ``>=0.5.9,<2.0`` to ``>=2.0``
+* removed the use of the ``schema`` attribute in the connection
+* removed ``db`` function since the database object can also be retrieved by calling ``cloudant_session['database_name']``
+
+For example:
+
+.. code-block:: python
+
+ from airflow.providers.cloudant.hooks.cloudant import CloudantHook
+
+ with CloudantHook().get_conn() as cloudant_session:
+ database = cloudant_session["database_name"]
+
+See the `docs `_ for details
New Features
-""""""""""""
+^^^^^^^^^^^^
+
- [AIRFLOW-4026] Add filter by DAG tags (#6489)
- [AIRFLOW-6613] Center dag on graph view load (#7238)
- [AIRFLOW-5843] Add conf option to Add DAG Run view (#7281)
- [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates (#7038)
Improvements
-""""""""""""
+^^^^^^^^^^^^
+
- [AIRFLOW-6438] Filter DAGs returned by blocked (#7019)
- [AIRFLOW-6666] Resolve js-yaml advisory (#7283)
- [AIRFLOW-6632] Bump dagre-d3 to resolve lodash CVE advisory (#7280)
@@ -1880,7 +5584,8 @@ Improvements
- [AIRFLOW-5385] spark hook does not work on spark 2.3/2.4 (#6976)
Bug Fixes
-"""""""""
+^^^^^^^^^
+
- [AIRFLOW-6345] Ensure arguments to ProxyFix are integers (#6901)
- [AIRFLOW-6576] Fix scheduler crash caused by deleted task with sla misses (#7187)
- [AIRFLOW-6686] Fix syntax error constructing list of process ids (#7298)
@@ -1919,7 +5624,8 @@ Bug Fixes
- [AIRFLOW-6425] Serialization: Add missing DAG parameters to JSON Schema (#7002)
Misc/Internal
-"""""""""""""
+^^^^^^^^^^^^^
+
- [AIRFLOW-6467] Use self.dag i/o creating a new one (#7067)
- [AIRFLOW-6490] Improve time delta comparison in local task job tests (#7083)
- [AIRFLOW-5814] Implementing Presto hook tests (#6491)
@@ -1961,7 +5667,8 @@ Misc/Internal
- [AIRFLOW-6383] Add no trailing-whitespace pre-commit hook (#6941)
Doc only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
+
- [AIRFLOW-XXXX] Consistency fixes in new documentation (#7207)
- [AIRFLOW-XXXX] Improve grammar and structure in FAQ doc (#7291)
- [AIRFLOW-XXXX] Fix email configuration link in CONTRIBUTING.rst (#7311)
@@ -1994,11 +5701,40 @@ Doc only changes
- [AIRFLOW-XXXX] Add versions_added field to configs
-Airflow 1.10.7, 2019-12-24
---------------------------
+Airflow 1.10.7 (2019-12-24)
+---------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+Changes in experimental API execution_date microseconds replacement
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The default behavior was to strip the microseconds (and milliseconds, etc) off of all dag runs triggered by
+by the experimental REST API. The default behavior will change when an explicit execution_date is
+passed in the request body. It will also now be possible to have the execution_date generated, but
+keep the microseconds by sending ``replace_microseconds=false`` in the request body. The default
+behavior can be overridden by sending ``replace_microseconds=true`` along with an explicit execution_date
+
+Infinite pool size and pool size query optimization
+"""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Pool size can now be set to -1 to indicate infinite size (it also includes
+optimization of pool query which lead to poor task n^2 performance of task
+pool queries in MySQL).
+
+Viewer won't have edit permissions on DAG view.
+"""""""""""""""""""""""""""""""""""""""""""""""
+
+Google Cloud Storage Hook
+"""""""""""""""""""""""""
+
+The ``GoogleCloudStorageDownloadOperator`` can either write to a supplied ``filename`` or
+return the content of a file via xcom through ``store_to_xcom_key`` - both options are mutually exclusive.
New Features
-""""""""""""
+^^^^^^^^^^^^
+
- [AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver scalability (#5992)
- [AIRFLOW-6083] Adding ability to pass custom configuration to AWS Lambda client. (#6678)
- [AIRFLOW-5117] Automatically refresh EKS API tokens when needed (#5731)
@@ -2018,7 +5754,8 @@ New Features
- [AIRFLOW-4758] Add GcsToGDriveOperator operator (#5822)
Improvements
-""""""""""""
+^^^^^^^^^^^^
+
- [AIRFLOW-3656] Show doc link for the current installed version (#6690)
- [AIRFLOW-5665] Add path_exists method to SFTPHook (#6344)
- [AIRFLOW-5729] Make InputDataConfig optional in Sagemaker's training config (#6398)
@@ -2038,7 +5775,7 @@ Improvements
- [AIRFLOW-5583] Extend the 'DAG Details' page to display the start_date / end_date (#6235)
- [AIRFLOW-6250] Ensure on_failure_callback always has a populated context (#6812)
- [AIRFLOW-6222] http hook logs response body for any failure (#6779)
-- [AIRFLOW-6260] Drive _cmd config option by env var (``AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD`` for example) (#6801)
+- [AIRFLOW-6260] Drive _cmd config option by env var (``AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_CMD`` for example) (#6801)
- [AIRFLOW-6168] Allow proxy_fix middleware of webserver to be configurable (#6723)
- [AIRFLOW-5931] Use os.fork when appropriate to speed up task execution. (#6627)
- [AIRFLOW-4145] Allow RBAC roles permissions, ViewMenu to be over-rideable (#4960)
@@ -2077,7 +5814,8 @@ Improvements
- [AIRFLOW-5870] Allow -1 for pool size and optimise pool query (#6520)
Bug Fixes
-"""""""""
+^^^^^^^^^
+
- [AIRFLOW-XXX] Bump Jira version to fix issue with async
- [AIRFLOW-XXX] Add encoding to fix Cyrillic output when reading back task logs (#6631)
- [AIRFLOW-5304] Fix extra links in BigQueryOperator with multiple queries (#5906)
@@ -2132,7 +5870,8 @@ Bug Fixes
- [AIRFLOW-6308] Unpin Kombu for Python 3
Misc/Internal
-"""""""""""""
+^^^^^^^^^^^^^
+
- [AIRFLOW-6009] Switch off travis_wait for regular tests (#6600)
- [AIRFLOW-6226] Always reset warnings in tests
- [AIRFLOW-XXX] Remove cyclic imports and pylint hacks in Serialization (#6601)
@@ -2152,7 +5891,8 @@ Misc/Internal
- [AIRFLOW-6309] Fix stable build on Travis
Doc only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
+
- [AIRFLOW-6211] Doc how to use conda for local virtualenv (#6766)
- [AIRFLOW-5855] Fix broken reference in custom operator doc (#6508)
- [AIRFLOW-5875] Fix typo in example_qubole_operator.py (#6525)
@@ -2186,11 +5926,52 @@ Doc only changes
- [AIRFLOW-XXX] Google Season of Docs updates to CONTRIBUTING doc (#6283)
-Airflow 1.10.6, 2019-10-28
---------------------------
+Airflow 1.10.6 (2019-10-28)
+---------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+BaseOperator::render_template function signature changed
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Previous versions of the ``BaseOperator::render_template`` function required an ``attr`` argument as the first
+positional argument, along with ``content`` and ``context``. This function signature was changed in 1.10.6 and
+the ``attr`` argument is no longer required (or accepted).
+
+In order to use this function in subclasses of the ``BaseOperator``\ , the ``attr`` argument must be removed:
+
+.. code-block:: python
+
+ result = self.render_template("myattr", self.myattr, context) # Pre-1.10.6 call
+ # ...
+ result = self.render_template(self.myattr, context) # Post-1.10.6 call
+
+Changes to ``aws_default`` Connection's default region
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The region of Airflow's default connection to AWS (\ ``aws_default``\ ) was previously
+set to ``us-east-1`` during installation.
+
+The region now needs to be set manually, either in the connection screens in
+Airflow, via the ``~/.aws`` config files, or via the ``AWS_DEFAULT_REGION`` environment
+variable.
+
+Some DAG Processing metrics have been renamed
+"""""""""""""""""""""""""""""""""""""""""""""
+
+The following metrics are deprecated and won't be emitted in Airflow 2.0:
+
+
+* ``scheduler.dagbag.errors`` and ``dagbag_import_errors`` -- use ``dag_processing.import_errors`` instead
+* ``dag_file_processor_timeouts`` -- use ``dag_processing.processor_timeouts`` instead
+* ``collect_dags`` -- use ``dag_processing.total_parse_time`` instead
+* ``dag.loading-duration.`` -- use ``dag_processing.last_duration.`` instead
+* ``dag_processing.last_runtime.`` -- use ``dag_processing.last_duration.`` instead
New Features
-""""""""""""
+^^^^^^^^^^^^
+
- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546)
- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407)
- [AIRFLOW-4939] Add default_task_retries config (#5570)
@@ -2199,7 +5980,8 @@ New Features
- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743)
Improvements
-""""""""""""
+^^^^^^^^^^^^
+
- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738)
- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737)
- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736)
@@ -2237,7 +6019,8 @@ Improvements
- [AIRFLOW-4835] Refactor operator render_template (#5461)
Bug Fixes
-"""""""""
+^^^^^^^^^
+
- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator (#6078)
- [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635)
- [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747)
@@ -2253,7 +6036,7 @@ Bug Fixes
- [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101)
- [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678)
- [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909)
-- [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185)
+- [AIRFLOW-5554] Require StatsD 3.3.0 minimum (#6185)
- [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904)
- [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841)
- [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237)
@@ -2268,7 +6051,7 @@ Bug Fixes
- [AIRFLOW-5342] Fix MSSQL breaking task_instance db migration (#6014)
- [AIRFLOW-5556] Add separate config for timeout from scheduler dag processing (#6186)
- [AIRFLOW-4858] Deprecate "Historical convenience functions" in airflow.configuration (#5495) (#6144)
-- [AIRFLOW-774] Fix long-broken DAG parsing Statsd metrics (#6157)
+- [AIRFLOW-774] Fix long-broken DAG parsing StatsD metrics (#6157)
- [AIRFLOW-5419] Use ``sudo`` to kill cleared tasks when running with impersonation (#6026) (#6176)
- [AIRFLOW-5537] Yamllint is not needed as dependency on host
- [AIRFLOW-5536] Better handling of temporary output files
@@ -2278,7 +6061,8 @@ Bug Fixes
- [AIRFLOW-5447] Scheduler stalls because second watcher thread in default args (#6129)
Doc-only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
+
- [AIRFLOW-5574] Fix Google Analytics script loading (#6218)
- [AIRFLOW-5588] Add Celery's architecture diagram (#6247)
- [AIRFLOW-5521] Fix link to GCP documentation (#6150)
@@ -2303,7 +6087,8 @@ Doc-only changes
- [AIRFLOW-XXX] Make Breeze The default integration test environment (#6001)
Misc/Internal
-"""""""""""""
+^^^^^^^^^^^^^
+
- [AIRFLOW-5687] Upgrade pip to 19.0.2 in CI build pipeline (#6358) (#6361)
- [AIRFLOW-5533] Fixed failing CRON build (#6167)
- [AIRFLOW-5130] Use GOOGLE_APPLICATION_CREDENTIALS constant from library (#5744)
@@ -2317,17 +6102,24 @@ Misc/Internal
- [AIRFLOW-5003] Making AWS Hooks pylint compatible (#5627)
- [AIRFLOW-5580] Add base class for system test (#6229)
-Airflow 1.10.5, 2019-09-04
---------------------------
+Airflow 1.10.5 (2019-09-04)
+---------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+No significant changes.
New Features
-""""""""""""
+^^^^^^^^^^^^
+
- [AIRFLOW-1498] Add feature for users to add Google Analytics to Airflow UI (#5850)
- [AIRFLOW-4074] Add option to add labels to Dataproc jobs (#5606)
- [AIRFLOW-4846] Allow specification of an existing secret containing git credentials for init containers (#5475)
Improvements
-""""""""""""
+^^^^^^^^^^^^
+
- [AIRFLOW-5335] Update GCSHook methods so they need min IAM perms (#5939)
- [AIRFLOW-2692] Allow AWS Batch Operator to use templates in job_name parameter (#3557)
- [AIRFLOW-4768] Add Timeout parameter in example_gcp_video_intelligence (#5862)
@@ -2347,7 +6139,8 @@ Improvements
- [AIRFLOW-4316] Support setting kubernetes_environment_variables config section from env var (#5668)
Bug fixes
-"""""""""
+^^^^^^^^^
+
- [AIRFLOW-5168] Fix Dataproc operators that failed in 1.10.4 (#5928)
- [AIRFLOW-5136] Fix Bug with Incorrect template_fields in DataProc{*} Operators (#5751)
- [AIRFLOW-5169] Pass GCP Project ID explicitly to StorageClient in GCSHook (#5783)
@@ -2373,7 +6166,8 @@ Bug fixes
- [AIRFLOW-5240] Latest version of Kombu is breaking Airflow for py2
Misc/Internal
-"""""""""""""
+^^^^^^^^^^^^^
+
- [AIRFLOW-5111] Remove apt-get upgrade from the Dockerfile (#5722)
- [AIRFLOW-5209] Fix Documentation build (#5814)
- [AIRFLOW-5083] Check licence image building can be faster and moved to before-install (#5695)
@@ -2402,16 +6196,131 @@ Misc/Internal
- [AIRFLOW-4027] Make experimental api tests more stateless (#4854)
Doc-only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
+
- [AIRFLOW-XXX] Fixed Azkaban link (#5865)
- [AIRFLOW-XXX] Remove duplicate lines from CONTRIBUTING.md (#5830)
- [AIRFLOW-XXX] Fix incorrect docstring parameter in SchedulerJob (#5729)
-Airflow 1.10.4, 2019-08-06
---------------------------
+Airflow 1.10.4 (2019-08-06)
+---------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+Export MySQL timestamps as UTC
+""""""""""""""""""""""""""""""
+
+``MySqlToGoogleCloudStorageOperator`` now exports TIMESTAMP columns as UTC
+by default, rather than using the default timezone of the MySQL server.
+This is the correct behavior for use with BigQuery, since BigQuery
+assumes that TIMESTAMP columns without time zones are in UTC. To
+preserve the previous behavior, set ``ensure_utc`` to ``False.``
+
+Changes to DatastoreHook
+""""""""""""""""""""""""
+
+
+* removed argument ``version`` from ``get_conn`` function and added it to the hook's ``__init__`` function instead and renamed it to ``api_version``
+* renamed the ``partialKeys`` argument of function ``allocate_ids`` to ``partial_keys``
+
+Changes to GoogleCloudStorageHook
+"""""""""""""""""""""""""""""""""
+
+
+* the discovery-based api (\ ``googleapiclient.discovery``\ ) used in ``GoogleCloudStorageHook`` is now replaced by the recommended client based api (\ ``google-cloud-storage``\ ). To know the difference between both the libraries, read https://cloud.google.com/apis/docs/client-libraries-explained. PR: `#5054 `_
+*
+ as a part of this replacement, the ``multipart`` & ``num_retries`` parameters for ``GoogleCloudStorageHook.upload`` method have been deprecated.
+
+ The client library uses multipart upload automatically if the object/blob size is more than 8 MB - `source code `_. The client also handles retries automatically
+
+*
+ the ``generation`` parameter is deprecated in ``GoogleCloudStorageHook.delete`` and ``GoogleCloudStorageHook.insert_object_acl``.
+
+Updating to ``google-cloud-storage >= 1.16`` changes the signature of the upstream ``client.get_bucket()`` method from ``get_bucket(bucket_name: str)`` to ``get_bucket(bucket_or_name: Union[str, Bucket])``. This method is not directly exposed by the airflow hook, but any code accessing the connection directly (\ ``GoogleCloudStorageHook().get_conn().get_bucket(...)`` or similar) will need to be updated.
+
+Changes in writing Logs to Elasticsearch
+""""""""""""""""""""""""""""""""""""""""
+
+The ``elasticsearch_`` prefix has been removed from all config items under the ``[elasticsearch]`` section. For example ``elasticsearch_host`` is now just ``host``.
+
+Removal of ``non_pooled_task_slot_count`` and ``non_pooled_backfill_task_slot_count``
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+``non_pooled_task_slot_count`` and ``non_pooled_backfill_task_slot_count``
+are removed in favor of a real pool, e.g. ``default_pool``.
+
+By default tasks are running in ``default_pool``.
+``default_pool`` is initialized with 128 slots and user can change the
+number of slots through UI/CLI. ``default_pool`` cannot be removed.
+
+``pool`` config option in Celery section to support different Celery pool implementation
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The new ``pool`` config option allows users to choose different pool
+implementation. Default value is "prefork", while choices include "prefork" (default),
+"eventlet", "gevent" or "solo". This may help users achieve better concurrency performance
+in different scenarios.
+
+For more details about Celery pool implementation, please refer to:
+
+
+* https://docs.celeryproject.org/en/latest/userguide/workers.html#concurrency
+* https://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html
+
+Change to method signature in ``BaseOperator`` and ``DAG`` classes
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The signature of the ``get_task_instances`` method in the ``BaseOperator`` and ``DAG`` classes has changed. The change does not change the behavior of the method in either case.
+
+For ``BaseOperator``
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Old signature:
+
+.. code-block:: python
+
+ def get_task_instances(self, session, start_date=None, end_date=None):
+ ...
+
+New signature:
+
+.. code-block:: python
+
+ @provide_session
+ def get_task_instances(self, start_date=None, end_date=None, session=None):
+ ...
+
+For ``DAG``
+~~~~~~~~~~~~~~~
+
+Old signature:
+
+.. code-block:: python
+
+ def get_task_instances(self, session, start_date=None, end_date=None, state=None):
+ ...
+
+New signature:
+
+.. code-block:: python
+
+ @provide_session
+ def get_task_instances(self, start_date=None, end_date=None, state=None, session=None):
+ ...
+
+In either case, it is necessary to rewrite calls to the ``get_task_instances`` method that currently provide the ``session`` positional argument. New calls to this method look like:
+
+.. code-block:: python
+
+ # if you can rely on @provide_session
+ dag.get_task_instances()
+ # if you need to provide the session
+ dag.get_task_instances(session=your_session)
New Features
-""""""""""""
+^^^^^^^^^^^^
+
- [AIRFLOW-4811] Implement GCP Data Loss Prevention Hook and Operators (#5539)
- [AIRFLOW-5035] Replace multiprocessing.Manager with a golang-"channel" style (#5615)
- [AIRFLOW-4883] Kill hung file process managers (#5605)
@@ -2459,7 +6368,8 @@ New Features
- [AIRFLOW-XXX] Change allowed version of Jinja2 to fix CVE-2019-10906 (#5075)
Improvement
-"""""""""""
+^^^^^^^^^^^
+
- [AIRFLOW-5022] Fix DockerHook for registries with port numbers (#5644)
- [AIRFLOW-4961] Insert TaskFail.duration as int match DB schema column type (#5593)
- [AIRFLOW-5038] skip pod deleted log message when pod deletion is disabled (#5656)
@@ -2569,7 +6479,8 @@ Improvement
- [AIRFLOW-4528] Cancel DataProc task on timeout (#5293)
Bug fixes
-"""""""""
+^^^^^^^^^
+
- [AIRFLOW-5089] Change version requirement on google-cloud-spanner to work around version incompatibility (#5703)
- [AIRFLOW-4289] fix spark_binary argument being ignored in SparkSubmitHook (#5564)
- [AIRFLOW-5075] Let HttpHook handle connections with empty host fields (#5686)
@@ -2663,7 +6574,8 @@ Bug fixes
- [AIRFLOW-3241] Remove Invalid template ext in GCS Sensors (#4076)
Misc/Internal
-"""""""""""""
+^^^^^^^^^^^^^
+
- [AIRFLOW-4338] Change k8s pod_request_factory to use yaml safe_load (#5120)
- [AIRFLOW-4869] Reorganize sql to gcs operators. (#5504)
- [AIRFLOW-5021] move gitpython into setup_requires (#5640)
@@ -2700,7 +6612,8 @@ Misc/Internal
- [AIRFLOW-3944] Remove code smells (#4762)
Doc-only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
+
- [AIRFLOW-XXX] Add missing doc for annotations param of KubernetesPodOperator (#5666)
- [AIRFLOW-XXX] Fix typos in CONTRIBUTING.md (#5626)
- [AIRFLOW-XXX] Correct BaseSensorOperator docs (#5562)
@@ -2735,11 +6648,228 @@ Doc-only changes
-Airflow 1.10.3, 2019-04-09
---------------------------
+Airflow 1.10.3 (2019-04-09)
+---------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+New ``dag_discovery_safe_mode`` config option
+"""""""""""""""""""""""""""""""""""""""""""""""""
+
+If ``dag_discovery_safe_mode`` is enabled, only check files for DAGs if
+they contain the strings "airflow" and "DAG". For backwards
+compatibility, this option is enabled by default.
+
+RedisPy dependency updated to v3 series
+"""""""""""""""""""""""""""""""""""""""
+
+If you are using the Redis Sensor or Hook you may have to update your code. See
+`redis-py porting instructions `_ to check if your code might be affected (MSET,
+MSETNX, ZADD, and ZINCRBY all were, but read the full doc).
+
+SLUGIFY_USES_TEXT_UNIDECODE or AIRFLOW_GPL_UNIDECODE no longer required
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+It is no longer required to set one of the environment variables to avoid
+a GPL dependency. Airflow will now always use ``text-unidecode`` if ``unidecode``
+was not installed before.
+
+New ``sync_parallelism`` config option in ``[celery]`` section
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The new ``sync_parallelism`` config option will control how many processes CeleryExecutor will use to
+fetch celery task state in parallel. Default value is max(1, number of cores - 1)
+
+Rename of BashTaskRunner to StandardTaskRunner
+""""""""""""""""""""""""""""""""""""""""""""""
+
+BashTaskRunner has been renamed to StandardTaskRunner. It is the default task runner
+so you might need to update your config.
+
+``task_runner = StandardTaskRunner``
+
+Modification to config file discovery
+"""""""""""""""""""""""""""""""""""""
+
+If the ``AIRFLOW_CONFIG`` environment variable was not set and the
+``~/airflow/airflow.cfg`` file existed, airflow previously used
+``~/airflow/airflow.cfg`` instead of ``$AIRFLOW_HOME/airflow.cfg``. Now airflow
+will discover its config file using the ``$AIRFLOW_CONFIG`` and ``$AIRFLOW_HOME``
+environment variables rather than checking for the presence of a file.
+
+Changes in Google Cloud related operators
+"""""""""""""""""""""""""""""""""""""""""
+
+Most GCP-related operators have now optional ``PROJECT_ID`` parameter. In case you do not specify it,
+the project id configured in
+`GCP Connection `_ is used.
+There will be an ``AirflowException`` thrown in case ``PROJECT_ID`` parameter is not specified and the
+connection used has no project id defined. This change should be backwards compatible as earlier version
+of the operators had ``PROJECT_ID`` mandatory.
+
+Operators involved:
+
+
+* GCP Compute Operators
+
+ * GceInstanceStartOperator
+ * GceInstanceStopOperator
+ * GceSetMachineTypeOperator
+
+* GCP Function Operators
+
+ * GcfFunctionDeployOperator
+
+* GCP Cloud SQL Operators
+
+ * CloudSqlInstanceCreateOperator
+ * CloudSqlInstancePatchOperator
+ * CloudSqlInstanceDeleteOperator
+ * CloudSqlInstanceDatabaseCreateOperator
+ * CloudSqlInstanceDatabasePatchOperator
+ * CloudSqlInstanceDatabaseDeleteOperator
+
+Other GCP operators are unaffected.
+
+Changes in Google Cloud related hooks
+"""""""""""""""""""""""""""""""""""""
+
+The change in GCP operators implies that GCP Hooks for those operators require now keyword parameters rather
+than positional ones in all methods where ``project_id`` is used. The methods throw an explanatory exception
+in case they are called using positional parameters.
+
+Hooks involved:
+
+
+* GceHook
+* GcfHook
+* CloudSqlHook
+
+Other GCP hooks are unaffected.
+
+Changed behaviour of using default value when accessing variables
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+It's now possible to use ``None`` as a default value with the ``default_var`` parameter when getting a variable, e.g.
+
+.. code-block:: python
+
+ foo = Variable.get("foo", default_var=None)
+ if foo is None:
+ handle_missing_foo()
+
+(Note: there is already ``Variable.setdefault()`` which me be helpful in some cases.)
+
+This changes the behaviour if you previously explicitly provided ``None`` as a default value. If your code expects a ``KeyError`` to be thrown, then don't pass the ``default_var`` argument.
+
+Removal of ``airflow_home`` config setting
+""""""""""""""""""""""""""""""""""""""""""""""
+
+There were previously two ways of specifying the Airflow "home" directory
+(\ ``~/airflow`` by default): the ``AIRFLOW_HOME`` environment variable, and the
+``airflow_home`` config setting in the ``[core]`` section.
+
+If they had two different values different parts of the code base would end up
+with different values. The config setting has been deprecated, and you should
+remove the value from the config file and set ``AIRFLOW_HOME`` environment
+variable if you need to use a non default value for this.
+
+(Since this setting is used to calculate what config file to load, it is not
+possible to keep just the config option)
+
+Change of two methods signatures in ``GCPTransferServiceHook``
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The signature of the ``create_transfer_job`` method in ``GCPTransferServiceHook``
+class has changed. The change does not change the behavior of the method.
+
+Old signature:
+
+.. code-block:: python
+
+ def create_transfer_job(self, description, schedule, transfer_spec, project_id=None):
+ ...
+
+New signature:
+
+.. code-block:: python
+
+ def create_transfer_job(self, body):
+ ...
+
+It is necessary to rewrite calls to method. The new call looks like this:
+
+.. code-block:: python
+
+ body = {
+ "status": "ENABLED",
+ "projectId": project_id,
+ "description": description,
+ "transferSpec": transfer_spec,
+ "schedule": schedule,
+ }
+ gct_hook.create_transfer_job(body)
+
+The change results from the unification of all hooks and adjust to
+`the official recommendations `_
+for the Google Cloud.
+
+The signature of ``wait_for_transfer_job`` method in ``GCPTransferServiceHook`` has changed.
+
+Old signature:
+
+.. code-block:: python
+
+ def wait_for_transfer_job(self, job):
+ ...
+
+New signature:
+
+.. code-block:: python
+
+ def wait_for_transfer_job(
+ self, job, expected_statuses=(GcpTransferOperationStatus.SUCCESS,)
+ ):
+ ...
+
+The behavior of ``wait_for_transfer_job`` has changed:
+
+Old behavior:
+
+``wait_for_transfer_job`` would wait for the SUCCESS status in specified jobs operations.
+
+New behavior:
+
+You can now specify an array of expected statuses. ``wait_for_transfer_job`` now waits for any of them.
+
+The default value of ``expected_statuses`` is SUCCESS so that change is backwards compatible.
+
+Moved two classes to different modules
+""""""""""""""""""""""""""""""""""""""
+
+The class ``GoogleCloudStorageToGoogleCloudStorageTransferOperator`` has been moved from
+``airflow.contrib.operators.gcs_to_gcs_transfer_operator`` to ``airflow.contrib.operators.gcp_transfer_operator``
+
+the class ``S3ToGoogleCloudStorageTransferOperator`` has been moved from
+``airflow.contrib.operators.s3_to_gcs_transfer_operator`` to ``airflow.contrib.operators.gcp_transfer_operator``
+
+The change was made to keep all the operators related to GCS Transfer Services in one file.
+
+The previous imports will continue to work until Airflow 2.0
+
+Fixed typo in --driver-class-path in SparkSubmitHook
+""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The ``driver_classapth`` argument to SparkSubmit Hook and Operator was
+generating ``--driver-classpath`` on the spark command line, but this isn't a
+valid option to spark.
+
+The argument has been renamed to ``driver_class_path`` and the option it
+generates has been fixed.
New Feature
-"""""""""""
+^^^^^^^^^^^
- [AIRFLOW-4232] Add ``none_skipped`` trigger rule (#5032)
- [AIRFLOW-3971] Add Google Cloud Natural Language operators (#4980)
@@ -2771,7 +6901,7 @@ New Feature
- [AIRFLOW-3556] Add cross join set dependency function (#4356)
Improvement
-"""""""""""
+^^^^^^^^^^^
- [AIRFLOW-3823] Exclude branch's downstream tasks from the tasks to skip (#4666)
- [AIRFLOW-3274] Add run_as_user and fs_group options for Kubernetes (#4648)
@@ -2882,7 +7012,7 @@ Improvement
- [AIRFLOW-3631] Update flake8 and fix lint. (#4436)
Bug fixes
-"""""""""
+^^^^^^^^^
- [AIRFLOW-4248] Fix ``FileExistsError`` makedirs race in file_processor_handler (#5047)
- [AIRFLOW-4240] State-changing actions should be POST requests (#5039)
@@ -2913,7 +7043,7 @@ Bug fixes
- [AIRFLOW-4037] Log response in SimpleHttpOperator even if the response check fails
- [AIRFLOW-4044] The documentation of ``query_params`` in ``BigQueryOperator`` is wrong. (#4876)
- [AIRFLOW-4015] Make missing API endpoints available in classic mode
-- [AIRFLOW-3153] Send DAG processing stats to statsd (#4748)
+- [AIRFLOW-3153] Send DAG processing stats to StatsD (#4748)
- [AIRFLOW-2966] Catch ApiException in the Kubernetes Executor (#4209)
- [AIRFLOW-4129] Escape HTML in generated tooltips (#4950)
- [AIRFLOW-4070] AirflowException -> log.warning for duplicate task dependencies (#4904)
@@ -2994,7 +7124,7 @@ Bug fixes
- [AIRFLOW-XXX] Fix flaky test - test_execution_unlimited_parallelism (#4988)
Misc/Internal
-"""""""""""""
+^^^^^^^^^^^^^
- [AIRFLOW-4144] Add description of is_delete_operator_pod (#4943)
- [AIRFLOW-3476][AIRFLOW-3477] Move Kube classes out of models.py (#4443)
@@ -3013,12 +7143,12 @@ Misc/Internal
- [AIRFLOW-XXX] Remove old/non-test files that nose ignores (#4930)
Doc-only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
- [AIRFLOW-3996] Add view source link to included fragments
- [AIRFLOW-3811] Automatic generation of API Reference in docs (#4788)
- [AIRFLOW-3810] Remove duplicate autoclass directive (#4656)
-- [AIRFLOW-XXX] Mention that statsd must be installed to gather metrics (#5038)
+- [AIRFLOW-XXX] Mention that StatsD must be installed to gather metrics (#5038)
- [AIRFLOW-XXX] Add contents to cli (#4825)
- [AIRFLOW-XXX] fix check docs failure on CI (#4998)
- [AIRFLOW-XXX] Fix syntax docs errors (#4789)
@@ -3055,11 +7185,121 @@ Doc-only changes
- [AIRFLOW-XXX] Speed up DagBagTest cases (#3974)
-Airflow 1.10.2, 2019-01-19
---------------------------
+Airflow 1.10.2 (2019-01-19)
+---------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+New ``dag_processor_manager_log_location`` config option
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The DAG parsing manager log now by default will be log into a file, where its location is
+controlled by the new ``dag_processor_manager_log_location`` config option in core section.
+
+DAG level Access Control for new RBAC UI
+""""""""""""""""""""""""""""""""""""""""
+
+Extend and enhance new Airflow RBAC UI to support DAG level ACL. Each dag now has two permissions(one for write, one for read) associated('can_dag_edit', 'can_dag_read').
+The admin will create new role, associate the dag permission with the target dag and assign that role to users. That user can only access / view the certain dags on the UI
+that he has permissions on. If a new role wants to access all the dags, the admin could associate dag permissions on an artificial view(\ ``all_dags``\ ) with that role.
+
+We also provide a new cli command(\ ``sync_perm``\ ) to allow admin to auto sync permissions.
+
+Modification to ``ts_nodash`` macro
+"""""""""""""""""""""""""""""""""""""""
+
+``ts_nodash`` previously contained TimeZone information along with execution date. For Example: ``20150101T000000+0000``. This is not user-friendly for file or folder names which was a popular use case for ``ts_nodash``. Hence this behavior has been changed and using ``ts_nodash`` will no longer contain TimeZone information, restoring the pre-1.10 behavior of this macro. And a new macro ``ts_nodash_with_tz`` has been added which can be used to get a string with execution date and timezone info without dashes.
+
+Examples:
+
+
+* ``ts_nodash``\ : ``20150101T000000``
+* ``ts_nodash_with_tz``\ : ``20150101T000000+0000``
+
+Semantics of next_ds/prev_ds changed for manually triggered runs
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+next_ds/prev_ds now map to execution_date instead of the next/previous schedule-aligned execution date for DAGs triggered in the UI.
+
+User model changes
+""""""""""""""""""
+
+This patch changes the ``User.superuser`` field from a hard-coded boolean to a ``Boolean()`` database column. ``User.superuser`` will default to ``False``\ , which means that this privilege will have to be granted manually to any users that may require it.
+
+For example, open a Python shell and
+
+.. code-block:: python
+
+ from airflow import models, settings
+
+ session = settings.Session()
+ users = session.query(models.User).all() # [admin, regular_user]
+
+ users[1].superuser # False
+
+ admin = users[0]
+ admin.superuser = True
+ session.add(admin)
+ session.commit()
+
+Custom auth backends interface change
+"""""""""""""""""""""""""""""""""""""
+
+We have updated the version of flask-login we depend upon, and as a result any
+custom auth backends might need a small change: ``is_active``\ ,
+``is_authenticated``\ , and ``is_anonymous`` should now be properties. What this means is if
+previously you had this in your user class
+
+.. code-block:: python
+
+ def is_active(self):
+ return self.active
+
+then you need to change it like this
+
+.. code-block:: python
+
+ @property
+ def is_active(self):
+ return self.active
+
+Support autodetected schemas to GoogleCloudStorageToBigQueryOperator
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+GoogleCloudStorageToBigQueryOperator is now support schema auto-detection is available when you load data into BigQuery. Unfortunately, changes can be required.
+
+If BigQuery tables are created outside of airflow and the schema is not defined in the task, multiple options are available:
+
+define a schema_fields:
+
+.. code-block:: python
+
+ gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
+ # ...
+ schema_fields={...}
+ )
+
+or define a schema_object:
+
+.. code-block:: python
+
+ gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
+ # ...
+ schema_object="path/to/schema/object"
+ )
+
+or enabled autodetect of schema:
+
+.. code-block:: python
+
+ gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
+ # ...
+ autodetect=True
+ )
New features
-""""""""""""
+^^^^^^^^^^^^
- [AIRFLOW-2658] Add GCP specific k8s pod operator (#3532)
- [AIRFLOW-2440] Google Cloud SQL import/export operator (#4251)
@@ -3095,7 +7335,7 @@ New features
- [AIRFLOW-3336] Add new TriggerRule for 0 upstream failures (#4182)
Improvements
-""""""""""""
+^^^^^^^^^^^^
- [AIRFLOW-3680] Consistency update in tests for All GCP-related operators (#4493)
- [AIRFLOW-3675] Use googleapiclient for google apis (#4484)
@@ -3202,7 +7442,7 @@ Improvements
Bug fixes
-"""""""""
+^^^^^^^^^
- [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun (#4037)
- [AIRFLOW-3657] Fix zendesk integration (#4466)
@@ -3284,7 +7524,7 @@ Bug fixes
- [AIRFLOW-1298] Clear UPSTREAM_FAILED using the clean cli (#3886)
Doc-only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
- [AIRFLOW-XXX] GCP operators documentation clarifications (#4273)
- [AIRFLOW-XXX] Docs: Fix paths to GCS transfer operator (#4479)
@@ -3325,11 +7565,48 @@ Doc-only changes
- [AIRFLOW-XXX] Fix SlackWebhookOperator execute method comment (#3963)
- [AIRFLOW-3070] Refine web UI authentication-related docs (#3863)
-Airflow 1.10.1, 2018-11-13
---------------------------
+Airflow 1.10.1 (2018-11-13)
+---------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+min_file_parsing_loop_time config option temporarily disabled
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The scheduler.min_file_parsing_loop_time config option has been temporarily removed due to
+some bugs.
+
+StatsD Metrics
+""""""""""""""
+
+The ``scheduler_heartbeat`` metric has been changed from a gauge to a counter. Each loop of the scheduler will increment the counter by 1. This provides a higher degree of visibility and allows for better integration with Prometheus using the `StatsD Exporter `_. The scheduler's activity status can be determined by graphing and alerting using a rate of change of the counter. If the scheduler goes down, the rate will drop to 0.
+
+EMRHook now passes all of connection's extra to CreateJobFlow API
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+EMRHook.create_job_flow has been changed to pass all keys to the create_job_flow API, rather than
+just specific known keys for greater flexibility.
+
+However prior to this release the "emr_default" sample connection that was created had invalid
+configuration, so creating EMR clusters might fail until your connection is updated. (Ec2KeyName,
+Ec2SubnetId, TerminationProtection and KeepJobFlowAliveWhenNoSteps were all top-level keys when they
+should be inside the "Instances" dict)
+
+LDAP Auth Backend now requires TLS
+""""""""""""""""""""""""""""""""""
+
+Connecting to an LDAP server over plain text is not supported anymore. The
+certificate presented by the LDAP server must be signed by a trusted
+certificate, or you must provide the ``cacert`` option under ``[ldap]`` in the
+config file.
+
+If you want to use LDAP auth backend without TLS then you will have to create a
+custom-auth backend based on
+https://github.com/apache/airflow/blob/1.10.0/airflow/contrib/auth/backends/ldap_auth.py
New features
-""""""""""""
+^^^^^^^^^^^^
- [AIRFLOW-2524] Airflow integration with AWS Sagemaker
- [AIRFLOW-2657] Add ability to delete DAG from web ui
@@ -3344,7 +7621,7 @@ New features
- [AIRFLOW-3276] Google Cloud SQL database create / patch / delete operators
Improvements
-""""""""""""
+^^^^^^^^^^^^
- [AIRFLOW-393] Add progress callbacks for FTP downloads
- [AIRFLOW-520] Show Airflow version on web page
@@ -3389,7 +7666,7 @@ Improvements
- [AIRFLOW-3265] Add support for "unix_socket" in connection extra for Mysql Hook
Doc-only changes
-""""""""""""""""
+^^^^^^^^^^^^^^^^
- [AIRFLOW-1441] Tutorial Inconsistencies Between Example Pipeline Definition and Recap
- [AIRFLOW-2682] Add how-to guide(s) for how to use basic operators like BashOperator and PythonOperator
@@ -3404,7 +7681,7 @@ Doc-only changes
- [AIRFLOW-2779] Add project version to license
Bug fixes
-"""""""""
+^^^^^^^^^
- [AIRFLOW-839] docker_operator.py attempts to log status key without first checking existence
- [AIRFLOW-1104] Concurrency check in scheduler should count queued tasks as well as running
@@ -3464,8 +7741,155 @@ Bug fixes
- [AIRFLOW-3295] Require encryption in DaskExecutor when certificates are configured.
- [AIRFLOW-3297] EmrStepSensor marks cancelled step as successful
-Airflow 1.10.0, 2018-08-03
---------------------------
+Airflow 1.10.0 (2018-08-03)
+---------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+Installation and upgrading requires setting ``SLUGIFY_USES_TEXT_UNIDECODE=yes`` in your environment or
+``AIRFLOW_GPL_UNIDECODE=yes``. In case of the latter a GPL runtime dependency will be installed due to a
+dependency ``(python-nvd3 -> python-slugify -> unidecode)``.
+
+Replace DataProcHook.await calls to DataProcHook.wait
+"""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+The method name was changed to be compatible with the Python 3.7 async/await keywords
+
+Setting UTF-8 as default mime_charset in email utils
+""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Add a configuration variable(default_dag_run_display_number) to control numbers of dag run for display
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Add a configuration variable(default_dag_run_display_number) under webserver section to control the number of dag runs to show in UI.
+
+Default executor for SubDagOperator is changed to SequentialExecutor
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+New Webserver UI with Role-Based Access Control
+"""""""""""""""""""""""""""""""""""""""""""""""
+
+The current webserver UI uses the Flask-Admin extension. The new webserver UI uses the `Flask-AppBuilder (FAB) `_ extension. FAB has built-in authentication support and Role-Based Access Control (RBAC), which provides configurable roles and permissions for individual users.
+
+To turn on this feature, in your airflow.cfg file (under [webserver]), set the configuration variable ``rbac = True``\ , and then run ``airflow`` command, which will generate the ``webserver_config.py`` file in your $AIRFLOW_HOME.
+
+Setting up Authentication
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+FAB has built-in authentication support for DB, OAuth, OpenID, LDAP, and REMOTE_USER. The default auth type is ``AUTH_DB``.
+
+For any other authentication type (OAuth, OpenID, LDAP, REMOTE_USER), see the `Authentication section of FAB docs `_ for how to configure variables in webserver_config.py file.
+
+Once you modify your config file, run ``airflow db init`` to generate new tables for RBAC support (these tables will have the prefix ``ab_``\ ).
+
+Creating an Admin Account
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once configuration settings have been updated and new tables have been generated, create an admin account with ``airflow create_user`` command.
+
+Using your new UI
+~~~~~~~~~~~~~~~~~
+
+Run ``airflow webserver`` to start the new UI. This will bring up a log in page, enter the recently created admin username and password.
+
+There are five roles created for Airflow by default: Admin, User, Op, Viewer, and Public. To configure roles/permissions, go to the ``Security`` tab and click ``List Roles`` in the new UI.
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+
+* AWS Batch Operator renamed property queue to job_queue to prevent conflict with the internal queue from CeleryExecutor - AIRFLOW-2542
+* Users created and stored in the old users table will not be migrated automatically. FAB's built-in authentication support must be reconfigured.
+* Airflow dag home page is now ``/home`` (instead of ``/admin``\ ).
+* All ModelViews in Flask-AppBuilder follow a different pattern from Flask-Admin. The ``/admin`` part of the URL path will no longer exist. For example: ``/admin/connection`` becomes ``/connection/list``\ , ``/admin/connection/new`` becomes ``/connection/add``\ , ``/admin/connection/edit`` becomes ``/connection/edit``\ , etc.
+* Due to security concerns, the new webserver will no longer support the features in the ``Data Profiling`` menu of old UI, including ``Ad Hoc Query``\ , ``Charts``\ , and ``Known Events``.
+* HiveServer2Hook.get_results() always returns a list of tuples, even when a single column is queried, as per Python API 2.
+* **UTC is now the default timezone**\ : Either reconfigure your workflows scheduling in UTC or set ``default_timezone`` as explained in https://airflow.apache.org/timezone.html#default-time-zone
+
+airflow.contrib.sensors.hdfs_sensors renamed to airflow.contrib.sensors.hdfs_sensor
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+We now rename airflow.contrib.sensors.hdfs_sensors to airflow.contrib.sensors.hdfs_sensor for consistency purpose.
+
+MySQL setting required
+""""""""""""""""""""""
+
+We now rely on more strict ANSI SQL settings for MySQL in order to have sane defaults. Make sure
+to have specified ``explicit_defaults_for_timestamp=1`` in your my.cnf under ``[mysqld]``
+
+Celery config
+"""""""""""""
+
+To make the config of Airflow compatible with Celery, some properties have been renamed:
+
+.. code-block::
+
+ celeryd_concurrency -> worker_concurrency
+ celery_result_backend -> result_backend
+ celery_ssl_active -> ssl_active
+ celery_ssl_cert -> ssl_cert
+ celery_ssl_key -> ssl_key
+
+Resulting in the same config parameters as Celery 4, with more transparency.
+
+GCP Dataflow Operators
+""""""""""""""""""""""
+
+Dataflow job labeling is now supported in Dataflow{Java,Python}Operator with a default
+"airflow-version" label, please upgrade your google-cloud-dataflow or apache-beam version
+to 2.2.0 or greater.
+
+BigQuery Hooks and Operator
+"""""""""""""""""""""""""""
+
+The ``bql`` parameter passed to ``BigQueryOperator`` and ``BigQueryBaseCursor.run_query`` has been deprecated and renamed to ``sql`` for consistency purposes. Using ``bql`` will still work (and raise a ``DeprecationWarning``\ ), but is no longer
+supported and will be removed entirely in Airflow 2.0
+
+Redshift to S3 Operator
+"""""""""""""""""""""""
+
+With Airflow 1.9 or lower, Unload operation always included header row. In order to include header row,
+we need to turn off parallel unload. It is preferred to perform unload operation using all nodes so that it is
+faster for larger tables. So, parameter called ``include_header`` is added and default is set to False.
+Header row will be added only if this parameter is set True and also in that case parallel will be automatically turned off (\ ``PARALLEL OFF``\ )
+
+Google cloud connection string
+""""""""""""""""""""""""""""""
+
+With Airflow 1.9 or lower, there were two connection strings for the Google Cloud operators, both ``google_cloud_storage_default`` and ``google_cloud_default``. This can be confusing and therefore the ``google_cloud_storage_default`` connection id has been replaced with ``google_cloud_default`` to make the connection id consistent across Airflow.
+
+Logging Configuration
+"""""""""""""""""""""
+
+With Airflow 1.9 or lower, ``FILENAME_TEMPLATE``\ , ``PROCESSOR_FILENAME_TEMPLATE``\ , ``LOG_ID_TEMPLATE``\ , ``END_OF_LOG_MARK`` were configured in ``airflow_local_settings.py``. These have been moved into the configuration file, and hence if you were using a custom configuration file the following defaults need to be added.
+
+.. code-block::
+
+ [core]
+ fab_logging_level = WARN
+ log_filename_template = {{{{ ti.dag_id }}}}/{{{{ ti.task_id }}}}/{{{{ ts }}}}/{{{{ try_number }}}}.log
+ log_processor_filename_template = {{{{ filename }}}}.log
+
+ [elasticsearch]
+ elasticsearch_log_id_template = {{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}
+ elasticsearch_end_of_log_mark = end_of_log
+
+The previous setting of ``log_task_reader`` is not needed in many cases now when using the default logging config with remote storage. (Previously it needed to be set to ``s3.task`` or similar. This is not needed with the default config anymore)
+
+Change of per-task log path
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+With the change to Airflow core to be timezone aware the default log path for task instances will now include timezone information. This will by default mean all previous task logs won't be found. You can get the old behaviour back by setting the following config options:
+
+.. code-block::
+
+ [core]
+ log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ execution_date.strftime("%%Y-%%m-%%dT%%H:%%M:%%S") }}/{{ try_number }}.log
+
+Changelog
+^^^^^^^^^
+
- [AIRFLOW-2870] Use abstract TaskInstance for migration
- [AIRFLOW-2859] Implement own UtcDateTime (#3708)
- [AIRFLOW-2140] Don't require kubernetes for the SparkSubmit hook
@@ -4245,8 +8669,260 @@ Airflow 1.10.0, 2018-08-03
- [AIRFLOW-1609] Fix gitignore to ignore all venvs
- [AIRFLOW-1601] Add configurable task cleanup time
-Airflow 1.9.0, 2018-01-02
--------------------------
+Airflow 1.9.0 (2018-01-02)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+SSH Hook updates, along with new SSH Operator & SFTP Operator
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+SSH Hook now uses the Paramiko library to create an ssh client connection, instead of the sub-process based ssh command execution previously (<1.9.0), so this is backward incompatible.
+
+
+* update SSHHook constructor
+* use SSHOperator class in place of SSHExecuteOperator which is removed now. Refer to test_ssh_operator.py for usage info.
+* SFTPOperator is added to perform secure file transfer from server A to server B. Refer to test_sftp_operator.py for usage info.
+* No updates are required if you are using ``ftpHook``, it will continue to work as is.
+
+S3Hook switched to use Boto3
+""""""""""""""""""""""""""""
+
+The airflow.hooks.S3_hook.S3Hook has been switched to use boto3 instead of the older boto (a.k.a. boto2). This results in a few backwards incompatible changes to the following classes: S3Hook:
+
+
+* the constructors no longer accepts ``s3_conn_id``. It is now called ``aws_conn_id``.
+* the default connection is now "aws_default" instead of "s3_default"
+* the return type of objects returned by ``get_bucket`` is now boto3.s3.Bucket
+* the return type of ``get_key``\ , and ``get_wildcard_key`` is now an boto3.S3.Object.
+
+If you are using any of these in your DAGs and specify a connection ID you will need to update the parameter name for the connection to "aws_conn_id": S3ToHiveTransfer, S3PrefixSensor, S3KeySensor, RedshiftToS3Transfer.
+
+Logging update
+""""""""""""""
+
+The logging structure of Airflow has been rewritten to make configuration easier and the logging system more transparent.
+
+A quick recap about logging
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A logger is the entry point into the logging system. Each logger is a named bucket to which messages can be written for processing. A logger is configured to have a log level. This log level describes the severity of the messages that the logger will handle. Python defines the following log levels: DEBUG, INFO, WARNING, ERROR or CRITICAL.
+
+Each message that is written to the logger is a Log Record. Each log record contains a log level indicating the severity of that specific message. A log record can also contain useful metadata that describes the event that is being logged. This can include details such as a stack trace or an error code.
+
+When a message is given to the logger, the log level of the message is compared to the log level of the logger. If the log level of the message meets or exceeds the log level of the logger itself, the message will undergo further processing. If it doesn’t, the message will be ignored.
+
+Once a logger has determined that a message needs to be processed, it is passed to a Handler. This configuration is now more flexible and can be easily be maintained in a single file.
+
+Changes in Airflow Logging
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Airflow's logging mechanism has been refactored to use Python’s built-in ``logging`` module to perform logging of the application. By extending classes with the existing ``LoggingMixin``\ , all the logging will go through a central logger. Also the ``BaseHook`` and ``BaseOperator`` already extend this class, so it is easily available to do logging.
+
+The main benefit is easier configuration of the logging by setting a single centralized python file. Disclaimer; there is still some inline configuration, but this will be removed eventually. The new logging class is defined by setting the dotted classpath in your ``~/airflow/airflow.cfg`` file:
+
+.. code-block::
+
+ # Logging class
+ # Specify the class that will specify the logging configuration
+ # This class has to be on the python classpath
+ logging_config_class = my.path.default_local_settings.LOGGING_CONFIG
+
+The logging configuration file needs to be on the ``PYTHONPATH``\ , for example ``$AIRFLOW_HOME/config``. This directory is loaded by default. Any directory may be added to the ``PYTHONPATH``\ , this might be handy when the config is in another directory or a volume is mounted in case of Docker.
+
+The config can be taken from ``airflow/config_templates/airflow_local_settings.py`` as a starting point. Copy the contents to ``${AIRFLOW_HOME}/config/airflow_local_settings.py``\ , and alter the config as is preferred.
+
+.. code-block::
+
+ #
+ # Licensed to the Apache Software Foundation (ASF) under one
+ # or more contributor license agreements. See the NOTICE file
+ # distributed with this work for additional information
+ # regarding copyright ownership. The ASF licenses this file
+ # to you under the Apache License, Version 2.0 (the
+ # "License"); you may not use this file except in compliance
+ # with the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing,
+ # software distributed under the License is distributed on an
+ # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ # KIND, either express or implied. See the License for the
+ # specific language governing permissions and limitations
+ # under the License.
+
+ import os
+
+ from airflow import configuration as conf
+
+ # TODO: Logging format and level should be configured
+ # in this file instead of from airflow.cfg. Currently
+ # there are other log format and level configurations in
+ # settings.py and cli.py. Please see AIRFLOW-1455.
+
+ LOG_LEVEL = conf.get('core', 'LOGGING_LEVEL').upper()
+ LOG_FORMAT = conf.get('core', 'log_format')
+
+ BASE_LOG_FOLDER = conf.get('core', 'BASE_LOG_FOLDER')
+ PROCESSOR_LOG_FOLDER = conf.get('scheduler', 'child_process_log_directory')
+
+ FILENAME_TEMPLATE = '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'
+ PROCESSOR_FILENAME_TEMPLATE = '{{ filename }}.log'
+
+ DEFAULT_LOGGING_CONFIG = {
+ 'version': 1,
+ 'disable_existing_loggers': False,
+ 'formatters': {
+ 'airflow.task': {
+ 'format': LOG_FORMAT,
+ },
+ 'airflow.processor': {
+ 'format': LOG_FORMAT,
+ },
+ },
+ 'handlers': {
+ 'console': {
+ 'class': 'logging.StreamHandler',
+ 'formatter': 'airflow.task',
+ 'stream': 'ext://sys.stdout'
+ },
+ 'file.task': {
+ 'class': 'airflow.utils.log.file_task_handler.FileTaskHandler',
+ 'formatter': 'airflow.task',
+ 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
+ 'filename_template': FILENAME_TEMPLATE,
+ },
+ 'file.processor': {
+ 'class': 'airflow.utils.log.file_processor_handler.FileProcessorHandler',
+ 'formatter': 'airflow.processor',
+ 'base_log_folder': os.path.expanduser(PROCESSOR_LOG_FOLDER),
+ 'filename_template': PROCESSOR_FILENAME_TEMPLATE,
+ }
+ # When using s3 or gcs, provide a customized LOGGING_CONFIG
+ # in airflow_local_settings within your PYTHONPATH, see UPDATING.md
+ # for details
+ # 's3.task': {
+ # 'class': 'airflow.utils.log.s3_task_handler.S3TaskHandler',
+ # 'formatter': 'airflow.task',
+ # 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
+ # 's3_log_folder': S3_LOG_FOLDER,
+ # 'filename_template': FILENAME_TEMPLATE,
+ # },
+ # 'gcs.task': {
+ # 'class': 'airflow.utils.log.gcs_task_handler.GCSTaskHandler',
+ # 'formatter': 'airflow.task',
+ # 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
+ # 'gcs_log_folder': GCS_LOG_FOLDER,
+ # 'filename_template': FILENAME_TEMPLATE,
+ # },
+ },
+ 'loggers': {
+ '': {
+ 'handlers': ['console'],
+ 'level': LOG_LEVEL
+ },
+ 'airflow': {
+ 'handlers': ['console'],
+ 'level': LOG_LEVEL,
+ 'propagate': False,
+ },
+ 'airflow.processor': {
+ 'handlers': ['file.processor'],
+ 'level': LOG_LEVEL,
+ 'propagate': True,
+ },
+ 'airflow.task': {
+ 'handlers': ['file.task'],
+ 'level': LOG_LEVEL,
+ 'propagate': False,
+ },
+ 'airflow.task_runner': {
+ 'handlers': ['file.task'],
+ 'level': LOG_LEVEL,
+ 'propagate': True,
+ },
+ }
+ }
+
+To customize the logging (for example, use logging rotate), define one or more of the logging handles that `Python has to offer `_. For more details about the Python logging, please refer to the `official logging documentation `_.
+
+Furthermore, this change also simplifies logging within the DAG itself:
+
+.. code-block::
+
+ root@ae1bc863e815:/airflow# python
+ Python 3.6.2 (default, Sep 13 2017, 14:26:54)
+ [GCC 4.9.2] on linux
+ Type "help", "copyright", "credits" or "license" for more information.
+ >>> from airflow.settings import *
+ >>>
+ >>> from datetime import datetime
+ >>> from airflow.models.dag import DAG
+ >>> from airflow.operators.dummy import DummyOperator
+ >>>
+ >>> dag = DAG('simple_dag', start_date=datetime(2017, 9, 1))
+ >>>
+ >>> task = DummyOperator(task_id='task_1', dag=dag)
+ >>>
+ >>> task.log.error('I want to say something..')
+ [2017-09-25 20:17:04,927] {:1} ERROR - I want to say something..
+
+Template path of the file_task_handler
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``file_task_handler`` logger has been made more flexible. The default format can be changed, ``{dag_id}/{task_id}/{execution_date}/{try_number}.log`` by supplying Jinja templating in the ``FILENAME_TEMPLATE`` configuration variable. See the ``file_task_handler`` for more information.
+
+I'm using S3Log or GCSLogs, what do I do!?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are logging to Google cloud storage, please see the `Google cloud platform documentation `_ for logging instructions.
+
+If you are using S3, the instructions should be largely the same as the Google cloud platform instructions above. You will need a custom logging config. The ``REMOTE_BASE_LOG_FOLDER`` configuration key in your airflow config has been removed, therefore you will need to take the following steps:
+
+
+* Copy the logging configuration from `airflow/config_templates/airflow_logging_settings.py `_.
+* Place it in a directory inside the Python import path ``PYTHONPATH``. If you are using Python 2.7, ensuring that any ``__init__.py`` files exist so that it is importable.
+* Update the config by setting the path of ``REMOTE_BASE_LOG_FOLDER`` explicitly in the config. The ``REMOTE_BASE_LOG_FOLDER`` key is not used anymore.
+* Set the ``logging_config_class`` to the filename and dict. For example, if you place ``custom_logging_config.py`` on the base of your ``PYTHONPATH``\ , you will need to set ``logging_config_class = custom_logging_config.LOGGING_CONFIG`` in your config as Airflow 1.8.
+
+New Features
+""""""""""""
+
+Dask Executor
+~~~~~~~~~~~~~
+
+A new DaskExecutor allows Airflow tasks to be run in Dask Distributed clusters.
+
+Deprecated Features
+"""""""""""""""""""
+
+These features are marked for deprecation. They may still work (and raise a ``DeprecationWarning``\ ), but are no longer
+supported and will be removed entirely in Airflow 2.0
+
+
+*
+ If you're using the ``google_cloud_conn_id`` or ``dataproc_cluster`` argument names explicitly in ``contrib.operators.Dataproc{*}Operator``\ (s), be sure to rename them to ``gcp_conn_id`` or ``cluster_name``\ , respectively. We've renamed these arguments for consistency. (AIRFLOW-1323)
+
+*
+ ``post_execute()`` hooks now take two arguments, ``context`` and ``result``
+ (AIRFLOW-886)
+
+ Previously, post_execute() only took one argument, ``context``.
+
+*
+ ``contrib.hooks.gcp_dataflow_hook.DataFlowHook`` starts to use ``--runner=DataflowRunner`` instead of ``DataflowPipelineRunner``\ , which is removed from the package ``google-cloud-dataflow-0.6.0``.
+
+*
+ The pickle type for XCom messages has been replaced by json to prevent RCE attacks.
+ Note that JSON serialization is stricter than pickling, so if you want to e.g. pass
+ raw bytes through XCom you must encode them using an encoding like base64.
+ By default pickling is still enabled until Airflow 2.0. To disable it
+ set enable_xcom_pickling = False in your Airflow config.
+
+Changelog
+^^^^^^^^^
- [AIRFLOW-1525] Fix minor LICENSE and NOTICE issues
- [AIRFLOW-XXX] Bump version to 1.9.0
@@ -4693,8 +9369,16 @@ Airflow 1.9.0, 2018-01-02
- [AIRFLOW-365] Set dag.fileloc explicitly and use for Code view
- [AIRFLOW-781] Allow DataFlowOperators to accept jobs stored in GCS
-Airflow 1.8.2, 2017-09-04
--------------------------
+Airflow 1.8.2 (2017-09-04)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+No significant changes.
+
+Changelog
+^^^^^^^^^
- [AIRFLOW-809][AIRFLOW-1] Use __eq__ ColumnOperator When Testing Booleans
- [AIRFLOW-1296] Propagate SKIPPED to all downstream tasks
@@ -4738,8 +9422,17 @@ Airflow 1.8.2, 2017-09-04
- [AIRFLOW-1168] Add closing() to all connections and cursors
- [AIRFLOW-XXX] Updating CHANGELOG, README, and UPDATING after 1.8.1 release
-Airflow 1.8.1, 2017-05-09
--------------------------
+Airflow 1.8.1 (2017-05-09)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+The Airflow package name was changed from ``airflow`` to ``apache-airflow`` during this release. You must uninstall
+a previously installed version of Airflow before installing 1.8.1.
+
+Changelog
+^^^^^^^^^
- [AIRFLOW-1142] SubDAG Tasks Not Executed Even Though All Dependencies Met
- [AIRFLOW-1138] Add licenses to files in scripts directory
@@ -4779,8 +9472,174 @@ Airflow 1.8.1, 2017-05-09
- [AIRFLOW-111] DAG concurrency is not honored
- [AIRFLOW-88] Improve clarity Travis CI reports
-Airflow 1.8.0, 2017-03-12
--------------------------
+Airflow 1.8.0 (2017-03-12)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+Database
+""""""""
+
+The database schema needs to be upgraded. Make sure to shutdown Airflow and make a backup of your database. To
+upgrade the schema issue ``airflow upgradedb``.
+
+Upgrade systemd unit files
+""""""""""""""""""""""""""
+
+Systemd unit files have been updated. If you use systemd please make sure to update these.
+
+..
+
+ Please note that the webserver does not detach properly, this will be fixed in a future version.
+
+
+Tasks not starting although dependencies are met due to stricter pool checking
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Airflow 1.7.1 has issues with being able to over subscribe to a pool, ie. more slots could be used than were
+available. This is fixed in Airflow 1.8.0, but due to past issue jobs may fail to start although their
+dependencies are met after an upgrade. To workaround either temporarily increase the amount of slots above
+the amount of queued tasks or use a new pool.
+
+Less forgiving scheduler on dynamic start_date
+""""""""""""""""""""""""""""""""""""""""""""""
+
+Using a dynamic start_date (e.g. ``start_date = datetime.now()``\ ) is not considered a best practice. The 1.8.0 scheduler
+is less forgiving in this area. If you encounter DAGs not being scheduled you can try using a fixed start_date and
+renaming your DAG. The last step is required to make sure you start with a clean slate, otherwise the old schedule can
+interfere.
+
+New and updated scheduler options
+"""""""""""""""""""""""""""""""""
+
+Please read through the new scheduler options, defaults have changed since 1.7.1.
+
+child_process_log_directory
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In order to increase the robustness of the scheduler, DAGs are now processed in their own process. Therefore each
+DAG has its own log file for the scheduler. These log files are placed in ``child_process_log_directory`` which defaults to
+``/scheduler/latest``. You will need to make sure these log files are removed.
+
+..
+
+ DAG logs or processor logs ignore and command line settings for log file locations.
+
+
+run_duration
+~~~~~~~~~~~~
+
+Previously the command line option ``num_runs`` was used to let the scheduler terminate after a certain amount of
+loops. This is now time bound and defaults to ``-1``\ , which means run continuously. See also num_runs.
+
+num_runs
+~~~~~~~~
+
+Previously ``num_runs`` was used to let the scheduler terminate after a certain amount of loops. Now num_runs specifies
+the number of times to try to schedule each DAG file within ``run_duration`` time. Defaults to ``-1``\ , which means try
+indefinitely. This is only available on the command line.
+
+min_file_process_interval
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+After how much time should an updated DAG be picked up from the filesystem.
+
+min_file_parsing_loop_time
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+CURRENTLY DISABLED DUE TO A BUG
+How many seconds to wait between file-parsing loops to prevent the logs from being spammed.
+
+dag_dir_list_interval
+~~~~~~~~~~~~~~~~~~~~~
+
+The frequency with which the scheduler should relist the contents of the DAG directory. If while developing +dags, they are not being picked up, have a look at this number and decrease it when necessary.
+
+catchup_by_default
+~~~~~~~~~~~~~~~~~~
+
+By default the scheduler will fill any missing interval DAG Runs between the last execution date and the current date.
+This setting changes that behavior to only execute the latest interval. This can also be specified per DAG as
+``catchup = False / True``. Command line backfills will still work.
+
+Faulty DAGs do not show an error in the Web UI
+""""""""""""""""""""""""""""""""""""""""""""""
+
+Due to changes in the way Airflow processes DAGs the Web UI does not show an error when processing a faulty DAG. To
+find processing errors go the ``child_process_log_directory`` which defaults to ``/scheduler/latest``.
+
+New DAGs are paused by default
+""""""""""""""""""""""""""""""
+
+Previously, new DAGs would be scheduled immediately. To retain the old behavior, add this to airflow.cfg:
+
+.. code-block::
+
+ [core]
+ dags_are_paused_at_creation = False
+
+Airflow Context variable are passed to Hive config if conf is specified
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+If you specify a hive conf to the run_cli command of the HiveHook, Airflow add some
+convenience variables to the config. In case you run a secure Hadoop setup it might be
+required to allow these variables by adjusting you hive configuration to add ``airflow\.ctx\..*`` to the regex
+of user-editable configuration properties. See
+`the Hive docs on Configuration Properties `_ for more info.
+
+Google Cloud Operator and Hook alignment
+""""""""""""""""""""""""""""""""""""""""
+
+All Google Cloud Operators and Hooks are aligned and use the same client library. Now you have a single connection
+type for all kinds of Google Cloud Operators.
+
+If you experience problems connecting with your operator make sure you set the connection type "Google Cloud".
+
+Also the old P12 key file type is not supported anymore and only the new JSON key files are supported as a service
+account.
+
+Deprecated Features
+"""""""""""""""""""
+
+These features are marked for deprecation. They may still work (and raise a ``DeprecationWarning``\ ), but are no longer
+supported and will be removed entirely in Airflow 2.0
+
+
+*
+ Hooks and operators must be imported from their respective submodules
+
+ ``airflow.operators.PigOperator`` is no longer supported; ``from airflow.operators.pig_operator import PigOperator`` is.
+ (AIRFLOW-31, AIRFLOW-200)
+
+*
+ Operators no longer accept arbitrary arguments
+
+ Previously, ``Operator.__init__()`` accepted any arguments (either positional ``*args`` or keyword ``**kwargs``\ ) without
+ complaint. Now, invalid arguments will be rejected. (https://github.com/apache/airflow/pull/1285)
+
+*
+ The config value secure_mode will default to True which will disable some insecure endpoints/features
+
+Known Issues
+""""""""""""
+
+There is a report that the default of "-1" for num_runs creates an issue where errors are reported while parsing tasks.
+It was not confirmed, but a workaround was found by changing the default back to ``None``.
+
+To do this edit ``cli.py``\ , find the following:
+
+.. code-block::
+
+ 'num_runs': Arg(
+ ("-n", "--num_runs"),
+ default=-1, type=int,
+ help="Set the number of runs to execute before exiting"),
+
+and change ``default=-1`` to ``default=None``. If you have this issue please report it on the mailing list.
+
+Changelog
+^^^^^^^^^
- [AIRFLOW-900] Double trigger should not kill original task instance
- [AIRFLOW-900] Fixes bugs in LocalTaskJob for double run protection
@@ -4989,8 +9848,50 @@ Airflow 1.8.0, 2017-03-12
- [AIRFLOW-475] make the segment granularity in Druid hook configurable
-Airflow 1.7.2
--------------
+Airflow 1.7.1.2 (2016-05-20)
+----------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+Changes to Configuration
+""""""""""""""""""""""""
+
+Email configuration change
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To continue using the default smtp email backend, change the email_backend line in your config file from:
+
+.. code-block::
+
+ [email]
+ email_backend = airflow.utils.send_email_smtp
+
+to:
+
+.. code-block::
+
+ [email]
+ email_backend = airflow.utils.email.send_email_smtp
+
+S3 configuration change
+~~~~~~~~~~~~~~~~~~~~~~~
+
+To continue using S3 logging, update your config file so:
+
+.. code-block::
+
+ s3_log_folder = s3://my-airflow-log-bucket/logs
+
+becomes:
+
+.. code-block::
+
+ remote_base_log_folder = s3://my-airflow-log-bucket/logs
+ remote_log_conn_id =
+
+Changelog
+^^^^^^^^^
- [AIRFLOW-463] Link Airflow icon to landing page
- [AIRFLOW-149] Task Dependency Engine + Why Isn't My Task Running View
@@ -5156,8 +10057,8 @@ Airflow 1.7.2
- [AIRFLOW-157] Make PR tool Py3-compat; add JIRA command
- [AIRFLOW-170] Add missing @apply_defaults
-Airflow 1.7.1, 2016-05-19
--------------------------
+Airflow 1.7.1 (2016-05-19)
+--------------------------
- Fix : Don't treat premature tasks as could_not_run tasks
- AIRFLOW-92 Avoid unneeded upstream_failed session closes apache/airflow#1485
@@ -5168,7 +10069,7 @@ Airflow 1.7.1, 2016-05-19
- [AIRFLOW-52] Fix bottlenecks when working with many tasks
- Add columns to toggle extra detail in the connection list view.
- Log the number of errors when importing DAGs
-- Log dagbag metrics duplicate messages in queue into Statsd (#1406)
+- Log dagbag metrics duplicate messages in queue into StatsD (#1406)
- Clean up issue template (#1419)
- correct missed arg.foreground to arg.daemon in cli
- Reinstate imports for github enterprise auth
diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
index 6121d694b62a9..dccade808e425 100644
--- a/STATIC_CODE_CHECKS.rst
+++ b/STATIC_CODE_CHECKS.rst
@@ -41,7 +41,7 @@ use. So, you can be sure your modifications will also work for CI if they pass
pre-commit hooks.
We have integrated the fantastic `pre-commit `__ framework
-in our development workflow. To install and use it, you need at least Python 3.6 locally.
+in our development workflow. To install and use it, you need at least Python 3.7 locally.
Installing pre-commit hooks
...........................
@@ -111,185 +111,215 @@ For details on advanced usage of the install method, use:
Available pre-commit checks
...........................
-This table lists pre-commit hooks used by Airflow. The ``Breeze`` column indicates which hooks
-require Breeze Docker images to be installed locally.
+This table lists pre-commit hooks used by Airflow. The ``Image`` column indicates which hooks
+require Breeze Docker image to be build locally.
.. note:: Disabling particular checks
- In case you have a problem with running particular ``pre-commit`` check you can still continue using the
- benefits of having ``pre-commit`` installed, with some of the checks disabled. In order to disable
- checks you need to set ``SKIP`` environment variable to coma-separated list of checks to skip. For example
- when you want to skip all checks that require Breeze Docker image to be installed, you should be able to
- do it by setting ``export SKIP=bat-in-container-tests,build,flake8,mypy``. You can also add this to your
- ``.bashrc`` or ``.zshrc`` if you do not want to set it manually every time you enter the terminal.
-
-==================================== ================================================================ ============
-**Checks** **Description** **Breeze**
-==================================== ================================================================ ============
-``airflow-config-yaml`` Checks that airflow config YAML is 1-1 with the code
------------------------------------- ---------------------------------------------------------------- ------------
-``airflow-providers-available`` Checks that providers are properly declared by extras
------------------------------------- ---------------------------------------------------------------- ------------
-``airflow-provider-yaml-files-ok`` Checks that providers YAML files are valid
------------------------------------- ---------------------------------------------------------------- ------------
-``autoflake`` Remove unused imports and unnecessary code
------------------------------------- ---------------------------------------------------------------- ------------
-``base-operator`` Checks that BaseOperator is imported properly
------------------------------------- ---------------------------------------------------------------- ------------
-``black`` Runs Black (the uncompromising Python code formatter)
------------------------------------- ---------------------------------------------------------------- ------------
-``blacken-docs`` Run black on python code blocks in documentation files
------------------------------------- ---------------------------------------------------------------- ------------
-``boring-cyborg`` Checks for Boring Cyborg configuration consistency
------------------------------------- ---------------------------------------------------------------- ------------
-``build-providers-dependencies`` Regenerates the JSON file with cross-provider dependencies
------------------------------------- ---------------------------------------------------------------- ------------
-``chart-schema-lint`` Lint chart/values.schema.json file
------------------------------------- ---------------------------------------------------------------- ------------
-``capitalized-breeze`` Breeze has to be Capitalized in Breeze2
------------------------------------- ---------------------------------------------------------------- ------------
-``changelog-duplicates`` Checks for duplicate changelog entries
------------------------------------- ---------------------------------------------------------------- ------------
-``check-apache-license`` Checks compatibility with Apache License requirements
------------------------------------- ---------------------------------------------------------------- ------------
-``check-builtin-literals`` Require literal syntax when initializing Python builtin types
------------------------------------- ---------------------------------------------------------------- ------------
-``check-executables-have-shebangs`` Checks that executables have shebang
------------------------------------- ---------------------------------------------------------------- ------------
-``check-extras-order`` Checks that extras in Dockerfile are sorted
------------------------------------- ---------------------------------------------------------------- ------------
-``check-hooks-apply`` Checks which hooks are applicable to the repository
------------------------------------- ---------------------------------------------------------------- ------------
-``check-integrations`` Checks if integration list is synchronized in code
------------------------------------- ---------------------------------------------------------------- ------------
-``check-merge-conflicts`` Checks that merge conflicts are not being committed
------------------------------------- ---------------------------------------------------------------- ------------
-``check-xml`` Checks XML files with xmllint
------------------------------------- ---------------------------------------------------------------- ------------
-``daysago-import-check`` Checks if daysago is properly imported
------------------------------------- ---------------------------------------------------------------- ------------
-``debug-statements`` Detects accidentally committed debug statements
------------------------------------- ---------------------------------------------------------------- ------------
-``detect-private-key`` Detects if private key is added to the repository
------------------------------------- ---------------------------------------------------------------- ------------
-``docstring-params`` Checks that param types not specified in docstring
------------------------------------- ---------------------------------------------------------------- ------------
-``doctoc`` Refreshes the table of contents for MD files
------------------------------------- ---------------------------------------------------------------- ------------
-``dont-use-safe-filter`` Don't use safe in templates
------------------------------------- ---------------------------------------------------------------- ------------
-``no-providers-in-core-examples`` Don't use providers imports in core example DAGs
------------------------------------- ---------------------------------------------------------------- ------------
-``no-relative-imports`` Use absolute imports, not relative
------------------------------------- ---------------------------------------------------------------- ------------
-``end-of-file-fixer`` Makes sure that there is an empty line at the end.
------------------------------------- ---------------------------------------------------------------- ------------
-``fix-encoding-pragma`` Removes encoding header from python files
------------------------------------- ---------------------------------------------------------------- ------------
-``forbidden-xcom-get-value`` Check if XCom.get_value is used in backwards-compatible way
------------------------------------- ---------------------------------------------------------------- ------------
-``pyupgrade`` Runs pyupgrade
------------------------------------- ---------------------------------------------------------------- ------------
-``flake8`` Runs flake8 *
------------------------------------- ---------------------------------------------------------------- ------------
-``flynt`` Runs flynt
------------------------------------- ---------------------------------------------------------------- ------------
-``codespell`` Checks for common misspellings in files.
------------------------------------- ---------------------------------------------------------------- ------------
-``forbid-tabs`` Fails if tabs are used in the project
------------------------------------- ---------------------------------------------------------------- ------------
-``helm-lint`` Verifies if helm lint passes for the chart
------------------------------------- ---------------------------------------------------------------- ------------
-``identity`` Prints inputs to the static check hooks for troubleshooting
------------------------------------- ---------------------------------------------------------------- ------------
-``incorrect-use-of-LoggingMixin`` Checks if LoggingMixin is properly imported
------------------------------------- ---------------------------------------------------------------- ------------
-``insert-license`` Adds licenses for most file types
------------------------------------- ---------------------------------------------------------------- ------------
-``isort`` Sorts imports in python files
------------------------------------- ---------------------------------------------------------------- ------------
-``json-schema`` Checks that the files complies with the JSON Schema spec
------------------------------------- ---------------------------------------------------------------- ------------
-``language-matters`` Check for language that we do not accept as community
------------------------------------- ---------------------------------------------------------------- ------------
-``lint-dockerfile`` Lints a dockerfile
------------------------------------- ---------------------------------------------------------------- ------------
-``lint-openapi`` Lints openapi specification
------------------------------------- ---------------------------------------------------------------- ------------
-``markdownlint`` Lints Markdown files
------------------------------------- ---------------------------------------------------------------- ------------
-``migration-reference`` Update migration reference doc and revision metadata
------------------------------------- ---------------------------------------------------------------- ------------
-``mixed-line-ending`` Detects if mixed line ending is used (\r vs. \r\n)
------------------------------------- ---------------------------------------------------------------- ------------
-``mypy`` Runs mypy *
------------------------------------- ---------------------------------------------------------------- ------------
-``persist-credentials-disabled`` Check that workflow files have persist-credentials disabled
------------------------------------- ---------------------------------------------------------------- ------------
-``pre-commit-descriptions`` Check if all pre-commits are described in docs
------------------------------------- ---------------------------------------------------------------- ------------
-``pre-commit-hook-names`` Check that hook names are not overly long
------------------------------------- ---------------------------------------------------------------- ------------
-``pretty-format-json`` Formats json files
------------------------------------- ---------------------------------------------------------------- ------------
-``provide-create-sessions`` Make sure provide-session and create-session imports are OK
------------------------------------- ---------------------------------------------------------------- ------------
-``providers-changelogs`` Updates documentation for providers changelogs
------------------------------------- ---------------------------------------------------------------- ------------
-``providers-subpackages-init-file`` Check that providers' subpackages __init__.py files are there
------------------------------------- ---------------------------------------------------------------- ------------
-``providers-init-file`` Check that provider's __init__.py file is removed
------------------------------------- ---------------------------------------------------------------- ------------
-``provider-yamls`` Checks that provider.yaml files have the correct content
------------------------------------- ---------------------------------------------------------------- ------------
-``pydevd`` Check for accidentally committed pydevd statements
------------------------------------- ---------------------------------------------------------------- ------------
-``pydocstyle`` Runs pydocstyle
------------------------------------- ---------------------------------------------------------------- ------------
-``python-no-log-warn`` Checks if there are no deprecate log warn
------------------------------------- ---------------------------------------------------------------- ------------
-``restrict-start_date`` 'start_date' should not be in default_args in example_dags
------------------------------------- ---------------------------------------------------------------- ------------
-``rst-backticks`` Checks if RST files use double backticks for code
------------------------------------- ---------------------------------------------------------------- ------------
-``setup-order`` Checks for an order of dependencies in setup.py
------------------------------------- ---------------------------------------------------------------- ------------
-``setup-extra-packages`` Checks if all the libraries in setup.py are listed in docs
------------------------------------- ---------------------------------------------------------------- ------------
-``shellcheck`` Checks shell files with shellcheck
------------------------------------- ---------------------------------------------------------------- ------------
-``sort-in-the-wild`` Sort INTHEWILD.md alphabetically
------------------------------------- ---------------------------------------------------------------- ------------
-``sort-spelling-wordlist`` Sort alphabetically and uniquify spelling_wordlist.txt
------------------------------------- ---------------------------------------------------------------- ------------
-``stylelint`` Checks CSS files with stylelint
------------------------------------- ---------------------------------------------------------------- ------------
-``trailing-whitespace`` Removes trailing whitespace at end of line
------------------------------------- ---------------------------------------------------------------- ------------
-``ui-lint`` Static checks of airflow/ui/ folder
------------------------------------- ---------------------------------------------------------------- ------------
-``update-breeze-file`` Update output of breeze command in BREEZE.rst
------------------------------------- ---------------------------------------------------------------- ------------
-``update-extras`` Updates extras in the documentation
------------------------------------- ---------------------------------------------------------------- ------------
-``update-local-yml-file`` Updates mounts in local.yml file
------------------------------------- ---------------------------------------------------------------- ------------
-``update-setup-cfg-file`` Update setup.cfg file with all licenses
------------------------------------- ---------------------------------------------------------------- ------------
-``update-supported-versions`` Updates supported versions in documentation
------------------------------------- ---------------------------------------------------------------- ------------
-``update-versions`` Updates latest versions in the documentation
------------------------------------- ---------------------------------------------------------------- ------------
-``vendor-k8s-json-schema`` Vendor k8s schema definitions in the helm chart schema file
------------------------------------- ---------------------------------------------------------------- ------------
-``verify-db-migrations-documented`` Verify DB Migrations have been documented
------------------------------------- ---------------------------------------------------------------- ------------
-``www-lint`` Static checks of js in airflow/www/static/js/ folder
------------------------------------- ---------------------------------------------------------------- ------------
-``yesqa`` Removes unnecessary noqa statements
------------------------------------- ---------------------------------------------------------------- ------------
-``yamllint`` Checks YAML files with yamllint
-==================================== ================================================================ ============
+ In case you have a problem with running particular ``pre-commit`` check you can still continue using the
+ benefits of having ``pre-commit`` installed, with some of the checks disabled. In order to disable
+ checks you might need to set ``SKIP`` environment variable to coma-separated list of checks to skip. For example
+ when you want to skip some checks (flake/mypy for example), you should be able to do it by setting
+ ``export SKIP=run-flake8,run-mypy``. You can also add this to your ``.bashrc`` or ``.zshrc`` if you
+ do not want to set it manually every time you enter the terminal.
+
+ In case you do not have breeze image configured locally, you can also disable all checks that require
+ the image by setting ``SKIP_IMAGE_PRE_COMMITS`` to "true". This will mark the tests as "green" automatically
+ when run locally (note that those checks will anyway run in CI).
+
+ .. BEGIN AUTO-GENERATED STATIC CHECK LIST
+
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| ID | Description | Image |
++========================================================+==================================================================+=========+
+| black | Run Black (the uncompromising Python code formatter) | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| blacken-docs | Run black on python code blocks in documentation files | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-airflow-2-1-compatibility | Check that providers are 2.1 compatible. | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-airflow-config-yaml-consistent | Checks for consistency between config.yml and default_config.cfg | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-airflow-providers-have-extras | Checks providers available when declared by extras in setup.py | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-apache-license-rat | Check if licenses are OK for Apache | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-base-operator-partial-arguments | Check BaseOperator and partial() arguments | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-base-operator-usage | * Check BaseOperator[Link] core imports | |
+| | * Check BaseOperator[Link] other imports | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-boring-cyborg-configuration | Checks for Boring Cyborg configuration consistency | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-breeze-top-dependencies-limited | Breeze should have small number of top-level dependencies | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-builtin-literals | Require literal syntax when initializing Python builtin types | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-changelog-has-no-duplicates | Check changelogs for duplicate entries | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-daysago-import-from-utils | Make sure days_ago is imported from airflow.utils.dates | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-docstring-param-types | Check that docstrings do not specify param types | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-executables-have-shebangs | Check that executables have shebang | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-extra-packages-references | Checks setup extra packages | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-extras-order | Check order of extras in Dockerfile | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-for-inclusive-language | Check for language that we do not accept as community | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-hooks-apply | Check if all hooks apply to the repository | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-incorrect-use-of-LoggingMixin | Make sure LoggingMixin is not used alone | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-integrations-are-consistent | Check if integration list is consistent in various places | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-merge-conflict | Check that merge conflicts are not being committed | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-newsfragments-are-valid | Check newsfragments are valid | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-no-providers-in-core-examples | No providers imports in core example DAGs | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-no-relative-imports | No relative imports | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-persist-credentials-disabled-in-github-workflows | Check that workflow files have persist-credentials disabled | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-pre-commit-information-consistent | Update information re pre-commit hooks and verify ids and names | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-provide-create-sessions-imports | Check provide_session and create_session imports | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-provider-yaml-valid | Validate providers.yaml files | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-providers-init-file-missing | Provider init file is missing | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-providers-subpackages-init-file-exist | Provider subpackage init files are there | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-pydevd-left-in-code | Check for pydevd debug statements accidentally left | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-revision-heads-map | Check that the REVISION_HEADS_MAP is up-to-date | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-safe-filter-usage-in-html | Don't use safe in templates | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-setup-order | Check order of dependencies in setup.cfg and setup.py | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-start-date-not-used-in-defaults | 'start_date' not to be defined in default_args in example_dags | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-system-tests-present | Check if system tests have required segments of code | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| check-xml | Check XML files with xmllint | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| codespell | Run codespell to check for common misspellings in files | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| debug-statements | Detect accidentally committed debug statements | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| detect-private-key | Detect if private key is added to the repository | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| doctoc | Add TOC for md and rst files | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| end-of-file-fixer | Make sure that there is an empty line at the end | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| fix-encoding-pragma | Remove encoding header from python files | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| flynt | Run flynt string format converter for Python | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| forbid-tabs | Fail if tabs are used in the project | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| identity | Print input to the static check hooks for troubleshooting | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| insert-license | * Add license for all SQL files | |
+| | * Add license for all rst files | |
+| | * Add license for all CSS/JS/PUML/TS/TSX files | |
+| | * Add license for all JINJA template files | |
+| | * Add license for all shell files | |
+| | * Add license for all Python files | |
+| | * Add license for all XML files | |
+| | * Add license for all YAML files | |
+| | * Add license for all md files | |
+| | * Add license for all other files | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| isort | Run isort to sort imports in Python files | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| lint-chart-schema | Lint chart/values.schema.json file | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| lint-css | stylelint | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| lint-dockerfile | Lint dockerfile | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| lint-helm-chart | Lint Helm Chart | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| lint-javascript | * ESLint against airflow/ui | * |
+| | * ESLint against current UI JavaScript files | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| lint-json-schema | * Lint JSON Schema files with JSON Schema | |
+| | * Lint NodePort Service with JSON Schema | |
+| | * Lint Docker compose files with JSON Schema | |
+| | * Lint chart/values.schema.json file with JSON Schema | |
+| | * Lint chart/values.yaml file with JSON Schema | |
+| | * Lint airflow/config_templates/config.yml file with JSON Schema | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| lint-markdown | Run markdownlint | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| lint-openapi | * Lint OpenAPI using spectral | |
+| | * Lint OpenAPI using openapi-spec-validator | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| mixed-line-ending | Detect if mixed line ending is used (\r vs. \r\n) | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| pretty-format-json | Format json files | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| pydocstyle | Run pydocstyle | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| python-no-log-warn | Check if there are no deprecate log warn | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| pyupgrade | Upgrade Python code automatically | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| rst-backticks | Check if RST files use double backticks for code | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| run-flake8 | Run flake8 | * |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| run-mypy | * Run mypy for dev | * |
+| | * Run mypy for core | |
+| | * Run mypy for providers | |
+| | * Run mypy for /docs/ folder | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| run-shellcheck | Check Shell scripts syntax correctness | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| static-check-autoflake | Remove all unused code | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| trailing-whitespace | Remove trailing whitespace at end of line | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| update-breeze-file | Update output of breeze commands in BREEZE.rst | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| update-breeze-readme-config-hash | Update Breeze README.md with config files hash | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| update-extras | Update extras in documentation | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| update-in-the-wild-to-be-sorted | Sort INTHEWILD.md alphabetically | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| update-inlined-dockerfile-scripts | Inline Dockerfile and Dockerfile.ci scripts | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| update-local-yml-file | Update mounts in the local yml file | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| update-migration-references | Update migration ref doc | * |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| update-providers-dependencies | Update cross-dependencies for providers packages | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| update-setup-cfg-file | Update setup.cfg file with all licenses | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| update-spelling-wordlist-to-be-sorted | Sort alphabetically and uniquify spelling_wordlist.txt | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| update-supported-versions | Updates supported versions in documentation | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| update-vendored-in-k8s-json-schema | Vendor k8s definitions into values.schema.json | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| update-version | Update version to the latest version in the documentation | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| yamllint | Check YAML files with yamllint | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+| yesqa | Remove unnecessary noqa statements | |
++--------------------------------------------------------+------------------------------------------------------------------+---------+
+
+ .. END AUTO-GENERATED STATIC CHECK LIST
Using pre-commit
................
@@ -307,13 +337,13 @@ code. But you can run pre-commit hooks manually as needed.
.. code-block:: bash
- pre-commit run mypy
+ pre-commit run run-mypy
- Run only mypy checks on all files by using:
.. code-block:: bash
- pre-commit run mypy --all-files
+ pre-commit run run-mypy --all-files
- Run all checks on all files by using:
@@ -341,7 +371,7 @@ code. But you can run pre-commit hooks manually as needed.
.. code-block:: bash
- SKIP=mypy,flake8,build pre-commit run --all-files
+ SKIP=run-mypy,run-flake8 pre-commit run --all-files
You can always skip running the tests by providing ``--no-verify`` flag to the
@@ -354,7 +384,7 @@ Running static code checks via Breeze
The static code checks can be launched using the Breeze environment.
-You run the static code checks via ``./breeze static-check`` or commands.
+You run the static code checks via ``breeze static-check`` or commands.
You can see the list of available static checks either via ``--help`` flag or by using the autocomplete
option. Note that the ``all`` static check runs all configured static checks.
@@ -363,49 +393,63 @@ Run the ``mypy`` check for the currently staged changes:
.. code-block:: bash
- ./breeze static-check mypy
+ breeze static-checks --type run-mypy
Run the ``mypy`` check for all files:
.. code-block:: bash
- ./breeze static-check mypy -- --all-files
+ breeze static-checks --type run-mypy --all-files
Run the ``flake8`` check for the ``tests.core.py`` file with verbose output:
.. code-block:: bash
- ./breeze static-check flake8 -- --files tests/core.py --verbose
+ breeze static-checks --type run-flake8 --file tests/core.py --verbose
Run the ``flake8`` check for the ``tests.core`` package with verbose output:
.. code-block:: bash
- ./breeze static-check flake8 -- --files tests/core/* --verbose
+ breeze static-checks --type run-flake8 --file tests/core/* --verbose
-Run all tests for the currently staged files:
+Run all checks for the currently staged files:
.. code-block:: bash
- ./breeze static-check all
+ breeze static-checks --type all
-Run all tests for all files:
+Run all checks for all files:
.. code-block:: bash
- ./breeze static-check all -- --all-files
+ breeze static-checks --type all --all-files
-Run all tests for last commit :
+Run all checks for last commit :
.. code-block:: bash
- ./breeze static-check all -- --from-ref HEAD^ --to-ref HEAD
+ breeze static-checks --type all --last-commit
+Debugging pre-commit check scripts requiring image
+--------------------------------------------------
-The ``license`` check is run via a separate script and a separate Docker image containing the
-Apache RAT verification tool that checks for Apache-compatibility of licenses within the codebase.
-It does not take pre-commit parameters as extra arguments.
+Those commits that use Breeze docker image might sometimes fail, depending on your operating system and
+docker setup, so sometimes it might be required to run debugging with the commands. This is done via
+two environment variables ``VERBOSE`` and ``DRY_RUN``. Setting them to "true" will respectively show the
+commands to run before running them or skip running the commands.
+
+Note that you need to run pre-commit with --verbose command to get the output regardless of the status
+of the static check (normally it will only show output on failure).
+
+Printing the commands while executing:
+
+.. code-block:: bash
+
+ VERBOSE="true" pre-commit run --verbose run-flake8
+
+Just performing dry run:
.. code-block:: bash
- ./breeze static-check licenses
+ DRY_RUN="true" pre-commit run --verbose run-flake8
diff --git a/TESTING.rst b/TESTING.rst
index 70273c23671c4..12983726e1ebb 100644
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -168,19 +168,19 @@ to breeze.
.. code-block:: bash
- ./breeze tests tests/providers/http/hooks/test_http.py tests/core/test_core.py --db-reset -- --log-cli-level=DEBUG
+ breeze tests tests/providers/http/hooks/test_http.py tests/core/test_core.py --db-reset --log-cli-level=DEBUG
You can run the whole test suite without adding the test target:
.. code-block:: bash
- ./breeze tests --db-reset
+ breeze tests --db-reset
You can also specify individual tests or a group of tests:
.. code-block:: bash
- ./breeze tests --db-reset tests/core/test_core.py::TestCore
+ breeze tests --db-reset tests/core/test_core.py::TestCore
Running Tests of a specified type from the Host
@@ -201,13 +201,13 @@ kinds of test types:
.. code-block:: bash
- ./breeze --test-type Core --db-reset tests
+ ./breeze-legacy --test-type Core --db-reset tests
Runs all provider tests:
.. code-block:: bash
- ./breeze --test-type Providers --db-reset tests
+ ./breeze-legacy --test-type Providers --db-reset tests
* Special kinds of tests - Integration, Quarantined, Postgres, MySQL, which are marked with pytest
marks and for those you need to select the type using test-type switch. If you want to run such tests
@@ -219,13 +219,13 @@ kinds of test types:
.. code-block:: bash
- ./breeze --test-type Quarantined tests tests/cli/commands/test_task_command.py --db-reset
+ ./breeze-legacy --test-type Quarantined tests tests/cli/commands/test_task_command.py --db-reset
Run all Quarantined tests:
.. code-block:: bash
- ./breeze --test-type Quarantined tests --db-reset
+ ./breeze-legacy --test-type Quarantined tests --db-reset
Helm Unit Tests
===============
@@ -273,7 +273,7 @@ To run tests using breeze run the following command
.. code-block:: bash
- ./breeze --test-type Helm tests
+ ./breeze-legacy --test-type Helm tests
Airflow Integration Tests
=========================
@@ -327,19 +327,19 @@ To start the ``mongo`` integration only, enter:
.. code-block:: bash
- ./breeze --integration mongo
+ breeze --integration mongo
To start ``mongo`` and ``cassandra`` integrations, enter:
.. code-block:: bash
- ./breeze --integration mongo --integration cassandra
+ breeze --integration mongo --integration cassandra
To start all integrations, enter:
.. code-block:: bash
- ./breeze --integration all
+ breeze --integration all
In the CI environment, integrations can be enabled by specifying the ``ENABLED_INTEGRATIONS`` variable
storing a space-separated list of integrations to start. Thanks to that, we can run integration and
@@ -559,7 +559,7 @@ need to run the following steps:
.. code-block:: bash
- ./breeze prepare-provider-packages [PACKAGE ...]
+ breeze prepare-provider-packages [PACKAGE ...]
If you run this command without packages, you will prepare all packages. However, You can specify
providers that you would like to build if you just want to build few provider packages.
@@ -570,7 +570,7 @@ before running, so you should run it before generating ``apache-airflow`` packag
.. code-block:: bash
- ./breeze prepare-airflow-packages
+ breeze prepare-airflow-package
This prepares airflow .whl package in the dist folder.
@@ -580,7 +580,7 @@ This installs airflow and enters
.. code-block:: bash
- ./breeze --use-airflow-version wheel --use-packages-from-dist --skip-mounting-local-sources
+ ./breeze-legacy --use-airflow-version wheel --use-packages-from-dist --skip-mounting-local-sources
@@ -602,7 +602,7 @@ For your testing, you manage Kind cluster with ``kind-cluster`` breeze command:
.. code-block:: bash
- ./breeze kind-cluster [ start | stop | recreate | status | deploy | test | shell | k9s ]
+ ./breeze-legacy kind-cluster [ start | stop | recreate | status | deploy | test | shell | k9s ]
The command allows you to start/stop/recreate/status Kind Kubernetes cluster, deploy Airflow via Helm
chart as well as interact with the cluster (via test and shell commands).
@@ -623,7 +623,7 @@ Deploying Airflow to the Kubernetes cluster created is also done via ``kind-clus
.. code-block:: bash
- ./breeze kind-cluster deploy
+ ./breeze-legacy kind-cluster deploy
The deploy command performs those steps:
@@ -640,10 +640,10 @@ You can also specify a different executor by providing the ``--executor`` option
.. code-block:: bash
- ./breeze kind-cluster deploy --executor CeleryExecutor
+ ./breeze-legacy kind-cluster deploy --executor CeleryExecutor
Note that when you specify the ``--executor`` option, it becomes the default. Therefore, every other operations
-on ``./breeze kind-cluster`` will default to using this executor. To change that, use the ``--executor`` option on the
+on ``./breeze-legacy kind-cluster`` will default to using this executor. To change that, use the ``--executor`` option on the
subsequent commands too.
@@ -667,15 +667,15 @@ Running Kubernetes tests via breeze:
.. code-block:: bash
- ./breeze kind-cluster test
- ./breeze kind-cluster test -- TEST TEST [TEST ...]
+ ./breeze-legacy kind-cluster test
+ ./breeze-legacy kind-cluster test -- TEST TEST [TEST ...]
Optionally add ``--executor``:
.. code-block:: bash
- ./breeze kind-cluster test --executor CeleryExecutor
- ./breeze kind-cluster test -- TEST TEST [TEST ...] --executor CeleryExecutor
+ ./breeze-legacy kind-cluster test --executor CeleryExecutor
+ ./breeze-legacy kind-cluster test -- TEST TEST [TEST ...] --executor CeleryExecutor
Entering shell with Kubernetes Cluster
--------------------------------------
@@ -701,13 +701,13 @@ You can enter the shell via those scripts
.. code-block:: bash
- ./breeze kind-cluster shell
+ ./breeze-legacy kind-cluster shell
Optionally add ``--executor``:
.. code-block:: bash
- ./breeze kind-cluster shell --executor CeleryExecutor
+ ./breeze-legacy kind-cluster shell --executor CeleryExecutor
K9s CLI - debug Kubernetes in style!
@@ -734,7 +734,7 @@ You can enter the k9s tool via breeze (after you deployed Airflow):
.. code-block:: bash
- ./breeze kind-cluster k9s
+ ./breeze-legacy kind-cluster k9s
You can exit k9s by pressing Ctrl-C.
@@ -747,7 +747,7 @@ The typical session for tests with Kubernetes looks like follows:
.. code-block:: bash
- ./breeze kind-cluster start
+ ./breeze-legacy kind-cluster start
Starts Kind Kubernetes cluster
@@ -787,7 +787,7 @@ The typical session for tests with Kubernetes looks like follows:
.. code-block:: bash
- ./breeze kind-cluster status
+ ./breeze-legacy kind-cluster status
Checks status of Kind Kubernetes cluster
@@ -807,7 +807,7 @@ The typical session for tests with Kubernetes looks like follows:
.. code-block:: bash
- ./breeze kind-cluster deploy
+ ./breeze-legacy kind-cluster deploy
4. Run Kubernetes tests
@@ -821,7 +821,7 @@ The virtualenv required will be created automatically when the scripts are run.
.. code-block:: bash
- ./breeze kind-cluster test
+ ./breeze-legacy kind-cluster test
4b) You can enter an interactive shell to run tests one-by-one
@@ -830,7 +830,7 @@ This prepares and enters the virtualenv in ``.build/.kubernetes_venv_``. For
+can be done by running ``./breeze-legacy prepare-provider-packages ``. For
example, the below command will build google, postgres and mysql wheel packages:
.. code-block:: bash
- ./breeze prepare-provider-packages -- google postgres mysql
+ breeze prepare-provider-packages google postgres mysql
Those packages will be prepared in ./dist folder. This folder is mapped to /dist folder
when you enter Breeze, so it is easy to automate installing those packages for testing.
@@ -1099,14 +1100,16 @@ Here is the typical session that you need to do to run system tests:
.. code-block:: bash
- ./breeze --python 3.6 --db-reset --forward-credentials restart
+ breeze stop
+ breeze --python 3.7 --db-reset --forward-credentials
This will:
-* restarts the whole environment (i.e. recreates metadata database from the scratch)
-* run Breeze with python 3.6 version
-* reset the Airflow database
-* forward your local credentials to Breeze
+* stop the whole environment (i.e. recreates metadata database from the scratch)
+* run Breeze with:
+ * python 3.7 version
+ * resetting the Airflow database
+ * forward your local credentials to Breeze
3. Run the tests:
@@ -1115,7 +1118,6 @@ This will:
pytest -o faulthandler_timeout=2400 \
--system=google tests/providers/google/cloud/operators/test_compute_system.py
-
Iteration with System Tests if your resources are slow to create
----------------------------------------------------------------
@@ -1154,7 +1156,8 @@ Breeze session. They are usually expensive to run.
.. code-block:: bash
- ./breeze --python 3.6 --db-reset --forward-credentials restart
+ breeze stop
+ breeze --python 3.7 --db-reset --forward-credentials
2. Run create action in helper (to create slowly created resources):
diff --git a/UPDATING.md b/UPDATING.md
deleted file mode 100644
index c929eced3de71..0000000000000
--- a/UPDATING.md
+++ /dev/null
@@ -1,3643 +0,0 @@
-
-
-# Updating Airflow
-
-This file documents any backwards-incompatible changes in Airflow and
-assists users migrating to a new version.
-
-
-
-**Table of contents**
-
-- [Main](#main)
-- [Airflow 2.2.4](#airflow-224)
-- [Airflow 2.2.3](#airflow-223)
-- [Airflow 2.2.2](#airflow-222)
-- [Airflow 2.2.1](#airflow-221)
-- [Airflow 2.2.0](#airflow-220)
-- [Airflow 2.1.4](#airflow-214)
-- [Airflow 2.1.3](#airflow-213)
-- [Airflow 2.1.2](#airflow-212)
-- [Airflow 2.1.1](#airflow-211)
-- [Airflow 2.1.0](#airflow-210)
-- [Airflow 2.0.2](#airflow-202)
-- [Airflow 2.0.1](#airflow-201)
-- [Airflow 2.0.0](#airflow-200)
-- [Airflow 1.10.15](#airflow-11015)
-- [Airflow 1.10.14](#airflow-11014)
-- [Airflow 1.10.13](#airflow-11013)
-- [Airflow 1.10.12](#airflow-11012)
-- [Airflow 1.10.11](#airflow-11011)
-- [Airflow 1.10.10](#airflow-11010)
-- [Airflow 1.10.9](#airflow-1109)
-- [Airflow 1.10.8](#airflow-1108)
-- [Airflow 1.10.7](#airflow-1107)
-- [Airflow 1.10.6](#airflow-1106)
-- [Airflow 1.10.5](#airflow-1105)
-- [Airflow 1.10.4](#airflow-1104)
-- [Airflow 1.10.3](#airflow-1103)
-- [Airflow 1.10.2](#airflow-1102)
-- [Airflow 1.10.1](#airflow-1101)
-- [Airflow 1.10](#airflow-110)
-- [Airflow 1.9](#airflow-19)
-- [Airflow 1.8.1](#airflow-181)
-- [Airflow 1.8](#airflow-18)
-- [Airflow 1.7.1.2](#airflow-1712)
-
-
-
-## Main
-
-
-
-### Minimum kubernetes version bumped from 3.0.0 to 21.7.0
-
-No change in behavior is expected. This was necessary in order to take advantage of a [bugfix](https://github.com/kubernetes-client/python-base/commit/70b78cd8488068c014b6d762a0c8d358273865b4) concerning refreshing of Kubernetes API tokens with EKS, which enabled the removal of some [workaround code](https://github.com/apache/airflow/pull/20759).
-
-### Deprecation: `Connection.extra` must be JSON-encoded dict
-
-#### TLDR
-
-From Airflow 3.0, the `extra` field in airflow connections must be a JSON-encoded Python dict.
-
-#### What, why, and when?
-
-Airflow's Connection is used for storing credentials. For storage of information that does not
-fit into user / password / host / schema / port, we have the `extra` string field. Its intention
-was always to provide for storage of arbitrary key-value pairs, like `no_host_key_check` in the SSH
-hook, or `keyfile_dict` in GCP.
-
-But since the field is string, it's technically been permissible to store any string value. For example
-one could have stored the string value `'my-website.com'` and used this in the hook. But this is a very
-bad practice. One reason is intelligibility: when you look at the value for `extra`, you don't have any idea
-what its purpose is. Better would be to store `{"api_host": "my-website.com"}` which at least tells you
-*something* about the value. Another reason is extensibility: if you store the API host as a simple string
-value, what happens if you need to add more information, such as the API endpoint, or credentials? Then
-you would need to convert the string to a dict, and this would be a breaking change.
-
-For these reason, starting in Airflow 3.0 we will require that the `Connection.extra` field store
-a JSON-encoded Python dict.
-
-#### How will I be affected?
-
-For users of providers that are included in the Airflow codebase, you should not have to make any changes
-because in the Airflow codebase we should not allow hooks to misuse the `Connection.extra` field in this way.
-
-However, if you have any custom hooks that store something other than JSON dict, you will have to update it.
-If you do, you should see a warning any time that this connection is retrieved or instantiated (e.g. it should show up in
-task logs).
-
-To see if you have any connections that will need to be updated, you can run this command:
-
-```shell
-airflow connections export - 2>&1 >/dev/null | grep 'non-JSON'
-```
-
-This will catch any warnings about connections that are storing something other than JSON-encoded Python dict in the `extra` field.
-
-### Zip files in the DAGs folder can no longer have a `.py` extension
-
-It was previously possible to have any extension for zip files in the DAGs folder. Now `.py` files are going to be loaded as modules without checking whether it is a zip file, as it leads to less IO. If a `.py` file in the DAGs folder is a zip compressed file, parsing it will fail with an exception.
-
-### You have to use `postgresql://` instead of `postgres://` in `sql_alchemy_conn` for SQLAlchemy 1.4.0+
-
-When you use SQLAlchemy 1.4.0+, you need to use `postgresql://` as the scheme in the `sql_alchemy_conn`.
-In the previous versions of SQLAlchemy it was possible to use `postgres://`, but using it in
-SQLAlchemy 1.4.0+ results in:
-
-```
-> raise exc.NoSuchModuleError(
- "Can't load plugin: %s:%s" % (self.group, name)
- )
-E sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgres
-```
-
-If you cannot change the scheme of your URL immediately, Airflow continues to work with SQLAlchemy
-1.3 and you can downgrade SQLAlchemy, but we recommend updating the scheme.
-Details in the [SQLAlchemy Changelog](https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-3687655465c25a39b968b4f5f6e9170b).
-
-### Passing `execution_date` to `XCom.set()`, `XCom.clear()`, `XCom.get_one()`, and `XCom.get_many()` is deprecated
-
-Continuing the effort to bind TaskInstance to a DagRun, XCom entries are now also tied to a DagRun. Use the ``run_id`` argument to specify the DagRun instead.
-
-### Non-JSON-serializable params deprecated.
-
-It was previously possible to use dag or task param defaults that were not JSON-serializable.
-
-For example this worked previously:
-
-```python
-@dag.task(params={"a": {1, 2, 3}, "b": pendulum.now()})
-def datetime_param(value):
- print(value)
-
-
-datetime_param("{{ params.a }} | {{ params.b }}")
-```
-
-Note the use of `set` and `datetime` types, which are not JSON-serializable. This behavior is problematic because to override these values in a dag run conf, you must use JSON, which could make these params non-overridable. Another problem is that the support for param validation assumes JSON. Use of non-JSON-serializable params will be removed in Airflow 3.0 and until then, use of them will produce a warning at parse time.
-
-### Task log templates are now read from the metadatabase instead of `airflow.cfg`
-
-Previously, a task’s log is dynamically rendered from the `[core] log_filename_template` and `[elasticsearch] log_id_template` config values at runtime. This resulted in unfortunate characteristics, e.g. it is impractical to modify the config value after an Airflow instance is running for a while, since all existing task logs have be saved under the previous format and cannot be found with the new config value.
-
-A new `log_template` table is introduced to solve this problem. This table is synchronised with the aforementioned config values every time Airflow starts, and a new field `log_template_id` is added to every DAG run to point to the format used by tasks (`NULL` indicates the first ever entry for compatibility).
-
-### Default templates for log filenames and elasticsearch log_id changed
-
-In order to support Dynamic Task Mapping the default templates for per-task instance logging has changed. If your config contains the old default values they will be upgraded-in-place.
-
-If you are happy with the new config values you should *remove* the setting in `airflow.cfg` and let the default value be used. Old default values were:
-
-
-- `[core] log_filename_template`: `{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log`
-- `[elasticsearch] log_id_template`: `{dag_id}-{task_id}-{execution_date}-{try_number}`
-
-`[core] log_filename_template` now uses "hive partition style" of `dag_id=/run_id=` by default, which may cause problems on some older FAT filesystems. If this affects you then you will have to change the log template.
-
-If you have customized the templates you should ensure that they contain `{{ ti.map_index }}` if you want to use dynamically mapped tasks.
-
-### `airflow.models.base.Operator` is removed
-
-Previously, there was an empty class `airflow.models.base.Operator` for “type hinting”. This class was never really useful for anything (everything it did could be done better with `airflow.models.baseoperator.BaseOperator`), and has been removed. If you are relying on the class’s existence, use `BaseOperator` (for concrete operators), `airflow.models.abstractoperator.AbstractOperator` (the base class of both `BaseOperator` and the AIP-42 `MappedOperator`), or `airflow.models.operator.Operator` (a union type `BaseOperator | MappedOperator` for type annotation).
-
-### XCom now define `run_id` instead of `execution_date`
-
-As a continuation to the TaskInstance-DagRun relation change started in Airflow 2.2, the `execution_date` columns on XCom has been removed from the database, and replaced by an [association proxy](https://docs.sqlalchemy.org/en/13/orm/extensions/associationproxy.html) field at the ORM level. If you access Airflow’s metadatabase directly, you should rewrite the implementation to use the `run_id` column instead.
-
-Note that Airflow’s metadatabase definition on both the database and ORM levels are considered implementation detail without strict backward compatibility guarantees.
-
-### `auth_backends` replaces `auth_backend` configuration setting
-
-Previously, only one backend was used to authorize use of the REST API. In 2.3 this was changed to support multiple backends, separated by whitespace. Each will be tried in turn until a successful response is returned.
-
-This setting is also used for the deprecated experimental API, which only uses the first option even if multiple are given.
-
-### `auth_backends` includes session
-
-To allow the Airflow UI to use the API, the previous default authorization backend `airflow.api.auth.backend.deny_all` is changed to `airflow.api.auth.backend.session`, and this is automatically added to the list of API authorization backends if a non-default value is set.
-
-### BaseOperatorLink's `get_link` method changed to take a `ti_key` keyword argument
-
-In v2.2 we "deprecated" passing an execution date to XCom.get methods, but there was no other option for operator links as they were only passed an execution_date.
-
-Now in 2.3 as part of Dynamic Task Mapping (AIP-42) we will need to add map_index to the XCom row to support the "reduce" part of the API.
-
-In order to support that cleanly we have changed the interface for BaseOperatorLink to take an TaskInstanceKey as the `ti_key` keyword argument (as execution_date + task is no longer unique for mapped operators).
-
-The existing signature will be detected (by the absence of the `ti_key` argument) and continue to work.
-
-### `ReadyToRescheduleDep` now only runs when `reschedule` is *True*
-
-When a `ReadyToRescheduleDep` is run, it now checks whether the `reschedule` attribute on the operator, and always reports itself as *passed* unless it is set to *True*. If you use this dep class on your custom operator, you will need to add this attribute to the operator class. Built-in operator classes that use this dep class (including sensors and all subclasses) already have this attribute and are not affected.
-
-### The `deps` attribute on an operator class should be a class level attribute
-
-To support operator-mapping (AIP 42), the `deps` attribute on operator class must be a set at the class level. This means that if a custom operator implements this as an instance-level variable, it will not be able to be used for operator-mapping. This does not affect existing code, but we highly recommend you to restructure the operator's dep logic in order to support the new feature.
-
-### The `tree` default view setting has been renamed to `grid`
-
-If you set the `dag_default_view` config option or the `default_view` argument to `DAG()` to `tree` you will need to update your deployment. The old name will continue to work but will issue warnings.
-
-## Airflow 2.2.4
-
-### Smart sensors deprecated
-
-Smart sensors, an "early access" feature added in Airflow 2, are now deprecated and will be removed in Airflow 2.4.0. They have been superseded by Deferrable Operators, added in Airflow 2.2.0.
-
-See [Migrating to Deferrable Operators](https://airflow.apache.org/docs/apache-airflow/2.2.4/concepts/smart-sensors.html#migrating-to-deferrable-operators) for details on how to migrate.
-
-## Airflow 2.2.3
-
-No breaking changes.
-
-## Airflow 2.2.2
-
-No breaking changes.
-
-## Airflow 2.2.1
-
-### ``Param``'s default value for ``default`` removed
-
-``Param``, introduced in Airflow 2.2.0, accidentally set the default value to ``None``. This default has been removed. If you want ``None`` as your default, explicitly set it as such. For example:
-
-```python
-Param(None, type=["null", "string"])
-```
-
-Now if you resolve a ``Param`` without a default and don't pass a value, you will get an ``TypeError``. For Example:
-
-```python
-Param().resolve() # raises TypeError
-```
-
-### `max_queued_runs_per_dag` configuration has been removed
-
-The `max_queued_runs_per_dag` configuration option in `[core]` section has been removed. Previously, this controlled the number of queued dagrun
-the scheduler can create in a dag. Now, the maximum number is controlled internally by the DAG's `max_active_runs`
-
-## Airflow 2.2.0
-
-Note: Upgrading the database to `2.2.0` or later can take some time to complete, particularly if you have a large `task_instance` table.
-
-### `worker_log_server_port` configuration has been moved to the ``logging`` section.
-
-The `worker_log_server_port` configuration option has been moved from `[celery]` section to `[logging]` section to allow for re-use between different executors.
-
-### `pandas` is now an optional dependency
-
-Previously `pandas` was a core requirement so when you run `pip install apache-airflow` it looked for `pandas`
-library and installed it if it does not exist.
-
-If you want to install `pandas` compatible with Airflow, you can use `[pandas]` extra while
-installing Airflow, example for Python 3.8 and Airflow 2.1.2:
-
-```shell
-pip install -U "apache-airflow[pandas]==2.1.2" \
- --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-3.8.txt"
-```
-
-### `none_failed_or_skipped` trigger rule has been deprecated
-
-`TriggerRule.NONE_FAILED_OR_SKIPPED` is replaced by `TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS`.
-This is only name change, no functionality changes made.
-This change is backward compatible however `TriggerRule.NONE_FAILED_OR_SKIPPED` will be removed in next major release.
-
-### Dummy trigger rule has been deprecated
-
-`TriggerRule.DUMMY` is replaced by `TriggerRule.ALWAYS`.
-This is only name change, no functionality changes made.
-This change is backward compatible however `TriggerRule.DUMMY` will be removed in next major release.
-
-### DAG concurrency settings have been renamed
-
-`[core] dag_concurrency` setting in `airflow.cfg` has been renamed to `[core] max_active_tasks_per_dag`
-for better understanding.
-
-It is the maximum number of task instances allowed to run concurrently in each DAG. To calculate
-the number of tasks that is running concurrently for a DAG, add up the number of running
-tasks for all DAG runs of the DAG.
-
-This is configurable at the DAG level with ``max_active_tasks`` and a default can be set in `airflow.cfg` as
-`[core] max_active_tasks_per_dag`.
-
-**Before**:
-
-```ini
-[core]
-dag_concurrency = 16
-```
-
-**Now**:
-
-```ini
-[core]
-max_active_tasks_per_dag = 16
-```
-
-Similarly, `DAG.concurrency` has been renamed to `DAG.max_active_tasks`.
-
-**Before**:
-
-```python
-dag = DAG(
- dag_id="example_dag",
- start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
- catchup=False,
- concurrency=3,
-)
-```
-
-**Now**:
-
-```python
-dag = DAG(
- dag_id="example_dag",
- start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
- catchup=False,
- max_active_tasks=3,
-)
-```
-
-If you are using DAGs Details API endpoint, use `max_active_tasks` instead of `concurrency`.
-
-### Task concurrency parameter has been renamed
-
-`BaseOperator.task_concurrency` has been deprecated and renamed to `max_active_tis_per_dag` for
-better understanding.
-
-This parameter controls the number of concurrent running task instances across ``dag_runs``
-per task.
-
-**Before**:
-
-```python
-with DAG(dag_id="task_concurrency_example"):
- BashOperator(task_id="t1", task_concurrency=2, bash_command="echo Hi")
-```
-
-**After**:
-
-```python
-with DAG(dag_id="task_concurrency_example"):
- BashOperator(task_id="t1", max_active_tis_per_dag=2, bash_command="echo Hi")
-```
-
-### `processor_poll_interval` config have been renamed to `scheduler_idle_sleep_time`
-
-`[scheduler] processor_poll_interval` setting in `airflow.cfg` has been renamed to `[scheduler] scheduler_idle_sleep_time`
-for better understanding.
-
-It controls the 'time to sleep' at the end of the Scheduler loop if nothing was scheduled inside `SchedulerJob`.
-
-**Before**:
-
-```ini
-[scheduler]
-processor_poll_interval = 16
-```
-
-**Now**:
-
-```ini
-[scheduler]
-scheduler_idle_sleep_time = 16
-```
-
-### Marking success/failed automatically clears failed downstream tasks
-
-When marking a task success/failed in Graph View, its downstream tasks that are in failed/upstream_failed state are automatically cleared.
-
-### `[core] store_dag_code` has been removed
-
-While DAG Serialization is a strict requirements since Airflow 2, we allowed users to control
-where the Webserver looked for when showing the **Code View**.
-
-If `[core] store_dag_code` was set to `True`, the Scheduler stored the code in the DAG file in the
-DB (in `dag_code` table) as a plain string, and the webserver just read it from the same table.
-If the value was set to `False`, the webserver read it from the DAG file.
-
-While this setting made sense for Airflow < 2, it caused some confusion to some users where they thought
-this setting controlled DAG Serialization.
-
-From Airflow 2.2, Airflow will only look for DB when a user clicks on **Code View** for a DAG.
-
-### Clearing a running task sets its state to `RESTARTING`
-
-Previously, clearing a running task sets its state to `SHUTDOWN`. The task gets killed and goes into `FAILED` state. After [#16681](https://github.com/apache/airflow/pull/16681), clearing a running task sets its state to `RESTARTING`. The task is eligible for retry without going into `FAILED` state.
-
-### Remove `TaskInstance.log_filepath` attribute
-
-This method returned incorrect values for a long time, because it did not take into account the different
-logger configuration and task retries. We have also started supporting more advanced tools that don't use
-files, so it is impossible to determine the correct file path in every case e.g. Stackdriver doesn't use files
-but identifies logs based on labels. For this reason, we decided to delete this attribute.
-
-If you need to read logs, you can use `airflow.utils.log.log_reader.TaskLogReader` class, which does not have
-the above restrictions.
-
-### If a sensor times out, it will not retry
-
-Previously, a sensor is retried when it times out until the number of ``retries`` are exhausted. So the effective timeout of a sensor is ``timeout * (retries + 1)``. This behaviour is now changed. A sensor will immediately fail without retrying if ``timeout`` is reached. If it's desirable to let the sensor continue running for longer time, set a larger ``timeout`` instead.
-
-### Default Task Pools Slots can be set using ``[core] default_pool_task_slot_count``
-
-By default tasks are running in `default_pool`. `default_pool` is initialized with `128` slots and user can change the
-number of slots through UI/CLI/API for an existing deployment.
-
-For new deployments, you can use `default_pool_task_slot_count` setting in `[core]` section. This setting would
-not have any effect in an existing deployment where the ``default_pool`` already exists.
-
-Previously this was controlled by `non_pooled_task_slot_count` in `[core]` section, which was not documented.
-
-### Webserver DAG refresh buttons removed
-
-Now that the DAG parser syncs DAG permissions there is no longer a need for manually refreshing DAGs. As such, the buttons to refresh a DAG have been removed from the UI.
-
-In addition, the `/refresh` and `/refresh_all` webserver endpoints have also been removed.
-
-### TaskInstances now *require* a DagRun
-
-Under normal operation every TaskInstance row in the database would have DagRun row too, but it was possible to manually delete the DagRun and Airflow would still schedule the TaskInstances.
-
-In Airflow 2.2 we have changed this and now there is a database-level foreign key constraint ensuring that every TaskInstance has a DagRun row.
-
-Before updating to this 2.2 release you will have to manually resolve any inconsistencies (add back DagRun rows, or delete TaskInstances) if you have any "dangling" TaskInstance" rows.
-
-As part of this change the `clean_tis_without_dagrun_interval` config option under `[scheduler]` section has been removed and has no effect.
-
-### TaskInstance and TaskReschedule now define `run_id` instead of `execution_date`
-
-As a part of the TaskInstance-DagRun relation change, the `execution_date` columns on TaskInstance and TaskReschedule have been removed from the database, and replaced by [association proxy](https://docs.sqlalchemy.org/en/13/orm/extensions/associationproxy.html) fields at the ORM level. If you access Airflow’s metadatabase directly, you should rewrite the implementation to use the `run_id` columns instead.
-
-Note that Airflow’s metadatabase definition on both the database and ORM levels are considered implementation detail without strict backward compatibility guarantees.
-
-### DaskExecutor - Dask Worker Resources and queues
-
-If dask workers are not started with complementary resources to match the specified queues, it will now result in an `AirflowException`, whereas before it would have just ignored the `queue` argument.
-
-## Airflow 2.1.4
-
-No breaking changes.
-
-## Airflow 2.1.3
-
-No breaking changes.
-
-## Airflow 2.1.2
-
-No breaking changes.
-
-## Airflow 2.1.1
-
-### `activate_dag_runs` argument of the function `clear_task_instances` is replaced with `dag_run_state`
-
-To achieve the previous default behaviour of `clear_task_instances` with `activate_dag_runs=True`, no change is needed. To achieve the previous behaviour of `activate_dag_runs=False`, pass `dag_run_state=False` instead. (The previous parameter is still accepted, but is deprecated)
-
-### `dag.set_dag_runs_state` is deprecated
-
-The method `set_dag_runs_state` is no longer needed after a bug fix in PR: [#15382](https://github.com/apache/airflow/pull/15382). This method is now deprecated and will be removed in a future version.
-
-## Airflow 2.1.0
-
-### New "deprecated_api" extra
-
-We have a new '[deprecated_api]' extra that should be used when installing airflow when the deprecated API
-is going to be used. This is now an optional feature of Airflow now because it pulls in `requests` which
-(as of 14 May 2021) pulls LGPL `chardet` dependency.
-
-### The `http` provider is not installed by default
-
-The `http` provider is now optional and not installed by default, until `chardet` becomes an optional
-dependency of `requests`.
-See [PR to replace chardet with charset-normalizer](https://github.com/psf/requests/pull/5797)
-
-### `@apply_default` decorator isn't longer necessary
-
-This decorator is now automatically added to all operators via the metaclass on BaseOperator
-
-### Change the configuration options for field masking
-
-We've improved masking for sensitive data in Web UI and logs. As part of it, the following configurations have been changed:
-
-* `hide_sensitive_variable_fields` option in `admin` section has been replaced by `hide_sensitive_var_conn_fields` section in `core` section,
-* `sensitive_variable_fields` option in `admin` section has been replaced by `sensitive_var_conn_names` section in `core` section.
-
-### Deprecated PodDefaults and add_xcom_sidecar in airflow.kubernetes.pod_generator
-
-We have moved PodDefaults from `airflow.kubernetes.pod_generator.PodDefaults` to
-`airflow.providers.cncf.kubernetes.utils.xcom_sidecar.PodDefaults` and moved add_xcom_sidecar
-from `airflow.kubernetes.pod_generator.PodGenerator.add_xcom_sidecar`to
-`airflow.providers.cncf.kubernetes.utils.xcom_sidecar.add_xcom_sidecar`.
-This change will allow us to modify the KubernetesPodOperator XCom functionality without requiring airflow upgrades.
-
-### Removed pod_launcher from core airflow
-
-Moved the pod launcher from `airflow.kubernetes.pod_launcher` to `airflow.providers.cncf.kubernetes.utils.pod_launcher`
-
-This will allow users to update the pod_launcher for the KubernetesPodOperator without requiring an airflow upgrade
-
-### Default `[webserver] worker_refresh_interval` is changed to `6000` seconds
-
-The default value for `[webserver] worker_refresh_interval` was `30` seconds for
-Airflow <=2.0.1. However, since Airflow 2.0 DAG Serialization is a hard requirement
-and the Webserver used the serialized DAGs, there is no need to kill an existing
-worker and create a new one as frequently as `30` seconds.
-
-This setting can be raised to an even higher value, currently it is
-set to `6000` seconds (100 minutes) to
-serve as a DagBag cache burst time.
-
-### `default_queue` configuration has been moved to the `operators` section.
-
-The `default_queue` configuration option has been moved from `[celery]` section to `[operators]` section to allow for re-use between different executors.
-
-## Airflow 2.0.2
-
-### Default `[kubernetes] enable_tcp_keepalive` is changed to `True`
-
-This allows Airflow to work more reliably with some environments (like Azure) by default.
-
-### `sync-perm` CLI no longer syncs DAG specific permissions by default
-
-The `sync-perm` CLI command will no longer sync DAG specific permissions by default as they are now being handled during
-DAG parsing. If you need or want the old behavior, you can pass `--include-dags` to have `sync-perm` also sync DAG
-specific permissions.
-
-## Airflow 2.0.1
-
-### Permission to view Airflow Configurations has been removed from `User` and `Viewer` role
-
-Previously, Users with `User` or `Viewer` role were able to get/view configurations using
-the REST API or in the Webserver. From Airflow 2.0.1, only users with `Admin` or `Op` role would be able
-to get/view Configurations.
-
-To allow users with other roles to view configuration, add `can read on Configurations` permissions to that role.
-
-Note that if `[webserver] expose_config` is set to `False`, the API will throw a `403` response even if
-the user has role with `can read on Configurations` permission.
-
-### Default `[celery] worker_concurrency` is changed to `16`
-
-The default value for `[celery] worker_concurrency` was `16` for Airflow <2.0.0.
-However, it was unintentionally changed to `8` in 2.0.0.
-
-From Airflow 2.0.1, we revert to the old default of `16`.
-
-### Default `[scheduler] min_file_process_interval` is changed to `30`
-
-The default value for `[scheduler] min_file_process_interval` was `0`,
-due to which the CPU Usage mostly stayed around 100% as the DAG files are parsed
-constantly.
-
-From Airflow 2.0.0, the scheduling decisions have been moved from
-DagFileProcessor to Scheduler, so we can keep the default a bit higher: `30`.
-
-## Airflow 2.0.0
-
-The 2.0 release of the Airflow is a significant upgrade, and includes substantial major changes,
-and some of them may be breaking. Existing code written for earlier versions of this project will may require updates
-to use this version. Sometimes necessary configuration changes are also required.
-This document describes the changes that have been made, and what you need to do to update your usage.
-
-If you experience issues or have questions, please file [an issue](https://github.com/apache/airflow/issues/new/choose).
-
-### Major changes
-
-This section describes the major changes that have been made in this release.
-
-### The experimental REST API is disabled by default
-
-The experimental REST API is disabled by default. To restore these APIs while migrating to
-the stable REST API, set `enable_experimental_api` option in `[api]` section to `True`.
-
-Please note that the experimental REST API do not have access control.
-The authenticated user has full access.
-
-### SparkJDBCHook default connection
-
-For SparkJDBCHook default connection was `spark-default`, and for SparkSubmitHook it was
-`spark_default`. Both hooks now use the `spark_default` which is a common pattern for the connection
-names used across all providers.
-
-### Changes to output argument in commands
-
-From Airflow 2.0, We are replacing [tabulate](https://pypi.org/project/tabulate/) with [rich](https://github.com/willmcgugan/rich) to render commands output. Due to this change, the `--output` argument
-will no longer accept formats of tabulate tables. Instead, it now accepts:
-
-- `table` - will render the output in predefined table
-- `json` - will render the output as a json
-- `yaml` - will render the output as yaml
-
-By doing this we increased consistency and gave users possibility to manipulate the
-output programmatically (when using json or yaml).
-
-Affected commands:
-
-- `airflow dags list`
-- `airflow dags report`
-- `airflow dags list-runs`
-- `airflow dags list-jobs`
-- `airflow connections list`
-- `airflow connections get`
-- `airflow pools list`
-- `airflow pools get`
-- `airflow pools set`
-- `airflow pools delete`
-- `airflow pools import`
-- `airflow pools export`
-- `airflow role list`
-- `airflow providers list`
-- `airflow providers get`
-- `airflow providers hooks`
-- `airflow tasks states-for-dag-run`
-- `airflow users list`
-- `airflow variables list`
-
-### Azure Wasb Hook does not work together with Snowflake hook
-
-The WasbHook in Apache Airflow use a legacy version of Azure library. While the conflict is not
-significant for most of the Azure hooks, it is a problem for Wasb Hook because the `blob` folders
-for both libraries overlap. Installing both Snowflake and Azure extra will result in non-importable
-WasbHook.
-
-### Rename `all` to `devel_all` extra
-
-The `all` extras were reduced to include only user-facing dependencies. This means
-that this extra does not contain development dependencies. If you were relying on
-`all` extra then you should use now `devel_all` or figure out if you need development
-extras at all.
-
-### Context variables `prev_execution_date_success` and `prev_execution_date_success` are now `pendulum.DateTime`
-
-### Rename policy to task_policy
-
-Because Airflow introduced DAG level policy (`dag_policy`) we decided to rename existing `policy`
-function to `task_policy` to make the distinction more profound and avoid any confusion.
-
-Users using cluster policy need to rename their `policy` functions in `airflow_local_settings.py`
-to `task_policy`.
-
-### Default value for `[celery] operation_timeout` has changed to `1.0`
-
-From Airflow 2, by default Airflow will retry 3 times to publish task to Celery broker. This is controlled by
-`[celery] task_publish_max_retries`. Because of this we can now have a lower Operation timeout that raises
-`AirflowTaskTimeout`. This generally occurs during network blips or intermittent DNS issues.
-
-### Adding Operators and Sensors via plugins is no longer supported
-
-Operators and Sensors should no longer be registered or imported via Airflow's plugin mechanism -- these types of classes are just treated as plain python classes by Airflow, so there is no need to register them with Airflow.
-
-If you previously had a `plugins/my_plugin.py` and you used it like this in a DAG:
-
-```
-from airflow.operators.my_plugin import MyOperator
-```
-
-You should instead import it as:
-
-```
-from my_plugin import MyOperator
-```
-
-The name under `airflow.operators.` was the plugin name, where as in the second example it is the python module name where the operator is defined.
-
-See https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html for more info.
-
-### Importing Hooks via plugins is no longer supported
-
-Importing hooks added in plugins via `airflow.hooks.` is no longer supported, and hooks should just be imported as regular python modules.
-
-```
-from airflow.hooks.my_plugin import MyHook
-```
-
-You should instead import it as:
-
-```
-from my_plugin import MyHook
-```
-
-It is still possible (but not required) to "register" hooks in plugins. This is to allow future support for dynamically populating the Connections form in the UI.
-
-See https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html for more info.
-
-### The default value for `[core] enable_xcom_pickling` has been changed to `False`
-
-The pickle type for XCom messages has been replaced to JSON by default to prevent RCE attacks.
-Note that JSON serialization is stricter than pickling, so for example if you want to pass
-raw bytes through XCom you must encode them using an encoding like ``base64``.
-If you understand the risk and still want to use [pickling](https://docs.python.org/3/library/pickle.html),
-set `enable_xcom_pickling = True` in your Airflow config's `core` section.
-
-### Airflowignore of base path
-
-There was a bug fixed in https://github.com/apache/airflow/pull/11993 that the "airflowignore" checked
-the base path of the dag folder for forbidden dags, not only the relative part. This had the effect
-that if the base path contained the excluded word the whole dag folder could have been excluded. For
-example if the airflowignore file contained x, and the dags folder was '/var/x/dags', then all dags in
-the folder would be excluded. The fix only matches the relative path only now which means that if you
-previously used full path as ignored, you should change it to relative one. For example if your dag
-folder was '/var/dags/' and your airflowignore contained '/var/dag/excluded/', you should change it
-to 'excluded/'.
-
-### `ExternalTaskSensor` provides all task context variables to `execution_date_fn` as keyword arguments
-
-The old syntax of passing `context` as a dictionary will continue to work with the caveat that the argument must be named `context`. The following will break. To fix it, change `ctx` to `context`.
-
-```python
-def execution_date_fn(execution_date, ctx):
- ...
-```
-
-`execution_date_fn` can take in any number of keyword arguments available in the task context dictionary. The following forms of `execution_date_fn` are all supported:
-
-```python
-def execution_date_fn(dt):
- ...
-
-
-def execution_date_fn(execution_date):
- ...
-
-
-def execution_date_fn(execution_date, ds_nodash):
- ...
-
-
-def execution_date_fn(execution_date, ds_nodash, dag):
- ...
-```
-
-### The default value for `[webserver] cookie_samesite` has been changed to `Lax`
-
-As [recommended](https://flask.palletsprojects.com/en/1.1.x/config/#SESSION_COOKIE_SAMESITE) by Flask, the
-`[webserver] cookie_samesite` has been changed to `Lax` from `''` (empty string) .
-
-#### Changes to import paths
-
-Formerly the core code was maintained by the original creators - Airbnb. The code that was in the contrib
-package was supported by the community. The project was passed to the Apache community and currently the
-entire code is maintained by the community, so now the division has no justification, and it is only due
-to historical reasons. In Airflow 2.0, we want to organize packages and move integrations
-with third party services to the ``airflow.providers`` package.
-
-All changes made are backward compatible, but if you use the old import paths you will
-see a deprecation warning. The old import paths can be abandoned in the future.
-
-According to [AIP-21](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths)
-`_operator` suffix has been removed from operators. A deprecation warning has also been raised for paths
-importing with the suffix.
-
-
-The following table shows changes in import paths.
-
-
-| Old path | New path |
-|-------------------------------------|----------------------------|
-| airflow.hooks.base_hook.BaseHook | airflow.hooks.base.BaseHook |
-| airflow.hooks.dbapi_hook.DbApiHook | airflow.hooks.dbapi.DbApiHook |
-| airflow.operators.dummy_operator.DummyOperator | airflow.operators.dummy.DummyOperator |
-| airflow.operators.dagrun_operator.TriggerDagRunOperator | airflow.operators.trigger_dagrun.TriggerDagRunOperator |
-| airflow.operators.branch_operator.BaseBranchOperator | airflow.operators.branch.BaseBranchOperator |
-| airflow.operators.subdag_operator.SubDagOperator | airflow.operators.subdag.SubDagOperator |
-| airflow.sensors.base_sensor_operator.BaseSensorOperator | airflow.sensors.base.BaseSensorOperator |
-| airflow.sensors.date_time_sensor.DateTimeSensor | airflow.sensors.date_time.DateTimeSensor |
-| airflow.sensors.external_task_sensor.ExternalTaskMarker | airflow.sensors.external_task.ExternalTaskMarker |
-| airflow.sensors.external_task_sensor.ExternalTaskSensor | airflow.sensors.external_task.ExternalTaskSensor |
-| airflow.sensors.sql_sensor.SqlSensor | airflow.sensors.sql.SqlSensor |
-| airflow.sensors.time_delta_sensor.TimeDeltaSensor | airflow.sensors.time_delta.TimeDeltaSensor |
-| airflow.contrib.sensors.weekday_sensor.DayOfWeekSensor | airflow.sensors.weekday.DayOfWeekSensor |
-
-
-### Database schema changes
-
-In order to migrate the database, you should use the command `airflow db upgrade`, but in
-some cases manual steps are required.
-
-#### Unique conn_id in connection table
-
-Previously, Airflow allowed users to add more than one connection with the same `conn_id` and on access it would choose one connection randomly. This acted as a basic load balancing and fault tolerance technique, when used in conjunction with retries.
-
-This behavior caused some confusion for users, and there was no clear evidence if it actually worked well or not.
-
-Now the `conn_id` will be unique. If you already have duplicates in your metadata database, you will have to manage those duplicate connections before upgrading the database.
-
-#### Not-nullable conn_type column in connection table
-
-The `conn_type` column in the `connection` table must contain content. Previously, this rule was enforced
-by application logic, but was not enforced by the database schema.
-
-If you made any modifications to the table directly, make sure you don't have
-null in the `conn_type` column.
-
-### Configuration changes
-
-This release contains many changes that require a change in the configuration of this application or
-other application that integrate with it.
-
-This section describes the changes that have been made, and what you need to do to.
-
-#### airflow.contrib.utils.log has been moved
-
-Formerly the core code was maintained by the original creators - Airbnb. The code that was in the contrib
-package was supported by the community. The project was passed to the Apache community and currently the
-entire code is maintained by the community, so now the division has no justification, and it is only due
-to historical reasons. In Airflow 2.0, we want to organize packages and move integrations
-with third party services to the ``airflow.providers`` package.
-
-To clean up, the following packages were moved:
-| Old package | New package |
-|-|-|
-| ``airflow.contrib.utils.log`` | ``airflow.utils.log`` |
-| ``airflow.utils.log.gcs_task_handler`` | ``airflow.providers.google.cloud.log.gcs_task_handler`` |
-| ``airflow.utils.log.wasb_task_handler`` | ``airflow.providers.microsoft.azure.log.wasb_task_handler`` |
-| ``airflow.utils.log.stackdriver_task_handler`` | ``airflow.providers.google.cloud.log.stackdriver_task_handler`` |
-| ``airflow.utils.log.s3_task_handler`` | ``airflow.providers.amazon.aws.log.s3_task_handler`` |
-| ``airflow.utils.log.es_task_handler`` | ``airflow.providers.elasticsearch.log.es_task_handler`` |
-| ``airflow.utils.log.cloudwatch_task_handler`` | ``airflow.providers.amazon.aws.log.cloudwatch_task_handler`` |
-
-You should update the import paths if you are setting log configurations with the ``logging_config_class`` option.
-The old import paths still works but can be abandoned.
-
-#### SendGrid emailer has been moved
-
-Formerly the core code was maintained by the original creators - Airbnb. The code that was in the contrib
-package was supported by the community. The project was passed to the Apache community and currently the
-entire code is maintained by the community, so now the division has no justification, and it is only due
-to historical reasons.
-
-To clean up, the `send_mail` function from the `airflow.contrib.utils.sendgrid` module has been moved.
-
-If your configuration file looks like this:
-
-```ini
-[email]
-email_backend = airflow.contrib.utils.sendgrid.send_email
-```
-
-It should look like this now:
-
-```ini
-[email]
-email_backend = airflow.providers.sendgrid.utils.emailer.send_email
-```
-
-The old configuration still works but can be abandoned.
-
-#### Unify `hostname_callable` option in `core` section
-
-The previous option used a colon(`:`) to split the module from function. Now the dot(`.`) is used.
-
-The change aims to unify the format of all options that refer to objects in the `airflow.cfg` file.
-
-
-#### Custom executors is loaded using full import path
-
-In previous versions of Airflow it was possible to use plugins to load custom executors. It is still
-possible, but the configuration has changed. Now you don't have to create a plugin to configure a
-custom executor, but you need to provide the full path to the module in the `executor` option
-in the `core` section. The purpose of this change is to simplify the plugin mechanism and make
-it easier to configure executor.
-
-If your module was in the path `my_acme_company.executors.MyCustomExecutor` and the plugin was
-called `my_plugin` then your configuration looks like this
-
-```ini
-[core]
-executor = my_plugin.MyCustomExecutor
-```
-
-And now it should look like this:
-
-```ini
-[core]
-executor = my_acme_company.executors.MyCustomExecutor
-```
-
-The old configuration is still works but can be abandoned at any time.
-
-#### Use `CustomSQLAInterface` instead of `SQLAInterface` for custom data models.
-
-From Airflow 2.0, if you want to define your own Flask App Builder data models you need to use CustomSQLAInterface
-instead of SQLAInterface.
-
-For Non-RBAC replace:
-
-```python
-from flask_appbuilder.models.sqla.interface import SQLAInterface
-
-datamodel = SQLAInterface(your_data_model)
-```
-
-with RBAC (in 1.10):
-
-```python
-from airflow.www_rbac.utils import CustomSQLAInterface
-
-datamodel = CustomSQLAInterface(your_data_model)
-```
-
-and in 2.0:
-
-```python
-from airflow.www.utils import CustomSQLAInterface
-
-datamodel = CustomSQLAInterface(your_data_model)
-```
-
-#### Drop plugin support for stat_name_handler
-
-In previous version, you could use plugins mechanism to configure ``stat_name_handler``. You should now use the `stat_name_handler`
-option in `[scheduler]` section to achieve the same effect.
-
-If your plugin looked like this and was available through the `test_plugin` path:
-
-```python
-def my_stat_name_handler(stat):
- return stat
-
-
-class AirflowTestPlugin(AirflowPlugin):
- name = "test_plugin"
- stat_name_handler = my_stat_name_handler
-```
-
-then your `airflow.cfg` file should look like this:
-
-```ini
-[scheduler]
-stat_name_handler=test_plugin.my_stat_name_handler
-```
-
-This change is intended to simplify the statsd configuration.
-
-#### Logging configuration has been moved to new section
-
-The following configurations have been moved from `[core]` to the new `[logging]` section.
-
-* `base_log_folder`
-* `remote_logging`
-* `remote_log_conn_id`
-* `remote_base_log_folder`
-* `encrypt_s3_logs`
-* `logging_level`
-* `fab_logging_level`
-* `logging_config_class`
-* `colored_console_log`
-* `colored_log_format`
-* `colored_formatter_class`
-* `log_format`
-* `simple_log_format`
-* `task_log_prefix_template`
-* `log_filename_template`
-* `log_processor_filename_template`
-* `dag_processor_manager_log_location`
-* `task_log_reader`
-
-#### Metrics configuration has been moved to new section
-
-The following configurations have been moved from `[scheduler]` to the new `[metrics]` section.
-
-- `statsd_on`
-- `statsd_host`
-- `statsd_port`
-- `statsd_prefix`
-- `statsd_allow_list`
-- `stat_name_handler`
-- `statsd_datadog_enabled`
-- `statsd_datadog_tags`
-- `statsd_custom_client_path`
-
-#### Changes to Elasticsearch logging provider
-
-When JSON output to stdout is enabled, log lines will now contain the `log_id` & `offset` fields, this should make reading task logs from elasticsearch on the webserver work out of the box. Example configuration:
-
-```ini
-[logging]
-remote_logging = True
-[elasticsearch]
-host = http://es-host:9200
-write_stdout = True
-json_format = True
-```
-
-Note that the webserver expects the log line data itself to be present in the `message` field of the document.
-
-#### Remove gcp_service_account_keys option in airflow.cfg file
-
-This option has been removed because it is no longer supported by the Google Kubernetes Engine. The new
-recommended service account keys for the Google Cloud management method is
-[Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity).
-
-#### Fernet is enabled by default
-
-The fernet mechanism is enabled by default to increase the security of the default installation. In order to
-restore the previous behavior, the user must consciously set an empty key in the ``fernet_key`` option of
-section ``[core]`` in the ``airflow.cfg`` file.
-
-At the same time, this means that the `apache-airflow[crypto]` extra-packages are always installed.
-However, this requires that your operating system has ``libffi-dev`` installed.
-
-#### Changes to propagating Kubernetes worker annotations
-
-`kubernetes_annotations` configuration section has been removed.
-A new key `worker_annotations` has been added to existing `kubernetes` section instead.
-That is to remove restriction on the character set for k8s annotation keys.
-All key/value pairs from `kubernetes_annotations` should now go to `worker_annotations` as a json. I.e. instead of e.g.
-
-```
-[kubernetes_annotations]
-annotation_key = annotation_value
-annotation_key2 = annotation_value2
-```
-
-it should be rewritten to
-
-```
-[kubernetes]
-worker_annotations = { "annotation_key" : "annotation_value", "annotation_key2" : "annotation_value2" }
-```
-
-#### Remove run_duration
-
-We should not use the `run_duration` option anymore. This used to be for restarting the scheduler from time to time, but right now the scheduler is getting more stable and therefore using this setting is considered bad and might cause an inconsistent state.
-
-#### Rename pool statsd metrics
-
-Used slot has been renamed to running slot to make the name self-explanatory
-and the code more maintainable.
-
-This means `pool.used_slots.` metric has been renamed to
-`pool.running_slots.`. The `Used Slots` column in Pools Web UI view
-has also been changed to `Running Slots`.
-
-#### Removal of Mesos Executor
-
-The Mesos Executor is removed from the code base as it was not widely used and not maintained. [Mailing List Discussion on deleting it](https://lists.apache.org/thread.html/daa9500026b820c6aaadeffd66166eae558282778091ebbc68819fb7@%3Cdev.airflow.apache.org%3E).
-
-#### Change dag loading duration metric name
-
-Change DAG file loading duration metric from
-`dag.loading-duration.` to `dag.loading-duration.`. This is to
-better handle the case when a DAG file has multiple DAGs.
-
-#### Sentry is disabled by default
-
-Sentry is disabled by default. To enable these integrations, you need set ``sentry_on`` option
-in ``[sentry]`` section to ``"True"``.
-
-#### Simplified GCSTaskHandler configuration
-
-In previous versions, in order to configure the service account key file, you had to create a connection entry.
-In the current version, you can configure ``google_key_path`` option in ``[logging]`` section to set
-the key file path.
-
-Users using Application Default Credentials (ADC) need not take any action.
-
-The change aims to simplify the configuration of logging, to prevent corruption of
-the instance configuration by changing the value controlled by the user - connection entry. If you
-configure a backend secret, it also means the webserver doesn't need to connect to it. This
-simplifies setups with multiple GCP projects, because only one project will require the Secret Manager API
-to be enabled.
-
-### Changes to the core operators/hooks
-
-We strive to ensure that there are no changes that may affect the end user and your files, but this
-release may contain changes that will require changes to your DAG files.
-
-This section describes the changes that have been made, and what you need to do to update your DAG File,
-if you use core operators or any other.
-
-#### BaseSensorOperator now respects the trigger_rule of downstream tasks
-
-Previously, BaseSensorOperator with setting `soft_fail=True` skips itself
-and skips all its downstream tasks unconditionally, when it fails i.e the trigger_rule of downstream tasks is not
-respected.
-
-In the new behavior, the trigger_rule of downstream tasks is respected.
-User can preserve/achieve the original behaviour by setting the trigger_rule of each downstream task to `all_success`.
-
-#### BaseOperator uses metaclass
-
-`BaseOperator` class uses a `BaseOperatorMeta` as a metaclass. This meta class is based on
-`abc.ABCMeta`. If your custom operator uses different metaclass then you will have to adjust it.
-
-#### Remove SQL support in BaseHook
-
-Remove ``get_records`` and ``get_pandas_df`` and ``run`` from BaseHook, which only apply for SQL-like hook,
-If want to use them, or your custom hook inherit them, please use ``airflow.hooks.dbapi.DbApiHook``
-
-#### Assigning task to a DAG using bitwise shift (bit-shift) operators are no longer supported
-
-Previously, you could assign a task to a DAG as follows:
-
-```python
-dag = DAG("my_dag")
-dummy = DummyOperator(task_id="dummy")
-
-dag >> dummy
-```
-
-This is no longer supported. Instead, we recommend using the DAG as context manager:
-
-```python
-with DAG("my_dag") as dag:
- dummy = DummyOperator(task_id="dummy")
-```
-
-#### Removed deprecated import mechanism
-
-The deprecated import mechanism has been removed so the import of modules becomes more consistent and explicit.
-
-For example: `from airflow.operators import BashOperator`
-becomes `from airflow.operators.bash_operator import BashOperator`
-
-#### Changes to sensor imports
-
-Sensors are now accessible via `airflow.sensors` and no longer via `airflow.operators.sensors`.
-
-For example: `from airflow.operators.sensors import BaseSensorOperator`
-becomes `from airflow.sensors.base import BaseSensorOperator`
-
-#### Skipped tasks can satisfy wait_for_downstream
-
-Previously, a task instance with `wait_for_downstream=True` will only run if the downstream task of
-the previous task instance is successful. Meanwhile, a task instance with `depends_on_past=True`
-will run if the previous task instance is either successful or skipped. These two flags are close siblings
-yet they have different behavior. This inconsistency in behavior made the API less intuitive to users.
-To maintain consistent behavior, both successful or skipped downstream task can now satisfy the
-`wait_for_downstream=True` flag.
-
-#### `airflow.utils.helpers.cross_downstream`
-
-#### `airflow.utils.helpers.chain`
-
-The `chain` and `cross_downstream` methods are now moved to airflow.models.baseoperator module from
-`airflow.utils.helpers` module.
-
-The `baseoperator` module seems to be a better choice to keep
-closely coupled methods together. Helpers module is supposed to contain standalone helper methods
-that can be imported by all classes.
-
-The `chain` method and `cross_downstream` method both use BaseOperator. If any other package imports
-any classes or functions from helpers module, then it automatically has an
-implicit dependency to BaseOperator. That can often lead to cyclic dependencies.
-
-More information in [AIRFLOW-6392](https://issues.apache.org/jira/browse/AIRFLOW-6392)
-
-In Airflow < 2.0 you imported those two methods like this:
-
-```python
-from airflow.utils.helpers import chain
-from airflow.utils.helpers import cross_downstream
-```
-
-In Airflow 2.0 it should be changed to:
-
-```python
-from airflow.models.baseoperator import chain
-from airflow.models.baseoperator import cross_downstream
-```
-
-#### `airflow.operators.python.BranchPythonOperator`
-
-`BranchPythonOperator` will now return a value equal to the `task_id` of the chosen branch,
-where previously it returned None. Since it inherits from BaseOperator it will do an
-`xcom_push` of this value if `do_xcom_push=True`. This is useful for downstream decision-making.
-
-#### `airflow.sensors.sql_sensor.SqlSensor`
-
-SQLSensor now consistent with python `bool()` function and the `allow_null` parameter has been removed.
-
-It will resolve after receiving any value that is casted to `True` with python `bool(value)`. That
-changes the previous response receiving `NULL` or `'0'`. Earlier `'0'` has been treated as success
-criteria. `NULL` has been treated depending on value of `allow_null`parameter. But all the previous
-behaviour is still achievable setting param `success` to `lambda x: x is None or str(x) not in ('0', '')`.
-
-#### `airflow.operators.trigger_dagrun.TriggerDagRunOperator`
-
-The TriggerDagRunOperator now takes a `conf` argument to which a dict can be provided as conf for the DagRun.
-As a result, the `python_callable` argument was removed. PR: https://github.com/apache/airflow/pull/6317.
-
-#### `airflow.operators.python.PythonOperator`
-
-`provide_context` argument on the PythonOperator was removed. The signature of the callable passed to the PythonOperator is now inferred and argument values are always automatically provided. There is no need to explicitly provide or not provide the context anymore. For example:
-
-```python
-def myfunc(execution_date):
- print(execution_date)
-
-
-python_operator = PythonOperator(task_id="mytask", python_callable=myfunc, dag=dag)
-```
-
-Notice you don't have to set provide_context=True, variables from the task context are now automatically detected and provided.
-
-All context variables can still be provided with a double-asterisk argument:
-
-```python
-def myfunc(**context):
- print(context) # all variables will be provided to context
-
-
-python_operator = PythonOperator(task_id="mytask", python_callable=myfunc)
-```
-
-The task context variable names are reserved names in the callable function, hence a clash with `op_args` and `op_kwargs` results in an exception:
-
-```python
-def myfunc(dag):
- # raises a ValueError because "dag" is a reserved name
- # valid signature example: myfunc(mydag)
- print("output")
-
-
-python_operator = PythonOperator(
- task_id="mytask",
- op_args=[1],
- python_callable=myfunc,
-)
-```
-
-The change is backwards compatible, setting `provide_context` will add the `provide_context` variable to the `kwargs` (but won't do anything).
-
-PR: [#5990](https://github.com/apache/airflow/pull/5990)
-
-#### `airflow.sensors.filesystem.FileSensor`
-
-FileSensor is now takes a glob pattern, not just a filename. If the filename you are looking for has `*`, `?`, or `[` in it then you should replace these with `[*]`, `[?]`, and `[[]`.
-
-#### `airflow.operators.subdag_operator.SubDagOperator`
-
-`SubDagOperator` is changed to use Airflow scheduler instead of backfill
-to schedule tasks in the subdag. User no longer need to specify the executor
-in `SubDagOperator`.
-
-
-#### `airflow.providers.google.cloud.operators.datastore.CloudDatastoreExportEntitiesOperator`
-
-#### `airflow.providers.google.cloud.operators.datastore.CloudDatastoreImportEntitiesOperator`
-
-#### `airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator`
-
-#### `airflow.providers.ssh.operators.ssh.SSHOperator`
-
-#### `airflow.providers.microsoft.winrm.operators.winrm.WinRMOperator`
-
-#### `airflow.operators.bash.BashOperator`
-
-#### `airflow.providers.docker.operators.docker.DockerOperator`
-
-#### `airflow.providers.http.operators.http.SimpleHttpOperator`
-
-The `do_xcom_push` flag (a switch to push the result of an operator to xcom or not) was appearing in different incarnations in different operators. It's function has been unified under a common name (`do_xcom_push`) on `BaseOperator`. This way it is also easy to globally disable pushing results to xcom.
-
-The following operators were affected:
-
-* DatastoreExportOperator (Backwards compatible)
-* DatastoreImportOperator (Backwards compatible)
-* KubernetesPodOperator (Not backwards compatible)
-* SSHOperator (Not backwards compatible)
-* WinRMOperator (Not backwards compatible)
-* BashOperator (Not backwards compatible)
-* DockerOperator (Not backwards compatible)
-* SimpleHttpOperator (Not backwards compatible)
-
-See [AIRFLOW-3249](https://jira.apache.org/jira/browse/AIRFLOW-3249) for details
-
-#### `airflow.operators.latest_only_operator.LatestOnlyOperator`
-
-In previous versions, the `LatestOnlyOperator` forcefully skipped all (direct and indirect) downstream tasks on its own. From this version on the operator will **only skip direct downstream** tasks and the scheduler will handle skipping any further downstream dependencies.
-
-No change is needed if only the default trigger rule `all_success` is being used.
-
-If the DAG relies on tasks with other trigger rules (i.e. `all_done`) being skipped by the `LatestOnlyOperator`, adjustments to the DAG need to be made to accommodate the change in behaviour, i.e. with additional edges from the `LatestOnlyOperator`.
-
-The goal of this change is to achieve a more consistent and configurable cascading behaviour based on the `BaseBranchOperator` (see [AIRFLOW-2923](https://jira.apache.org/jira/browse/AIRFLOW-2923) and [AIRFLOW-1784](https://jira.apache.org/jira/browse/AIRFLOW-1784)).
-
-### Changes to the core Python API
-
-We strive to ensure that there are no changes that may affect the end user, and your Python files, but this
-release may contain changes that will require changes to your plugins, DAG File or other integration.
-
-Only changes unique to this provider are described here. You should still pay attention to the changes that
-have been made to the core (including core operators) as they can affect the integration behavior
-of this provider.
-
-This section describes the changes that have been made, and what you need to do to update your Python files.
-
-#### Removed sub-package imports from `airflow/__init__.py`
-
-The imports `LoggingMixin`, `conf`, and `AirflowException` have been removed from `airflow/__init__.py`.
-All implicit references of these objects will no longer be valid. To migrate, all usages of each old path must be
-replaced with its corresponding new path.
-
-| Old Path (Implicit Import) | New Path (Explicit Import) |
-|------------------------------|--------------------------------------------------|
-| ``airflow.LoggingMixin`` | ``airflow.utils.log.logging_mixin.LoggingMixin`` |
-| ``airflow.conf`` | ``airflow.configuration.conf`` |
-| ``airflow.AirflowException`` | ``airflow.exceptions.AirflowException`` |
-
-#### Variables removed from the task instance context
-
-The following variables were removed from the task instance context:
-
-- end_date
-- latest_date
-- tables
-
-
-#### `airflow.contrib.utils.Weekday`
-
-Formerly the core code was maintained by the original creators - Airbnb. The code that was in the contrib
-package was supported by the community. The project was passed to the Apache community and currently the
-entire code is maintained by the community, so now the division has no justification, and it is only due
-to historical reasons.
-
-To clean up, `Weekday` enum has been moved from `airflow.contrib.utils` into `airflow.utils` module.
-
-#### `airflow.models.connection.Connection`
-
-The connection module has new deprecated methods:
-
-- `Connection.parse_from_uri`
-- `Connection.log_info`
-- `Connection.debug_info`
-
-and one deprecated function:
-
-- `parse_netloc_to_hostname`
-
-Previously, users could create a connection object in two ways
-
-```
-conn_1 = Connection(conn_id="conn_a", uri="mysql://AAA/")
-# or
-conn_2 = Connection(conn_id="conn_a")
-conn_2.parse_uri(uri="mysql://AAA/")
-```
-
-Now the second way is not supported.
-
-`Connection.log_info` and `Connection.debug_info` method have been deprecated. Read each Connection field individually or use the
-default representation (`__repr__`).
-
-The old method is still works but can be abandoned at any time. The changes are intended to delete method
-that are rarely used.
-
-#### `airflow.models.dag.DAG.create_dagrun`
-
-DAG.create_dagrun accepts run_type and does not require run_id
-This change is caused by adding `run_type` column to `DagRun`.
-
-Previous signature:
-
-```python
-def create_dagrun(
- self,
- run_id,
- state,
- execution_date=None,
- start_date=None,
- external_trigger=False,
- conf=None,
- session=None,
-):
- ...
-```
-
-current:
-
-```python
-def create_dagrun(
- self,
- state,
- execution_date=None,
- run_id=None,
- start_date=None,
- external_trigger=False,
- conf=None,
- run_type=None,
- session=None,
-):
- ...
-```
-
-If user provides `run_id` then the `run_type` will be derived from it by checking prefix, allowed types
-: `manual`, `scheduled`, `backfill` (defined by `airflow.utils.types.DagRunType`).
-
-If user provides `run_type` and `execution_date` then `run_id` is constructed as
-`{run_type}__{execution_data.isoformat()}`.
-
-Airflow should construct dagruns using `run_type` and `execution_date`, creation using
-`run_id` is preserved for user actions.
-
-
-#### `airflow.models.dagrun.DagRun`
-
-Use DagRunType.SCHEDULED.value instead of DagRun.ID_PREFIX
-
-All the run_id prefixes for different kind of DagRuns have been grouped into a single
-enum in `airflow.utils.types.DagRunType`.
-
-Previously, there were defined in various places, example as `ID_PREFIX` class variables for
-`DagRun`, `BackfillJob` and in `_trigger_dag` function.
-
-Was:
-
-```pycon
->> from airflow.models.dagrun import DagRun
->> DagRun.ID_PREFIX
-scheduled__
-```
-
-Replaced by:
-
-```pycon
->> from airflow.utils.types import DagRunType
->> DagRunType.SCHEDULED.value
-scheduled
-```
-
-
-#### `airflow.utils.file.TemporaryDirectory`
-
-We remove `airflow.utils.file.TemporaryDirectory`
-Since Airflow dropped support for Python < 3.5 there's no need to have this custom
-implementation of `TemporaryDirectory` because the same functionality is provided by
-`tempfile.TemporaryDirectory`.
-
-Now users instead of `import from airflow.utils.files import TemporaryDirectory` should
-do `from tempfile import TemporaryDirectory`. Both context managers provide the same
-interface, thus no additional changes should be required.
-
-#### `airflow.AirflowMacroPlugin`
-
-We removed `airflow.AirflowMacroPlugin` class. The class was there in airflow package but it has not been used (apparently since 2015).
-It has been removed.
-
-#### `airflow.settings.CONTEXT_MANAGER_DAG`
-
-CONTEXT_MANAGER_DAG was removed from settings. It's role has been taken by `DagContext` in
-'airflow.models.dag'. One of the reasons was that settings should be rather static than store
-dynamic context from the DAG, but the main one is that moving the context out of settings allowed to
-untangle cyclic imports between DAG, BaseOperator, SerializedDAG, SerializedBaseOperator which was
-part of AIRFLOW-6010.
-
-#### `airflow.utils.log.logging_mixin.redirect_stderr`
-
-#### `airflow.utils.log.logging_mixin.redirect_stdout`
-
-Function `redirect_stderr` and `redirect_stdout` from `airflow.utils.log.logging_mixin` module has
-been deleted because it can be easily replaced by the standard library.
-The functions of the standard library are more flexible and can be used in larger cases.
-
-The code below
-
-```python
-import logging
-
-from airflow.utils.log.logging_mixin import redirect_stderr, redirect_stdout
-
-logger = logging.getLogger("custom-logger")
-with redirect_stdout(logger, logging.INFO), redirect_stderr(logger, logging.WARN):
- print("I love Airflow")
-```
-
-can be replaced by the following code:
-
-```python
-from contextlib import redirect_stdout, redirect_stderr
-import logging
-
-from airflow.utils.log.logging_mixin import StreamLogWriter
-
-logger = logging.getLogger("custom-logger")
-
-with redirect_stdout(StreamLogWriter(logger, logging.INFO)), redirect_stderr(
- StreamLogWriter(logger, logging.WARN)
-):
- print("I Love Airflow")
-```
-
-#### `airflow.models.baseoperator.BaseOperator`
-
-Now, additional arguments passed to BaseOperator cause an exception. Previous versions of Airflow took additional arguments and displayed a message on the console. When the
-message was not noticed by users, it caused very difficult to detect errors.
-
-In order to restore the previous behavior, you must set an ``True`` in the ``allow_illegal_arguments``
-option of section ``[operators]`` in the ``airflow.cfg`` file. In the future it is possible to completely
-delete this option.
-
-#### `airflow.models.dagbag.DagBag`
-
-Passing `store_serialized_dags` argument to `DagBag.__init__` and accessing `DagBag.store_serialized_dags` property
-are deprecated and will be removed in future versions.
-
-
-**Previous signature**:
-
-```python
-def __init__(
- dag_folder=None,
- include_examples=conf.getboolean("core", "LOAD_EXAMPLES"),
- safe_mode=conf.getboolean("core", "DAG_DISCOVERY_SAFE_MODE"),
- store_serialized_dags=False,
-):
- ...
-```
-
-**current**:
-
-```python
-def __init__(
- dag_folder=None,
- include_examples=conf.getboolean("core", "LOAD_EXAMPLES"),
- safe_mode=conf.getboolean("core", "DAG_DISCOVERY_SAFE_MODE"),
- read_dags_from_db=False,
-):
- ...
-```
-
-If you were using positional arguments, it requires no change but if you were using keyword
-arguments, please change `store_serialized_dags` to `read_dags_from_db`.
-
-Similarly, if you were using `DagBag().store_serialized_dags` property, change it to
-`DagBag().read_dags_from_db`.
-
-### Changes in `google` provider package
-
-We strive to ensure that there are no changes that may affect the end user and your Python files, but this
-release may contain changes that will require changes to your configuration, DAG Files or other integration
-e.g. custom operators.
-
-Only changes unique to this provider are described here. You should still pay attention to the changes that
-have been made to the core (including core operators) as they can affect the integration behavior
-of this provider.
-
-This section describes the changes that have been made, and what you need to do to update your if
-you use operators or hooks which integrate with Google services (including Google Cloud - GCP).
-
-#### Direct impersonation added to operators communicating with Google services
-
-[Directly impersonating a service account](https://cloud.google.com/iam/docs/understanding-service-accounts#directly_impersonating_a_service_account)
-has been made possible for operators communicating with Google services via new argument called `impersonation_chain`
-(`google_impersonation_chain` in case of operators that also communicate with services of other cloud providers).
-As a result, GCSToS3Operator no longer derivatives from GCSListObjectsOperator.
-
-#### Normalize gcp_conn_id for Google Cloud
-
-Previously not all hooks and operators related to Google Cloud use
-`gcp_conn_id` as parameter for GCP connection. There is currently one parameter
-which apply to most services. Parameters like ``datastore_conn_id``, ``bigquery_conn_id``,
-``google_cloud_storage_conn_id`` and similar have been deprecated. Operators that require two connections are not changed.
-
-Following components were affected by normalization:
-
- * airflow.providers.google.cloud.hooks.datastore.DatastoreHook
- * airflow.providers.google.cloud.hooks.bigquery.BigQueryHook
- * airflow.providers.google.cloud.hooks.gcs.GoogleCloudStorageHook
- * airflow.providers.google.cloud.operators.bigquery.BigQueryCheckOperator
- * airflow.providers.google.cloud.operators.bigquery.BigQueryValueCheckOperator
- * airflow.providers.google.cloud.operators.bigquery.BigQueryIntervalCheckOperator
- * airflow.providers.google.cloud.operators.bigquery.BigQueryGetDataOperator
- * airflow.providers.google.cloud.operators.bigquery.BigQueryOperator
- * airflow.providers.google.cloud.operators.bigquery.BigQueryDeleteDatasetOperator
- * airflow.providers.google.cloud.operators.bigquery.BigQueryCreateEmptyDatasetOperator
- * airflow.providers.google.cloud.operators.bigquery.BigQueryTableDeleteOperator
- * airflow.providers.google.cloud.operators.gcs.GoogleCloudStorageCreateBucketOperator
- * airflow.providers.google.cloud.operators.gcs.GoogleCloudStorageListOperator
- * airflow.providers.google.cloud.operators.gcs.GoogleCloudStorageDownloadOperator
- * airflow.providers.google.cloud.operators.gcs.GoogleCloudStorageDeleteOperator
- * airflow.providers.google.cloud.operators.gcs.GoogleCloudStorageBucketCreateAclEntryOperator
- * airflow.providers.google.cloud.operators.gcs.GoogleCloudStorageObjectCreateAclEntryOperator
- * airflow.operators.sql_to_gcs.BaseSQLToGoogleCloudStorageOperator
- * airflow.operators.adls_to_gcs.AdlsToGoogleCloudStorageOperator
- * airflow.operators.gcs_to_s3.GoogleCloudStorageToS3Operator
- * airflow.operators.gcs_to_gcs.GoogleCloudStorageToGoogleCloudStorageOperator
- * airflow.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator
- * airflow.operators.local_to_gcs.FileToGoogleCloudStorageOperator
- * airflow.operators.cassandra_to_gcs.CassandraToGoogleCloudStorageOperator
- * airflow.operators.bigquery_to_bigquery.BigQueryToBigQueryOperator
-
-#### Changes to import paths and names of GCP operators and hooks
-
-According to [AIP-21](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths)
-operators related to Google Cloud has been moved from contrib to core.
-The following table shows changes in import paths.
-
-| Old path | New path |
-|------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------|
-|airflow.contrib.hooks.bigquery_hook.BigQueryHook |airflow.providers.google.cloud.hooks.bigquery.BigQueryHook |
-|airflow.contrib.hooks.datastore_hook.DatastoreHook |airflow.providers.google.cloud.hooks.datastore.DatastoreHook |
-|airflow.contrib.hooks.gcp_bigtable_hook.BigtableHook |airflow.providers.google.cloud.hooks.bigtable.BigtableHook |
-|airflow.contrib.hooks.gcp_cloud_build_hook.CloudBuildHook |airflow.providers.google.cloud.hooks.cloud_build.CloudBuildHook |
-|airflow.contrib.hooks.gcp_container_hook.GKEClusterHook |airflow.providers.google.cloud.hooks.kubernetes_engine.GKEHook |
-|airflow.contrib.hooks.gcp_compute_hook.GceHook |airflow.providers.google.cloud.hooks.compute.ComputeEngineHook |
-|airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook |airflow.providers.google.cloud.hooks.dataflow.DataflowHook |
-|airflow.contrib.hooks.gcp_dataproc_hook.DataProcHook |airflow.providers.google.cloud.hooks.dataproc.DataprocHook |
-|airflow.contrib.hooks.gcp_dlp_hook.CloudDLPHook |airflow.providers.google.cloud.hooks.dlp.CloudDLPHook |
-|airflow.contrib.hooks.gcp_function_hook.GcfHook |airflow.providers.google.cloud.hooks.functions.CloudFunctionsHook |
-|airflow.contrib.hooks.gcp_kms_hook.GoogleCloudKMSHook |airflow.providers.google.cloud.hooks.kms.CloudKMSHook |
-|airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook |airflow.providers.google.cloud.hooks.mlengine.MLEngineHook |
-|airflow.contrib.hooks.gcp_natural_language_hook.CloudNaturalLanguageHook |airflow.providers.google.cloud.hooks.natural_language.CloudNaturalLanguageHook |
-|airflow.contrib.hooks.gcp_pubsub_hook.PubSubHook |airflow.providers.google.cloud.hooks.pubsub.PubSubHook |
-|airflow.contrib.hooks.gcp_speech_to_text_hook.GCPSpeechToTextHook |airflow.providers.google.cloud.hooks.speech_to_text.CloudSpeechToTextHook |
-|airflow.contrib.hooks.gcp_spanner_hook.CloudSpannerHook |airflow.providers.google.cloud.hooks.spanner.SpannerHook |
-|airflow.contrib.hooks.gcp_sql_hook.CloudSqlDatabaseHook |airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLDatabaseHook |
-|airflow.contrib.hooks.gcp_sql_hook.CloudSqlHook |airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLHook |
-|airflow.contrib.hooks.gcp_tasks_hook.CloudTasksHook |airflow.providers.google.cloud.hooks.tasks.CloudTasksHook |
-|airflow.contrib.hooks.gcp_text_to_speech_hook.GCPTextToSpeechHook |airflow.providers.google.cloud.hooks.text_to_speech.CloudTextToSpeechHook |
-|airflow.contrib.hooks.gcp_transfer_hook.GCPTransferServiceHook |airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.CloudDataTransferServiceHook |
-|airflow.contrib.hooks.gcp_translate_hook.CloudTranslateHook |airflow.providers.google.cloud.hooks.translate.CloudTranslateHook |
-|airflow.contrib.hooks.gcp_video_intelligence_hook.CloudVideoIntelligenceHook |airflow.providers.google.cloud.hooks.video_intelligence.CloudVideoIntelligenceHook |
-|airflow.contrib.hooks.gcp_vision_hook.CloudVisionHook |airflow.providers.google.cloud.hooks.vision.CloudVisionHook |
-|airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook |airflow.providers.google.cloud.hooks.gcs.GCSHook |
-|airflow.contrib.operators.adls_to_gcs.AdlsToGoogleCloudStorageOperator |airflow.operators.adls_to_gcs.AdlsToGoogleCloudStorageOperator |
-|airflow.contrib.operators.bigquery_check_operator.BigQueryCheckOperator |airflow.providers.google.cloud.operators.bigquery.BigQueryCheckOperator |
-|airflow.contrib.operators.bigquery_check_operator.BigQueryIntervalCheckOperator |airflow.providers.google.cloud.operators.bigquery.BigQueryIntervalCheckOperator |
-|airflow.contrib.operators.bigquery_check_operator.BigQueryValueCheckOperator |airflow.providers.google.cloud.operators.bigquery.BigQueryValueCheckOperator |
-|airflow.contrib.operators.bigquery_get_data.BigQueryGetDataOperator |airflow.providers.google.cloud.operators.bigquery.BigQueryGetDataOperator |
-|airflow.contrib.operators.bigquery_operator.BigQueryCreateEmptyDatasetOperator |airflow.providers.google.cloud.operators.bigquery.BigQueryCreateEmptyDatasetOperator |
-|airflow.contrib.operators.bigquery_operator.BigQueryCreateEmptyTableOperator |airflow.providers.google.cloud.operators.bigquery.BigQueryCreateEmptyTableOperator |
-|airflow.contrib.operators.bigquery_operator.BigQueryCreateExternalTableOperator |airflow.providers.google.cloud.operators.bigquery.BigQueryCreateExternalTableOperator |
-|airflow.contrib.operators.bigquery_operator.BigQueryDeleteDatasetOperator |airflow.providers.google.cloud.operators.bigquery.BigQueryDeleteDatasetOperator |
-|airflow.contrib.operators.bigquery_operator.BigQueryOperator |airflow.providers.google.cloud.operators.bigquery.BigQueryExecuteQueryOperator |
-|airflow.contrib.operators.bigquery_table_delete_operator.BigQueryTableDeleteOperator |airflow.providers.google.cloud.operators.bigquery.BigQueryDeleteTableOperator |
-|airflow.contrib.operators.bigquery_to_bigquery.BigQueryToBigQueryOperator |airflow.operators.bigquery_to_bigquery.BigQueryToBigQueryOperator |
-|airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator |airflow.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator |
-|airflow.contrib.operators.bigquery_to_mysql_operator.BigQueryToMySqlOperator |airflow.operators.bigquery_to_mysql.BigQueryToMySqlOperator |
-|airflow.contrib.operators.dataflow_operator.DataFlowJavaOperator |airflow.providers.google.cloud.operators.dataflow.DataFlowJavaOperator |
-|airflow.contrib.operators.dataflow_operator.DataFlowPythonOperator |airflow.providers.google.cloud.operators.dataflow.DataFlowPythonOperator |
-|airflow.contrib.operators.dataflow_operator.DataflowTemplateOperator |airflow.providers.google.cloud.operators.dataflow.DataflowTemplateOperator |
-|airflow.contrib.operators.dataproc_operator.DataProcHadoopOperator |airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHadoopJobOperator |
-|airflow.contrib.operators.dataproc_operator.DataProcHiveOperator |airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHiveJobOperator |
-|airflow.contrib.operators.dataproc_operator.DataProcJobBaseOperator |airflow.providers.google.cloud.operators.dataproc.DataprocJobBaseOperator |
-|airflow.contrib.operators.dataproc_operator.DataProcPigOperator |airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPigJobOperator |
-|airflow.contrib.operators.dataproc_operator.DataProcPySparkOperator |airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPySparkJobOperator |
-|airflow.contrib.operators.dataproc_operator.DataProcSparkOperator |airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkJobOperator |
-|airflow.contrib.operators.dataproc_operator.DataProcSparkSqlOperator |airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkSqlJobOperator |
-|airflow.contrib.operators.dataproc_operator.DataprocClusterCreateOperator |airflow.providers.google.cloud.operators.dataproc.DataprocCreateClusterOperator |
-|airflow.contrib.operators.dataproc_operator.DataprocClusterDeleteOperator |airflow.providers.google.cloud.operators.dataproc.DataprocDeleteClusterOperator |
-|airflow.contrib.operators.dataproc_operator.DataprocClusterScaleOperator |airflow.providers.google.cloud.operators.dataproc.DataprocScaleClusterOperator |
-|airflow.contrib.operators.dataproc_operator.DataprocOperationBaseOperator |airflow.providers.google.cloud.operators.dataproc.DataprocOperationBaseOperator |
-|airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateInstantiateInlineOperator |airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateInlineWorkflowTemplateOperator |
-|airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateInstantiateOperator |airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateWorkflowTemplateOperator |
-|airflow.contrib.operators.datastore_export_operator.DatastoreExportOperator |airflow.providers.google.cloud.operators.datastore.DatastoreExportOperator |
-|airflow.contrib.operators.datastore_import_operator.DatastoreImportOperator |airflow.providers.google.cloud.operators.datastore.DatastoreImportOperator |
-|airflow.contrib.operators.file_to_gcs.FileToGoogleCloudStorageOperator |airflow.providers.google.cloud.transfers.local_to_gcs.FileToGoogleCloudStorageOperator |
-|airflow.contrib.operators.gcp_bigtable_operator.BigtableClusterUpdateOperator |airflow.providers.google.cloud.operators.bigtable.BigtableUpdateClusterOperator |
-|airflow.contrib.operators.gcp_bigtable_operator.BigtableInstanceCreateOperator |airflow.providers.google.cloud.operators.bigtable.BigtableCreateInstanceOperator |
-|airflow.contrib.operators.gcp_bigtable_operator.BigtableInstanceDeleteOperator |airflow.providers.google.cloud.operators.bigtable.BigtableDeleteInstanceOperator |
-|airflow.contrib.operators.gcp_bigtable_operator.BigtableTableCreateOperator |airflow.providers.google.cloud.operators.bigtable.BigtableCreateTableOperator |
-|airflow.contrib.operators.gcp_bigtable_operator.BigtableTableDeleteOperator |airflow.providers.google.cloud.operators.bigtable.BigtableDeleteTableOperator |
-|airflow.contrib.operators.gcp_bigtable_operator.BigtableTableWaitForReplicationSensor |airflow.providers.google.cloud.sensors.bigtable.BigtableTableReplicationCompletedSensor |
-|airflow.contrib.operators.gcp_cloud_build_operator.CloudBuildCreateBuildOperator |airflow.providers.google.cloud.operators.cloud_build.CloudBuildCreateBuildOperator |
-|airflow.contrib.operators.gcp_compute_operator.GceBaseOperator |airflow.providers.google.cloud.operators.compute.GceBaseOperator |
-|airflow.contrib.operators.gcp_compute_operator.GceInstanceGroupManagerUpdateTemplateOperator |airflow.providers.google.cloud.operators.compute.GceInstanceGroupManagerUpdateTemplateOperator |
-|airflow.contrib.operators.gcp_compute_operator.GceInstanceStartOperator |airflow.providers.google.cloud.operators.compute.GceInstanceStartOperator |
-|airflow.contrib.operators.gcp_compute_operator.GceInstanceStopOperator |airflow.providers.google.cloud.operators.compute.GceInstanceStopOperator |
-|airflow.contrib.operators.gcp_compute_operator.GceInstanceTemplateCopyOperator |airflow.providers.google.cloud.operators.compute.GceInstanceTemplateCopyOperator |
-|airflow.contrib.operators.gcp_compute_operator.GceSetMachineTypeOperator |airflow.providers.google.cloud.operators.compute.GceSetMachineTypeOperator |
-|airflow.contrib.operators.gcp_container_operator.GKEClusterCreateOperator |airflow.providers.google.cloud.operators.kubernetes_engine.GKECreateClusterOperator |
-|airflow.contrib.operators.gcp_container_operator.GKEClusterDeleteOperator |airflow.providers.google.cloud.operators.kubernetes_engine.GKEDeleteClusterOperator |
-|airflow.contrib.operators.gcp_container_operator.GKEPodOperator |airflow.providers.google.cloud.operators.kubernetes_engine.GKEStartPodOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPCancelDLPJobOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPCancelDLPJobOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPCreateDLPJobOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPCreateDLPJobOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPCreateDeidentifyTemplateOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPCreateDeidentifyTemplateOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPCreateInspectTemplateOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPCreateInspectTemplateOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPCreateJobTriggerOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPCreateJobTriggerOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPCreateStoredInfoTypeOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPCreateStoredInfoTypeOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPDeidentifyContentOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPDeidentifyContentOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPDeleteDeidentifyTemplateOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPDeleteDeidentifyTemplateOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPDeleteDlpJobOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPDeleteDLPJobOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPDeleteInspectTemplateOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPDeleteInspectTemplateOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPDeleteJobTriggerOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPDeleteJobTriggerOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPDeleteStoredInfoTypeOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPDeleteStoredInfoTypeOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPGetDeidentifyTemplateOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPGetDeidentifyTemplateOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPGetDlpJobOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPGetDLPJobOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPGetInspectTemplateOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPGetInspectTemplateOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPGetJobTripperOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPGetJobTriggerOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPGetStoredInfoTypeOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPGetStoredInfoTypeOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPInspectContentOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPInspectContentOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPListDeidentifyTemplatesOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPListDeidentifyTemplatesOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPListDlpJobsOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPListDLPJobsOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPListInfoTypesOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPListInfoTypesOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPListInspectTemplatesOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPListInspectTemplatesOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPListJobTriggersOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPListJobTriggersOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPListStoredInfoTypesOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPListStoredInfoTypesOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPRedactImageOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPRedactImageOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPReidentifyContentOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPReidentifyContentOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPUpdateDeidentifyTemplateOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPUpdateDeidentifyTemplateOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPUpdateInspectTemplateOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPUpdateInspectTemplateOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPUpdateJobTriggerOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPUpdateJobTriggerOperator |
-|airflow.contrib.operators.gcp_dlp_operator.CloudDLPUpdateStoredInfoTypeOperator |airflow.providers.google.cloud.operators.dlp.CloudDLPUpdateStoredInfoTypeOperator |
-|airflow.contrib.operators.gcp_function_operator.GcfFunctionDeleteOperator |airflow.providers.google.cloud.operators.functions.GcfFunctionDeleteOperator |
-|airflow.contrib.operators.gcp_function_operator.GcfFunctionDeployOperator |airflow.providers.google.cloud.operators.functions.GcfFunctionDeployOperator |
-|airflow.contrib.operators.gcp_natural_language_operator.CloudNaturalLanguageAnalyzeEntitiesOperator |airflow.providers.google.cloud.operators.natural_language.CloudNaturalLanguageAnalyzeEntitiesOperator |
-|airflow.contrib.operators.gcp_natural_language_operator.CloudNaturalLanguageAnalyzeEntitySentimentOperator |airflow.providers.google.cloud.operators.natural_language.CloudNaturalLanguageAnalyzeEntitySentimentOperator |
-|airflow.contrib.operators.gcp_natural_language_operator.CloudNaturalLanguageAnalyzeSentimentOperator |airflow.providers.google.cloud.operators.natural_language.CloudNaturalLanguageAnalyzeSentimentOperator |
-|airflow.contrib.operators.gcp_natural_language_operator.CloudNaturalLanguageClassifyTextOperator |airflow.providers.google.cloud.operators.natural_language.CloudNaturalLanguageClassifyTextOperator |
-|airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseDeleteOperator |airflow.providers.google.cloud.operators.spanner.SpannerDeleteDatabaseInstanceOperator |
-|airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseDeployOperator |airflow.providers.google.cloud.operators.spanner.SpannerDeployDatabaseInstanceOperator |
-|airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseQueryOperator |airflow.providers.google.cloud.operators.spanner.SpannerQueryDatabaseInstanceOperator |
-|airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDatabaseUpdateOperator |airflow.providers.google.cloud.operators.spanner.SpannerUpdateDatabaseInstanceOperator |
-|airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDeleteOperator |airflow.providers.google.cloud.operators.spanner.SpannerDeleteInstanceOperator |
-|airflow.contrib.operators.gcp_spanner_operator.CloudSpannerInstanceDeployOperator |airflow.providers.google.cloud.operators.spanner.SpannerDeployInstanceOperator |
-|airflow.contrib.operators.gcp_speech_to_text_operator.GcpSpeechToTextRecognizeSpeechOperator |airflow.providers.google.cloud.operators.speech_to_text.CloudSpeechToTextRecognizeSpeechOperator |
-|airflow.contrib.operators.gcp_text_to_speech_operator.GcpTextToSpeechSynthesizeOperator |airflow.providers.google.cloud.operators.text_to_speech.CloudTextToSpeechSynthesizeOperator |
-|airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceJobCreateOperator |airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceCreateJobOperator |
-|airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceJobDeleteOperator |airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceDeleteJobOperator |
-|airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceJobUpdateOperator |airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceUpdateJobOperator |
-|airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceOperationCancelOperator |airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceCancelOperationOperator |
-|airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceOperationGetOperator |airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceGetOperationOperator |
-|airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceOperationPauseOperator |airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServicePauseOperationOperator |
-|airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceOperationResumeOperator |airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceResumeOperationOperator |
-|airflow.contrib.operators.gcp_transfer_operator.GcpTransferServiceOperationsListOperator |airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceListOperationsOperator |
-|airflow.contrib.operators.gcp_transfer_operator.GoogleCloudStorageToGoogleCloudStorageTransferOperator |airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceGCSToGCSOperator |
-|airflow.contrib.operators.gcp_translate_operator.CloudTranslateTextOperator |airflow.providers.google.cloud.operators.translate.CloudTranslateTextOperator |
-|airflow.contrib.operators.gcp_translate_speech_operator.GcpTranslateSpeechOperator |airflow.providers.google.cloud.operators.translate_speech.GcpTranslateSpeechOperator |
-|airflow.contrib.operators.gcp_video_intelligence_operator.CloudVideoIntelligenceDetectVideoExplicitContentOperator|airflow.providers.google.cloud.operators.video_intelligence.CloudVideoIntelligenceDetectVideoExplicitContentOperator |
-|airflow.contrib.operators.gcp_video_intelligence_operator.CloudVideoIntelligenceDetectVideoLabelsOperator |airflow.providers.google.cloud.operators.video_intelligence.CloudVideoIntelligenceDetectVideoLabelsOperator |
-|airflow.contrib.operators.gcp_video_intelligence_operator.CloudVideoIntelligenceDetectVideoShotsOperator |airflow.providers.google.cloud.operators.video_intelligence.CloudVideoIntelligenceDetectVideoShotsOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionAddProductToProductSetOperator |airflow.providers.google.cloud.operators.vision.CloudVisionAddProductToProductSetOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionAnnotateImageOperator |airflow.providers.google.cloud.operators.vision.CloudVisionImageAnnotateOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionDetectDocumentTextOperator |airflow.providers.google.cloud.operators.vision.CloudVisionTextDetectOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionDetectImageLabelsOperator |airflow.providers.google.cloud.operators.vision.CloudVisionDetectImageLabelsOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionDetectImageSafeSearchOperator |airflow.providers.google.cloud.operators.vision.CloudVisionDetectImageSafeSearchOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionDetectTextOperator |airflow.providers.google.cloud.operators.vision.CloudVisionDetectTextOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionProductCreateOperator |airflow.providers.google.cloud.operators.vision.CloudVisionCreateProductOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionProductDeleteOperator |airflow.providers.google.cloud.operators.vision.CloudVisionDeleteProductOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionProductGetOperator |airflow.providers.google.cloud.operators.vision.CloudVisionGetProductOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionProductSetCreateOperator |airflow.providers.google.cloud.operators.vision.CloudVisionCreateProductSetOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionProductSetDeleteOperator |airflow.providers.google.cloud.operators.vision.CloudVisionDeleteProductSetOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionProductSetGetOperator |airflow.providers.google.cloud.operators.vision.CloudVisionGetProductSetOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionProductSetUpdateOperator |airflow.providers.google.cloud.operators.vision.CloudVisionUpdateProductSetOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionProductUpdateOperator |airflow.providers.google.cloud.operators.vision.CloudVisionUpdateProductOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionReferenceImageCreateOperator |airflow.providers.google.cloud.operators.vision.CloudVisionCreateReferenceImageOperator |
-|airflow.contrib.operators.gcp_vision_operator.CloudVisionRemoveProductFromProductSetOperator |airflow.providers.google.cloud.operators.vision.CloudVisionRemoveProductFromProductSetOperator |
-|airflow.contrib.operators.gcs_acl_operator.GoogleCloudStorageBucketCreateAclEntryOperator |airflow.providers.google.cloud.operators.gcs.GCSBucketCreateAclEntryOperator |
-|airflow.contrib.operators.gcs_acl_operator.GoogleCloudStorageObjectCreateAclEntryOperator |airflow.providers.google.cloud.operators.gcs.GCSObjectCreateAclEntryOperator |
-|airflow.contrib.operators.gcs_delete_operator.GoogleCloudStorageDeleteOperator |airflow.providers.google.cloud.operators.gcs.GCSDeleteObjectsOperator |
-|airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator |airflow.providers.google.cloud.operators.gcs.GCSToLocalFilesystemOperator |
-|airflow.contrib.operators.gcs_list_operator.GoogleCloudStorageListOperator |airflow.providers.google.cloud.operators.gcs.GCSListObjectsOperator |
-|airflow.contrib.operators.gcs_operator.GoogleCloudStorageCreateBucketOperator |airflow.providers.google.cloud.operators.gcs.GCSCreateBucketOperator |
-|airflow.contrib.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator |airflow.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator |
-|airflow.contrib.operators.gcs_to_gcs.GoogleCloudStorageToGoogleCloudStorageOperator |airflow.operators.gcs_to_gcs.GoogleCloudStorageToGoogleCloudStorageOperator |
-|airflow.contrib.operators.gcs_to_s3.GoogleCloudStorageToS3Operator |airflow.operators.gcs_to_s3.GCSToS3Operator |
-|airflow.contrib.operators.mlengine_operator.MLEngineBatchPredictionOperator |airflow.providers.google.cloud.operators.mlengine.MLEngineStartBatchPredictionJobOperator |
-|airflow.contrib.operators.mlengine_operator.MLEngineModelOperator |airflow.providers.google.cloud.operators.mlengine.MLEngineManageModelOperator |
-|airflow.contrib.operators.mlengine_operator.MLEngineTrainingOperator |airflow.providers.google.cloud.operators.mlengine.MLEngineStartTrainingJobOperator |
-|airflow.contrib.operators.mlengine_operator.MLEngineVersionOperator |airflow.providers.google.cloud.operators.mlengine.MLEngineManageVersionOperator |
-|airflow.contrib.operators.mssql_to_gcs.MsSqlToGoogleCloudStorageOperator |airflow.operators.mssql_to_gcs.MsSqlToGoogleCloudStorageOperator |
-|airflow.contrib.operators.mysql_to_gcs.MySqlToGoogleCloudStorageOperator |airflow.operators.mysql_to_gcs.MySqlToGoogleCloudStorageOperator |
-|airflow.contrib.operators.postgres_to_gcs_operator.PostgresToGoogleCloudStorageOperator |airflow.operators.postgres_to_gcs.PostgresToGoogleCloudStorageOperator |
-|airflow.contrib.operators.pubsub_operator.PubSubPublishOperator |airflow.providers.google.cloud.operators.pubsub.PubSubPublishMessageOperator |
-|airflow.contrib.operators.pubsub_operator.PubSubSubscriptionCreateOperator |airflow.providers.google.cloud.operators.pubsub.PubSubCreateSubscriptionOperator |
-|airflow.contrib.operators.pubsub_operator.PubSubSubscriptionDeleteOperator |airflow.providers.google.cloud.operators.pubsub.PubSubDeleteSubscriptionOperator |
-|airflow.contrib.operators.pubsub_operator.PubSubTopicCreateOperator |airflow.providers.google.cloud.operators.pubsub.PubSubCreateTopicOperator |
-|airflow.contrib.operators.pubsub_operator.PubSubTopicDeleteOperator |airflow.providers.google.cloud.operators.pubsub.PubSubDeleteTopicOperator |
-|airflow.contrib.operators.sql_to_gcs.BaseSQLToGoogleCloudStorageOperator |airflow.operators.sql_to_gcs.BaseSQLToGoogleCloudStorageOperator |
-|airflow.contrib.sensors.bigquery_sensor.BigQueryTableSensor |airflow.providers.google.cloud.sensors.bigquery.BigQueryTableExistenceSensor |
-|airflow.contrib.sensors.gcp_transfer_sensor.GCPTransferServiceWaitForJobStatusSensor |airflow.providers.google.cloud.sensors.cloud_storage_transfer_service.DataTransferServiceJobStatusSensor |
-|airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectSensor |airflow.providers.google.cloud.sensors.gcs.GCSObjectExistenceSensor |
-|airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectUpdatedSensor |airflow.providers.google.cloud.sensors.gcs.GCSObjectUpdateSensor |
-|airflow.contrib.sensors.gcs_sensor.GoogleCloudStoragePrefixSensor |airflow.providers.google.cloud.sensors.gcs.GCSObjectsWithPrefixExistenceSensor |
-|airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageUploadSessionCompleteSensor |airflow.providers.google.cloud.sensors.gcs.GCSUploadSessionCompleteSensor |
-|airflow.contrib.sensors.pubsub_sensor.PubSubPullSensor |airflow.providers.google.cloud.sensors.pubsub.PubSubPullSensor |
-
-#### Unify default conn_id for Google Cloud
-
-Previously not all hooks and operators related to Google Cloud use
-``google_cloud_default`` as a default conn_id. There is currently one default
-variant. Values like ``google_cloud_storage_default``, ``bigquery_default``,
-``google_cloud_datastore_default`` have been deprecated. The configuration of
-existing relevant connections in the database have been preserved. To use those
-deprecated GCP conn_id, you need to explicitly pass their conn_id into
-operators/hooks. Otherwise, ``google_cloud_default`` will be used as GCP's conn_id
-by default.
-
-#### `airflow.providers.google.cloud.hooks.dataflow.DataflowHook`
-
-#### `airflow.providers.google.cloud.operators.dataflow.DataflowCreateJavaJobOperator`
-
-#### `airflow.providers.google.cloud.operators.dataflow.DataflowTemplatedJobStartOperator`
-
-#### `airflow.providers.google.cloud.operators.dataflow.DataflowCreatePythonJobOperator`
-
-To use project_id argument consistently across GCP hooks and operators, we did the following changes:
-
-- Changed order of arguments in DataflowHook.start_python_dataflow. Uses
- with positional arguments may break.
-- Changed order of arguments in DataflowHook.is_job_dataflow_running. Uses
- with positional arguments may break.
-- Changed order of arguments in DataflowHook.cancel_job. Uses
- with positional arguments may break.
-- Added optional project_id argument to DataflowCreateJavaJobOperator
- constructor.
-- Added optional project_id argument to DataflowTemplatedJobStartOperator
- constructor.
-- Added optional project_id argument to DataflowCreatePythonJobOperator
- constructor.
-
-#### `airflow.providers.google.cloud.sensors.gcs.GCSUploadSessionCompleteSensor`
-
-To provide more precise control in handling of changes to objects in
-underlying GCS Bucket the constructor of this sensor now has changed.
-
-- Old Behavior: This constructor used to optionally take ``previous_num_objects: int``.
-- New replacement constructor kwarg: ``previous_objects: Optional[Set[str]]``.
-
-Most users would not specify this argument because the bucket begins empty
-and the user wants to treat any files as new.
-
-Example of Updating usage of this sensor:
-Users who used to call:
-
-``GCSUploadSessionCompleteSensor(bucket='my_bucket', prefix='my_prefix', previous_num_objects=1)``
-
-Will now call:
-
-``GCSUploadSessionCompleteSensor(bucket='my_bucket', prefix='my_prefix', previous_num_objects={'.keep'})``
-
-Where '.keep' is a single file at your prefix that the sensor should not consider new.
-
-#### `airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor`
-
-#### `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook`
-
-To simplify BigQuery operators (no need of `Cursor`) and standardize usage of hooks within all GCP integration methods from `BiqQueryBaseCursor`
-were moved to `BigQueryHook`. Using them by from `Cursor` object is still possible due to preserved backward compatibility but they will raise `DeprecationWarning`.
-The following methods were moved:
-
-| Old path | New path |
-|------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------|
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.cancel_query | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.cancel_query |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.create_empty_dataset | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.create_empty_dataset |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.create_empty_table | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.create_empty_table |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.create_external_table | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.create_external_table |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.delete_dataset | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.delete_dataset |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_dataset | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_dataset |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_dataset_tables | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_dataset_tables |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_dataset_tables_list | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_dataset_tables_list |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_datasets_list | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_datasets_list |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_schema | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_schema |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_tabledata | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.get_tabledata |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.insert_all | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.insert_all |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.patch_dataset | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.patch_dataset |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.patch_table | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.patch_table |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.poll_job_complete | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.poll_job_complete |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_copy | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_copy |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_extract | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_extract |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_grant_dataset_view_access | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_grant_dataset_view_access |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_load | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_load |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_query | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_query |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_table_delete | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_table_delete |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_table_upsert | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_table_upsert |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_with_configuration | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_with_configuration |
-| airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.update_dataset | airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.update_dataset |
-
-#### `airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor`
-
-Since BigQuery is the part of the GCP it was possible to simplify the code by handling the exceptions
-by usage of the `airflow.providers.google.common.hooks.base.GoogleBaseHook.catch_http_exception` decorator however it changes
-exceptions raised by the following methods:
-
-* `airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.run_table_delete` raises `AirflowException` instead of `Exception`.
-* `airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.create_empty_dataset` raises `AirflowException` instead of `ValueError`.
-* `airflow.providers.google.cloud.hooks.bigquery.BigQueryBaseCursor.get_dataset` raises `AirflowException` instead of `ValueError`.
-
-#### `airflow.providers.google.cloud.operators.bigquery.BigQueryCreateEmptyTableOperator`
-
-#### `airflow.providers.google.cloud.operators.bigquery.BigQueryCreateEmptyDatasetOperator`
-
-Idempotency was added to `BigQueryCreateEmptyTableOperator` and `BigQueryCreateEmptyDatasetOperator`.
-But to achieve that try / except clause was removed from `create_empty_dataset` and `create_empty_table`
-methods of `BigQueryHook`.
-
-#### `airflow.providers.google.cloud.hooks.dataflow.DataflowHook`
-
-#### `airflow.providers.google.cloud.hooks.mlengine.MLEngineHook`
-
-#### `airflow.providers.google.cloud.hooks.pubsub.PubSubHook`
-
-The change in GCP operators implies that GCP Hooks for those operators require now keyword parameters rather
-than positional ones in all methods where `project_id` is used. The methods throw an explanatory exception
-in case they are called using positional parameters.
-
-Other GCP hooks are unaffected.
-
-#### `airflow.providers.google.cloud.hooks.pubsub.PubSubHook`
-
-#### `airflow.providers.google.cloud.operators.pubsub.PubSubTopicCreateOperator`
-
-#### `airflow.providers.google.cloud.operators.pubsub.PubSubSubscriptionCreateOperator`
-
-#### `airflow.providers.google.cloud.operators.pubsub.PubSubTopicDeleteOperator`
-
-#### `airflow.providers.google.cloud.operators.pubsub.PubSubSubscriptionDeleteOperator`
-
-#### `airflow.providers.google.cloud.operators.pubsub.PubSubPublishOperator`
-
-#### `airflow.providers.google.cloud.sensors.pubsub.PubSubPullSensor`
-
-In the `PubSubPublishOperator` and `PubSubHook.publsh` method the data field in a message should be bytestring (utf-8 encoded) rather than base64 encoded string.
-
-Due to the normalization of the parameters within GCP operators and hooks a parameters like `project` or `topic_project`
-are deprecated and will be substituted by parameter `project_id`.
-In `PubSubHook.create_subscription` hook method in the parameter `subscription_project` is replaced by `subscription_project_id`.
-Template fields are updated accordingly and old ones may not work.
-
-It is required now to pass key-word only arguments to `PubSub` hook.
-
-These changes are not backward compatible.
-
-#### `airflow.providers.google.cloud.operators.kubernetes_engine.GKEStartPodOperator`
-
-The gcp_conn_id parameter in GKEPodOperator is required. In previous versions, it was possible to pass
-the `None` value to the `gcp_conn_id` in the GKEStartPodOperator
-operator, which resulted in credentials being determined according to the
-[Application Default Credentials](https://cloud.google.com/docs/authentication/production) strategy.
-
-Now this parameter requires a value. To restore the previous behavior, configure the connection without
-specifying the service account.
-
-Detailed information about connection management is available:
-[Google Cloud Connection](https://airflow.apache.org/howto/connection/gcp.html).
-
-
-#### `airflow.providers.google.cloud.hooks.gcs.GCSHook`
-
-* The following parameters have been replaced in all the methods in GCSHook:
- * `bucket` is changed to `bucket_name`
- * `object` is changed to `object_name`
-
-* The `maxResults` parameter in `GoogleCloudStorageHook.list` has been renamed to `max_results` for consistency.
-
-#### `airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPigJobOperator`
-
-#### `airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHiveJobOperator`
-
-#### `airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkSqlJobOperator`
-
-#### `airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkJobOperator`
-
-#### `airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHadoopJobOperator`
-
-#### `airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPySparkJobOperator`
-
-The 'properties' and 'jars' properties for the Dataproc related operators (`DataprocXXXOperator`) have been renamed from
-`dataproc_xxxx_properties` and `dataproc_xxx_jars` to `dataproc_properties`
-and `dataproc_jars`respectively.
-Arguments for dataproc_properties dataproc_jars
-
-#### `airflow.providers.google.cloud.operators.cloud_storage_transfer_service.CloudDataTransferServiceCreateJobOperator`
-
-To obtain pylint compatibility the `filter` argument in `CloudDataTransferServiceCreateJobOperator`
-has been renamed to `request_filter`.
-
-#### `airflow.providers.google.cloud.hooks.cloud_storage_transfer_service.CloudDataTransferServiceHook`
-
- To obtain pylint compatibility the `filter` argument in `CloudDataTransferServiceHook.list_transfer_job` and
- `CloudDataTransferServiceHook.list_transfer_operations` has been renamed to `request_filter`.
-
-#### `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook`
-
-In general all hook methods are decorated with `@GoogleBaseHook.fallback_to_default_project_id` thus
-parameters to hook can only be passed via keyword arguments.
-
-- `create_empty_table` method accepts now `table_resource` parameter. If provided all
-other parameters are ignored.
-- `create_empty_dataset` will now use values from `dataset_reference` instead of raising error
-if parameters were passed in `dataset_reference` and as arguments to method. Additionally validation
-of `dataset_reference` is done using `Dataset.from_api_repr`. Exception and log messages has been
-changed.
-- `update_dataset` requires now new `fields` argument (breaking change)
-- `delete_dataset` has new signature (dataset_id, project_id, ...)
-previous one was (project_id, dataset_id, ...) (breaking change)
-- `get_tabledata` returns list of rows instead of API response in dict format. This method is deprecated in
- favor of `list_rows`. (breaking change)
-
-#### `airflow.providers.google.cloud.hooks.cloud_build.CloudBuildHook`
-
-#### `airflow.providers.google.cloud.operators.cloud_build.CloudBuildCreateBuildOperator`
-
-The `api_version` has been removed and will not be used since we migrate `CloudBuildHook` from using
- Discovery API to native google-cloud-build python library.
-
-The `body` parameter in `CloudBuildCreateBuildOperator` has been deprecated.
- Instead, you should pass body using the `build` parameter.
-
-#### `airflow.providers.google.cloud.hooks.dataflow.DataflowHook.start_python_dataflow`
-
-#### `airflow.providers.google.cloud.hooks.dataflow.DataflowHook.start_python_dataflow`
-
-#### `airflow.providers.google.cloud.operators.dataflow.DataflowCreatePythonJobOperator`
-
-Change python3 as Dataflow Hooks/Operators default interpreter
-
-Now the `py_interpreter` argument for DataFlow Hooks/Operators has been changed from python2 to python3.
-
-#### `airflow.providers.google.common.hooks.base_google.GoogleBaseHook`
-
-To simplify the code, the decorator provide_gcp_credential_file has been moved from the inner-class.
-
-Instead of `@GoogleBaseHook._Decorators.provide_gcp_credential_file`,
-you should write `@GoogleBaseHook.provide_gcp_credential_file`
-
-#### `airflow.providers.google.cloud.operators.dataproc.DataprocCreateClusterOperator`
-
-It is highly recommended to have 1TB+ disk size for Dataproc to have sufficient throughput:
-https://cloud.google.com/compute/docs/disks/performance
-
-Hence, the default value for `master_disk_size` in `DataprocCreateClusterOperator` has been changed from 500GB to 1TB.
-
-##### Generating Cluster Config
-
-If you are upgrading from Airflow 1.10.x and are not using **CLUSTER_CONFIG**,
-You can easily generate config using **make()** of `airflow.providers.google.cloud.operators.dataproc.ClusterGenerator`
-
-This has been proved specially useful if you are using **metadata** argument from older API, refer [AIRFLOW-16911](https://github.com/apache/airflow/issues/16911) for details.
-
-eg. your cluster creation may look like this in **v1.10.x**
-
-```python
-path = f"gs://goog-dataproc-initialization-actions-us-central1/python/pip-install.sh"
-
-create_cluster = DataprocClusterCreateOperator(
- task_id="create_dataproc_cluster",
- cluster_name="test",
- project_id="test",
- zone="us-central1-a",
- region="us-central1",
- master_machine_type="n1-standard-4",
- worker_machine_type="n1-standard-4",
- num_workers=2,
- storage_bucket="test_bucket",
- init_actions_uris=[path],
- metadata={"PIP_PACKAGES": "pyyaml requests pandas openpyxl"},
-)
-```
-
-After upgrading to **v2.x.x** and using **CLUSTER_CONFIG**, it will look like followed:
-
-```python
-path = f"gs://goog-dataproc-initialization-actions-us-central1/python/pip-install.sh"
-
-CLUSTER_CONFIG = ClusterGenerator(
- project_id="test",
- zone="us-central1-a",
- master_machine_type="n1-standard-4",
- worker_machine_type="n1-standard-4",
- num_workers=2,
- storage_bucket="test",
- init_actions_uris=[path],
- metadata={"PIP_PACKAGES": "pyyaml requests pandas openpyxl"},
-).make()
-
-create_cluster_operator = DataprocClusterCreateOperator(
- task_id="create_dataproc_cluster",
- cluster_name="test",
- project_id="test",
- region="us-central1",
- cluster_config=CLUSTER_CONFIG,
-)
-```
-
-#### `airflow.providers.google.cloud.operators.bigquery.BigQueryGetDatasetTablesOperator`
-
-We changed signature of `BigQueryGetDatasetTablesOperator`.
-
-Before:
-
-```python
-def __init__(
- dataset_id: str,
- dataset_resource: dict,
- # ...
-):
- ...
-```
-
-After:
-
-```python
-def __init__(
- dataset_resource: dict,
- dataset_id: Optional[str] = None,
- # ...
-):
- ...
-```
-
-### Changes in `amazon` provider package
-
-We strive to ensure that there are no changes that may affect the end user, and your Python files, but this
-release may contain changes that will require changes to your configuration, DAG Files or other integration
-e.g. custom operators.
-
-Only changes unique to this provider are described here. You should still pay attention to the changes that
-have been made to the core (including core operators) as they can affect the integration behavior
-of this provider.
-
-This section describes the changes that have been made, and what you need to do to update your if
-you use operators or hooks which integrate with Amazon services (including Amazon Web Service - AWS).
-
-#### Migration of AWS components
-
-All AWS components (hooks, operators, sensors, example DAGs) will be grouped together as decided in
-[AIP-21](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths). Migrated
-components remain backwards compatible but raise a `DeprecationWarning` when imported from the old module.
-Migrated are:
-
-| Old path | New path |
-| ------------------------------------------------------------ | -------------------------------------------------------- |
-| airflow.hooks.S3_hook.S3Hook | airflow.providers.amazon.aws.hooks.s3.S3Hook |
-| airflow.contrib.hooks.aws_athena_hook.AWSAthenaHook | airflow.providers.amazon.aws.hooks.athena.AWSAthenaHook |
-| airflow.contrib.hooks.aws_lambda_hook.AwsLambdaHook | airflow.providers.amazon.aws.hooks.lambda_function.AwsLambdaHook |
-| airflow.contrib.hooks.aws_sqs_hook.SQSHook | airflow.providers.amazon.aws.hooks.sqs.SQSHook |
-| airflow.contrib.hooks.aws_sns_hook.AwsSnsHook | airflow.providers.amazon.aws.hooks.sns.AwsSnsHook |
-| airflow.contrib.operators.aws_athena_operator.AWSAthenaOperator | airflow.providers.amazon.aws.operators.athena.AWSAthenaOperator |
-| airflow.contrib.operators.awsbatch.AWSBatchOperator | airflow.providers.amazon.aws.operators.batch.AwsBatchOperator |
-| airflow.contrib.operators.awsbatch.BatchProtocol | airflow.providers.amazon.aws.hooks.batch_client.AwsBatchProtocol |
-| private attrs and methods on AWSBatchOperator | airflow.providers.amazon.aws.hooks.batch_client.AwsBatchClient |
-| n/a | airflow.providers.amazon.aws.hooks.batch_waiters.AwsBatchWaiters |
-| airflow.contrib.operators.aws_sqs_publish_operator.SQSPublishOperator | airflow.providers.amazon.aws.operators.sqs.SQSPublishOperator |
-| airflow.contrib.operators.aws_sns_publish_operator.SnsPublishOperator | airflow.providers.amazon.aws.operators.sns.SnsPublishOperator |
-| airflow.contrib.sensors.aws_athena_sensor.AthenaSensor | airflow.providers.amazon.aws.sensors.athena.AthenaSensor |
-| airflow.contrib.sensors.aws_sqs_sensor.SQSSensor | airflow.providers.amazon.aws.sensors.sqs.SQSSensor |
-
-#### `airflow.providers.amazon.aws.hooks.emr.EmrHook`
-
-#### `airflow.providers.amazon.aws.operators.emr_add_steps.EmrAddStepsOperator`
-
-#### `airflow.providers.amazon.aws.operators.emr_create_job_flow.EmrCreateJobFlowOperator`
-
-#### `airflow.providers.amazon.aws.operators.emr_terminate_job_flow.EmrTerminateJobFlowOperator`
-
-The default value for the [aws_conn_id](https://airflow.apache.org/howto/manage-connections.html#amazon-web-services) was accidentally set to 's3_default' instead of 'aws_default' in some of the emr operators in previous
-versions. This was leading to EmrStepSensor not being able to find their corresponding emr cluster. With the new
-changes in the EmrAddStepsOperator, EmrTerminateJobFlowOperator and EmrCreateJobFlowOperator this issue is
-solved.
-
-#### `airflow.providers.amazon.aws.operators.batch.AwsBatchOperator`
-
-The `AwsBatchOperator` was refactored to extract an `AwsBatchClient` (and inherit from it). The
-changes are mostly backwards compatible and clarify the public API for these classes; some
-private methods on `AwsBatchOperator` for polling a job status were relocated and renamed
-to surface new public methods on `AwsBatchClient` (and via inheritance on `AwsBatchOperator`). A
-couple of job attributes are renamed on an instance of `AwsBatchOperator`; these were mostly
-used like private attributes but they were surfaced in the public API, so any use of them needs
-to be updated as follows:
-
-- `AwsBatchOperator().jobId` -> `AwsBatchOperator().job_id`
-- `AwsBatchOperator().jobName` -> `AwsBatchOperator().job_name`
-
-The `AwsBatchOperator` gets a new option to define a custom model for waiting on job status changes.
-The `AwsBatchOperator` can use a new `waiters` parameter, an instance of `AwsBatchWaiters`, to
-specify that custom job waiters will be used to monitor a batch job. See the latest API
-documentation for details.
-
-#### `airflow.providers.amazon.aws.sensors.athena.AthenaSensor`
-
-Replace parameter `max_retires` with `max_retries` to fix typo.
-
-#### `airflow.providers.amazon.aws.hooks.s3.S3Hook`
-
-Note: The order of arguments has changed for `check_for_prefix`.
-The `bucket_name` is now optional. It falls back to the `connection schema` attribute.
-The `delete_objects` now returns `None` instead of a response, since the method now makes multiple api requests when the keys list length is > 1000.
-
-### Changes in other provider packages
-
-We strive to ensure that there are no changes that may affect the end user and your Python files, but this
-release may contain changes that will require changes to your configuration, DAG Files or other integration
-e.g. custom operators.
-
-Only changes unique to providers are described here. You should still pay attention to the changes that
-have been made to the core (including core operators) as they can affect the integration behavior
-of this provider.
-
-This section describes the changes that have been made, and what you need to do to update your if
-you use any code located in `airflow.providers` package.
-
-#### Changed return type of `list_prefixes` and `list_keys` methods in `S3Hook`
-
-Previously, the `list_prefixes` and `list_keys` methods returned `None` when there were no
-results. The behavior has been changed to return an empty list instead of `None` in this
-case.
-
-#### Removed HipChat integration
-
-HipChat has reached end of life and is no longer available.
-
-For more information please see
-https://community.atlassian.com/t5/Stride-articles/Stride-and-Hipchat-Cloud-have-reached-End-of-Life-updated/ba-p/940248
-
-#### `airflow.providers.salesforce.hooks.salesforce.SalesforceHook`
-
-Replace parameter ``sandbox`` with ``domain``. According to change in simple-salesforce package.
-
-Rename `sign_in` function to `get_conn`.
-
-#### `airflow.providers.apache.pinot.hooks.pinot.PinotAdminHook.create_segment`
-
-Rename parameter name from ``format`` to ``segment_format`` in PinotAdminHook function create_segment for pylint compatible
-
-#### `airflow.providers.apache.hive.hooks.hive.HiveMetastoreHook.get_partitions`
-
-Rename parameter name from ``filter`` to ``partition_filter`` in HiveMetastoreHook function get_partitions for pylint compatible
-
-#### `airflow.providers.ftp.hooks.ftp.FTPHook.list_directory`
-
-Remove unnecessary parameter ``nlst`` in FTPHook function ``list_directory`` for pylint compatible
-
-#### `airflow.providers.postgres.hooks.postgres.PostgresHook.copy_expert`
-
-Remove unnecessary parameter ``open`` in PostgresHook function ``copy_expert`` for pylint compatible
-
-#### `airflow.providers.opsgenie.operators.opsgenie_alert.OpsgenieAlertOperator`
-
-Change parameter name from ``visibleTo`` to ``visible_to`` in OpsgenieAlertOperator for pylint compatible
-
-#### `airflow.providers.imap.hooks.imap.ImapHook`
-
-#### `airflow.providers.imap.sensors.imap_attachment.ImapAttachmentSensor`
-
-ImapHook:
-
-* The order of arguments has changed for `has_mail_attachment`,
-`retrieve_mail_attachments` and `download_mail_attachments`.
-* A new `mail_filter` argument has been added to each of those.
-
-
-#### `airflow.providers.http.hooks.http.HttpHook`
-
-The HTTPHook is now secured by default: `verify=True` (before: `verify=False`)
-This can be overwritten by using the extra_options param as `{'verify': False}`.
-
-#### `airflow.providers.cloudant.hooks.cloudant.CloudantHook`
-
-* upgraded cloudant version from `>=0.5.9,<2.0` to `>=2.0`
-* removed the use of the `schema` attribute in the connection
-* removed `db` function since the database object can also be retrieved by calling `cloudant_session['database_name']`
-
-For example:
-
-```python
-from airflow.providers.cloudant.hooks.cloudant import CloudantHook
-
-with CloudantHook().get_conn() as cloudant_session:
- database = cloudant_session["database_name"]
-```
-
-See the [docs](https://python-cloudant.readthedocs.io/en/latest/) for more information on how to use the new cloudant version.
-
-#### `airflow.providers.snowflake`
-
-When initializing a Snowflake hook or operator, the value used for `snowflake_conn_id` was always `snowflake_conn_id`, regardless of whether or not you specified a value for it. The default `snowflake_conn_id` value is now switched to `snowflake_default` for consistency and will be properly overridden when specified.
-
-### Other changes
-
-This release also includes changes that fall outside any of the sections above.
-
-#### Standardized "extra" requirements
-
-We standardized the Extras names and synchronized providers package names with the main airflow extras.
-
-We deprecated a number of extras in 2.0.
-
-| Deprecated extras | New extras |
-|---------------------------|------------------|
-| atlas | apache.atlas |
-| aws | amazon |
-| azure | microsoft.azure |
-| azure_blob_storage | microsoft.azure |
-| azure_data_lake | microsoft.azure |
-| azure_cosmos | microsoft.azure |
-| azure_container_instances | microsoft.azure |
-| cassandra | apache.cassandra |
-| druid | apache.druid |
-| gcp | google |
-| gcp_api | google |
-| hdfs | apache.hdfs |
-| hive | apache.hive |
-| kubernetes | cncf.kubernetes |
-| mssql | microsoft.mssql |
-| pinot | apache.pinot |
-| webhdfs | apache.webhdfs |
-| winrm | apache.winrm |
-
-For example:
-
-If you want to install integration for Apache Atlas, then instead of `pip install apache-airflow[atlas]`
-you should use `pip install apache-airflow[apache.atlas]`.
-
-
-NOTE!
-
-If you want to install integration for Microsoft Azure, then instead of
-
-```
-pip install 'apache-airflow[azure_blob_storage,azure_data_lake,azure_cosmos,azure_container_instances]'
-```
-
-you should run `pip install 'apache-airflow[microsoft.azure]'`
-
-If you want to install integration for Amazon Web Services, then instead of
-`pip install 'apache-airflow[s3,emr]'`, you should execute `pip install 'apache-airflow[aws]'`
-
-The deprecated extras will be removed in 3.0.
-
-#### Simplify the response payload of endpoints /dag_stats and /task_stats
-
-The response of endpoints `/dag_stats` and `/task_stats` help UI fetch brief statistics about DAGs and Tasks. The format was like
-
-```json
-{
- "example_http_operator": [
- {
- "state": "success",
- "count": 0,
- "dag_id": "example_http_operator",
- "color": "green"
- },
- {
- "state": "running",
- "count": 0,
- "dag_id": "example_http_operator",
- "color": "lime"
- },
- ...
-],
-...
-}
-```
-
-The `dag_id` was repeated in the payload, which makes the response payload unnecessarily bigger.
-
-Now the `dag_id` will not appear repeated in the payload, and the response format is like
-
-```json
-{
- "example_http_operator": [
- {
- "state": "success",
- "count": 0,
- "color": "green"
- },
- {
- "state": "running",
- "count": 0,
- "color": "lime"
- },
- ...
-],
-...
-}
-```
-
-## Airflow 1.10.15
-
-No breaking changes.
-
-## Airflow 1.10.14
-
-### `[scheduler] max_threads` config has been renamed to `[scheduler] parsing_processes`
-
-From Airflow 1.10.14, `max_threads` config under `[scheduler]` section has been renamed to `parsing_processes`.
-
-This is to align the name with the actual code where the Scheduler launches the number of processes defined by
-`[scheduler] parsing_processes` to parse the DAG files.
-
-### Airflow CLI changes in line with 2.0
-
-The Airflow CLI has been organized so that related commands are grouped together as subcommands,
-which means that if you use these commands in your scripts, they will now raise a DeprecationWarning and
-you have to make changes to them before you upgrade to Airflow 2.0.
-
-This section describes the changes that have been made, and what you need to do to update your script.
-
-The ability to manipulate users from the command line has been changed. ``airflow create_user``, ``airflow delete_user``
- and ``airflow list_users`` has been grouped to a single command `airflow users` with optional flags `create`, `list` and `delete`.
-
-The `airflow list_dags` command is now `airflow dags list`, `airflow pause` is `airflow dags pause`, etc.
-
-In Airflow 1.10 and 2.0 there is an `airflow config` command but there is a difference in behavior. In Airflow 1.10,
-it prints all config options while in Airflow 2.0, it's a command group. `airflow config` is now `airflow config list`.
-You can check other options by running the command `airflow config --help`
-
-Compatibility with the old CLI has been maintained, but they will no longer appear in the help
-
-You can learn about the commands by running ``airflow --help``. For example to get help about the ``celery`` group command,
-you have to run the help command: ``airflow celery --help``.
-
-| Old command | New command | Group |
-|-------------------------------|------------------------------------|--------------------|
-| ``airflow worker`` | ``airflow celery worker`` | ``celery`` |
-| ``airflow flower`` | ``airflow celery flower`` | ``celery`` |
-| ``airflow trigger_dag`` | ``airflow dags trigger`` | ``dags`` |
-| ``airflow delete_dag`` | ``airflow dags delete`` | ``dags`` |
-| ``airflow show_dag`` | ``airflow dags show`` | ``dags`` |
-| ``airflow list_dag`` | ``airflow dags list`` | ``dags`` |
-| ``airflow dag_status`` | ``airflow dags status`` | ``dags`` |
-| ``airflow backfill`` | ``airflow dags backfill`` | ``dags`` |
-| ``airflow list_dag_runs`` | ``airflow dags list-runs`` | ``dags`` |
-| ``airflow pause`` | ``airflow dags pause`` | ``dags`` |
-| ``airflow unpause`` | ``airflow dags unpause`` | ``dags`` |
-| ``airflow next_execution`` | ``airflow dags next-execution`` | ``dags`` |
-| ``airflow test`` | ``airflow tasks test`` | ``tasks`` |
-| ``airflow clear`` | ``airflow tasks clear`` | ``tasks`` |
-| ``airflow list_tasks`` | ``airflow tasks list`` | ``tasks`` |
-| ``airflow task_failed_deps`` | ``airflow tasks failed-deps`` | ``tasks`` |
-| ``airflow task_state`` | ``airflow tasks state`` | ``tasks`` |
-| ``airflow run`` | ``airflow tasks run`` | ``tasks`` |
-| ``airflow render`` | ``airflow tasks render`` | ``tasks`` |
-| ``airflow initdb`` | ``airflow db init`` | ``db`` |
-| ``airflow resetdb`` | ``airflow db reset`` | ``db`` |
-| ``airflow upgradedb`` | ``airflow db upgrade`` | ``db`` |
-| ``airflow checkdb`` | ``airflow db check`` | ``db`` |
-| ``airflow shell`` | ``airflow db shell`` | ``db`` |
-| ``airflow pool`` | ``airflow pools`` | ``pools`` |
-| ``airflow create_user`` | ``airflow users create`` | ``users`` |
-| ``airflow delete_user`` | ``airflow users delete`` | ``users`` |
-| ``airflow list_users`` | ``airflow users list`` | ``users`` |
-| ``airflow rotate_fernet_key`` | ``airflow rotate-fernet-key`` | |
-| ``airflow sync_perm`` | ``airflow sync-perm`` | |
-
-## Airflow 1.10.13
-
-### TimeSensor is now timezone aware
-
-Previously `TimeSensor` always compared the `target_time` with the current time in UTC.
-
-Now it will compare `target_time` with the current time in the timezone of the DAG,
-defaulting to the `default_timezone` in the global config.
-
-### Removed Kerberos support for HDFS hook
-
-The HDFS hook's Kerberos support has been removed due to removed `python-krbV` dependency from PyPI
-and generally lack of support for SSL in Python3 (Snakebite-py3 we use as dependency has no
-support for SSL connection to HDFS).
-
-SSL support still works for WebHDFS hook.
-
-### Unify user session lifetime configuration
-
-In previous version of Airflow user session lifetime could be configured by
-`session_lifetime_days` and `force_log_out_after` options. In practice only `session_lifetime_days`
-had impact on session lifetime, but it was limited to values in day.
-We have removed mentioned options and introduced new `session_lifetime_minutes`
-option which simplify session lifetime configuration.
-
-Before
-
- ```ini
-[webserver]
-force_log_out_after = 0
-session_lifetime_days = 30
- ```
-
-After
-
- ```ini
-[webserver]
-session_lifetime_minutes = 43200
- ```
-
-### Adding Operators, Hooks and Sensors via Airflow Plugins is deprecated
-
-The ability to import Operators, Hooks and Sensors via the plugin mechanism has been deprecated and will raise warnings
-in Airflow 1.10.13 and will be removed completely in Airflow 2.0.
-
-Check https://airflow.apache.org/docs/1.10.13/howto/custom-operator.html to see how you can create and import
-Custom Hooks, Operators and Sensors.
-
-## Airflow 1.10.12
-
-### Clearing tasks skipped by SkipMixin will skip them
-
-Previously, when tasks skipped by SkipMixin (such as BranchPythonOperator, BaseBranchOperator and ShortCircuitOperator) are cleared, they execute. Since 1.10.12, when such skipped tasks are cleared,
-they will be skipped again by the newly introduced NotPreviouslySkippedDep.
-
-### The pod_mutation_hook function will now accept a kubernetes V1Pod object
-
-As of airflow 1.10.12, using the `airflow.contrib.kubernetes.Pod` class in the `pod_mutation_hook` is now deprecated. Instead we recommend that users
-treat the `pod` parameter as a `kubernetes.client.models.V1Pod` object. This means that users now have access to the full Kubernetes API
-when modifying airflow pods
-
-### pod_template_file option now available in the KubernetesPodOperator
-
-Users can now offer a path to a yaml for the KubernetesPodOperator using the `pod_template_file` parameter.
-
-## Airflow 1.10.11
-
-### Use NULL as default value for dag.description
-
-Now use NULL as default value for dag.description in dag table
-
-### Restrict editing DagRun State in the old UI (Flask-admin based UI)
-
-Before 1.10.11 it was possible to edit DagRun State in the `/admin/dagrun/` page
- to any text.
-
-In Airflow 1.10.11+, the user can only choose the states from the list.
-
-### Experimental API will deny all request by default.
-
-The previous default setting was to allow all API requests without authentication, but this poses security
-risks to users who miss this fact. This changes the default for new installs to deny all requests by default.
-
-**Note**: This will not change the behavior for existing installs, please update check your airflow.cfg
-
-If you wish to have the experimental API work, and aware of the risks of enabling this without authentication
-(or if you have your own authentication layer in front of Airflow) you can get
-the previous behaviour on a new install by setting this in your airflow.cfg:
-
-```
-[api]
-auth_backend = airflow.api.auth.backend.default
-```
-
-### XCom Values can no longer be added or changed from the Webserver
-
-Since XCom values can contain pickled data, we would no longer allow adding or
-changing XCom values from the UI.
-
-### Default for `run_as_user` configured has been changed to 50000 from 0
-
-The UID to run the first process of the Worker PODs when using has been changed to `50000`
-from the previous default of `0`. The previous default was an empty string but the code used `0` if it was
-empty string.
-
-**Before**:
-
-```ini
-[kubernetes]
-run_as_user =
-```
-
-**After**:
-
-```ini
-[kubernetes]
-run_as_user = 50000
-```
-
-This is done to avoid running the container as `root` user.
-
-## Airflow 1.10.10
-
-### Setting Empty string to a Airflow Variable will return an empty string
-
-Previously when you set an Airflow Variable with an empty string (`''`), the value you used to get
-back was ``None``. This will now return an empty string (`'''`)
-
-Example:
-
-```python
-Variable.set("test_key", "")
-Variable.get("test_key")
-```
-
-The above code returned `None` previously, now it will return `''`.
-
-### Make behavior of `none_failed` trigger rule consistent with documentation
-
-The behavior of the `none_failed` trigger rule is documented as "all parents have not failed (`failed` or
- `upstream_failed`) i.e. all parents have succeeded or been skipped." As previously implemented, the actual behavior
- would skip if all parents of a task had also skipped.
-
-### Add new trigger rule `none_failed_or_skipped`
-
-The fix to `none_failed` trigger rule breaks workflows that depend on the previous behavior.
- If you need the old behavior, you should change the tasks with `none_failed` trigger rule to `none_failed_or_skipped`.
-
-### Success Callback will be called when a task in marked as success from UI
-
-When a task is marked as success by a user from Airflow UI - `on_success_callback` will be called
-
-## Airflow 1.10.9
-
-No breaking changes.
-
-## Airflow 1.10.8
-
-### Failure callback will be called when task is marked failed
-
-When task is marked failed by user or task fails due to system failures - on failure call back will be called as part of clean up
-
-See [AIRFLOW-5621](https://jira.apache.org/jira/browse/AIRFLOW-5621) for details
-
-## Airflow 1.10.7
-
-### Changes in experimental API execution_date microseconds replacement
-
-The default behavior was to strip the microseconds (and milliseconds, etc) off of all dag runs triggered by
-by the experimental REST API. The default behavior will change when an explicit execution_date is
-passed in the request body. It will also now be possible to have the execution_date generated, but
-keep the microseconds by sending `replace_microseconds=false` in the request body. The default
-behavior can be overridden by sending `replace_microseconds=true` along with an explicit execution_date
-
-### Infinite pool size and pool size query optimization
-
-Pool size can now be set to -1 to indicate infinite size (it also includes
-optimization of pool query which lead to poor task n^2 performance of task
-pool queries in MySQL).
-
-### Viewer won't have edit permissions on DAG view.
-
-### Google Cloud Storage Hook
-
-The `GoogleCloudStorageDownloadOperator` can either write to a supplied `filename` or
-return the content of a file via xcom through `store_to_xcom_key` - both options are mutually exclusive.
-
-## Airflow 1.10.6
-
-### BaseOperator::render_template function signature changed
-
-Previous versions of the `BaseOperator::render_template` function required an `attr` argument as the first
-positional argument, along with `content` and `context`. This function signature was changed in 1.10.6 and
-the `attr` argument is no longer required (or accepted).
-
-In order to use this function in subclasses of the `BaseOperator`, the `attr` argument must be removed:
-
-```python
-result = self.render_template("myattr", self.myattr, context) # Pre-1.10.6 call
-# ...
-result = self.render_template(self.myattr, context) # Post-1.10.6 call
-```
-
-### Changes to `aws_default` Connection's default region
-
-The region of Airflow's default connection to AWS (`aws_default`) was previously
-set to `us-east-1` during installation.
-
-The region now needs to be set manually, either in the connection screens in
-Airflow, via the `~/.aws` config files, or via the `AWS_DEFAULT_REGION` environment
-variable.
-
-### Some DAG Processing metrics have been renamed
-
-The following metrics are deprecated and won't be emitted in Airflow 2.0:
-
-- `scheduler.dagbag.errors` and `dagbag_import_errors` -- use `dag_processing.import_errors` instead
-- `dag_file_processor_timeouts` -- use `dag_processing.processor_timeouts` instead
-- `collect_dags` -- use `dag_processing.total_parse_time` instead
-- `dag.loading-duration.` -- use `dag_processing.last_duration.` instead
-- `dag_processing.last_runtime.` -- use `dag_processing.last_duration.` instead
-
-## Airflow 1.10.5
-
-No breaking changes.
-
-## Airflow 1.10.4
-
-### Export MySQL timestamps as UTC
-
-`MySqlToGoogleCloudStorageOperator` now exports TIMESTAMP columns as UTC
-by default, rather than using the default timezone of the MySQL server.
-This is the correct behavior for use with BigQuery, since BigQuery
-assumes that TIMESTAMP columns without time zones are in UTC. To
-preserve the previous behavior, set `ensure_utc` to `False.`
-
-### Changes to DatastoreHook
-
-* removed argument `version` from `get_conn` function and added it to the hook's `__init__` function instead and renamed it to `api_version`
-* renamed the `partialKeys` argument of function `allocate_ids` to `partial_keys`
-
-### Changes to GoogleCloudStorageHook
-
-* the discovery-based api (`googleapiclient.discovery`) used in `GoogleCloudStorageHook` is now replaced by the recommended client based api (`google-cloud-storage`). To know the difference between both the libraries, read https://cloud.google.com/apis/docs/client-libraries-explained. PR: [#5054](https://github.com/apache/airflow/pull/5054)
-* as a part of this replacement, the `multipart` & `num_retries` parameters for `GoogleCloudStorageHook.upload` method have been deprecated.
-
- The client library uses multipart upload automatically if the object/blob size is more than 8 MB - [source code](https://github.com/googleapis/google-cloud-python/blob/11c543ce7dd1d804688163bc7895cf592feb445f/storage/google/cloud/storage/blob.py#L989-L997). The client also handles retries automatically
-
-* the `generation` parameter is deprecated in `GoogleCloudStorageHook.delete` and `GoogleCloudStorageHook.insert_object_acl`.
-
-Updating to `google-cloud-storage >= 1.16` changes the signature of the upstream `client.get_bucket()` method from `get_bucket(bucket_name: str)` to `get_bucket(bucket_or_name: Union[str, Bucket])`. This method is not directly exposed by the airflow hook, but any code accessing the connection directly (`GoogleCloudStorageHook().get_conn().get_bucket(...)` or similar) will need to be updated.
-
-### Changes in writing Logs to Elasticsearch
-
-The `elasticsearch_` prefix has been removed from all config items under the `[elasticsearch]` section. For example `elasticsearch_host` is now just `host`.
-
-### Removal of `non_pooled_task_slot_count` and `non_pooled_backfill_task_slot_count`
-
-`non_pooled_task_slot_count` and `non_pooled_backfill_task_slot_count`
-are removed in favor of a real pool, e.g. `default_pool`.
-
-By default tasks are running in `default_pool`.
-`default_pool` is initialized with 128 slots and user can change the
-number of slots through UI/CLI. `default_pool` cannot be removed.
-
-### `pool` config option in Celery section to support different Celery pool implementation
-
-The new `pool` config option allows users to choose different pool
-implementation. Default value is "prefork", while choices include "prefork" (default),
-"eventlet", "gevent" or "solo". This may help users achieve better concurrency performance
-in different scenarios.
-
-For more details about Celery pool implementation, please refer to:
-
-- https://docs.celeryproject.org/en/latest/userguide/workers.html#concurrency
-- https://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html
-
-
-### Change to method signature in `BaseOperator` and `DAG` classes
-
-The signature of the `get_task_instances` method in the `BaseOperator` and `DAG` classes has changed. The change does not change the behavior of the method in either case.
-
-#### For `BaseOperator`
-
-Old signature:
-
-```python
-def get_task_instances(self, session, start_date=None, end_date=None):
- ...
-```
-
-New signature:
-
-```python
-@provide_session
-def get_task_instances(self, start_date=None, end_date=None, session=None):
- ...
-```
-
-#### For `DAG`
-
-Old signature:
-
-```python
-def get_task_instances(self, session, start_date=None, end_date=None, state=None):
- ...
-```
-
-New signature:
-
-```python
-@provide_session
-def get_task_instances(self, start_date=None, end_date=None, state=None, session=None):
- ...
-```
-
-In either case, it is necessary to rewrite calls to the `get_task_instances` method that currently provide the `session` positional argument. New calls to this method look like:
-
-```python
-# if you can rely on @provide_session
-dag.get_task_instances()
-# if you need to provide the session
-dag.get_task_instances(session=your_session)
-```
-
-## Airflow 1.10.3
-
-### New `dag_discovery_safe_mode` config option
-
-If `dag_discovery_safe_mode` is enabled, only check files for DAGs if
-they contain the strings "airflow" and "DAG". For backwards
-compatibility, this option is enabled by default.
-
-### RedisPy dependency updated to v3 series
-
-If you are using the Redis Sensor or Hook you may have to update your code. See
-[redis-py porting instructions] to check if your code might be affected (MSET,
-MSETNX, ZADD, and ZINCRBY all were, but read the full doc).
-
-[redis-py porting instructions]: https://github.com/andymccurdy/redis-py/tree/3.2.0#upgrading-from-redis-py-2x-to-30
-
-### SLUGIFY_USES_TEXT_UNIDECODE or AIRFLOW_GPL_UNIDECODE no longer required
-
-It is no longer required to set one of the environment variables to avoid
-a GPL dependency. Airflow will now always use `text-unidecode` if `unidecode`
-was not installed before.
-
-### New `sync_parallelism` config option in `[celery]` section
-
-The new `sync_parallelism` config option will control how many processes CeleryExecutor will use to
-fetch celery task state in parallel. Default value is max(1, number of cores - 1)
-
-### Rename of BashTaskRunner to StandardTaskRunner
-
-BashTaskRunner has been renamed to StandardTaskRunner. It is the default task runner
-so you might need to update your config.
-
-`task_runner = StandardTaskRunner`
-
-### Modification to config file discovery
-
-If the `AIRFLOW_CONFIG` environment variable was not set and the
-`~/airflow/airflow.cfg` file existed, airflow previously used
-`~/airflow/airflow.cfg` instead of `$AIRFLOW_HOME/airflow.cfg`. Now airflow
-will discover its config file using the `$AIRFLOW_CONFIG` and `$AIRFLOW_HOME`
-environment variables rather than checking for the presence of a file.
-
-### Changes in Google Cloud related operators
-
-Most GCP-related operators have now optional `PROJECT_ID` parameter. In case you do not specify it,
-the project id configured in
-[GCP Connection](https://airflow.apache.org/howto/manage-connections.html#connection-type-gcp) is used.
-There will be an `AirflowException` thrown in case `PROJECT_ID` parameter is not specified and the
-connection used has no project id defined. This change should be backwards compatible as earlier version
-of the operators had `PROJECT_ID` mandatory.
-
-Operators involved:
-
- * GCP Compute Operators
- * GceInstanceStartOperator
- * GceInstanceStopOperator
- * GceSetMachineTypeOperator
- * GCP Function Operators
- * GcfFunctionDeployOperator
- * GCP Cloud SQL Operators
- * CloudSqlInstanceCreateOperator
- * CloudSqlInstancePatchOperator
- * CloudSqlInstanceDeleteOperator
- * CloudSqlInstanceDatabaseCreateOperator
- * CloudSqlInstanceDatabasePatchOperator
- * CloudSqlInstanceDatabaseDeleteOperator
-
-Other GCP operators are unaffected.
-
-### Changes in Google Cloud related hooks
-
-The change in GCP operators implies that GCP Hooks for those operators require now keyword parameters rather
-than positional ones in all methods where `project_id` is used. The methods throw an explanatory exception
-in case they are called using positional parameters.
-
-Hooks involved:
-
- * GceHook
- * GcfHook
- * CloudSqlHook
-
-Other GCP hooks are unaffected.
-
-### Changed behaviour of using default value when accessing variables
-
-It's now possible to use `None` as a default value with the `default_var` parameter when getting a variable, e.g.
-
-```python
-foo = Variable.get("foo", default_var=None)
-if foo is None:
- handle_missing_foo()
-```
-
-(Note: there is already `Variable.setdefault()` which me be helpful in some cases.)
-
-This changes the behaviour if you previously explicitly provided `None` as a default value. If your code expects a `KeyError` to be thrown, then don't pass the `default_var` argument.
-
-### Removal of `airflow_home` config setting
-
-There were previously two ways of specifying the Airflow "home" directory
-(`~/airflow` by default): the `AIRFLOW_HOME` environment variable, and the
-`airflow_home` config setting in the `[core]` section.
-
-If they had two different values different parts of the code base would end up
-with different values. The config setting has been deprecated, and you should
-remove the value from the config file and set `AIRFLOW_HOME` environment
-variable if you need to use a non default value for this.
-
-(Since this setting is used to calculate what config file to load, it is not
-possible to keep just the config option)
-
-### Change of two methods signatures in `GCPTransferServiceHook`
-
-The signature of the `create_transfer_job` method in `GCPTransferServiceHook`
-class has changed. The change does not change the behavior of the method.
-
-Old signature:
-
-```python
-def create_transfer_job(self, description, schedule, transfer_spec, project_id=None):
- ...
-```
-
-New signature:
-
-```python
-def create_transfer_job(self, body):
- ...
-```
-
-It is necessary to rewrite calls to method. The new call looks like this:
-
-```python
-body = {
- "status": "ENABLED",
- "projectId": project_id,
- "description": description,
- "transferSpec": transfer_spec,
- "schedule": schedule,
-}
-gct_hook.create_transfer_job(body)
-```
-
-The change results from the unification of all hooks and adjust to
-[the official recommendations](https://lists.apache.org/thread.html/e8534d82be611ae7bcb21ba371546a4278aad117d5e50361fd8f14fe@%3Cdev.airflow.apache.org%3E)
-for the Google Cloud.
-
-The signature of `wait_for_transfer_job` method in `GCPTransferServiceHook` has changed.
-
-Old signature:
-
-```python
-def wait_for_transfer_job(self, job):
- ...
-```
-
-New signature:
-
-```python
-def wait_for_transfer_job(
- self, job, expected_statuses=(GcpTransferOperationStatus.SUCCESS,)
-):
- ...
-```
-
-The behavior of `wait_for_transfer_job` has changed:
-
-Old behavior:
-
-`wait_for_transfer_job` would wait for the SUCCESS status in specified jobs operations.
-
-New behavior:
-
-You can now specify an array of expected statuses. `wait_for_transfer_job` now waits for any of them.
-
-The default value of `expected_statuses` is SUCCESS so that change is backwards compatible.
-
-### Moved two classes to different modules
-
-The class `GoogleCloudStorageToGoogleCloudStorageTransferOperator` has been moved from
-`airflow.contrib.operators.gcs_to_gcs_transfer_operator` to `airflow.contrib.operators.gcp_transfer_operator`
-
-the class `S3ToGoogleCloudStorageTransferOperator` has been moved from
-`airflow.contrib.operators.s3_to_gcs_transfer_operator` to `airflow.contrib.operators.gcp_transfer_operator`
-
-The change was made to keep all the operators related to GCS Transfer Services in one file.
-
-The previous imports will continue to work until Airflow 2.0
-
-### Fixed typo in --driver-class-path in SparkSubmitHook
-
-The `driver_classapth` argument to SparkSubmit Hook and Operator was
-generating `--driver-classpath` on the spark command line, but this isn't a
-valid option to spark.
-
-The argument has been renamed to `driver_class_path` and the option it
-generates has been fixed.
-
-## Airflow 1.10.2
-
-### New `dag_processor_manager_log_location` config option
-
-The DAG parsing manager log now by default will be log into a file, where its location is
-controlled by the new `dag_processor_manager_log_location` config option in core section.
-
-### DAG level Access Control for new RBAC UI
-
-Extend and enhance new Airflow RBAC UI to support DAG level ACL. Each dag now has two permissions(one for write, one for read) associated('can_dag_edit', 'can_dag_read').
-The admin will create new role, associate the dag permission with the target dag and assign that role to users. That user can only access / view the certain dags on the UI
-that he has permissions on. If a new role wants to access all the dags, the admin could associate dag permissions on an artificial view(``all_dags``) with that role.
-
-We also provide a new cli command(``sync_perm``) to allow admin to auto sync permissions.
-
-### Modification to `ts_nodash` macro
-
-`ts_nodash` previously contained TimeZone information along with execution date. For Example: `20150101T000000+0000`. This is not user-friendly for file or folder names which was a popular use case for `ts_nodash`. Hence this behavior has been changed and using `ts_nodash` will no longer contain TimeZone information, restoring the pre-1.10 behavior of this macro. And a new macro `ts_nodash_with_tz` has been added which can be used to get a string with execution date and timezone info without dashes.
-
-Examples:
-
- * `ts_nodash`: `20150101T000000`
- * `ts_nodash_with_tz`: `20150101T000000+0000`
-
-### Semantics of next_ds/prev_ds changed for manually triggered runs
-
-next_ds/prev_ds now map to execution_date instead of the next/previous schedule-aligned execution date for DAGs triggered in the UI.
-
-### User model changes
-
-This patch changes the `User.superuser` field from a hard-coded boolean to a `Boolean()` database column. `User.superuser` will default to `False`, which means that this privilege will have to be granted manually to any users that may require it.
-
-For example, open a Python shell and
-
-```python
-from airflow import models, settings
-
-session = settings.Session()
-users = session.query(models.User).all() # [admin, regular_user]
-
-users[1].superuser # False
-
-admin = users[0]
-admin.superuser = True
-session.add(admin)
-session.commit()
-```
-
-### Custom auth backends interface change
-
-We have updated the version of flask-login we depend upon, and as a result any
-custom auth backends might need a small change: `is_active`,
-`is_authenticated`, and `is_anonymous` should now be properties. What this means is if
-previously you had this in your user class
-
-```python
-def is_active(self):
- return self.active
-```
-
-then you need to change it like this
-
-```python
-@property
-def is_active(self):
- return self.active
-```
-
-### Support autodetected schemas to GoogleCloudStorageToBigQueryOperator
-
-GoogleCloudStorageToBigQueryOperator is now support schema auto-detection is available when you load data into BigQuery. Unfortunately, changes can be required.
-
-If BigQuery tables are created outside of airflow and the schema is not defined in the task, multiple options are available:
-
-define a schema_fields:
-
-```python
-gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
- # ...
- schema_fields={...}
-)
-```
-
-or define a schema_object:
-
-```python
-gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
- # ...
- schema_object="path/to/schema/object"
-)
-```
-
-or enabled autodetect of schema:
-
-```python
-gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
- # ...
- autodetect=True
-)
-```
-
-## Airflow 1.10.1
-
-### min_file_parsing_loop_time config option temporarily disabled
-
-The scheduler.min_file_parsing_loop_time config option has been temporarily removed due to
-some bugs.
-
-### StatsD Metrics
-
-The `scheduler_heartbeat` metric has been changed from a gauge to a counter. Each loop of the scheduler will increment the counter by 1. This provides a higher degree of visibility and allows for better integration with Prometheus using the [StatsD Exporter](https://github.com/prometheus/statsd_exporter). The scheduler's activity status can be determined by graphing and alerting using a rate of change of the counter. If the scheduler goes down, the rate will drop to 0.
-
-### EMRHook now passes all of connection's extra to CreateJobFlow API
-
-EMRHook.create_job_flow has been changed to pass all keys to the create_job_flow API, rather than
-just specific known keys for greater flexibility.
-
-However prior to this release the "emr_default" sample connection that was created had invalid
-configuration, so creating EMR clusters might fail until your connection is updated. (Ec2KeyName,
-Ec2SubnetId, TerminationProtection and KeepJobFlowAliveWhenNoSteps were all top-level keys when they
-should be inside the "Instances" dict)
-
-### LDAP Auth Backend now requires TLS
-
-Connecting to an LDAP server over plain text is not supported anymore. The
-certificate presented by the LDAP server must be signed by a trusted
-certificate, or you must provide the `cacert` option under `[ldap]` in the
-config file.
-
-If you want to use LDAP auth backend without TLS then you will have to create a
-custom-auth backend based on
-https://github.com/apache/airflow/blob/1.10.0/airflow/contrib/auth/backends/ldap_auth.py
-
-## Airflow 1.10
-
-Installation and upgrading requires setting `SLUGIFY_USES_TEXT_UNIDECODE=yes` in your environment or
-`AIRFLOW_GPL_UNIDECODE=yes`. In case of the latter a GPL runtime dependency will be installed due to a
-dependency `(python-nvd3 -> python-slugify -> unidecode)`.
-
-### Replace DataProcHook.await calls to DataProcHook.wait
-
-The method name was changed to be compatible with the Python 3.7 async/await keywords
-
-### Setting UTF-8 as default mime_charset in email utils
-
-### Add a configuration variable(default_dag_run_display_number) to control numbers of dag run for display
-
-Add a configuration variable(default_dag_run_display_number) under webserver section to control the number of dag runs to show in UI.
-
-### Default executor for SubDagOperator is changed to SequentialExecutor
-
-### New Webserver UI with Role-Based Access Control
-
-The current webserver UI uses the Flask-Admin extension. The new webserver UI uses the [Flask-AppBuilder (FAB)](https://github.com/dpgaspar/Flask-AppBuilder) extension. FAB has built-in authentication support and Role-Based Access Control (RBAC), which provides configurable roles and permissions for individual users.
-
-To turn on this feature, in your airflow.cfg file (under [webserver]), set the configuration variable `rbac = True`, and then run `airflow` command, which will generate the `webserver_config.py` file in your $AIRFLOW_HOME.
-
-#### Setting up Authentication
-
-FAB has built-in authentication support for DB, OAuth, OpenID, LDAP, and REMOTE_USER. The default auth type is `AUTH_DB`.
-
-For any other authentication type (OAuth, OpenID, LDAP, REMOTE_USER), see the [Authentication section of FAB docs](http://flask-appbuilder.readthedocs.io/en/latest/security.html#authentication-methods) for how to configure variables in webserver_config.py file.
-
-Once you modify your config file, run `airflow db init` to generate new tables for RBAC support (these tables will have the prefix `ab_`).
-
-#### Creating an Admin Account
-
-Once configuration settings have been updated and new tables have been generated, create an admin account with `airflow create_user` command.
-
-#### Using your new UI
-
-Run `airflow webserver` to start the new UI. This will bring up a log in page, enter the recently created admin username and password.
-
-There are five roles created for Airflow by default: Admin, User, Op, Viewer, and Public. To configure roles/permissions, go to the `Security` tab and click `List Roles` in the new UI.
-
-#### Breaking changes
-
-- AWS Batch Operator renamed property queue to job_queue to prevent conflict with the internal queue from CeleryExecutor - AIRFLOW-2542
-- Users created and stored in the old users table will not be migrated automatically. FAB's built-in authentication support must be reconfigured.
-- Airflow dag home page is now `/home` (instead of `/admin`).
-- All ModelViews in Flask-AppBuilder follow a different pattern from Flask-Admin. The `/admin` part of the URL path will no longer exist. For example: `/admin/connection` becomes `/connection/list`, `/admin/connection/new` becomes `/connection/add`, `/admin/connection/edit` becomes `/connection/edit`, etc.
-- Due to security concerns, the new webserver will no longer support the features in the `Data Profiling` menu of old UI, including `Ad Hoc Query`, `Charts`, and `Known Events`.
-- HiveServer2Hook.get_results() always returns a list of tuples, even when a single column is queried, as per Python API 2.
-- **UTC is now the default timezone**: Either reconfigure your workflows scheduling in UTC or set `default_timezone` as explained in https://airflow.apache.org/timezone.html#default-time-zone
-
-### airflow.contrib.sensors.hdfs_sensors renamed to airflow.contrib.sensors.hdfs_sensor
-
-We now rename airflow.contrib.sensors.hdfs_sensors to airflow.contrib.sensors.hdfs_sensor for consistency purpose.
-
-### MySQL setting required
-
-We now rely on more strict ANSI SQL settings for MySQL in order to have sane defaults. Make sure
-to have specified `explicit_defaults_for_timestamp=1` in your my.cnf under `[mysqld]`
-
-### Celery config
-
-To make the config of Airflow compatible with Celery, some properties have been renamed:
-
-```
-celeryd_concurrency -> worker_concurrency
-celery_result_backend -> result_backend
-celery_ssl_active -> ssl_active
-celery_ssl_cert -> ssl_cert
-celery_ssl_key -> ssl_key
-```
-
-Resulting in the same config parameters as Celery 4, with more transparency.
-
-### GCP Dataflow Operators
-
-Dataflow job labeling is now supported in Dataflow{Java,Python}Operator with a default
-"airflow-version" label, please upgrade your google-cloud-dataflow or apache-beam version
-to 2.2.0 or greater.
-
-### BigQuery Hooks and Operator
-
-The `bql` parameter passed to `BigQueryOperator` and `BigQueryBaseCursor.run_query` has been deprecated and renamed to `sql` for consistency purposes. Using `bql` will still work (and raise a `DeprecationWarning`), but is no longer
-supported and will be removed entirely in Airflow 2.0
-
-### Redshift to S3 Operator
-
-With Airflow 1.9 or lower, Unload operation always included header row. In order to include header row,
-we need to turn off parallel unload. It is preferred to perform unload operation using all nodes so that it is
-faster for larger tables. So, parameter called `include_header` is added and default is set to False.
-Header row will be added only if this parameter is set True and also in that case parallel will be automatically turned off (`PARALLEL OFF`)
-
-### Google cloud connection string
-
-With Airflow 1.9 or lower, there were two connection strings for the Google Cloud operators, both `google_cloud_storage_default` and `google_cloud_default`. This can be confusing and therefore the `google_cloud_storage_default` connection id has been replaced with `google_cloud_default` to make the connection id consistent across Airflow.
-
-### Logging Configuration
-
-With Airflow 1.9 or lower, `FILENAME_TEMPLATE`, `PROCESSOR_FILENAME_TEMPLATE`, `LOG_ID_TEMPLATE`, `END_OF_LOG_MARK` were configured in `airflow_local_settings.py`. These have been moved into the configuration file, and hence if you were using a custom configuration file the following defaults need to be added.
-
-```
-[core]
-fab_logging_level = WARN
-log_filename_template = {{{{ ti.dag_id }}}}/{{{{ ti.task_id }}}}/{{{{ ts }}}}/{{{{ try_number }}}}.log
-log_processor_filename_template = {{{{ filename }}}}.log
-
-[elasticsearch]
-elasticsearch_log_id_template = {{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}
-elasticsearch_end_of_log_mark = end_of_log
-```
-
-The previous setting of `log_task_reader` is not needed in many cases now when using the default logging config with remote storage. (Previously it needed to be set to `s3.task` or similar. This is not needed with the default config anymore)
-
-#### Change of per-task log path
-
-With the change to Airflow core to be timezone aware the default log path for task instances will now include timezone information. This will by default mean all previous task logs won't be found. You can get the old behaviour back by setting the following config options:
-
-```
-[core]
-log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ execution_date.strftime("%%Y-%%m-%%dT%%H:%%M:%%S") }}/{{ try_number }}.log
-```
-
-## Airflow 1.9
-
-### SSH Hook updates, along with new SSH Operator & SFTP Operator
-
-SSH Hook now uses the Paramiko library to create an ssh client connection, instead of the sub-process based ssh command execution previously (<1.9.0), so this is backward incompatible.
-
-- update SSHHook constructor
-- use SSHOperator class in place of SSHExecuteOperator which is removed now. Refer to test_ssh_operator.py for usage info.
-- SFTPOperator is added to perform secure file transfer from server A to server B. Refer to test_sftp_operator.py for usage info.
-- No updates are required if you are using `ftpHook`, it will continue to work as is.
-
-### S3Hook switched to use Boto3
-
-The airflow.hooks.S3_hook.S3Hook has been switched to use boto3 instead of the older boto (a.k.a. boto2). This results in a few backwards incompatible changes to the following classes: S3Hook:
-
-- the constructors no longer accepts `s3_conn_id`. It is now called `aws_conn_id`.
-- the default connection is now "aws_default" instead of "s3_default"
-- the return type of objects returned by `get_bucket` is now boto3.s3.Bucket
-- the return type of `get_key`, and `get_wildcard_key` is now an boto3.S3.Object.
-
-If you are using any of these in your DAGs and specify a connection ID you will need to update the parameter name for the connection to "aws_conn_id": S3ToHiveTransfer, S3PrefixSensor, S3KeySensor, RedshiftToS3Transfer.
-
-### Logging update
-
-The logging structure of Airflow has been rewritten to make configuration easier and the logging system more transparent.
-
-#### A quick recap about logging
-
-A logger is the entry point into the logging system. Each logger is a named bucket to which messages can be written for processing. A logger is configured to have a log level. This log level describes the severity of the messages that the logger will handle. Python defines the following log levels: DEBUG, INFO, WARNING, ERROR or CRITICAL.
-
-Each message that is written to the logger is a Log Record. Each log record contains a log level indicating the severity of that specific message. A log record can also contain useful metadata that describes the event that is being logged. This can include details such as a stack trace or an error code.
-
-When a message is given to the logger, the log level of the message is compared to the log level of the logger. If the log level of the message meets or exceeds the log level of the logger itself, the message will undergo further processing. If it doesn’t, the message will be ignored.
-
-Once a logger has determined that a message needs to be processed, it is passed to a Handler. This configuration is now more flexible and can be easily be maintained in a single file.
-
-#### Changes in Airflow Logging
-
-Airflow's logging mechanism has been refactored to use Python’s built-in `logging` module to perform logging of the application. By extending classes with the existing `LoggingMixin`, all the logging will go through a central logger. Also the `BaseHook` and `BaseOperator` already extend this class, so it is easily available to do logging.
-
-The main benefit is easier configuration of the logging by setting a single centralized python file. Disclaimer; there is still some inline configuration, but this will be removed eventually. The new logging class is defined by setting the dotted classpath in your `~/airflow/airflow.cfg` file:
-
-```
-# Logging class
-# Specify the class that will specify the logging configuration
-# This class has to be on the python classpath
-logging_config_class = my.path.default_local_settings.LOGGING_CONFIG
-```
-
-The logging configuration file needs to be on the `PYTHONPATH`, for example `$AIRFLOW_HOME/config`. This directory is loaded by default. Any directory may be added to the `PYTHONPATH`, this might be handy when the config is in another directory or a volume is mounted in case of Docker.
-
-The config can be taken from `airflow/config_templates/airflow_local_settings.py` as a starting point. Copy the contents to `${AIRFLOW_HOME}/config/airflow_local_settings.py`, and alter the config as is preferred.
-
-```
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-import os
-
-from airflow import configuration as conf
-
-# TODO: Logging format and level should be configured
-# in this file instead of from airflow.cfg. Currently
-# there are other log format and level configurations in
-# settings.py and cli.py. Please see AIRFLOW-1455.
-
-LOG_LEVEL = conf.get('core', 'LOGGING_LEVEL').upper()
-LOG_FORMAT = conf.get('core', 'log_format')
-
-BASE_LOG_FOLDER = conf.get('core', 'BASE_LOG_FOLDER')
-PROCESSOR_LOG_FOLDER = conf.get('scheduler', 'child_process_log_directory')
-
-FILENAME_TEMPLATE = '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'
-PROCESSOR_FILENAME_TEMPLATE = '{{ filename }}.log'
-
-DEFAULT_LOGGING_CONFIG = {
- 'version': 1,
- 'disable_existing_loggers': False,
- 'formatters': {
- 'airflow.task': {
- 'format': LOG_FORMAT,
- },
- 'airflow.processor': {
- 'format': LOG_FORMAT,
- },
- },
- 'handlers': {
- 'console': {
- 'class': 'logging.StreamHandler',
- 'formatter': 'airflow.task',
- 'stream': 'ext://sys.stdout'
- },
- 'file.task': {
- 'class': 'airflow.utils.log.file_task_handler.FileTaskHandler',
- 'formatter': 'airflow.task',
- 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
- 'filename_template': FILENAME_TEMPLATE,
- },
- 'file.processor': {
- 'class': 'airflow.utils.log.file_processor_handler.FileProcessorHandler',
- 'formatter': 'airflow.processor',
- 'base_log_folder': os.path.expanduser(PROCESSOR_LOG_FOLDER),
- 'filename_template': PROCESSOR_FILENAME_TEMPLATE,
- }
- # When using s3 or gcs, provide a customized LOGGING_CONFIG
- # in airflow_local_settings within your PYTHONPATH, see UPDATING.md
- # for details
- # 's3.task': {
- # 'class': 'airflow.utils.log.s3_task_handler.S3TaskHandler',
- # 'formatter': 'airflow.task',
- # 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
- # 's3_log_folder': S3_LOG_FOLDER,
- # 'filename_template': FILENAME_TEMPLATE,
- # },
- # 'gcs.task': {
- # 'class': 'airflow.utils.log.gcs_task_handler.GCSTaskHandler',
- # 'formatter': 'airflow.task',
- # 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
- # 'gcs_log_folder': GCS_LOG_FOLDER,
- # 'filename_template': FILENAME_TEMPLATE,
- # },
- },
- 'loggers': {
- '': {
- 'handlers': ['console'],
- 'level': LOG_LEVEL
- },
- 'airflow': {
- 'handlers': ['console'],
- 'level': LOG_LEVEL,
- 'propagate': False,
- },
- 'airflow.processor': {
- 'handlers': ['file.processor'],
- 'level': LOG_LEVEL,
- 'propagate': True,
- },
- 'airflow.task': {
- 'handlers': ['file.task'],
- 'level': LOG_LEVEL,
- 'propagate': False,
- },
- 'airflow.task_runner': {
- 'handlers': ['file.task'],
- 'level': LOG_LEVEL,
- 'propagate': True,
- },
- }
-}
-```
-
-To customize the logging (for example, use logging rotate), define one or more of the logging handles that [Python has to offer](https://docs.python.org/3/library/logging.handlers.html). For more details about the Python logging, please refer to the [official logging documentation](https://docs.python.org/3/library/logging.html).
-
-Furthermore, this change also simplifies logging within the DAG itself:
-
-```
-root@ae1bc863e815:/airflow# python
-Python 3.6.2 (default, Sep 13 2017, 14:26:54)
-[GCC 4.9.2] on linux
-Type "help", "copyright", "credits" or "license" for more information.
->>> from airflow.settings import *
->>>
->>> from datetime import datetime
->>> from airflow.models.dag import DAG
->>> from airflow.operators.dummy import DummyOperator
->>>
->>> dag = DAG('simple_dag', start_date=pendulum.datetime(2017, 9, 1, tz="UTC"))
->>>
->>> task = DummyOperator(task_id='task_1', dag=dag)
->>>
->>> task.log.error('I want to say something..')
-[2017-09-25 20:17:04,927] {:1} ERROR - I want to say something..
-```
-
-#### Template path of the file_task_handler
-
-The `file_task_handler` logger has been made more flexible. The default format can be changed, `{dag_id}/{task_id}/{execution_date}/{try_number}.log` by supplying Jinja templating in the `FILENAME_TEMPLATE` configuration variable. See the `file_task_handler` for more information.
-
-#### I'm using S3Log or GCSLogs, what do I do!?
-
-If you are logging to Google cloud storage, please see the [Google cloud platform documentation](https://airflow.apache.org/integration.html#gcp-google-cloud-platform) for logging instructions.
-
-If you are using S3, the instructions should be largely the same as the Google cloud platform instructions above. You will need a custom logging config. The `REMOTE_BASE_LOG_FOLDER` configuration key in your airflow config has been removed, therefore you will need to take the following steps:
-
-- Copy the logging configuration from [`airflow/config_templates/airflow_logging_settings.py`](https://github.com/apache/airflow/blob/main/airflow/config_templates/airflow_local_settings.py).
-- Place it in a directory inside the Python import path `PYTHONPATH`. If you are using Python 2.7, ensuring that any `__init__.py` files exist so that it is importable.
-- Update the config by setting the path of `REMOTE_BASE_LOG_FOLDER` explicitly in the config. The `REMOTE_BASE_LOG_FOLDER` key is not used anymore.
-- Set the `logging_config_class` to the filename and dict. For example, if you place `custom_logging_config.py` on the base of your `PYTHONPATH`, you will need to set `logging_config_class = custom_logging_config.LOGGING_CONFIG` in your config as Airflow 1.8.
-
-### New Features
-
-#### Dask Executor
-
-A new DaskExecutor allows Airflow tasks to be run in Dask Distributed clusters.
-
-### Deprecated Features
-
-These features are marked for deprecation. They may still work (and raise a `DeprecationWarning`), but are no longer
-supported and will be removed entirely in Airflow 2.0
-
-- If you're using the `google_cloud_conn_id` or `dataproc_cluster` argument names explicitly in `contrib.operators.Dataproc{*}Operator`(s), be sure to rename them to `gcp_conn_id` or `cluster_name`, respectively. We've renamed these arguments for consistency. (AIRFLOW-1323)
-
-- `post_execute()` hooks now take two arguments, `context` and `result`
- (AIRFLOW-886)
-
- Previously, post_execute() only took one argument, `context`.
-
-- `contrib.hooks.gcp_dataflow_hook.DataFlowHook` starts to use `--runner=DataflowRunner` instead of `DataflowPipelineRunner`, which is removed from the package `google-cloud-dataflow-0.6.0`.
-
-- The pickle type for XCom messages has been replaced by json to prevent RCE attacks.
- Note that JSON serialization is stricter than pickling, so if you want to e.g. pass
- raw bytes through XCom you must encode them using an encoding like base64.
- By default pickling is still enabled until Airflow 2.0. To disable it
- set enable_xcom_pickling = False in your Airflow config.
-
-## Airflow 1.8.1
-
-The Airflow package name was changed from `airflow` to `apache-airflow` during this release. You must uninstall
-a previously installed version of Airflow before installing 1.8.1.
-
-## Airflow 1.8
-
-### Database
-
-The database schema needs to be upgraded. Make sure to shutdown Airflow and make a backup of your database. To
-upgrade the schema issue `airflow upgradedb`.
-
-### Upgrade systemd unit files
-
-Systemd unit files have been updated. If you use systemd please make sure to update these.
-
-> Please note that the webserver does not detach properly, this will be fixed in a future version.
-
-### Tasks not starting although dependencies are met due to stricter pool checking
-
-Airflow 1.7.1 has issues with being able to over subscribe to a pool, ie. more slots could be used than were
-available. This is fixed in Airflow 1.8.0, but due to past issue jobs may fail to start although their
-dependencies are met after an upgrade. To workaround either temporarily increase the amount of slots above
-the amount of queued tasks or use a new pool.
-
-### Less forgiving scheduler on dynamic start_date
-
-Using a dynamic start_date (e.g. `start_date = datetime.now()`) is not considered a best practice. The 1.8.0 scheduler
-is less forgiving in this area. If you encounter DAGs not being scheduled you can try using a fixed start_date and
-renaming your DAG. The last step is required to make sure you start with a clean slate, otherwise the old schedule can
-interfere.
-
-### New and updated scheduler options
-
-Please read through the new scheduler options, defaults have changed since 1.7.1.
-
-#### child_process_log_directory
-
-In order to increase the robustness of the scheduler, DAGs are now processed in their own process. Therefore each
-DAG has its own log file for the scheduler. These log files are placed in `child_process_log_directory` which defaults to
-`/scheduler/latest`. You will need to make sure these log files are removed.
-
-> DAG logs or processor logs ignore and command line settings for log file locations.
-
-#### run_duration
-
-Previously the command line option `num_runs` was used to let the scheduler terminate after a certain amount of
-loops. This is now time bound and defaults to `-1`, which means run continuously. See also num_runs.
-
-#### num_runs
-
-Previously `num_runs` was used to let the scheduler terminate after a certain amount of loops. Now num_runs specifies
-the number of times to try to schedule each DAG file within `run_duration` time. Defaults to `-1`, which means try
-indefinitely. This is only available on the command line.
-
-#### min_file_process_interval
-
-After how much time should an updated DAG be picked up from the filesystem.
-
-#### min_file_parsing_loop_time
-
-CURRENTLY DISABLED DUE TO A BUG
-How many seconds to wait between file-parsing loops to prevent the logs from being spammed.
-
-#### dag_dir_list_interval
-
-The frequency with which the scheduler should relist the contents of the DAG directory. If while developing +dags, they are not being picked up, have a look at this number and decrease it when necessary.
-
-#### catchup_by_default
-
-By default the scheduler will fill any missing interval DAG Runs between the last execution date and the current date.
-This setting changes that behavior to only execute the latest interval. This can also be specified per DAG as
-`catchup = False / True`. Command line backfills will still work.
-
-### Faulty DAGs do not show an error in the Web UI
-
-Due to changes in the way Airflow processes DAGs the Web UI does not show an error when processing a faulty DAG. To
-find processing errors go the `child_process_log_directory` which defaults to `/scheduler/latest`.
-
-### New DAGs are paused by default
-
-Previously, new DAGs would be scheduled immediately. To retain the old behavior, add this to airflow.cfg:
-
-```
-[core]
-dags_are_paused_at_creation = False
-```
-
-### Airflow Context variable are passed to Hive config if conf is specified
-
-If you specify a hive conf to the run_cli command of the HiveHook, Airflow add some
-convenience variables to the config. In case you run a secure Hadoop setup it might be
-required to allow these variables by adjusting you hive configuration to add `airflow\.ctx\..*` to the regex
-of user-editable configuration properties. See
-[the Hive docs on Configuration Properties][hive.security.authorization.sqlstd] for more info.
-
-[hive.security.authorization.sqlstd]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=82903061#ConfigurationProperties-SQLStandardBasedAuthorization.1
-
-### Google Cloud Operator and Hook alignment
-
-All Google Cloud Operators and Hooks are aligned and use the same client library. Now you have a single connection
-type for all kinds of Google Cloud Operators.
-
-If you experience problems connecting with your operator make sure you set the connection type "Google Cloud".
-
-Also the old P12 key file type is not supported anymore and only the new JSON key files are supported as a service
-account.
-
-### Deprecated Features
-
-These features are marked for deprecation. They may still work (and raise a `DeprecationWarning`), but are no longer
-supported and will be removed entirely in Airflow 2.0
-
-- Hooks and operators must be imported from their respective submodules
-
- `airflow.operators.PigOperator` is no longer supported; `from airflow.operators.pig_operator import PigOperator` is.
- (AIRFLOW-31, AIRFLOW-200)
-
-- Operators no longer accept arbitrary arguments
-
- Previously, `Operator.__init__()` accepted any arguments (either positional `*args` or keyword `**kwargs`) without
- complaint. Now, invalid arguments will be rejected. (https://github.com/apache/airflow/pull/1285)
-
-- The config value secure_mode will default to True which will disable some insecure endpoints/features
-
-### Known Issues
-
-There is a report that the default of "-1" for num_runs creates an issue where errors are reported while parsing tasks.
-It was not confirmed, but a workaround was found by changing the default back to `None`.
-
-To do this edit `cli.py`, find the following:
-
-```
- 'num_runs': Arg(
- ("-n", "--num_runs"),
- default=-1, type=int,
- help="Set the number of runs to execute before exiting"),
-```
-
-and change `default=-1` to `default=None`. If you have this issue please report it on the mailing list.
-
-## Airflow 1.7.1.2
-
-### Changes to Configuration
-
-#### Email configuration change
-
-To continue using the default smtp email backend, change the email_backend line in your config file from:
-
-```
-[email]
-email_backend = airflow.utils.send_email_smtp
-```
-
-to:
-
-```
-[email]
-email_backend = airflow.utils.email.send_email_smtp
-```
-
-#### S3 configuration change
-
-To continue using S3 logging, update your config file so:
-
-```
-s3_log_folder = s3://my-airflow-log-bucket/logs
-```
-
-becomes:
-
-```
-remote_base_log_folder = s3://my-airflow-log-bucket/logs
-remote_log_conn_id =
-```
diff --git a/airflow/__init__.py b/airflow/__init__.py
index 342122093fa6b..cbbb03dd1be29 100644
--- a/airflow/__init__.py
+++ b/airflow/__init__.py
@@ -37,7 +37,7 @@
__version__ = version.version
-__all__ = ['__version__', 'login', 'DAG', 'PY36', 'PY37', 'PY38', 'PY39', 'PY310']
+__all__ = ['__version__', 'login', 'DAG', 'PY36', 'PY37', 'PY38', 'PY39', 'PY310', 'XComArg']
# Make `airflow` an namespace package, supporting installing
# airflow.providers.* in different locations (i.e. one in site, and one in user
@@ -54,18 +54,31 @@
PY39 = sys.version_info >= (3, 9)
PY310 = sys.version_info >= (3, 10)
+# Things to lazy import in form 'name': 'path.to.module'
+__lazy_imports = {
+ 'DAG': 'airflow.models.dag',
+ 'XComArg': 'airflow.models.xcom_arg',
+ 'AirflowException': 'airflow.exceptions',
+}
+
def __getattr__(name):
# PEP-562: Lazy loaded attributes on python modules
- if name == "DAG":
- from airflow.models.dag import DAG
+ path = __lazy_imports.get(name)
+ if not path:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ import operator
- return DAG
- if name == "AirflowException":
- from airflow.exceptions import AirflowException
+ # Strip off the "airflow." prefix because of how `__import__` works (it always returns the top level
+ # module)
+ without_prefix = path.split('.', 1)[-1]
- return AirflowException
- raise AttributeError(f"module {__name__} has no attribute {name}")
+ getter = operator.attrgetter(f'{without_prefix}.{name}')
+ val = getter(__import__(path))
+ # Store for next time
+ globals()[name] = val
+ return val
if not settings.LAZY_LOAD_PLUGINS:
@@ -89,10 +102,5 @@ def __getattr__(name):
globals()['kcah_acitats'[::-1].upper()] = False
if STATICA_HACK: # pragma: no cover
from airflow.models.dag import DAG
+ from airflow.models.xcom_arg import XComArg
from airflow.exceptions import AirflowException
-
-
-if not PY37:
- from pep562 import Pep562
-
- Pep562(__name__)
diff --git a/airflow/__main__.py b/airflow/__main__.py
index 50f156a3acf29..334126b2d930b 100644
--- a/airflow/__main__.py
+++ b/airflow/__main__.py
@@ -20,14 +20,11 @@
"""Main executable module"""
import os
-import warnings
import argcomplete
-from airflow import PY310
from airflow.cli import cli_parser
from airflow.configuration import conf
-from airflow.utils.docs import get_docs_url
def main():
@@ -35,13 +32,6 @@ def main():
if conf.get("core", "security") == 'kerberos':
os.environ['KRB5CCNAME'] = conf.get('kerberos', 'ccache')
os.environ['KRB5_KTNAME'] = conf.get('kerberos', 'keytab')
- if PY310:
- docs_url = get_docs_url('installation/prerequisites.html')
- warnings.warn(
- "Python v3.10 is not official supported on this version of Airflow. Please be careful. "
- f"For details, see: {docs_url}"
- )
-
parser = cli_parser.get_parser()
argcomplete.autocomplete(parser)
args = parser.parse_args()
diff --git a/airflow/_vendor/README.md b/airflow/_vendor/README.md
new file mode 100644
index 0000000000000..e708f1e507160
--- /dev/null
+++ b/airflow/_vendor/README.md
@@ -0,0 +1,37 @@
+# Vendor package
+
+## What vendored packages are for
+
+The `airflow._vendor` package is foreseen for vendoring in packages, that we have to modify ourselves
+because authors of the packages do not have time to modify them themselves. This is often temporary
+and once the packages implement fixes that we need, and then we remove the packages from
+the `_vendor` package.
+
+All Vendored libraries must follow these rules:
+
+1. Vendored libraries must be pure Python--no compiling (so that we do not have to release multi-platform airflow packages on PyPI).
+2. Source code for the libary is included in this directory.
+3. License must be included in this repo and in the [LICENSE](../../LICENSE) file and in the
+ [licenses](../../licenses) folder.
+4. Requirements of the library become requirements of airflow core.
+5. Version of the library should be included in the [vendor.md](vendor.md) file.
+6. No modifications to the library may be made in the initial commit.
+7. Apply the fixes necessary to use the vendored library as separate commits - each package separately,
+ so that they can be cherry-picked later if we upgrade the vendored package. Changes to airflow code to
+ use the vendored packages should be applied as separate commits/PRs.
+8. The `_vendor` packages should be excluded from any refactorings, static checks and automated fixes.
+
+## Adding and upgrading a vendored package
+
+Way to vendor a library or update a version:
+
+1. Update ``vendor.txt`` with the library, version, and SHA256 (`pypi` provides hashes as of recently)
+2. Remove all old files and directories of the old version.
+3. Replace them with new files (only replace relevant python packages:move LICENSE )
+ * move licence files to [licenses](../../licenses) folder
+ * remove README and any other supporting files (they can be found in PyPI)
+ * make sure to add requirements from setup.py to airflow's setup.py with appropriate comment stating
+ why the requirements are added and when they should be removed
+4. If you replace previous version, re-apply historical fixes from the "package" folder by
+ cherry-picking them.
+
diff --git a/airflow/_vendor/vendor.md b/airflow/_vendor/vendor.md
new file mode 100644
index 0000000000000..ab102ab08353f
--- /dev/null
+++ b/airflow/_vendor/vendor.md
@@ -0,0 +1,2 @@
+| Package | Version | File | SHA256 |
+|-----------|---------|------------------------|------------------------------------------------------------------|
diff --git a/airflow/api/__init__.py b/airflow/api/__init__.py
index a9c7619e64557..da07429869877 100644
--- a/airflow/api/__init__.py
+++ b/airflow/api/__init__.py
@@ -34,9 +34,9 @@ def load_auth():
pass
backends = []
- for backend in auth_backends.split():
+ for backend in auth_backends.split(","):
try:
- auth = import_module(backend)
+ auth = import_module(backend.strip())
log.info("Loaded API auth backend: %s", backend)
backends.append(auth)
except ImportError as err:
diff --git a/airflow/api/client/__init__.py b/airflow/api/client/__init__.py
index b7f9c78d30a49..49224b5336a32 100644
--- a/airflow/api/client/__init__.py
+++ b/airflow/api/client/__init__.py
@@ -26,7 +26,7 @@
def get_current_api_client() -> Client:
"""Return current API Client based on current Airflow configuration"""
- api_module = import_module(conf.get('cli', 'api_client')) # type: Any
+ api_module = import_module(conf.get_mandatory_value('cli', 'api_client')) # type: Any
auth_backends = api.load_auth()
session = None
for backend in auth_backends:
diff --git a/airflow/api/common/mark_tasks.py b/airflow/api/common/mark_tasks.py
index 22cde7febf78d..83bdb2081f0b8 100644
--- a/airflow/api/common/mark_tasks.py
+++ b/airflow/api/common/mark_tasks.py
@@ -18,10 +18,10 @@
"""Marks tasks APIs."""
from datetime import datetime
-from typing import TYPE_CHECKING, Iterable, Iterator, List, NamedTuple, Optional, Tuple
+from typing import TYPE_CHECKING, Collection, Iterable, Iterator, List, NamedTuple, Optional, Tuple, Union
from sqlalchemy import or_
-from sqlalchemy.orm import contains_eager
+from sqlalchemy.orm import lazyload
from sqlalchemy.orm.session import Session as SASession
from airflow.models.dag import DAG
@@ -78,7 +78,7 @@ def _create_dagruns(
@provide_session
def set_state(
*,
- tasks: Iterable[Operator],
+ tasks: Collection[Union[Operator, Tuple[Operator, int]]],
run_id: Optional[str] = None,
execution_date: Optional[datetime] = None,
upstream: bool = False,
@@ -96,9 +96,10 @@ def set_state(
tasks that did not exist. It will not create dag runs that are missing
on the schedule (but it will as for subdag dag runs if needed).
- :param tasks: the iterable of tasks from which to work. task.task.dag needs to be set
+ :param tasks: the iterable of tasks or (task, map_index) tuples from which to work.
+ ``task.dag`` needs to be set
:param run_id: the run_id of the dagrun to start looking from
- :param execution_date: the execution date from which to start looking(deprecated)
+ :param execution_date: the execution date from which to start looking (deprecated)
:param upstream: Mark all parents (upstream tasks)
:param downstream: Mark all siblings (downstream tasks) of task_id, including SubDags
:param future: Mark all future tasks on the interval of the dag up until
@@ -118,7 +119,7 @@ def set_state(
if execution_date and not timezone.is_localized(execution_date):
raise ValueError(f"Received non-localized date {execution_date}")
- task_dags = {task.dag for task in tasks}
+ task_dags = {task[0].dag if isinstance(task, tuple) else task.dag for task in tasks}
if len(task_dags) > 1:
raise ValueError(f"Received tasks from multiple DAGs: {task_dags}")
dag = next(iter(task_dags))
@@ -126,15 +127,15 @@ def set_state(
raise ValueError("Received tasks with no DAG")
if execution_date:
- run_id = dag.get_dagrun(execution_date=execution_date).run_id
+ run_id = dag.get_dagrun(execution_date=execution_date, session=session).run_id
if not run_id:
raise ValueError("Received tasks with no run_id")
- dag_run_ids = get_run_ids(dag, run_id, future, past)
-
- task_ids = list(find_task_relatives(tasks, downstream, upstream))
+ dag_run_ids = get_run_ids(dag, run_id, future, past, session=session)
+ task_id_map_index_list = list(find_task_relatives(tasks, downstream, upstream))
+ task_ids = [task_id if isinstance(task_id, str) else task_id[0] for task_id in task_id_map_index_list]
- confirmed_infos = list(_iter_existing_dag_run_infos(dag, dag_run_ids))
+ confirmed_infos = list(_iter_existing_dag_run_infos(dag, dag_run_ids, session=session))
confirmed_dates = [info.logical_date for info in confirmed_infos]
sub_dag_run_ids = list(
@@ -143,7 +144,7 @@ def set_state(
# now look for the task instances that are affected
- qry_dag = get_all_dag_task_query(dag, session, state, task_ids, confirmed_dates)
+ qry_dag = get_all_dag_task_query(dag, session, state, task_id_map_index_list, dag_run_ids)
if commit:
tis_altered = qry_dag.with_for_update().all()
@@ -151,7 +152,8 @@ def set_state(
qry_sub_dag = all_subdag_tasks_query(sub_dag_run_ids, session, state, confirmed_dates)
tis_altered += qry_sub_dag.with_for_update().all()
for task_instance in tis_altered:
- task_instance.set_state(state)
+ task_instance.set_state(state, session=session)
+ session.flush()
else:
tis_altered = qry_dag.all()
if sub_dag_run_ids:
@@ -179,20 +181,18 @@ def get_all_dag_task_query(
dag: DAG,
session: SASession,
state: TaskInstanceState,
- task_ids: List[str],
- confirmed_dates: Iterable[datetime],
+ task_ids: List[Union[str, Tuple[str, int]]],
+ run_ids: Iterable[str],
):
"""Get all tasks of the main dag that will be affected by a state change"""
- qry_dag = (
- session.query(TaskInstance)
- .join(TaskInstance.dag_run)
- .filter(
- TaskInstance.dag_id == dag.dag_id,
- DagRun.execution_date.in_(confirmed_dates),
- TaskInstance.task_id.in_(task_ids),
- )
- .filter(or_(TaskInstance.state.is_(None), TaskInstance.state != state))
- .options(contains_eager(TaskInstance.dag_run))
+ qry_dag = session.query(TaskInstance).filter(
+ TaskInstance.dag_id == dag.dag_id,
+ TaskInstance.run_id.in_(run_ids),
+ TaskInstance.ti_selector_condition(task_ids),
+ )
+
+ qry_dag = qry_dag.filter(or_(TaskInstance.state.is_(None), TaskInstance.state != state)).options(
+ lazyload(TaskInstance.dag_run)
)
return qry_dag
@@ -261,17 +261,22 @@ def verify_dagruns(
session.merge(dag_run)
-def _iter_existing_dag_run_infos(dag: DAG, run_ids: List[str]) -> Iterator[_DagRunInfo]:
- for dag_run in DagRun.find(dag_id=dag.dag_id, run_id=run_ids):
+def _iter_existing_dag_run_infos(dag: DAG, run_ids: List[str], session: SASession) -> Iterator[_DagRunInfo]:
+ for dag_run in DagRun.find(dag_id=dag.dag_id, run_id=run_ids, session=session):
dag_run.dag = dag
- dag_run.verify_integrity()
+ dag_run.verify_integrity(session=session)
yield _DagRunInfo(dag_run.logical_date, dag.get_run_data_interval(dag_run))
def find_task_relatives(tasks, downstream, upstream):
"""Yield task ids and optionally ancestor and descendant ids."""
- for task in tasks:
- yield task.task_id
+ for item in tasks:
+ if isinstance(item, tuple):
+ task, map_index = item
+ yield task.task_id, map_index
+ else:
+ task = item
+ yield task.task_id
if downstream:
for relative in task.get_flat_relatives(upstream=False):
yield relative.task_id
@@ -313,8 +318,8 @@ def get_execution_dates(
@provide_session
def get_run_ids(dag: DAG, run_id: str, future: bool, past: bool, session: SASession = NEW_SESSION):
"""Returns run_ids of DAG execution"""
- last_dagrun = dag.get_last_dagrun(include_externally_triggered=True)
- current_dagrun = dag.get_dagrun(run_id=run_id)
+ last_dagrun = dag.get_last_dagrun(include_externally_triggered=True, session=session)
+ current_dagrun = dag.get_dagrun(run_id=run_id, session=session)
first_dagrun = (
session.query(DagRun)
.filter(DagRun.dag_id == dag.dag_id)
@@ -331,7 +336,7 @@ def get_run_ids(dag: DAG, run_id: str, future: bool, past: bool, session: SASess
if not dag.timetable.can_run:
# If the DAG never schedules, need to look at existing DagRun if the user wants future or
# past runs.
- dag_runs = dag.get_dagruns_between(start_date=start_date, end_date=end_date)
+ dag_runs = dag.get_dagruns_between(start_date=start_date, end_date=end_date, session=session)
run_ids = sorted({d.run_id for d in dag_runs})
elif not dag.timetable.periodic:
run_ids = [run_id]
@@ -339,7 +344,7 @@ def get_run_ids(dag: DAG, run_id: str, future: bool, past: bool, session: SASess
dates = [
info.logical_date for info in dag.iter_dagrun_infos_between(start_date, end_date, align=False)
]
- run_ids = [dr.run_id for dr in DagRun.find(dag_id=dag.dag_id, execution_date=dates)]
+ run_ids = [dr.run_id for dr in DagRun.find(dag_id=dag.dag_id, execution_date=dates, session=session)]
return run_ids
@@ -468,7 +473,20 @@ def set_dag_run_state_to_failed(
task.dag = dag
tasks.append(task)
- return set_state(tasks=tasks, run_id=run_id, state=State.FAILED, commit=commit, session=session)
+ # Mark non-finished tasks as SKIPPED.
+ tis = session.query(TaskInstance).filter(
+ TaskInstance.dag_id == dag.dag_id,
+ TaskInstance.run_id == run_id,
+ TaskInstance.state.not_in(State.finished),
+ TaskInstance.state.not_in(State.running),
+ )
+
+ tis = [ti for ti in tis]
+ if commit:
+ for ti in tis:
+ ti.set_state(State.SKIPPED)
+
+ return tis + set_state(tasks=tasks, run_id=run_id, state=State.FAILED, commit=commit, session=session)
def __set_dag_run_state_to_running_or_queued(
diff --git a/airflow/api_connexion/endpoints/config_endpoint.py b/airflow/api_connexion/endpoints/config_endpoint.py
index 9514621447609..bdd2b3a959547 100644
--- a/airflow/api_connexion/endpoints/config_endpoint.py
+++ b/airflow/api_connexion/endpoints/config_endpoint.py
@@ -15,6 +15,8 @@
# specific language governing permissions and limitations
# under the License.
+from http import HTTPStatus
+
from flask import Response, request
from airflow.api_connexion import security
@@ -72,7 +74,7 @@ def get_config() -> Response:
}
return_type = request.accept_mimetypes.best_match(serializer.keys())
if return_type not in serializer:
- return Response(status=406)
+ return Response(status=HTTPStatus.NOT_ACCEPTABLE)
elif conf.getboolean("webserver", "expose_config"):
conf_dict = conf.as_dict(display_source=False, display_sensitive=True)
config = _conf_dict_to_config(conf_dict)
diff --git a/airflow/api_connexion/endpoints/connection_endpoint.py b/airflow/api_connexion/endpoints/connection_endpoint.py
index f9be9c227e3f3..b196b3236b911 100644
--- a/airflow/api_connexion/endpoints/connection_endpoint.py
+++ b/airflow/api_connexion/endpoints/connection_endpoint.py
@@ -16,6 +16,7 @@
# under the License.
import os
+from http import HTTPStatus
from connexion import NoContent
from flask import request
@@ -51,7 +52,7 @@ def delete_connection(*, connection_id: str, session: Session = NEW_SESSION) ->
detail=f"The Connection with connection_id: `{connection_id}` was not found",
)
session.delete(connection)
- return NoContent, 204
+ return NoContent, HTTPStatus.NO_CONTENT
@security.requires_access([(permissions.ACTION_CAN_READ, permissions.RESOURCE_CONNECTION)])
diff --git a/airflow/api_connexion/endpoints/dag_endpoint.py b/airflow/api_connexion/endpoints/dag_endpoint.py
index e94707b127a69..0505f864ee333 100644
--- a/airflow/api_connexion/endpoints/dag_endpoint.py
+++ b/airflow/api_connexion/endpoints/dag_endpoint.py
@@ -15,6 +15,7 @@
# specific language governing permissions and limitations
# under the License.
+from http import HTTPStatus
from typing import Collection, Optional
from connexion import NoContent
@@ -176,4 +177,4 @@ def delete_dag(dag_id: str, session: Session = NEW_SESSION) -> APIResponse:
except AirflowException:
raise AlreadyExists(detail=f"Task instances of dag with id: '{dag_id}' are still running")
- return NoContent, 204
+ return NoContent, HTTPStatus.NO_CONTENT
diff --git a/airflow/api_connexion/endpoints/dag_run_endpoint.py b/airflow/api_connexion/endpoints/dag_run_endpoint.py
index a83ca223b07ac..e510126534b12 100644
--- a/airflow/api_connexion/endpoints/dag_run_endpoint.py
+++ b/airflow/api_connexion/endpoints/dag_run_endpoint.py
@@ -14,6 +14,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+from http import HTTPStatus
from typing import List, Optional, Tuple
import pendulum
@@ -23,17 +24,26 @@
from sqlalchemy import or_
from sqlalchemy.orm import Query, Session
-from airflow.api.common.mark_tasks import set_dag_run_state_to_failed, set_dag_run_state_to_success
+from airflow.api.common.mark_tasks import (
+ set_dag_run_state_to_failed,
+ set_dag_run_state_to_queued,
+ set_dag_run_state_to_success,
+)
from airflow.api_connexion import security
from airflow.api_connexion.exceptions import AlreadyExists, BadRequest, NotFound
from airflow.api_connexion.parameters import apply_sorting, check_limit, format_datetime, format_parameters
from airflow.api_connexion.schemas.dag_run_schema import (
DAGRunCollection,
+ clear_dagrun_form_schema,
dagrun_collection_schema,
dagrun_schema,
dagruns_batch_form_schema,
set_dagrun_state_form_schema,
)
+from airflow.api_connexion.schemas.task_instance_schema import (
+ TaskInstanceReferenceCollection,
+ task_instance_reference_collection_schema,
+)
from airflow.api_connexion.types import APIResponse
from airflow.models import DagModel, DagRun
from airflow.security import permissions
@@ -53,7 +63,7 @@ def delete_dag_run(*, dag_id: str, dag_run_id: str, session: Session = NEW_SESSI
"""Delete a DAG Run"""
if session.query(DagRun).filter(DagRun.dag_id == dag_id, DagRun.run_id == dag_run_id).delete() == 0:
raise NotFound(detail=f"DAGRun with DAG ID: '{dag_id}' and DagRun ID: '{dag_run_id}' not found")
- return NoContent, 204
+ return NoContent, HTTPStatus.NO_CONTENT
@security.requires_access(
@@ -308,7 +318,60 @@ def update_dag_run_state(*, dag_id: str, dag_run_id: str, session: Session = NEW
dag = current_app.dag_bag.get_dag(dag_id)
if state == DagRunState.SUCCESS:
set_dag_run_state_to_success(dag=dag, run_id=dag_run.run_id, commit=True)
+ elif state == DagRunState.QUEUED:
+ set_dag_run_state_to_queued(dag=dag, run_id=dag_run.run_id, commit=True)
else:
set_dag_run_state_to_failed(dag=dag, run_id=dag_run.run_id, commit=True)
dag_run = session.query(DagRun).get(dag_run.id)
return dagrun_schema.dump(dag_run)
+
+
+@security.requires_access(
+ [
+ (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG),
+ (permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG_RUN),
+ ],
+)
+@provide_session
+def clear_dag_run(*, dag_id: str, dag_run_id: str, session: Session = NEW_SESSION) -> APIResponse:
+ """Clear a dag run."""
+ dag_run: Optional[DagRun] = (
+ session.query(DagRun).filter(DagRun.dag_id == dag_id, DagRun.run_id == dag_run_id).one_or_none()
+ )
+ if dag_run is None:
+ error_message = f'Dag Run id {dag_run_id} not found in dag {dag_id}'
+ raise NotFound(error_message)
+ try:
+ post_body = clear_dagrun_form_schema.load(request.json)
+ except ValidationError as err:
+ raise BadRequest(detail=str(err))
+
+ dry_run = post_body.get('dry_run', False)
+ dag = current_app.dag_bag.get_dag(dag_id)
+ start_date = dag_run.logical_date
+ end_date = dag_run.logical_date
+
+ if dry_run:
+ task_instances = dag.clear(
+ start_date=start_date,
+ end_date=end_date,
+ task_ids=None,
+ include_subdags=True,
+ include_parentdag=True,
+ only_failed=False,
+ dry_run=True,
+ )
+ return task_instance_reference_collection_schema.dump(
+ TaskInstanceReferenceCollection(task_instances=task_instances)
+ )
+ else:
+ dag.clear(
+ start_date=start_date,
+ end_date=end_date,
+ task_ids=None,
+ include_subdags=True,
+ include_parentdag=True,
+ only_failed=False,
+ )
+ dag_run.refresh_from_db()
+ return dagrun_schema.dump(dag_run)
diff --git a/airflow/api_connexion/endpoints/dag_source_endpoint.py b/airflow/api_connexion/endpoints/dag_source_endpoint.py
index 74c3496a2c208..ad6209221e523 100644
--- a/airflow/api_connexion/endpoints/dag_source_endpoint.py
+++ b/airflow/api_connexion/endpoints/dag_source_endpoint.py
@@ -15,6 +15,8 @@
# specific language governing permissions and limitations
# under the License.
+from http import HTTPStatus
+
from flask import Response, current_app, request
from itsdangerous import BadSignature, URLSafeSerializer
@@ -42,4 +44,4 @@ def get_dag_source(*, file_token: str) -> Response:
if return_type == 'application/json':
content = dag_source_schema.dumps(dict(content=dag_source))
return Response(content, headers={'Content-Type': return_type})
- return Response("Not Allowed Accept Header", status=406)
+ return Response("Not Allowed Accept Header", status=HTTPStatus.NOT_ACCEPTABLE)
diff --git a/airflow/api_connexion/endpoints/pool_endpoint.py b/airflow/api_connexion/endpoints/pool_endpoint.py
index e9c8aee252bec..1d24fea63d756 100644
--- a/airflow/api_connexion/endpoints/pool_endpoint.py
+++ b/airflow/api_connexion/endpoints/pool_endpoint.py
@@ -14,6 +14,8 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+
+from http import HTTPStatus
from typing import Optional
from flask import Response, request
@@ -41,7 +43,7 @@ def delete_pool(*, pool_name: str, session: Session = NEW_SESSION) -> APIRespons
affected_count = session.query(Pool).filter(Pool.pool == pool_name).delete()
if affected_count == 0:
raise NotFound(detail=f"Pool with name:'{pool_name}' not found")
- return Response(status=204)
+ return Response(status=HTTPStatus.NO_CONTENT)
@security.requires_access([(permissions.ACTION_CAN_READ, permissions.RESOURCE_POOL)])
diff --git a/airflow/api_connexion/endpoints/provider_endpoint.py b/airflow/api_connexion/endpoints/provider_endpoint.py
index 9c994bf3de5c1..7526e284beb06 100644
--- a/airflow/api_connexion/endpoints/provider_endpoint.py
+++ b/airflow/api_connexion/endpoints/provider_endpoint.py
@@ -34,9 +34,9 @@ def _remove_rst_syntax(value: str) -> str:
def _provider_mapper(provider: ProviderInfo) -> Provider:
return Provider(
- package_name=provider[1]["package-name"],
- description=_remove_rst_syntax(provider[1]["description"]),
- version=provider[0],
+ package_name=provider.data["package-name"],
+ description=_remove_rst_syntax(provider.data["description"]),
+ version=provider.version,
)
diff --git a/airflow/api_connexion/endpoints/role_and_permission_endpoint.py b/airflow/api_connexion/endpoints/role_and_permission_endpoint.py
index a25856e111b2c..88a68341c129e 100644
--- a/airflow/api_connexion/endpoints/role_and_permission_endpoint.py
+++ b/airflow/api_connexion/endpoints/role_and_permission_endpoint.py
@@ -15,6 +15,7 @@
# specific language governing permissions and limitations
# under the License.
+from http import HTTPStatus
from typing import List, Optional, Tuple
from connexion import NoContent
@@ -104,7 +105,7 @@ def delete_role(*, role_name: str) -> APIResponse:
if not role:
raise NotFound(title="Role not found", detail=f"Role with name {role_name!r} was not found")
ab_security_manager.delete_role(role_name=role_name)
- return NoContent, 204
+ return NoContent, HTTPStatus.NO_CONTENT
@security.requires_access([(permissions.ACTION_CAN_EDIT, permissions.RESOURCE_ROLE)])
diff --git a/airflow/api_connexion/endpoints/task_instance_endpoint.py b/airflow/api_connexion/endpoints/task_instance_endpoint.py
index a2ee6334d4929..c2416ab0d9d44 100644
--- a/airflow/api_connexion/endpoints/task_instance_endpoint.py
+++ b/airflow/api_connexion/endpoints/task_instance_endpoint.py
@@ -137,6 +137,118 @@ def get_mapped_task_instance(
return task_instance_schema.dump(task_instance)
+@format_parameters(
+ {
+ "execution_date_gte": format_datetime,
+ "execution_date_lte": format_datetime,
+ "start_date_gte": format_datetime,
+ "start_date_lte": format_datetime,
+ "end_date_gte": format_datetime,
+ "end_date_lte": format_datetime,
+ },
+)
+@security.requires_access(
+ [
+ (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG),
+ (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG_RUN),
+ (permissions.ACTION_CAN_READ, permissions.RESOURCE_TASK_INSTANCE),
+ ],
+)
+@provide_session
+def get_mapped_task_instances(
+ *,
+ dag_id: str,
+ dag_run_id: str,
+ task_id: str,
+ execution_date_gte: Optional[str] = None,
+ execution_date_lte: Optional[str] = None,
+ start_date_gte: Optional[str] = None,
+ start_date_lte: Optional[str] = None,
+ end_date_gte: Optional[str] = None,
+ end_date_lte: Optional[str] = None,
+ duration_gte: Optional[float] = None,
+ duration_lte: Optional[float] = None,
+ state: Optional[List[str]] = None,
+ pool: Optional[List[str]] = None,
+ queue: Optional[List[str]] = None,
+ limit: Optional[int] = None,
+ offset: Optional[int] = None,
+ order_by: Optional[str] = None,
+ session: Session = NEW_SESSION,
+) -> APIResponse:
+ """Get list of task instances."""
+ # Because state can be 'none'
+ states = _convert_state(state)
+
+ base_query = (
+ session.query(TI)
+ .filter(TI.dag_id == dag_id, TI.run_id == dag_run_id, TI.task_id == task_id, TI.map_index >= 0)
+ .join(TI.dag_run)
+ )
+
+ # 0 can mean a mapped TI that expanded to an empty list, so it is not an automatic 404
+ if base_query.with_entities(func.count('*')).scalar() == 0:
+ dag = current_app.dag_bag.get_dag(dag_id)
+ if not dag:
+ error_message = f"DAG {dag_id} not found"
+ raise NotFound(error_message)
+ task = dag.get_task(task_id)
+ if not task:
+ error_message = f"Task id {task_id} not found"
+ raise NotFound(error_message)
+ if not task.is_mapped:
+ error_message = f"Task id {task_id} is not mapped"
+ raise NotFound(error_message)
+
+ # Other search criteria
+ query = _apply_range_filter(
+ base_query,
+ key=DR.execution_date,
+ value_range=(execution_date_gte, execution_date_lte),
+ )
+ query = _apply_range_filter(query, key=TI.start_date, value_range=(start_date_gte, start_date_lte))
+ query = _apply_range_filter(query, key=TI.end_date, value_range=(end_date_gte, end_date_lte))
+ query = _apply_range_filter(query, key=TI.duration, value_range=(duration_gte, duration_lte))
+ query = _apply_array_filter(query, key=TI.state, values=states)
+ query = _apply_array_filter(query, key=TI.pool, values=pool)
+ query = _apply_array_filter(query, key=TI.queue, values=queue)
+
+ # Count elements before joining extra columns
+ total_entries = query.with_entities(func.count('*')).scalar()
+
+ # Add SLA miss
+ query = (
+ query.join(
+ SlaMiss,
+ and_(
+ SlaMiss.dag_id == TI.dag_id,
+ SlaMiss.task_id == TI.task_id,
+ SlaMiss.execution_date == DR.execution_date,
+ ),
+ isouter=True,
+ )
+ .add_entity(SlaMiss)
+ .options(joinedload(TI.rendered_task_instance_fields))
+ )
+
+ if order_by:
+ if order_by == 'state':
+ query = query.order_by(TI.state.asc(), TI.map_index.asc())
+ elif order_by == '-state':
+ query = query.order_by(TI.state.desc(), TI.map_index.asc())
+ elif order_by == '-map_index':
+ query = query.order_by(TI.map_index.desc())
+ else:
+ raise BadRequest(detail=f"Ordering with '{order_by}' is not supported")
+ else:
+ query = query.order_by(TI.map_index.asc())
+
+ task_instances = query.offset(offset).limit(limit).all()
+ return task_instance_collection_schema.dump(
+ TaskInstanceCollection(task_instances=task_instances, total_entries=total_entries)
+ )
+
+
def _convert_state(states: Optional[Iterable[str]]) -> Optional[List[Optional[str]]]:
if not states:
return None
diff --git a/airflow/api_connexion/endpoints/user_endpoint.py b/airflow/api_connexion/endpoints/user_endpoint.py
index 82375cebcaf16..6b4e984a69559 100644
--- a/airflow/api_connexion/endpoints/user_endpoint.py
+++ b/airflow/api_connexion/endpoints/user_endpoint.py
@@ -14,6 +14,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+from http import HTTPStatus
from typing import List, Optional
from connexion import NoContent
@@ -204,4 +205,4 @@ def delete_user(*, username: str) -> APIResponse:
security_manager.get_session.delete(user)
security_manager.get_session.commit()
- return NoContent, 204
+ return NoContent, HTTPStatus.NO_CONTENT
diff --git a/airflow/api_connexion/endpoints/variable_endpoint.py b/airflow/api_connexion/endpoints/variable_endpoint.py
index 067d163401454..487d2cc486c83 100644
--- a/airflow/api_connexion/endpoints/variable_endpoint.py
+++ b/airflow/api_connexion/endpoints/variable_endpoint.py
@@ -14,6 +14,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+from http import HTTPStatus
from typing import Optional
from flask import Response, request
@@ -36,7 +37,7 @@ def delete_variable(*, variable_key: str) -> Response:
"""Delete variable"""
if Variable.delete(variable_key) == 0:
raise NotFound("Variable not found")
- return Response(status=204)
+ return Response(status=HTTPStatus.NO_CONTENT)
@security.requires_access([(permissions.ACTION_CAN_READ, permissions.RESOURCE_VARIABLE)])
diff --git a/airflow/api_connexion/exceptions.py b/airflow/api_connexion/exceptions.py
index 0c6c4fa0d3a8f..8fb7f2e78883b 100644
--- a/airflow/api_connexion/exceptions.py
+++ b/airflow/api_connexion/exceptions.py
@@ -14,6 +14,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+from http import HTTPStatus
from typing import Any, Dict, Optional
import flask
@@ -80,7 +81,7 @@ def __init__(
**kwargs: Any,
) -> None:
super().__init__(
- status=404,
+ status=HTTPStatus.NOT_FOUND,
type=EXCEPTIONS_LINK_MAP[404],
title=title,
detail=detail,
@@ -100,7 +101,7 @@ def __init__(
**kwargs: Any,
) -> None:
super().__init__(
- status=400,
+ status=HTTPStatus.BAD_REQUEST,
type=EXCEPTIONS_LINK_MAP[400],
title=title,
detail=detail,
@@ -120,7 +121,7 @@ def __init__(
**kwargs: Any,
):
super().__init__(
- status=401,
+ status=HTTPStatus.UNAUTHORIZED,
type=EXCEPTIONS_LINK_MAP[401],
title=title,
detail=detail,
@@ -140,7 +141,7 @@ def __init__(
**kwargs: Any,
) -> None:
super().__init__(
- status=403,
+ status=HTTPStatus.FORBIDDEN,
type=EXCEPTIONS_LINK_MAP[403],
title=title,
detail=detail,
@@ -160,7 +161,7 @@ def __init__(
**kwargs: Any,
):
super().__init__(
- status=409,
+ status=HTTPStatus.CONFLICT,
type=EXCEPTIONS_LINK_MAP[409],
title=title,
detail=detail,
@@ -180,7 +181,7 @@ def __init__(
**kwargs: Any,
) -> None:
super().__init__(
- status=500,
+ status=HTTPStatus.INTERNAL_SERVER_ERROR,
type=EXCEPTIONS_LINK_MAP[500],
title=title,
detail=detail,
diff --git a/airflow/api_connexion/openapi/v1.yaml b/airflow/api_connexion/openapi/v1.yaml
index a25ad735adada..fc71fbb7f5e3d 100644
--- a/airflow/api_connexion/openapi/v1.yaml
+++ b/airflow/api_connexion/openapi/v1.yaml
@@ -781,6 +781,43 @@ paths:
'404':
$ref: '#/components/responses/NotFound'
+ /dags/{dag_id}/dagRuns/{dag_run_id}/clear:
+ parameters:
+ - $ref: '#/components/parameters/DAGID'
+ - $ref: '#/components/parameters/DAGRunID'
+
+ post:
+ summary: Clear a DAG run
+ description: |
+ Clear a DAG run.
+
+ *New in version 2.4.0*
+ x-openapi-router-controller: airflow.api_connexion.endpoints.dag_run_endpoint
+ operationId: clear_dag_run
+ tags: [DAGRun]
+ requestBody:
+ required: true
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/ClearDagRun'
+
+ responses:
+ '200':
+ description: Success.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/DAGRun'
+ '400':
+ $ref: '#/components/responses/BadRequest'
+ '401':
+ $ref: '#/components/responses/Unauthenticated'
+ '403':
+ $ref: '#/components/responses/PermissionDenied'
+ '404':
+ $ref: '#/components/responses/NotFound'
+
/eventLogs:
get:
summary: List log entries
@@ -1109,6 +1146,50 @@ paths:
'404':
$ref: '#/components/responses/NotFound'
+ /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/listMapped:
+ parameters:
+ - $ref: '#/components/parameters/DAGID'
+ - $ref: '#/components/parameters/DAGRunID'
+ - $ref: '#/components/parameters/TaskID'
+
+ get:
+ summary: List mapped task instances
+ description: |
+ Get details of all mapped task instances.
+
+ *New in version 2.3.0*
+ x-openapi-router-controller: airflow.api_connexion.endpoints.task_instance_endpoint
+ operationId: get_mapped_task_instances
+ tags: [TaskInstance]
+ parameters:
+ - $ref: '#/components/parameters/PageLimit'
+ - $ref: '#/components/parameters/PageOffset'
+ - $ref: '#/components/parameters/FilterExecutionDateGTE'
+ - $ref: '#/components/parameters/FilterExecutionDateLTE'
+ - $ref: '#/components/parameters/FilterStartDateGTE'
+ - $ref: '#/components/parameters/FilterStartDateLTE'
+ - $ref: '#/components/parameters/FilterEndDateGTE'
+ - $ref: '#/components/parameters/FilterEndDateLTE'
+ - $ref: '#/components/parameters/FilterDurationGTE'
+ - $ref: '#/components/parameters/FilterDurationLTE'
+ - $ref: '#/components/parameters/FilterState'
+ - $ref: '#/components/parameters/FilterPool'
+ - $ref: '#/components/parameters/FilterQueue'
+ - $ref: '#/components/parameters/OrderBy'
+ responses:
+ '200':
+ description: Success.
+ content:
+ application/json:
+ schema:
+ $ref: '#/components/schemas/TaskInstance'
+ '401':
+ $ref: '#/components/responses/Unauthenticated'
+ '403':
+ $ref: '#/components/responses/PermissionDenied'
+ '404':
+ $ref: '#/components/responses/NotFound'
+
/dags/~/dagRuns/~/taskInstances/list:
post:
summary: List task instances (batch)
@@ -2111,6 +2192,58 @@ components:
description: Whether the DAG is SubDAG.
type: boolean
readOnly: true
+ last_parsed_time:
+ type: string
+ format: date-time
+ readOnly: true
+ nullable: true
+ description: |
+ The last time the DAG was parsed.
+
+ *New in version 2.3.0*
+ last_pickled:
+ type: string
+ format: date-time
+ readOnly: true
+ nullable: true
+ description: |
+ The last time the DAG was pickled.
+
+ *New in version 2.3.0*
+ last_expired:
+ type: string
+ format: date-time
+ readOnly: true
+ nullable: true
+ description: |
+ Time when the DAG last received a refresh signal
+ (e.g. the DAG's "refresh" button was clicked in the web UI)
+
+ *New in version 2.3.0*
+ scheduler_lock:
+ type: boolean
+ readOnly: true
+ nullable: true
+ description: |
+ Whether (one of) the scheduler is scheduling this DAG at the moment
+
+ *New in version 2.3.0*
+ pickle_id:
+ type: string
+ readOnly: true
+ nullable: true
+ description: |
+ Foreign key to the latest pickle_id
+
+ *New in version 2.3.0*
+ default_view:
+ type: string
+ nullable: true
+ readOnly: true
+ description: |
+ Default view of the DAG inside the webserver
+
+ *New in version 2.3.0*
fileloc:
description: The absolute path to the file.
type: string
@@ -2136,6 +2269,14 @@ components:
describe DAG contents.
schedule_interval:
$ref: '#/components/schemas/ScheduleInterval'
+ timetable_description:
+ type: string
+ readOnly: true
+ nullable: true
+ description: |
+ Timetable/Schedule Interval description.
+
+ *New in version 2.3.0*
tags:
description: List of tags.
type: array
@@ -2143,6 +2284,74 @@ components:
items:
$ref: '#/components/schemas/Tag'
readOnly: true
+ max_active_tasks:
+ type: integer
+ nullable: true
+ readOnly: true
+ description: |
+ Maximum number of active tasks that can be run on the DAG
+
+ *New in version 2.3.0*
+ max_active_runs:
+ type: integer
+ nullable: true
+ readOnly: true
+ description: |
+ Maximum number of active DAG runs for the DAG
+
+ *New in version 2.3.0*
+ has_task_concurrency_limits:
+ type: boolean
+ nullable: true
+ readOnly: true
+ description: |
+ Whether the DAG has task concurrency limits
+
+ *New in version 2.3.0*
+ has_import_errors:
+ type: boolean
+ nullable: true
+ readOnly: true
+ description: |
+ Whether the DAG has import errors
+
+ *New in version 2.3.0*
+ next_dagrun:
+ type: string
+ format: date-time
+ readOnly: true
+ nullable: true
+ description: |
+ The logical date of the next dag run.
+
+ *New in version 2.3.0*
+ next_dagrun_data_interval_start:
+ type: string
+ format: date-time
+ readOnly: true
+ nullable: true
+ description: |
+ The start of the interval of the next dag run.
+
+ *New in version 2.3.0*
+ next_dagrun_data_interval_end:
+ type: string
+ format: date-time
+ readOnly: true
+ nullable: true
+ description: |
+ The end of the interval of the next dag run.
+
+ *New in version 2.3.0*
+ next_dagrun_create_after:
+ type: string
+ format: date-time
+ readOnly: true
+ nullable: true
+ description: |
+ Earliest time at which this ``next_dagrun`` can be created.
+
+ *New in version 2.3.0*
DAGCollection:
description: |
@@ -2220,6 +2429,28 @@ components:
format: date-time
readOnly: true
nullable: true
+ data_interval_start:
+ type: string
+ format: date-time
+ readOnly: true
+ nullable: true
+ data_interval_end:
+ type: string
+ format: date-time
+ readOnly: true
+ nullable: true
+ last_scheduling_decision:
+ type: string
+ format: date-time
+ readOnly: true
+ nullable: true
+ run_type:
+ type: string
+ readOnly: true
+ enum:
+ - backfill
+ - manual
+ - scheduled
state:
$ref: '#/components/schemas/DagState'
readOnly: true
@@ -2248,6 +2479,7 @@ components:
enum:
- success
- failed
+ - queued
DAGRunCollection:
type: object
@@ -2740,7 +2972,49 @@ components:
User-specified DAG params.
*New in version 2.0.1*
+ end_date:
+ type: string
+ format: 'date-time'
+ readOnly: true
+ nullable: true
+ description: |
+ The DAG's end date.
+
+ *New in version 2.3.0*.
+ is_paused_upon_creation:
+ type: boolean
+ readOnly: true
+ nullable: true
+ description: |
+ Whether the DAG is paused upon creation.
+ *New in version 2.3.0*
+ last_parsed:
+ type: string
+ format: date-time
+ nullable: true
+ readOnly: true
+ description: |
+ The last time the DAG was parsed.
+
+ *New in version 2.3.0*
+ template_search_path:
+ type: array
+ nullable: true
+ items:
+ type: string
+ description: |
+ The template search path.
+
+ *New in version 2.3.0*
+ render_template_as_native_obj:
+ type: boolean
+ nullable: true
+ readOnly: true
+ description: |
+ Whether to render templates as native Python objects.
+
+ *New in version 2.3.0*
ExtraLink:
type: object
description: Additional links containing additional information about the task.
@@ -2799,6 +3073,9 @@ components:
depends_on_past:
type: boolean
readOnly: true
+ is_mapped:
+ type: boolean
+ readOnly: true
wait_for_downstream:
type: boolean
readOnly: true
@@ -3071,6 +3348,16 @@ components:
nullable: true
# Form
+ ClearDagRun:
+ type: object
+ properties:
+ dry_run:
+ description: |
+ If set, don't actually run this operation. The response will contain a list of task instances
+ planned to be cleaned, but not modified in any way.
+ type: boolean
+ default: true
+
ClearTaskInstance:
type: object
properties:
@@ -3348,7 +3635,7 @@ components:
Schedule interval. Defines how often DAG runs, this object gets added to your latest task instance's
execution_date to figure out the next schedule.
readOnly: true
- oneOf:
+ anyOf:
- $ref: '#/components/schemas/TimeDelta'
- $ref: '#/components/schemas/RelativeDelta'
- $ref: '#/components/schemas/CronExpression'
diff --git a/airflow/api_connexion/schemas/connection_schema.py b/airflow/api_connexion/schemas/connection_schema.py
index 140e8b1de3515..f06da92bacdab 100644
--- a/airflow/api_connexion/schemas/connection_schema.py
+++ b/airflow/api_connexion/schemas/connection_schema.py
@@ -45,7 +45,7 @@ class ConnectionSchema(ConnectionCollectionItemSchema):
"""Connection schema"""
password = auto_field(load_only=True)
- extra = fields.Method('serialize_extra', deserialize='deserialize_extra')
+ extra = fields.Method('serialize_extra', deserialize='deserialize_extra', allow_none=True)
@staticmethod
def serialize_extra(obj: Connection):
diff --git a/airflow/api_connexion/schemas/dag_run_schema.py b/airflow/api_connexion/schemas/dag_run_schema.py
index 4bd91dbe372ba..5cd79b20228fb 100644
--- a/airflow/api_connexion/schemas/dag_run_schema.py
+++ b/airflow/api_connexion/schemas/dag_run_schema.py
@@ -67,6 +67,10 @@ class Meta:
state = DagStateField(dump_only=True)
external_trigger = auto_field(dump_default=True, dump_only=True)
conf = ConfObject()
+ data_interval_start = auto_field(dump_only=True)
+ data_interval_end = auto_field(dump_only=True)
+ last_scheduling_decision = auto_field(dump_only=True)
+ run_type = auto_field(dump_only=True)
@pre_load
def autogenerate(self, data, **kwargs):
@@ -108,7 +112,17 @@ def autofill(self, data, **kwargs):
class SetDagRunStateFormSchema(Schema):
"""Schema for handling the request of setting state of DAG run"""
- state = DagStateField(validate=validate.OneOf([DagRunState.SUCCESS.value, DagRunState.FAILED.value]))
+ state = DagStateField(
+ validate=validate.OneOf(
+ [DagRunState.SUCCESS.value, DagRunState.FAILED.value, DagRunState.QUEUED.value]
+ )
+ )
+
+
+class ClearDagRunStateFormSchema(Schema):
+ """Schema for handling the request of clearing a DAG run"""
+
+ dry_run = fields.Boolean(load_default=True)
class DAGRunCollection(NamedTuple):
@@ -150,4 +164,5 @@ class Meta:
dagrun_schema = DAGRunSchema()
dagrun_collection_schema = DAGRunCollectionSchema()
set_dagrun_state_form_schema = SetDagRunStateFormSchema()
+clear_dagrun_form_schema = ClearDagRunStateFormSchema()
dagruns_batch_form_schema = DagRunsBatchFormSchema()
diff --git a/airflow/api_connexion/schemas/dag_schema.py b/airflow/api_connexion/schemas/dag_schema.py
index 3a543bbb86410..2f369113290d9 100644
--- a/airflow/api_connexion/schemas/dag_schema.py
+++ b/airflow/api_connexion/schemas/dag_schema.py
@@ -51,12 +51,27 @@ class Meta:
is_paused = auto_field()
is_active = auto_field(dump_only=True)
is_subdag = auto_field(dump_only=True)
+ last_parsed_time = auto_field(dump_only=True)
+ last_pickled = auto_field(dump_only=True)
+ last_expired = auto_field(dump_only=True)
+ scheduler_lock = auto_field(dump_only=True)
+ pickle_id = auto_field(dump_only=True)
+ default_view = auto_field(dump_only=True)
fileloc = auto_field(dump_only=True)
file_token = fields.Method("get_token", dump_only=True)
owners = fields.Method("get_owners", dump_only=True)
description = auto_field(dump_only=True)
schedule_interval = fields.Nested(ScheduleIntervalSchema)
+ timetable_description = auto_field(dump_only=True)
tags = fields.List(fields.Nested(DagTagSchema), dump_only=True)
+ max_active_tasks = auto_field(dump_only=True)
+ max_active_runs = auto_field(dump_only=True)
+ has_task_concurrency_limits = auto_field(dump_only=True)
+ has_import_errors = auto_field(dump_only=True)
+ next_dagrun = auto_field(dump_only=True)
+ next_dagrun_data_interval_start = auto_field(dump_only=True)
+ next_dagrun_data_interval_end = auto_field(dump_only=True)
+ next_dagrun_create_after = auto_field(dump_only=True)
@staticmethod
def get_owners(obj: DagModel):
@@ -89,6 +104,11 @@ class DAGDetailSchema(DAGSchema):
tags = fields.Method("get_tags", dump_only=True) # type: ignore
is_paused = fields.Method("get_is_paused", dump_only=True)
is_active = fields.Method("get_is_active", dump_only=True)
+ is_paused_upon_creation = fields.Boolean()
+ end_date = fields.DateTime(dump_only=True)
+ template_search_path = fields.String(dump_only=True)
+ render_template_as_native_obj = fields.Boolean(dump_only=True)
+ last_loaded = fields.DateTime(dump_only=True, data_key='last_parsed')
@staticmethod
def get_concurrency(obj: DAG):
diff --git a/airflow/api_connexion/schemas/task_schema.py b/airflow/api_connexion/schemas/task_schema.py
index 600d28ac0d6c6..aa9a4703088b0 100644
--- a/airflow/api_connexion/schemas/task_schema.py
+++ b/airflow/api_connexion/schemas/task_schema.py
@@ -58,6 +58,7 @@ class TaskSchema(Schema):
sub_dag = fields.Nested(DAGSchema, dump_only=True)
downstream_task_ids = fields.List(fields.String(), dump_only=True)
params = fields.Method('get_params', dump_only=True)
+ is_mapped = fields.Boolean(dump_only=True)
def _get_class_reference(self, obj):
result = ClassReferenceSchema().dump(obj)
diff --git a/airflow/cli/cli_parser.py b/airflow/cli/cli_parser.py
index 9f8dda47cb50b..60c77c7a37feb 100644
--- a/airflow/cli/cli_parser.py
+++ b/airflow/cli/cli_parser.py
@@ -28,7 +28,7 @@
import lazy_object_proxy
-from airflow import PY37, settings
+from airflow import settings
from airflow.cli.commands.legacy_commands import check_legacy_command
from airflow.configuration import conf
from airflow.exceptions import AirflowException
@@ -97,9 +97,6 @@ def _check_value(self, action, value):
"To do it, run: pip install 'apache-airflow[cncf.kubernetes]'"
)
raise ArgumentError(action, message)
- if action.dest == 'subcommand' and value == 'triggerer':
- if not PY37:
- raise ArgumentError(action, 'triggerer subcommand only works with Python 3.7+')
if action.choices is not None and value not in action.choices:
check_legacy_command(action, value)
@@ -337,6 +334,11 @@ def string_lower_type(val):
),
action="store_true",
)
+ARG_CONTINUE_ON_FAILURES = Arg(
+ ("--continue-on-failures",),
+ help=("if set, the backfill will keep going even if some of the tasks failed"),
+ action="store_true",
+)
ARG_RUN_BACKWARDS = Arg(
(
"-B",
@@ -510,7 +512,6 @@ def string_lower_type(val):
("--ship-dag",), help="Pickles (serializes) the DAG and ships it to the worker", action="store_true"
)
ARG_PICKLE = Arg(("-p", "--pickle"), help="Serialized pickle object of the entire dag (used internally)")
-ARG_ERROR_FILE = Arg(("--error-file",), help="File to store task failure error")
ARG_JOB_ID = Arg(("-j", "--job-id"), help=argparse.SUPPRESS)
ARG_CFG_PATH = Arg(("--cfg-path",), help="Path to config file to use instead of airflow.cfg")
ARG_MAP_INDEX = Arg(('--map-index',), type=int, default=-1, help="Mapped task index")
@@ -524,18 +525,18 @@ def string_lower_type(val):
default=60,
)
ARG_DB_VERSION__UPGRADE = Arg(
- ("-n", "--version"),
+ ("-n", "--to-version"),
help=(
"(Optional) The airflow version to upgrade to. Note: must provide either "
- "`--revision` or `--version`."
+ "`--to-revision` or `--to-version`."
),
)
ARG_DB_REVISION__UPGRADE = Arg(
- ("-r", "--revision"),
- help="(Optional) If provided, only run migrations up to and including this revision.",
+ ("-r", "--to-revision"),
+ help="(Optional) If provided, only run migrations up to and including this Alembic revision.",
)
ARG_DB_VERSION__DOWNGRADE = Arg(
- ("-n", "--version"),
+ ("-n", "--to-version"),
help="(Optional) If provided, only run migrations up to this version.",
)
ARG_DB_FROM_VERSION = Arg(
@@ -543,12 +544,12 @@ def string_lower_type(val):
help="(Optional) If generating sql, may supply a *from* version",
)
ARG_DB_REVISION__DOWNGRADE = Arg(
- ("-r", "--revision"),
- help="The airflow revision to downgrade to. Note: must provide either `--revision` or `--version`.",
+ ("-r", "--to-revision"),
+ help="The Alembic revision to downgrade to. Note: must provide either `--to-revision` or `--to-version`.",
)
ARG_DB_FROM_REVISION = Arg(
("--from-revision",),
- help="(Optional) If generating sql, may supply a *from* revision",
+ help="(Optional) If generating sql, may supply a *from* Alembic revision",
)
ARG_DB_SQL_ONLY = Arg(
("-s", "--show-sql-only"),
@@ -557,6 +558,12 @@ def string_lower_type(val):
action="store_true",
default=False,
)
+ARG_DB_SKIP_INIT = Arg(
+ ("-s", "--skip-init"),
+ help="Only remove tables; do not perform db init.",
+ action="store_true",
+ default=False,
+)
# webserver
ARG_PORT = Arg(
@@ -1111,6 +1118,7 @@ class GroupCommand(NamedTuple):
ARG_LOCAL,
ARG_DONOT_PICKLE,
ARG_YES,
+ ARG_CONTINUE_ON_FAILURES,
ARG_BF_IGNORE_DEPENDENCIES,
ARG_BF_IGNORE_FIRST_DEPENDS_ON_PAST,
ARG_SUBDIR,
@@ -1255,7 +1263,6 @@ class GroupCommand(NamedTuple):
ARG_PICKLE,
ARG_JOB_ID,
ARG_INTERACTIVE,
- ARG_ERROR_FILE,
ARG_SHUT_DOWN_LOGGING,
ARG_MAP_INDEX,
),
@@ -1381,7 +1388,7 @@ class GroupCommand(NamedTuple):
name='reset',
help="Burn down and rebuild the metadata database",
func=lazy_load_command('airflow.cli.commands.db_command.resetdb'),
- args=(ARG_YES,),
+ args=(ARG_YES, ARG_DB_SKIP_INIT),
),
ActionCommand(
name='upgrade',
@@ -1391,7 +1398,7 @@ class GroupCommand(NamedTuple):
"To print but not execute commands, use option ``--show-sql-only``. "
"If using options ``--from-revision`` or ``--from-version``, you must also use "
"``--show-sql-only``, because if actually *running* migrations, we should only "
- "migrate from the *current* revision."
+ "migrate from the *current* Alembic revision."
),
func=lazy_load_command('airflow.cli.commands.db_command.upgradedb'),
args=(
@@ -1407,10 +1414,11 @@ class GroupCommand(NamedTuple):
help="Downgrade the schema of the metadata database.",
description=(
"Downgrade the schema of the metadata database. "
- "You must provide either `--revision` or `--version`. "
+ "You must provide either `--to-revision` or `--to-version`. "
"To print but not execute commands, use option `--show-sql-only`. "
"If using options `--from-revision` or `--from-version`, you must also use `--show-sql-only`, "
- "because if actually *running* migrations, we should only migrate from the *current* revision."
+ "because if actually *running* migrations, we should only migrate from the *current* Alembic "
+ "revision."
),
func=lazy_load_command('airflow.cli.commands.db_command.downgrade'),
args=(
@@ -1863,6 +1871,21 @@ class GroupCommand(NamedTuple):
ARG_CAPACITY,
),
),
+ ActionCommand(
+ name='dag-processor',
+ help="Start a standalone Dag Processor instance",
+ func=lazy_load_command('airflow.cli.commands.dag_processor_command.dag_processor'),
+ args=(
+ ARG_PID,
+ ARG_DAEMON,
+ ARG_SUBDIR,
+ ARG_NUM_RUNS,
+ ARG_DO_PICKLE,
+ ARG_STDOUT,
+ ARG_STDERR,
+ ARG_LOG_FILE,
+ ),
+ ),
ActionCommand(
name='version',
help="Show the version",
diff --git a/airflow/cli/commands/celery_command.py b/airflow/cli/commands/celery_command.py
index a962d4cacbc68..affe032411c79 100644
--- a/airflow/cli/commands/celery_command.py
+++ b/airflow/cli/commands/celery_command.py
@@ -23,7 +23,7 @@
import daemon
import psutil
import sqlalchemy.exc
-from celery import maybe_patch_concurrency
+from celery import maybe_patch_concurrency # type: ignore[attr-defined]
from daemon.pidfile import TimeoutPIDLockFile
from lockfile.pidlockfile import read_pid_from_pidfile, remove_existing_pidfile
@@ -100,6 +100,8 @@ def _run_worker(options, skip_serve_logs):
@cli_utils.action_cli
def worker(args):
"""Starts Airflow Celery worker"""
+ # Disable connection pool so that celery worker does not hold an unnecessary db connection
+ settings.reconfigure_orm(disable_connection_pool=True)
if not settings.validate_session():
raise SystemExit("Worker exiting, database connection precheck failed.")
diff --git a/airflow/cli/commands/connection_command.py b/airflow/cli/commands/connection_command.py
index c07cea72f46ed..8a0c0a3acb80a 100644
--- a/airflow/cli/commands/connection_command.py
+++ b/airflow/cli/commands/connection_command.py
@@ -27,9 +27,11 @@
from sqlalchemy.orm import exc
from airflow.cli.simple_table import AirflowConsole
+from airflow.compat.functools import cache
from airflow.exceptions import AirflowNotFoundException
from airflow.hooks.base import BaseHook
from airflow.models import Connection
+from airflow.providers_manager import ProvidersManager
from airflow.secrets.local_filesystem import load_connections_dict
from airflow.utils import cli as cli_utils, yaml
from airflow.utils.cli import suppress_logs_and_warning
@@ -133,6 +135,21 @@ def _valid_uri(uri: str) -> bool:
return uri_parts.scheme != '' and uri_parts.netloc != ''
+@cache
+def _get_connection_types():
+ """Returns connection types available."""
+ _connection_types = ['fs', 'mesos_framework-id', 'email', 'generic']
+ providers_manager = ProvidersManager()
+ for connection_type, provider_info in providers_manager.hooks.items():
+ if provider_info:
+ _connection_types.append(connection_type)
+ return _connection_types
+
+
+def _valid_conn_type(conn_type: str) -> bool:
+ return conn_type in _get_connection_types()
+
+
def connections_export(args):
"""Exports all connections to a file"""
file_formats = ['.yaml', '.json', '.env']
@@ -195,6 +212,14 @@ def connections_add(args):
if has_json and has_uri:
raise SystemExit('Cannot supply both conn-uri and conn-json')
+ if has_type and not (args.conn_type in _get_connection_types()):
+ warnings.warn(f'The type provided to --conn-type is invalid: {args.conn_type}')
+ warnings.warn(
+ f'Supported --conn-types are:{_get_connection_types()}.'
+ 'Hence overriding the conn-type with generic'
+ )
+ args.conn_type = 'generic'
+
if has_uri or has_json:
invalid_args = []
if has_uri and not _valid_uri(args.conn_uri):
@@ -245,12 +270,8 @@ def connections_add(args):
or urlunparse(
(
new_conn.conn_type,
- '{login}:{password}@{host}:{port}'.format(
- login=new_conn.login or '',
- password='******' if new_conn.password else '',
- host=new_conn.host or '',
- port=new_conn.port or '',
- ),
+ f"{new_conn.login or ''}:{'******' if new_conn.password else ''}"
+ f"@{new_conn.host or ''}:{new_conn.port or ''}",
new_conn.schema or '',
'',
'',
diff --git a/airflow/cli/commands/dag_command.py b/airflow/cli/commands/dag_command.py
index 57df9cceca30b..a06c5a706907a 100644
--- a/airflow/cli/commands/dag_command.py
+++ b/airflow/cli/commands/dag_command.py
@@ -118,6 +118,7 @@ def dag_backfill(args, dag=None):
conf=run_conf,
rerun_failed_tasks=args.rerun_failed_tasks,
run_backwards=args.run_backwards,
+ continue_on_failures=args.continue_on_failures,
)
except ValueError as vr:
print(str(vr))
diff --git a/airflow/cli/commands/dag_processor_command.py b/airflow/cli/commands/dag_processor_command.py
new file mode 100644
index 0000000000000..d57e26510c974
--- /dev/null
+++ b/airflow/cli/commands/dag_processor_command.py
@@ -0,0 +1,77 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""DagProcessor command"""
+import logging
+from datetime import timedelta
+
+import daemon
+from daemon.pidfile import TimeoutPIDLockFile
+
+from airflow.configuration import conf
+from airflow.dag_processing.manager import DagFileProcessorManager
+from airflow.utils import cli as cli_utils
+from airflow.utils.cli import setup_locations, setup_logging
+
+log = logging.getLogger(__name__)
+
+
+def _create_dag_processor_manager(args) -> DagFileProcessorManager:
+ """Creates DagFileProcessorProcess instance."""
+ processor_timeout_seconds: int = conf.getint('core', 'dag_file_processor_timeout')
+ processor_timeout = timedelta(seconds=processor_timeout_seconds)
+ return DagFileProcessorManager(
+ dag_directory=args.subdir,
+ max_runs=args.num_runs,
+ processor_timeout=processor_timeout,
+ dag_ids=[],
+ pickle_dags=args.do_pickle,
+ )
+
+
+@cli_utils.action_cli
+def dag_processor(args):
+ """Starts Airflow Dag Processor Job"""
+ if not conf.getboolean("scheduler", "standalone_dag_processor"):
+ raise SystemExit('The option [scheduler/standalone_dag_processor] must be True.')
+
+ sql_conn: str = conf.get('database', 'sql_alchemy_conn').lower()
+ if sql_conn.startswith('sqlite'):
+ raise SystemExit('Standalone DagProcessor is not supported when using sqlite.')
+
+ manager = _create_dag_processor_manager(args)
+
+ if args.daemon:
+ pid, stdout, stderr, log_file = setup_locations(
+ "dag-processor", args.pid, args.stdout, args.stderr, args.log_file
+ )
+ handle = setup_logging(log_file)
+ with open(stdout, 'w+') as stdout_handle, open(stderr, 'w+') as stderr_handle:
+ ctx = daemon.DaemonContext(
+ pidfile=TimeoutPIDLockFile(pid, -1),
+ files_preserve=[handle],
+ stdout=stdout_handle,
+ stderr=stderr_handle,
+ )
+ with ctx:
+ try:
+ manager.start()
+ finally:
+ manager.terminate()
+ manager.end()
+ else:
+ manager.start()
diff --git a/airflow/cli/commands/db_command.py b/airflow/cli/commands/db_command.py
index 02f34738c4ca0..c9201ad59ba80 100644
--- a/airflow/cli/commands/db_command.py
+++ b/airflow/cli/commands/db_command.py
@@ -39,25 +39,24 @@ def initdb(args):
def resetdb(args):
"""Resets the metadata database"""
print("DB: " + repr(settings.engine.url))
- if args.yes or input("This will drop existing tables if they exist. Proceed? (y/n)").upper() == "Y":
- db.resetdb()
- else:
- print("Cancelled")
+ if not (args.yes or input("This will drop existing tables if they exist. Proceed? (y/n)").upper() == "Y"):
+ raise SystemExit("Cancelled")
+ db.resetdb(skip_init=args.skip_init)
@cli_utils.action_cli(check_db=False)
def upgradedb(args):
"""Upgrades the metadata database"""
print("DB: " + repr(settings.engine.url))
- if args.revision and args.version:
- raise SystemExit("Cannot supply both `--revision` and `--version`.")
+ if args.to_revision and args.to_version:
+ raise SystemExit("Cannot supply both `--to-revision` and `--to-version`.")
if args.from_version and args.from_revision:
raise SystemExit("Cannot supply both `--from-revision` and `--from-version`")
if (args.from_revision or args.from_version) and not args.show_sql_only:
raise SystemExit(
"Args `--from-revision` and `--from-version` may only be used with `--show-sql-only`"
)
- revision = None
+ to_revision = None
from_revision = None
if args.from_revision:
from_revision = args.from_revision
@@ -68,19 +67,19 @@ def upgradedb(args):
if not from_revision:
raise SystemExit(f"Unknown version {args.from_version!r} supplied as `--from-version`.")
- if args.version:
- revision = REVISION_HEADS_MAP.get(args.version)
- if not revision:
- raise SystemExit(f"Upgrading to version {args.version} is not supported.")
- elif args.revision:
- revision = args.revision
+ if args.to_version:
+ to_revision = REVISION_HEADS_MAP.get(args.to_version)
+ if not to_revision:
+ raise SystemExit(f"Upgrading to version {args.to_version} is not supported.")
+ elif args.to_revision:
+ to_revision = args.to_revision
if not args.show_sql_only:
print("Performing upgrade with database " + repr(settings.engine.url))
else:
print("Generating sql for upgrade -- upgrade commands will *not* be submitted.")
- db.upgradedb(to_revision=revision, from_revision=from_revision, show_sql_only=args.show_sql_only)
+ db.upgradedb(to_revision=to_revision, from_revision=from_revision, show_sql_only=args.show_sql_only)
if not args.show_sql_only:
print("Upgrades done")
@@ -88,29 +87,29 @@ def upgradedb(args):
@cli_utils.action_cli(check_db=False)
def downgrade(args):
"""Downgrades the metadata database"""
- if args.revision and args.version:
- raise SystemExit("Cannot supply both `revision` and `version`.")
+ if args.to_revision and args.to_version:
+ raise SystemExit("Cannot supply both `--to-revision` and `--to-version`.")
if args.from_version and args.from_revision:
raise SystemExit("`--from-revision` may not be combined with `--from-version`")
if (args.from_revision or args.from_version) and not args.show_sql_only:
raise SystemExit(
"Args `--from-revision` and `--from-version` may only be used with `--show-sql-only`"
)
- if not (args.version or args.revision):
- raise SystemExit("Must provide either revision or version.")
+ if not (args.to_version or args.to_revision):
+ raise SystemExit("Must provide either --to-revision or --to-version.")
from_revision = None
if args.from_revision:
from_revision = args.from_revision
elif args.from_version:
from_revision = REVISION_HEADS_MAP.get(args.from_version)
if not from_revision:
- raise SystemExit(f"Unknown version {args.version!r} supplied as `--from-version`.")
- if args.version:
- revision = REVISION_HEADS_MAP.get(args.version)
- if not revision:
- raise SystemExit(f"Downgrading to version {args.version} is not supported.")
- elif args.revision:
- revision = args.revision
+ raise SystemExit(f"Unknown version {args.from_version!r} supplied as `--from-version`.")
+ if args.to_version:
+ to_revision = REVISION_HEADS_MAP.get(args.to_version)
+ if not to_revision:
+ raise SystemExit(f"Downgrading to version {args.to_version} is not supported.")
+ elif args.to_revision:
+ to_revision = args.to_revision
if not args.show_sql_only:
print("Performing downgrade with database " + repr(settings.engine.url))
else:
@@ -125,7 +124,7 @@ def downgrade(args):
).upper()
== "Y"
):
- db.downgrade(to_revision=revision, from_revision=from_revision, show_sql_only=args.show_sql_only)
+ db.downgrade(to_revision=to_revision, from_revision=from_revision, show_sql_only=args.show_sql_only)
if not args.show_sql_only:
print("Downgrade complete")
else:
diff --git a/airflow/cli/commands/info_command.py b/airflow/cli/commands/info_command.py
index bb474fc3ab184..fc03615210af3 100644
--- a/airflow/cli/commands/info_command.py
+++ b/airflow/cli/commands/info_command.py
@@ -221,7 +221,7 @@ def get_fullname(o):
def _airflow_info(self):
executor = configuration.conf.get("core", "executor")
sql_alchemy_conn = self.anonymizer.process_url(
- configuration.conf.get("core", "SQL_ALCHEMY_CONN", fallback="NOT AVAILABLE")
+ configuration.conf.get("database", "SQL_ALCHEMY_CONN", fallback="NOT AVAILABLE")
)
dags_folder = self.anonymizer.process_path(
configuration.conf.get("core", "dags_folder", fallback="NOT AVAILABLE")
@@ -304,7 +304,7 @@ def _paths_info(self):
@property
def _providers_info(self):
- return [(p.provider_info['package-name'], p.version) for p in ProvidersManager().providers.values()]
+ return [(p.data['package-name'], p.version) for p in ProvidersManager().providers.values()]
def show(self, output: str, console: Optional[AirflowConsole] = None) -> None:
"""Shows information about Airflow instance"""
diff --git a/airflow/cli/commands/pool_command.py b/airflow/cli/commands/pool_command.py
index 684cd7ea5e742..e435c2a4833bc 100644
--- a/airflow/cli/commands/pool_command.py
+++ b/airflow/cli/commands/pool_command.py
@@ -64,7 +64,7 @@ def pool_set(args):
"""Creates new pool with a given name and slots"""
api_client = get_current_api_client()
api_client.create_pool(name=args.pool, slots=args.slots, description=args.description)
- print("Pool created")
+ print(f"Pool {args.pool} created")
@cli_utils.action_cli
@@ -74,7 +74,7 @@ def pool_delete(args):
api_client = get_current_api_client()
try:
api_client.delete_pool(name=args.pool)
- print("Pool deleted")
+ print(f"Pool {args.pool} deleted")
except PoolNotFound:
raise SystemExit(f"Pool {args.pool} does not exist")
@@ -84,7 +84,7 @@ def pool_delete(args):
def pool_import(args):
"""Imports pools from the file"""
if not os.path.exists(args.file):
- raise SystemExit("Missing pools file.")
+ raise SystemExit(f"Missing pools file {args.file}")
pools, failed = pool_import_helper(args.file)
if len(failed) > 0:
raise SystemExit(f"Failed to update pool(s): {', '.join(failed)}")
@@ -106,7 +106,7 @@ def pool_import_helper(filepath):
try:
pools_json = json.loads(data)
except JSONDecodeError as e:
- raise SystemExit("Invalid json file: " + str(e))
+ raise SystemExit(f"Invalid json file: {e}")
pools = []
failed = []
for k, v in pools_json.items():
diff --git a/airflow/cli/commands/provider_command.py b/airflow/cli/commands/provider_command.py
index 08ec94a2cbaa7..b89f77fb5d2a4 100644
--- a/airflow/cli/commands/provider_command.py
+++ b/airflow/cli/commands/provider_command.py
@@ -34,7 +34,7 @@ def provider_get(args):
providers = ProvidersManager().providers
if args.provider_name in providers:
provider_version = providers[args.provider_name].version
- provider_info = providers[args.provider_name].provider_info
+ provider_info = providers[args.provider_name].data
if args.full:
provider_info["description"] = _remove_rst_syntax(provider_info["description"])
AirflowConsole().print_as(
@@ -55,9 +55,9 @@ def providers_list(args):
data=list(ProvidersManager().providers.values()),
output=args.output,
mapper=lambda x: {
- "package_name": x[1]["package-name"],
- "description": _remove_rst_syntax(x[1]["description"]),
- "version": x[0],
+ "package_name": x.data["package-name"],
+ "description": _remove_rst_syntax(x.data["description"]),
+ "version": x.version,
},
)
diff --git a/airflow/cli/commands/standalone_command.py b/airflow/cli/commands/standalone_command.py
index 82a082e853f6b..3860942adb056 100644
--- a/airflow/cli/commands/standalone_command.py
+++ b/airflow/cli/commands/standalone_command.py
@@ -157,7 +157,7 @@ def calculate_env(self):
executor_constants.LOCAL_EXECUTOR,
executor_constants.SEQUENTIAL_EXECUTOR,
]:
- if "sqlite" in conf.get("core", "sql_alchemy_conn"):
+ if "sqlite" in conf.get("database", "sql_alchemy_conn"):
self.print_output("standalone", "Forcing executor to SequentialExecutor")
env["AIRFLOW__CORE__EXECUTOR"] = executor_constants.SEQUENTIAL_EXECUTOR
else:
diff --git a/airflow/cli/commands/task_command.py b/airflow/cli/commands/task_command.py
index 3c82740383d0f..f8caf08487187 100644
--- a/airflow/cli/commands/task_command.py
+++ b/airflow/cli/commands/task_command.py
@@ -23,7 +23,7 @@
import os
import textwrap
from contextlib import contextmanager, redirect_stderr, redirect_stdout
-from typing import List, Optional
+from typing import Dict, Generator, List, Optional, Tuple, Union
from pendulum.parsing.exceptions import ParserError
from sqlalchemy.orm.exc import NoResultFound
@@ -39,12 +39,13 @@
from airflow.models.baseoperator import BaseOperator
from airflow.models.dag import DAG
from airflow.models.dagrun import DagRun
-from airflow.models.xcom import IN_MEMORY_RUN_ID
from airflow.ti_deps.dep_context import DepContext
from airflow.ti_deps.dependencies_deps import SCHEDULER_QUEUED_DEPS
+from airflow.typing_compat import Literal
from airflow.utils import cli as cli_utils
from airflow.utils.cli import (
get_dag,
+ get_dag_by_deserialization,
get_dag_by_file_location,
get_dag_by_pickle,
get_dags,
@@ -54,15 +55,29 @@
from airflow.utils.log.logging_mixin import StreamLogWriter
from airflow.utils.net import get_hostname
from airflow.utils.session import NEW_SESSION, create_session, provide_session
+from airflow.utils.state import DagRunState
+
+log = logging.getLogger(__name__)
+
+CreateIfNecessary = Union[Literal[False], Literal["db"], Literal["memory"]]
+
+
+def _generate_temporary_run_id() -> str:
+ """Generate a ``run_id`` for a DAG run that will be created temporarily.
+
+ This is used mostly by ``airflow task test`` to create a DAG run that will
+ be deleted after the task is run.
+ """
+ return f"__airflow_temporary_run_{timezone.utcnow().isoformat()}__"
def _get_dag_run(
*,
dag: DAG,
exec_date_or_run_id: str,
- create_if_necessary: bool,
+ create_if_necessary: CreateIfNecessary,
session: Session,
-) -> DagRun:
+) -> Tuple[DagRun, bool]:
"""Try to retrieve a DAG run from a string representing either a run ID or logical date.
This checks DAG runs like this:
@@ -78,7 +93,7 @@ def _get_dag_run(
"""
dag_run = dag.get_dagrun(run_id=exec_date_or_run_id, session=session)
if dag_run:
- return dag_run
+ return dag_run, False
try:
execution_date: Optional[datetime.datetime] = timezone.parse(exec_date_or_run_id)
@@ -86,7 +101,7 @@ def _get_dag_run(
execution_date = None
try:
- return (
+ dag_run = (
session.query(DagRun)
.filter(DagRun.dag_id == dag.dag_id, DagRun.execution_date == execution_date)
.one()
@@ -96,10 +111,25 @@ def _get_dag_run(
raise DagRunNotFound(
f"DagRun for {dag.dag_id} with run_id or execution_date of {exec_date_or_run_id!r} not found"
) from None
+ else:
+ return dag_run, False
if execution_date is not None:
- return DagRun(dag.dag_id, run_id=IN_MEMORY_RUN_ID, execution_date=execution_date)
- return DagRun(dag.dag_id, run_id=exec_date_or_run_id, execution_date=timezone.utcnow())
+ dag_run_execution_date = execution_date
+ else:
+ dag_run_execution_date = timezone.utcnow()
+ if create_if_necessary == "memory":
+ dag_run = DagRun(dag.dag_id, run_id=exec_date_or_run_id, execution_date=dag_run_execution_date)
+ return dag_run, True
+ elif create_if_necessary == "db":
+ dag_run = dag.create_dagrun(
+ state=DagRunState.QUEUED,
+ execution_date=dag_run_execution_date,
+ run_id=_generate_temporary_run_id(),
+ session=session,
+ )
+ return dag_run, True
+ raise ValueError(f"unknown create_if_necessary value: {create_if_necessary!r}")
@provide_session
@@ -108,16 +138,17 @@ def _get_ti(
exec_date_or_run_id: str,
map_index: int,
*,
- create_if_necessary: bool = False,
+ pool: Optional[str] = None,
+ create_if_necessary: CreateIfNecessary = False,
session: Session = NEW_SESSION,
-) -> TaskInstance:
+) -> Tuple[TaskInstance, bool]:
"""Get the task instance through DagRun.run_id, if that fails, get the TI the old way"""
if task.is_mapped:
if map_index < 0:
raise RuntimeError("No map_index passed to mapped task")
elif map_index >= 0:
raise RuntimeError("map_index passed to non-mapped task")
- dag_run = _get_dag_run(
+ dag_run, dr_created = _get_dag_run(
dag=task.dag,
exec_date_or_run_id=exec_date_or_run_id,
create_if_necessary=create_if_necessary,
@@ -136,8 +167,8 @@ def _get_ti(
ti.dag_run = dag_run
else:
ti = ti_or_none
- ti.refresh_from_task(task)
- return ti
+ ti.refresh_from_task(task, pool_override=pool)
+ return ti, dr_created
def _run_task_by_selected_method(args, dag: DAG, ti: TaskInstance) -> None:
@@ -228,7 +259,6 @@ def _run_raw_task(args, ti: TaskInstance) -> None:
mark_success=args.mark_success,
job_id=args.job_id,
pool=args.pool,
- error_file=args.error_file,
)
@@ -239,7 +269,7 @@ def _extract_external_executor_id(args) -> Optional[str]:
@contextmanager
-def _capture_task_logs(ti):
+def _capture_task_logs(ti: TaskInstance) -> Generator[None, None, None]:
"""Manage logging context for a task run
- Replace the root logger configuration with the airflow.task configuration
@@ -327,17 +357,24 @@ def task_run(args, dag=None):
print(f'Loading pickle id: {args.pickle}')
dag = get_dag_by_pickle(args.pickle)
elif not dag:
- dag = get_dag(args.subdir, args.dag_id)
+ if args.local:
+ try:
+ dag = get_dag_by_deserialization(args.dag_id)
+ except AirflowException:
+ print(f'DAG {args.dag_id} does not exist in the database, trying to parse the dag_file')
+ dag = get_dag(args.subdir, args.dag_id)
+ else:
+ dag = get_dag(args.subdir, args.dag_id)
else:
# Use DAG from parameter
pass
task = dag.get_task(task_id=args.task_id)
- ti = _get_ti(task, args.execution_date_or_run_id, args.map_index)
+ ti, _ = _get_ti(task, args.execution_date_or_run_id, args.map_index, pool=args.pool)
ti.init_run_context(raw=args.raw)
hostname = get_hostname()
- print(f"Running {ti} on host {hostname}")
+ log.info("Running %s on host %s", ti, hostname)
if args.interactive:
_run_task_by_selected_method(args, dag, ti)
@@ -360,7 +397,7 @@ def task_failed_deps(args):
"""
dag = get_dag(args.subdir, args.dag_id)
task = dag.get_task(task_id=args.task_id)
- ti = _get_ti(task, args.execution_date_or_run_id, args.map_index)
+ ti, _ = _get_ti(task, args.execution_date_or_run_id, args.map_index)
dep_context = DepContext(deps=SCHEDULER_QUEUED_DEPS)
failed_deps = list(ti.get_failed_dep_statuses(dep_context=dep_context))
@@ -383,7 +420,7 @@ def task_state(args):
"""
dag = get_dag(args.subdir, args.dag_id)
task = dag.get_task(task_id=args.task_id)
- ti = _get_ti(task, args.execution_date_or_run_id, args.map_index)
+ ti, _ = _get_ti(task, args.execution_date_or_run_id, args.map_index)
print(ti.current_state())
@@ -454,18 +491,22 @@ def task_states_for_dag_run(args, session=None):
"not found"
)
- AirflowConsole().print_as(
- data=dag_run.task_instances,
- output=args.output,
- mapper=lambda ti: {
+ has_mapped_instances = any(ti.map_index >= 0 for ti in dag_run.task_instances)
+
+ def format_task_instance(ti: TaskInstance) -> Dict[str, str]:
+ data = {
"dag_id": ti.dag_id,
"execution_date": dag_run.execution_date.isoformat(),
"task_id": ti.task_id,
"state": ti.state,
"start_date": ti.start_date.isoformat() if ti.start_date else "",
"end_date": ti.end_date.isoformat() if ti.end_date else "",
- },
- )
+ }
+ if has_mapped_instances:
+ data["map_index"] = str(ti.map_index) if ti.map_index >= 0 else ""
+ return data
+
+ AirflowConsole().print_as(data=dag_run.task_instances, output=args.output, mapper=format_task_instance)
@cli_utils.action_cli(check_db=False)
@@ -502,7 +543,7 @@ def task_test(args, dag=None):
if task.params:
task.params.validate()
- ti = _get_ti(task, args.execution_date_or_run_id, args.map_index, create_if_necessary=True)
+ ti, dr_created = _get_ti(task, args.execution_date_or_run_id, args.map_index, create_if_necessary="db")
try:
if args.dry_run:
@@ -520,6 +561,9 @@ def task_test(args, dag=None):
# Make sure to reset back to normal. When run for CLI this doesn't
# matter, but it does for test suite
logging.getLogger('airflow.task').propagate = False
+ if dr_created:
+ with create_session() as session:
+ session.delete(ti.dag_run)
@cli_utils.action_cli(check_db=False)
@@ -528,7 +572,7 @@ def task_render(args):
"""Renders and displays templated fields for a given task"""
dag = get_dag(args.subdir, args.dag_id)
task = dag.get_task(task_id=args.task_id)
- ti = _get_ti(task, args.execution_date_or_run_id, args.map_index, create_if_necessary=True)
+ ti, _ = _get_ti(task, args.execution_date_or_run_id, args.map_index, create_if_necessary="memory")
ti.render_templates()
for attr in task.__class__.template_fields:
print(
diff --git a/airflow/cli/commands/triggerer_command.py b/airflow/cli/commands/triggerer_command.py
index 82e7fde129698..8bf419268059b 100644
--- a/airflow/cli/commands/triggerer_command.py
+++ b/airflow/cli/commands/triggerer_command.py
@@ -24,7 +24,7 @@
from airflow import settings
from airflow.jobs.triggerer_job import TriggererJob
from airflow.utils import cli as cli_utils
-from airflow.utils.cli import setup_locations, setup_logging, sigint_handler, sigquit_handler
+from airflow.utils.cli import setup_locations, setup_logging, sigquit_handler
@cli_utils.action_cli
@@ -50,7 +50,19 @@ def triggerer(args):
job.run()
else:
- signal.signal(signal.SIGINT, sigint_handler)
- signal.signal(signal.SIGTERM, sigint_handler)
+ # There is a bug in CPython (fixed in March 2022 but not yet released) that
+ # makes async.io handle SIGTERM improperly by using async unsafe
+ # functions and hanging the triggerer receive SIGPIPE while handling
+ # SIGTERN/SIGINT and deadlocking itself. Until the bug is handled
+ # we should rather rely on standard handling of the signals rather than
+ # adding our own signal handlers. Seems that even if our signal handler
+ # just run exit(0) - it caused a race condition that led to the hanging.
+ #
+ # More details:
+ # * https://bugs.python.org/issue39622
+ # * https://github.com/python/cpython/issues/83803
+ #
+ # signal.signal(signal.SIGINT, sigint_handler)
+ # signal.signal(signal.SIGTERM, sigint_handler)
signal.signal(signal.SIGQUIT, sigquit_handler)
job.run()
diff --git a/airflow/config_templates/airflow_local_settings.py b/airflow/config_templates/airflow_local_settings.py
index 14fa529991e9b..b2752c2be7c25 100644
--- a/airflow/config_templates/airflow_local_settings.py
+++ b/airflow/config_templates/airflow_local_settings.py
@@ -19,7 +19,7 @@
import os
from pathlib import Path
-from typing import Any, Dict, Union
+from typing import Any, Dict, Optional, Union
from urllib.parse import urlparse
from airflow.configuration import conf
@@ -29,30 +29,32 @@
# in this file instead of from airflow.cfg. Currently
# there are other log format and level configurations in
# settings.py and cli.py. Please see AIRFLOW-1455.
-LOG_LEVEL: str = conf.get('logging', 'LOGGING_LEVEL').upper()
+LOG_LEVEL: str = conf.get_mandatory_value('logging', 'LOGGING_LEVEL').upper()
# Flask appbuilder's info level log is very verbose,
# so it's set to 'WARN' by default.
-FAB_LOG_LEVEL: str = conf.get('logging', 'FAB_LOGGING_LEVEL').upper()
+FAB_LOG_LEVEL: str = conf.get_mandatory_value('logging', 'FAB_LOGGING_LEVEL').upper()
-LOG_FORMAT: str = conf.get('logging', 'LOG_FORMAT')
+LOG_FORMAT: str = conf.get_mandatory_value('logging', 'LOG_FORMAT')
-COLORED_LOG_FORMAT: str = conf.get('logging', 'COLORED_LOG_FORMAT')
+COLORED_LOG_FORMAT: str = conf.get_mandatory_value('logging', 'COLORED_LOG_FORMAT')
COLORED_LOG: bool = conf.getboolean('logging', 'COLORED_CONSOLE_LOG')
-COLORED_FORMATTER_CLASS: str = conf.get('logging', 'COLORED_FORMATTER_CLASS')
+COLORED_FORMATTER_CLASS: str = conf.get_mandatory_value('logging', 'COLORED_FORMATTER_CLASS')
-BASE_LOG_FOLDER: str = conf.get('logging', 'BASE_LOG_FOLDER')
+BASE_LOG_FOLDER: str = conf.get_mandatory_value('logging', 'BASE_LOG_FOLDER')
-PROCESSOR_LOG_FOLDER: str = conf.get('scheduler', 'CHILD_PROCESS_LOG_DIRECTORY')
+PROCESSOR_LOG_FOLDER: str = conf.get_mandatory_value('scheduler', 'CHILD_PROCESS_LOG_DIRECTORY')
-DAG_PROCESSOR_MANAGER_LOG_LOCATION: str = conf.get('logging', 'DAG_PROCESSOR_MANAGER_LOG_LOCATION')
+DAG_PROCESSOR_MANAGER_LOG_LOCATION: str = conf.get_mandatory_value(
+ 'logging', 'DAG_PROCESSOR_MANAGER_LOG_LOCATION'
+)
-FILENAME_TEMPLATE: str = conf.get('logging', 'LOG_FILENAME_TEMPLATE')
+FILENAME_TEMPLATE: str = conf.get_mandatory_value('logging', 'LOG_FILENAME_TEMPLATE')
-PROCESSOR_FILENAME_TEMPLATE: str = conf.get('logging', 'LOG_PROCESSOR_FILENAME_TEMPLATE')
+PROCESSOR_FILENAME_TEMPLATE: str = conf.get_mandatory_value('logging', 'LOG_PROCESSOR_FILENAME_TEMPLATE')
DEFAULT_LOGGING_CONFIG: Dict[str, Any] = {
'version': 1,
@@ -116,7 +118,7 @@
},
}
-EXTRA_LOGGER_NAMES: str = conf.get('logging', 'EXTRA_LOGGER_NAMES', fallback=None)
+EXTRA_LOGGER_NAMES: Optional[str] = conf.get('logging', 'EXTRA_LOGGER_NAMES', fallback=None)
if EXTRA_LOGGER_NAMES:
new_loggers = {
logger_name.strip(): {
@@ -171,7 +173,7 @@
if REMOTE_LOGGING:
- ELASTICSEARCH_HOST: str = conf.get('elasticsearch', 'HOST')
+ ELASTICSEARCH_HOST: Optional[str] = conf.get('elasticsearch', 'HOST')
# Storage bucket URL for remote logging
# S3 buckets should start with "s3://"
@@ -179,7 +181,7 @@
# GCS buckets should start with "gs://"
# WASB buckets should start with "wasb"
# just to help Airflow select correct handler
- REMOTE_BASE_LOG_FOLDER: str = conf.get('logging', 'REMOTE_BASE_LOG_FOLDER')
+ REMOTE_BASE_LOG_FOLDER: str = conf.get_mandatory_value('logging', 'REMOTE_BASE_LOG_FOLDER')
if REMOTE_BASE_LOG_FOLDER.startswith('s3://'):
S3_REMOTE_HANDLERS: Dict[str, Dict[str, str]] = {
@@ -207,7 +209,7 @@
DEFAULT_LOGGING_CONFIG['handlers'].update(CLOUDWATCH_REMOTE_HANDLERS)
elif REMOTE_BASE_LOG_FOLDER.startswith('gs://'):
- key_path = conf.get('logging', 'GOOGLE_KEY_PATH', fallback=None)
+ key_path = conf.get_mandatory_value('logging', 'GOOGLE_KEY_PATH', fallback=None)
GCS_REMOTE_HANDLERS: Dict[str, Dict[str, str]] = {
'task': {
'class': 'airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler',
@@ -235,7 +237,7 @@
DEFAULT_LOGGING_CONFIG['handlers'].update(WASB_REMOTE_HANDLERS)
elif REMOTE_BASE_LOG_FOLDER.startswith('stackdriver://'):
- key_path = conf.get('logging', 'GOOGLE_KEY_PATH', fallback=None)
+ key_path = conf.get_mandatory_value('logging', 'GOOGLE_KEY_PATH', fallback=None)
# stackdriver:///airflow-tasks => airflow-tasks
log_name = urlparse(REMOTE_BASE_LOG_FOLDER).path[1:]
STACKDRIVER_REMOTE_HANDLERS = {
@@ -260,14 +262,14 @@
}
DEFAULT_LOGGING_CONFIG['handlers'].update(OSS_REMOTE_HANDLERS)
elif ELASTICSEARCH_HOST:
- ELASTICSEARCH_LOG_ID_TEMPLATE: str = conf.get('elasticsearch', 'LOG_ID_TEMPLATE')
- ELASTICSEARCH_END_OF_LOG_MARK: str = conf.get('elasticsearch', 'END_OF_LOG_MARK')
- ELASTICSEARCH_FRONTEND: str = conf.get('elasticsearch', 'frontend')
+ ELASTICSEARCH_LOG_ID_TEMPLATE: str = conf.get_mandatory_value('elasticsearch', 'LOG_ID_TEMPLATE')
+ ELASTICSEARCH_END_OF_LOG_MARK: str = conf.get_mandatory_value('elasticsearch', 'END_OF_LOG_MARK')
+ ELASTICSEARCH_FRONTEND: str = conf.get_mandatory_value('elasticsearch', 'frontend')
ELASTICSEARCH_WRITE_STDOUT: bool = conf.getboolean('elasticsearch', 'WRITE_STDOUT')
ELASTICSEARCH_JSON_FORMAT: bool = conf.getboolean('elasticsearch', 'JSON_FORMAT')
- ELASTICSEARCH_JSON_FIELDS: str = conf.get('elasticsearch', 'JSON_FIELDS')
- ELASTICSEARCH_HOST_FIELD: str = conf.get('elasticsearch', 'HOST_FIELD')
- ELASTICSEARCH_OFFSET_FIELD: str = conf.get('elasticsearch', 'OFFSET_FIELD')
+ ELASTICSEARCH_JSON_FIELDS: str = conf.get_mandatory_value('elasticsearch', 'JSON_FIELDS')
+ ELASTICSEARCH_HOST_FIELD: str = conf.get_mandatory_value('elasticsearch', 'HOST_FIELD')
+ ELASTICSEARCH_OFFSET_FIELD: str = conf.get_mandatory_value('elasticsearch', 'OFFSET_FIELD')
ELASTIC_REMOTE_HANDLERS: Dict[str, Dict[str, Union[str, bool]]] = {
'task': {
diff --git a/airflow/config_templates/config.yml b/airflow/config_templates/config.yml
index d43e1fb508592..f884959a88840 100644
--- a/airflow/config_templates/config.yml
+++ b/airflow/config_templates/config.yml
@@ -60,116 +60,12 @@
type: string
example: ~
default: "SequentialExecutor"
- - name: sql_alchemy_conn
- description: |
- The SqlAlchemy connection string to the metadata database.
- SqlAlchemy supports many different database engines.
- More information here:
- http://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#database-uri
- version_added: ~
- type: string
- sensitive: true
- example: ~
- default: "sqlite:///{AIRFLOW_HOME}/airflow.db"
- - name: sql_alchemy_engine_args
- description: |
- Extra engine specific keyword args passed to SQLAlchemy's create_engine, as a JSON-encoded value
- version_added: 2.3.0
- type: string
- sensitive: true
- example: '{"arg1": True}'
- default: ~
- - name: sql_engine_encoding
- description: |
- The encoding for the databases
- version_added: 1.10.1
- type: string
- example: ~
- default: "utf-8"
- - name: sql_engine_collation_for_ids
- description: |
- Collation for ``dag_id``, ``task_id``, ``key`` columns in case they have different encoding.
- By default this collation is the same as the database collation, however for ``mysql`` and ``mariadb``
- the default is ``utf8mb3_bin`` so that the index sizes of our index keys will not exceed
- the maximum size of allowed index when collation is set to ``utf8mb4`` variant
- (see https://github.com/apache/airflow/pull/17603#issuecomment-901121618).
- version_added: 2.0.0
- type: string
- example: ~
- default: ~
- - name: sql_alchemy_pool_enabled
- description: |
- If SqlAlchemy should pool database connections.
- version_added: ~
- type: string
- example: ~
- default: "True"
- - name: sql_alchemy_pool_size
- description: |
- The SqlAlchemy pool size is the maximum number of database connections
- in the pool. 0 indicates no limit.
- version_added: ~
- type: string
- example: ~
- default: "5"
- - name: sql_alchemy_max_overflow
- description: |
- The maximum overflow size of the pool.
- When the number of checked-out connections reaches the size set in pool_size,
- additional connections will be returned up to this limit.
- When those additional connections are returned to the pool, they are disconnected and discarded.
- It follows then that the total number of simultaneous connections the pool will allow
- is pool_size + max_overflow,
- and the total number of "sleeping" connections the pool will allow is pool_size.
- max_overflow can be set to ``-1`` to indicate no overflow limit;
- no limit will be placed on the total number of concurrent connections. Defaults to ``10``.
- version_added: 1.10.4
- type: string
- example: ~
- default: "10"
- - name: sql_alchemy_pool_recycle
- description: |
- The SqlAlchemy pool recycle is the number of seconds a connection
- can be idle in the pool before it is invalidated. This config does
- not apply to sqlite. If the number of DB connections is ever exceeded,
- a lower config value will allow the system to recover faster.
- version_added: ~
- type: string
- example: ~
- default: "1800"
- - name: sql_alchemy_pool_pre_ping
- description: |
- Check connection at the start of each connection pool checkout.
- Typically, this is a simple statement like "SELECT 1".
- More information here:
- https://docs.sqlalchemy.org/en/13/core/pooling.html#disconnect-handling-pessimistic
- version_added: 1.10.6
- type: string
- example: ~
- default: "True"
- - name: sql_alchemy_schema
- description: |
- The schema to use for the metadata database.
- SqlAlchemy supports databases with the concept of multiple schemas.
- version_added: 1.10.3
- type: string
- example: ~
- default: ""
- - name: sql_alchemy_connect_args
- description: |
- Import path for connect args in SqlAlchemy. Defaults to an empty dict.
- This is useful when you want to configure db engine args that SqlAlchemy won't parse
- in connection string.
- See https://docs.sqlalchemy.org/en/13/core/engines.html#sqlalchemy.create_engine.params.connect_args
- version_added: 1.10.11
- type: string
- example: ~
- default: ~
- name: parallelism
description: |
- This defines the maximum number of task instances that can run concurrently in Airflow
- regardless of scheduler count and worker count. Generally, this value is reflective of
- the number of task instances with the running state in the metadata database.
+ This defines the maximum number of task instances that can run concurrently per scheduler in
+ Airflow, regardless of the worker count. Generally this value, multiplied by the number of
+ schedulers in your cluster, is the maximum number of task instances with the running
+ state in the metadata database.
version_added: ~
type: string
example: ~
@@ -212,15 +108,6 @@
type: string
example: ~
default: "True"
- - name: load_default_connections
- description: |
- Whether to load the default connections that ship with Airflow. It's good to
- get started, but you probably want to set this to ``False`` in a production
- environment
- version_added: 1.10.10
- type: string
- example: ~
- default: "True"
- name: plugins_folder
description: |
Path to the folder containing Airflow plugins
@@ -347,6 +234,14 @@
type: string
example: ~
default: "True"
+ - name: dag_ignore_file_syntax
+ description: |
+ The pattern syntax used in the ".airflowignore" files in the DAG directories. Valid values are
+ ``regexp`` or ``glob``.
+ version_added: 2.3.0
+ type: string
+ example: ~
+ default: "regexp"
- name: default_task_retries
description: |
The number of retries each task is going to have by default. Can be overridden at dag or task level.
@@ -354,6 +249,14 @@
type: string
example: ~
default: "0"
+ - name: default_task_retry_delay
+ description: |
+ The number of seconds each task is going to wait by default between retries. Can be overridden at
+ dag or task level.
+ version_added: 2.3.2
+ type: integer
+ example: ~
+ default: "300"
- name: default_task_weight_rule
description: |
The weighting method used for the effective total priority weight of the task
@@ -361,6 +264,15 @@
type: string
example: ~
default: "downstream"
+ - name: default_task_execution_timeout
+ description: |
+ The default task execution_timeout value for the operators. Expected an integer value to
+ be passed into timedelta as seconds. If not specified, then the value is considered as None,
+ meaning that the operators are never timed out by default.
+ version_added: 2.3.0
+ type: integer
+ example: ~
+ default: ""
- name: min_serialized_dag_update_interval
description: |
Updating serialized DAG can not be faster than a minimum interval to reduce database write rate.
@@ -426,15 +338,6 @@
type: boolean
example: ~
default: "True"
- - name: max_db_retries
- description: |
- Number of times the code should be retried in case of DB Operational Errors.
- Not all transactions will be retried as it can cause undesired state.
- Currently it is only used in ``DagFileProcessor.process_file`` to retry ``dagbag.sync_to_db``.
- version_added: 2.0.0
- type: integer
- example: ~
- default: "3"
- name: hide_sensitive_var_conn_fields
description: |
Hide sensitive Variables or Connection extra json keys from UI and task logs when set to True
@@ -471,6 +374,133 @@
example: ~
default: "1024"
+- name: database
+ description: ~
+ options:
+ - name: sql_alchemy_conn
+ description: |
+ The SqlAlchemy connection string to the metadata database.
+ SqlAlchemy supports many different database engines.
+ More information here:
+ http://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#database-uri
+ version_added: 2.3.0
+ type: string
+ sensitive: true
+ example: ~
+ default: "sqlite:///{AIRFLOW_HOME}/airflow.db"
+ - name: sql_alchemy_engine_args
+ description: |
+ Extra engine specific keyword args passed to SQLAlchemy's create_engine, as a JSON-encoded value
+ version_added: 2.3.0
+ type: string
+ sensitive: true
+ example: '{"arg1": True}'
+ default: ~
+ - name: sql_engine_encoding
+ description: |
+ The encoding for the databases
+ version_added: 2.3.0
+ type: string
+ example: ~
+ default: "utf-8"
+ - name: sql_engine_collation_for_ids
+ description: |
+ Collation for ``dag_id``, ``task_id``, ``key`` columns in case they have different encoding.
+ By default this collation is the same as the database collation, however for ``mysql`` and ``mariadb``
+ the default is ``utf8mb3_bin`` so that the index sizes of our index keys will not exceed
+ the maximum size of allowed index when collation is set to ``utf8mb4`` variant
+ (see https://github.com/apache/airflow/pull/17603#issuecomment-901121618).
+ version_added: 2.3.0
+ type: string
+ example: ~
+ default: ~
+ - name: sql_alchemy_pool_enabled
+ description: |
+ If SqlAlchemy should pool database connections.
+ version_added: 2.3.0
+ type: string
+ example: ~
+ default: "True"
+ - name: sql_alchemy_pool_size
+ description: |
+ The SqlAlchemy pool size is the maximum number of database connections
+ in the pool. 0 indicates no limit.
+ version_added: 2.3.0
+ type: string
+ example: ~
+ default: "5"
+ - name: sql_alchemy_max_overflow
+ description: |
+ The maximum overflow size of the pool.
+ When the number of checked-out connections reaches the size set in pool_size,
+ additional connections will be returned up to this limit.
+ When those additional connections are returned to the pool, they are disconnected and discarded.
+ It follows then that the total number of simultaneous connections the pool will allow
+ is pool_size + max_overflow,
+ and the total number of "sleeping" connections the pool will allow is pool_size.
+ max_overflow can be set to ``-1`` to indicate no overflow limit;
+ no limit will be placed on the total number of concurrent connections. Defaults to ``10``.
+ version_added: 2.3.0
+ type: string
+ example: ~
+ default: "10"
+ - name: sql_alchemy_pool_recycle
+ description: |
+ The SqlAlchemy pool recycle is the number of seconds a connection
+ can be idle in the pool before it is invalidated. This config does
+ not apply to sqlite. If the number of DB connections is ever exceeded,
+ a lower config value will allow the system to recover faster.
+ version_added: 2.3.0
+ type: string
+ example: ~
+ default: "1800"
+ - name: sql_alchemy_pool_pre_ping
+ description: |
+ Check connection at the start of each connection pool checkout.
+ Typically, this is a simple statement like "SELECT 1".
+ More information here:
+ https://docs.sqlalchemy.org/en/13/core/pooling.html#disconnect-handling-pessimistic
+ version_added: 2.3.0
+ type: string
+ example: ~
+ default: "True"
+ - name: sql_alchemy_schema
+ description: |
+ The schema to use for the metadata database.
+ SqlAlchemy supports databases with the concept of multiple schemas.
+ version_added: 2.3.0
+ type: string
+ example: ~
+ default: ""
+ - name: sql_alchemy_connect_args
+ description: |
+ Import path for connect args in SqlAlchemy. Defaults to an empty dict.
+ This is useful when you want to configure db engine args that SqlAlchemy won't parse
+ in connection string.
+ See https://docs.sqlalchemy.org/en/13/core/engines.html#sqlalchemy.create_engine.params.connect_args
+ version_added: 2.3.0
+ type: string
+ example: ~
+ default: ~
+ - name: load_default_connections
+ description: |
+ Whether to load the default connections that ship with Airflow. It's good to
+ get started, but you probably want to set this to ``False`` in a production
+ environment
+ version_added: 2.3.0
+ type: string
+ example: ~
+ default: "True"
+ - name: max_db_retries
+ description: |
+ Number of times the code should be retried in case of DB Operational Errors.
+ Not all transactions will be retried as it can cause undesired state.
+ Currently it is only used in ``DagFileProcessor.process_file`` to retry ``dagbag.sync_to_db``.
+ version_added: 2.3.0
+ type: integer
+ example: ~
+ default: "3"
+
- name: logging
description: ~
options:
@@ -496,7 +526,8 @@
- name: remote_log_conn_id
description: |
Users must supply an Airflow connection id that provides access to the storage
- location. Note: This is only used for reading logs, not writing logs
+ location. Depending on your remote logging service, this may only be used for
+ reading logs, not writing them.
version_added: 2.0.0
type: string
example: ~
@@ -699,7 +730,7 @@
default: ""
- name: stat_name_handler
description: |
- A function that validate the statsd stat name, apply changes to the stat name if necessary and return
+ A function that validate the StatsD stat name, apply changes to the stat name if necessary and return
the transformed stat name.
The function should have the following signature:
@@ -724,7 +755,7 @@
default: ""
- name: statsd_custom_client_path
description: |
- If you want to utilise your own custom Statsd client set the relevant
+ If you want to utilise your own custom StatsD client set the relevant
module path below.
Note: The module path must exist on your PYTHONPATH for Airflow to pick it up
version_added: 2.0.0
@@ -797,17 +828,17 @@
deprecated since version 2.0. Please consider using
`the Stable REST API `__.
For more information on migration, see
- `UPDATING.md `_
+ `RELEASE_NOTES.rst `_
version_added: 2.0.0
type: boolean
example: ~
default: "False"
- name: auth_backends
description: |
- How to authenticate users of the API. See
- https://airflow.apache.org/docs/apache-airflow/stable/security.html for possible values.
+ Comma separated list of auth backends to authenticate users of the API. See
+ https://airflow.apache.org/docs/apache-airflow/stable/security/api.html for possible values.
("airflow.api.auth.backend.default" allows all requests for historic reasons)
- version_added: ~
+ version_added: 2.3.0
type: string
example: ~
default: "airflow.api.auth.backend.session"
@@ -1745,12 +1776,23 @@
default: "True"
- name: task_adoption_timeout
description: |
- Time in seconds after which Adopted tasks are cleared by CeleryExecutor. This is helpful to clear
- stalled tasks.
+ Time in seconds after which adopted tasks which are queued in celery are assumed to be stalled,
+ and are automatically rescheduled. This setting does the same thing as ``stalled_task_timeout`` but
+ applies specifically to adopted tasks only. When set to 0, the ``stalled_task_timeout`` setting
+ also applies to adopted tasks.
version_added: 2.0.0
type: integer
example: ~
default: "600"
+ - name: stalled_task_timeout
+ description: |
+ Time in seconds after which tasks queued in celery are assumed to be stalled, and are automatically
+ rescheduled. Adopted tasks will instead use the ``task_adoption_timeout`` setting if specified.
+ When set to 0, automatic clearing of stalled tasks is disabled.
+ version_added: 2.3.1
+ type: integer
+ example: ~
+ default: "0"
- name: task_publish_max_retries
description: |
The Maximum number of retries for publishing task messages to the broker when failing
@@ -1766,13 +1808,6 @@
type: string
example: ~
default: "False"
- - name: stuck_queued_task_check_interval
- description: |
- How often to check for stuck queued task (in seconds)
- version_added: 2.2.4
- type: integer
- example: ~
- default: "300"
- name: celery_broker_transport_options
description: |
This section is for specifying options which can be passed to the
@@ -1870,6 +1905,14 @@
type: string
example: ~
default: "30"
+ - name: deactivate_stale_dags_interval
+ description: |
+ How often (in seconds) to check for stale DAGs (DAGs which are no longer present in
+ the expected files) which should be deactivated.
+ version_added: 2.2.5
+ type: integer
+ example: ~
+ default: "60"
- name: dag_dir_list_interval
description: |
How often (in seconds) to scan the DAGs directory for new files. Default to 5 minutes.
@@ -1886,7 +1929,7 @@
default: "30"
- name: pool_metrics_interval
description: |
- How often (in seconds) should pool usage stats be sent to statsd (if statsd_on is enabled)
+ How often (in seconds) should pool usage stats be sent to StatsD (if statsd_on is enabled)
version_added: 2.0.0
type: float
example: ~
@@ -1941,6 +1984,17 @@
type: string
example: ~
default: "True"
+ - name: ignore_first_depends_on_past_by_default
+ description: |
+ Setting this to True will make first task instance of a task
+ ignore depends_on_past setting. A task instance will be considered
+ as the first task instance of a task when there is no task instance
+ in the DB with an execution_date earlier than it., i.e. no manual marking
+ success will be needed for a newly added task to be scheduled.
+ version_added: 2.3.0
+ type: string
+ example: ~
+ default: "True"
- name: max_tis_per_query
description: |
This changes the batch size of queries in the scheduling main loop.
@@ -2114,15 +2168,6 @@
type: boolean
example: ~
default: "True"
-- name: github_enterprise
- description: ~
- options:
- - name: api_rev
- description: ~
- version_added: ~
- type: string
- example: ~
- default: "v3"
- name: elasticsearch
description: ~
options:
@@ -2396,7 +2441,7 @@
- name: default_timeout
description: |
Sensor default timeout, 7 days by default (7 * 24 * 60 * 60).
- version_added: 2.2.1
+ version_added: 2.3.0
type: float
example: ~
default: "604800"
diff --git a/airflow/config_templates/config.yml.schema.json b/airflow/config_templates/config.yml.schema.json
index 9fef382fa448d..1b433060900c0 100644
--- a/airflow/config_templates/config.yml.schema.json
+++ b/airflow/config_templates/config.yml.schema.json
@@ -36,7 +36,7 @@
},
"sensitive": {
"type": "boolean",
- "description": "When true, this option is sensitive and can be specified using AIRFLOW__{section}___{name}__SECRET or AIRFLOW__{section}___{name}__CMD environment variables. See: airflow.configuration.AirflowConfigParser.sensitive_config_values"
+ "description": "When true, this option is sensitive and can be specified using AIRFLOW__{section}___{name}__SECRET or AIRFLOW__{section}___{name}_CMD environment variables. See: airflow.configuration.AirflowConfigParser.sensitive_config_values"
}
},
"required": [
diff --git a/airflow/config_templates/default_airflow.cfg b/airflow/config_templates/default_airflow.cfg
index c8d34bb2f94d2..b5c1d4290bd60 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -53,69 +53,10 @@ default_timezone = utc
# full import path to the class when using a custom executor.
executor = SequentialExecutor
-# The SqlAlchemy connection string to the metadata database.
-# SqlAlchemy supports many different database engines.
-# More information here:
-# http://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#database-uri
-sql_alchemy_conn = sqlite:///{AIRFLOW_HOME}/airflow.db
-
-# Extra engine specific keyword args passed to SQLAlchemy's create_engine, as a JSON-encoded value
-# Example: sql_alchemy_engine_args = {{"arg1": True}}
-# sql_alchemy_engine_args =
-
-# The encoding for the databases
-sql_engine_encoding = utf-8
-
-# Collation for ``dag_id``, ``task_id``, ``key`` columns in case they have different encoding.
-# By default this collation is the same as the database collation, however for ``mysql`` and ``mariadb``
-# the default is ``utf8mb3_bin`` so that the index sizes of our index keys will not exceed
-# the maximum size of allowed index when collation is set to ``utf8mb4`` variant
-# (see https://github.com/apache/airflow/pull/17603#issuecomment-901121618).
-# sql_engine_collation_for_ids =
-
-# If SqlAlchemy should pool database connections.
-sql_alchemy_pool_enabled = True
-
-# The SqlAlchemy pool size is the maximum number of database connections
-# in the pool. 0 indicates no limit.
-sql_alchemy_pool_size = 5
-
-# The maximum overflow size of the pool.
-# When the number of checked-out connections reaches the size set in pool_size,
-# additional connections will be returned up to this limit.
-# When those additional connections are returned to the pool, they are disconnected and discarded.
-# It follows then that the total number of simultaneous connections the pool will allow
-# is pool_size + max_overflow,
-# and the total number of "sleeping" connections the pool will allow is pool_size.
-# max_overflow can be set to ``-1`` to indicate no overflow limit;
-# no limit will be placed on the total number of concurrent connections. Defaults to ``10``.
-sql_alchemy_max_overflow = 10
-
-# The SqlAlchemy pool recycle is the number of seconds a connection
-# can be idle in the pool before it is invalidated. This config does
-# not apply to sqlite. If the number of DB connections is ever exceeded,
-# a lower config value will allow the system to recover faster.
-sql_alchemy_pool_recycle = 1800
-
-# Check connection at the start of each connection pool checkout.
-# Typically, this is a simple statement like "SELECT 1".
-# More information here:
-# https://docs.sqlalchemy.org/en/13/core/pooling.html#disconnect-handling-pessimistic
-sql_alchemy_pool_pre_ping = True
-
-# The schema to use for the metadata database.
-# SqlAlchemy supports databases with the concept of multiple schemas.
-sql_alchemy_schema =
-
-# Import path for connect args in SqlAlchemy. Defaults to an empty dict.
-# This is useful when you want to configure db engine args that SqlAlchemy won't parse
-# in connection string.
-# See https://docs.sqlalchemy.org/en/13/core/engines.html#sqlalchemy.create_engine.params.connect_args
-# sql_alchemy_connect_args =
-
-# This defines the maximum number of task instances that can run concurrently in Airflow
-# regardless of scheduler count and worker count. Generally, this value is reflective of
-# the number of task instances with the running state in the metadata database.
+# This defines the maximum number of task instances that can run concurrently per scheduler in
+# Airflow, regardless of the worker count. Generally this value, multiplied by the number of
+# schedulers in your cluster, is the maximum number of task instances with the running
+# state in the metadata database.
parallelism = 32
# The maximum number of task instances allowed to run concurrently in each DAG. To calculate
@@ -140,11 +81,6 @@ max_active_runs_per_dag = 16
# environment
load_examples = True
-# Whether to load the default connections that ship with Airflow. It's good to
-# get started, but you probably want to set this to ``False`` in a production
-# environment
-load_default_connections = True
-
# Path to the folder containing Airflow plugins
plugins_folder = {AIRFLOW_HOME}/plugins
@@ -204,12 +140,25 @@ dag_run_conf_overrides_params = True
# When discovering DAGs, ignore any files that don't contain the strings ``DAG`` and ``airflow``.
dag_discovery_safe_mode = True
+# The pattern syntax used in the ".airflowignore" files in the DAG directories. Valid values are
+# ``regexp`` or ``glob``.
+dag_ignore_file_syntax = regexp
+
# The number of retries each task is going to have by default. Can be overridden at dag or task level.
default_task_retries = 0
+# The number of seconds each task is going to wait by default between retries. Can be overridden at
+# dag or task level.
+default_task_retry_delay = 300
+
# The weighting method used for the effective total priority weight of the task
default_task_weight_rule = downstream
+# The default task execution_timeout value for the operators. Expected an integer value to
+# be passed into timedelta as seconds. If not specified, then the value is considered as None,
+# meaning that the operators are never timed out by default.
+default_task_execution_timeout =
+
# Updating serialized DAG can not be faster than a minimum interval to reduce database write rate.
min_serialized_dag_update_interval = 30
@@ -244,11 +193,6 @@ lazy_load_plugins = True
# loaded from module.
lazy_discover_providers = True
-# Number of times the code should be retried in case of DB Operational Errors.
-# Not all transactions will be retried as it can cause undesired state.
-# Currently it is only used in ``DagFileProcessor.process_file`` to retry ``dagbag.sync_to_db``.
-max_db_retries = 3
-
# Hide sensitive Variables or Connection extra json keys from UI and task logs when set to True
#
# (Connection passwords are always hidden in logs)
@@ -268,6 +212,77 @@ default_pool_task_slot_count = 128
# mapped tasks from clogging the scheduler.
max_map_length = 1024
+[database]
+# The SqlAlchemy connection string to the metadata database.
+# SqlAlchemy supports many different database engines.
+# More information here:
+# http://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#database-uri
+sql_alchemy_conn = sqlite:///{AIRFLOW_HOME}/airflow.db
+
+# Extra engine specific keyword args passed to SQLAlchemy's create_engine, as a JSON-encoded value
+# Example: sql_alchemy_engine_args = {{"arg1": True}}
+# sql_alchemy_engine_args =
+
+# The encoding for the databases
+sql_engine_encoding = utf-8
+
+# Collation for ``dag_id``, ``task_id``, ``key`` columns in case they have different encoding.
+# By default this collation is the same as the database collation, however for ``mysql`` and ``mariadb``
+# the default is ``utf8mb3_bin`` so that the index sizes of our index keys will not exceed
+# the maximum size of allowed index when collation is set to ``utf8mb4`` variant
+# (see https://github.com/apache/airflow/pull/17603#issuecomment-901121618).
+# sql_engine_collation_for_ids =
+
+# If SqlAlchemy should pool database connections.
+sql_alchemy_pool_enabled = True
+
+# The SqlAlchemy pool size is the maximum number of database connections
+# in the pool. 0 indicates no limit.
+sql_alchemy_pool_size = 5
+
+# The maximum overflow size of the pool.
+# When the number of checked-out connections reaches the size set in pool_size,
+# additional connections will be returned up to this limit.
+# When those additional connections are returned to the pool, they are disconnected and discarded.
+# It follows then that the total number of simultaneous connections the pool will allow
+# is pool_size + max_overflow,
+# and the total number of "sleeping" connections the pool will allow is pool_size.
+# max_overflow can be set to ``-1`` to indicate no overflow limit;
+# no limit will be placed on the total number of concurrent connections. Defaults to ``10``.
+sql_alchemy_max_overflow = 10
+
+# The SqlAlchemy pool recycle is the number of seconds a connection
+# can be idle in the pool before it is invalidated. This config does
+# not apply to sqlite. If the number of DB connections is ever exceeded,
+# a lower config value will allow the system to recover faster.
+sql_alchemy_pool_recycle = 1800
+
+# Check connection at the start of each connection pool checkout.
+# Typically, this is a simple statement like "SELECT 1".
+# More information here:
+# https://docs.sqlalchemy.org/en/13/core/pooling.html#disconnect-handling-pessimistic
+sql_alchemy_pool_pre_ping = True
+
+# The schema to use for the metadata database.
+# SqlAlchemy supports databases with the concept of multiple schemas.
+sql_alchemy_schema =
+
+# Import path for connect args in SqlAlchemy. Defaults to an empty dict.
+# This is useful when you want to configure db engine args that SqlAlchemy won't parse
+# in connection string.
+# See https://docs.sqlalchemy.org/en/13/core/engines.html#sqlalchemy.create_engine.params.connect_args
+# sql_alchemy_connect_args =
+
+# Whether to load the default connections that ship with Airflow. It's good to
+# get started, but you probably want to set this to ``False`` in a production
+# environment
+load_default_connections = True
+
+# Number of times the code should be retried in case of DB Operational Errors.
+# Not all transactions will be retried as it can cause undesired state.
+# Currently it is only used in ``DagFileProcessor.process_file`` to retry ``dagbag.sync_to_db``.
+max_db_retries = 3
+
[logging]
# The folder where airflow should store its log files.
# This path must be absolute.
@@ -281,7 +296,8 @@ base_log_folder = {AIRFLOW_HOME}/logs
remote_logging = False
# Users must supply an Airflow connection id that provides access to the storage
-# location. Note: This is only used for reading logs, not writing logs
+# location. Depending on your remote logging service, this may only be used for
+# reading logs, not writing them.
remote_log_conn_id =
# Path to Google Credential JSON file. If omitted, authorization based on `the Application Default
@@ -377,7 +393,7 @@ statsd_prefix = airflow
# start with the elements of the list (e.g: "scheduler,executor,dagrun")
statsd_allow_list =
-# A function that validate the statsd stat name, apply changes to the stat name if necessary and return
+# A function that validate the StatsD stat name, apply changes to the stat name if necessary and return
# the transformed stat name.
#
# The function should have the following signature:
@@ -390,7 +406,7 @@ statsd_datadog_enabled = False
# List of datadog tags attached to all metrics(e.g: key1:value1,key2:value2)
statsd_datadog_tags =
-# If you want to utilise your own custom Statsd client set the relevant
+# If you want to utilise your own custom StatsD client set the relevant
# module path below.
# Note: The module path must exist on your PYTHONPATH for Airflow to pick it up
# statsd_custom_client_path =
@@ -432,11 +448,11 @@ fail_fast = False
# deprecated since version 2.0. Please consider using
# `the Stable REST API `__.
# For more information on migration, see
-# `UPDATING.md `_
+# `RELEASE_NOTES.rst `_
enable_experimental_api = False
-# How to authenticate users of the API. See
-# https://airflow.apache.org/docs/apache-airflow/stable/security.html for possible values.
+# Comma separated list of auth backends to authenticate users of the API. See
+# https://airflow.apache.org/docs/apache-airflow/stable/security/api.html for possible values.
# ("airflow.api.auth.backend.default" allows all requests for historic reasons)
auth_backends = airflow.api.auth.backend.session
@@ -876,10 +892,17 @@ operation_timeout = 1.0
# or run in HA mode, it can adopt the orphan tasks launched by previous SchedulerJob.
task_track_started = True
-# Time in seconds after which Adopted tasks are cleared by CeleryExecutor. This is helpful to clear
-# stalled tasks.
+# Time in seconds after which adopted tasks which are queued in celery are assumed to be stalled,
+# and are automatically rescheduled. This setting does the same thing as ``stalled_task_timeout`` but
+# applies specifically to adopted tasks only. When set to 0, the ``stalled_task_timeout`` setting
+# also applies to adopted tasks.
task_adoption_timeout = 600
+# Time in seconds after which tasks queued in celery are assumed to be stalled, and are automatically
+# rescheduled. Adopted tasks will instead use the ``task_adoption_timeout`` setting if specified.
+# When set to 0, automatic clearing of stalled tasks is disabled.
+stalled_task_timeout = 0
+
# The Maximum number of retries for publishing task messages to the broker when failing
# due to ``AirflowTaskTimeout`` error before giving up and marking Task as failed.
task_publish_max_retries = 3
@@ -887,9 +910,6 @@ task_publish_max_retries = 3
# Worker initialisation check to validate Metadata Database connection
worker_precheck = False
-# How often to check for stuck queued task (in seconds)
-stuck_queued_task_check_interval = 300
-
[celery_broker_transport_options]
# This section is for specifying options which can be passed to the
@@ -942,13 +962,17 @@ scheduler_idle_sleep_time = 1
# this interval. Keeping this number low will increase CPU usage.
min_file_process_interval = 30
+# How often (in seconds) to check for stale DAGs (DAGs which are no longer present in
+# the expected files) which should be deactivated.
+deactivate_stale_dags_interval = 60
+
# How often (in seconds) to scan the DAGs directory for new files. Default to 5 minutes.
dag_dir_list_interval = 300
# How often should stats be printed to the logs. Setting to 0 will disable printing stats
print_stats_interval = 30
-# How often (in seconds) should pool usage stats be sent to statsd (if statsd_on is enabled)
+# How often (in seconds) should pool usage stats be sent to StatsD (if statsd_on is enabled)
pool_metrics_interval = 5.0
# If the last scheduler heartbeat happened more than scheduler_health_check_threshold
@@ -976,6 +1000,13 @@ zombie_detection_interval = 10.0
# DAG definition (catchup)
catchup_by_default = True
+# Setting this to True will make first task instance of a task
+# ignore depends_on_past setting. A task instance will be considered
+# as the first task instance of a task when there is no task instance
+# in the DB with an execution_date earlier than it., i.e. no manual marking
+# success will be needed for a newly added task to be scheduled.
+ignore_first_depends_on_past_by_default = True
+
# This changes the batch size of queries in the scheduling main loop.
# If this is too high, SQL query performance may be impacted by
# complexity of query predicate, and/or excessive locking.
@@ -1056,9 +1087,6 @@ forwardable = True
# Allow to remove source IP from token, useful when using token behind NATted Docker host.
include_ip = True
-[github_enterprise]
-api_rev = v3
-
[elasticsearch]
# Elasticsearch host
host =
diff --git a/airflow/config_templates/default_celery.py b/airflow/config_templates/default_celery.py
index bbc8aa6a7faee..9d81c6353fba2 100644
--- a/airflow/config_templates/default_celery.py
+++ b/airflow/config_templates/default_celery.py
@@ -59,14 +59,14 @@ def _broker_supports_visibility_timeout(url):
try:
if celery_ssl_active:
- if 'amqp://' in broker_url:
+ if broker_url and 'amqp://' in broker_url:
broker_use_ssl = {
'keyfile': conf.get('celery', 'SSL_KEY'),
'certfile': conf.get('celery', 'SSL_CERT'),
'ca_certs': conf.get('celery', 'SSL_CACERT'),
'cert_reqs': ssl.CERT_REQUIRED,
}
- elif 'redis://' in broker_url:
+ elif broker_url and 'redis://' in broker_url:
broker_use_ssl = {
'ssl_keyfile': conf.get('celery', 'SSL_KEY'),
'ssl_certfile': conf.get('celery', 'SSL_CERT'),
@@ -92,7 +92,7 @@ def _broker_supports_visibility_timeout(url):
f'all necessary certs and key ({e}).'
)
-result_backend = DEFAULT_CELERY_CONFIG['result_backend']
+result_backend = str(DEFAULT_CELERY_CONFIG['result_backend'])
if 'amqp://' in result_backend or 'redis://' in result_backend or 'rpc://' in result_backend:
log.warning(
"You have configured a result_backend of %s, it is highly recommended "
diff --git a/airflow/config_templates/default_test.cfg b/airflow/config_templates/default_test.cfg
index 09430e55d74dd..2f9b6fa264b13 100644
--- a/airflow/config_templates/default_test.cfg
+++ b/airflow/config_templates/default_test.cfg
@@ -33,9 +33,7 @@ unit_test_mode = True
dags_folder = {TEST_DAGS_FOLDER}
plugins_folder = {TEST_PLUGINS_FOLDER}
executor = SequentialExecutor
-sql_alchemy_conn = sqlite:///{AIRFLOW_HOME}/unittests.db
load_examples = True
-load_default_connections = True
donot_pickle = True
max_active_tasks_per_dag = 16
dags_are_paused_at_creation = False
@@ -47,6 +45,10 @@ default_task_retries = 0
# This is a hack, too many tests assume DAGs are already in the DB. We need to fix those tests instead
store_serialized_dags = False
+[database]
+sql_alchemy_conn = sqlite:///{AIRFLOW_HOME}/unittests.db
+load_default_connections = True
+
[logging]
base_log_folder = {AIRFLOW_HOME}/logs
logging_level = INFO
diff --git a/airflow/configuration.py b/airflow/configuration.py
index 584158e260744..93612160c8290 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -14,7 +14,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
+import datetime
import functools
import json
import logging
@@ -33,7 +33,11 @@
from configparser import _UNSET, ConfigParser, NoOptionError, NoSectionError # type: ignore
from contextlib import suppress
from json.decoder import JSONDecodeError
-from typing import Any, Dict, List, Optional, Tuple, Union
+from re import Pattern
+from typing import IO, Any, Dict, Iterable, List, Optional, Set, Tuple, Union
+from urllib.parse import urlparse
+
+from typing_extensions import overload
from airflow.exceptions import AirflowConfigException
from airflow.secrets import DEFAULT_SECRETS_SEARCH_PATH, BaseSecretsBackend
@@ -50,6 +54,13 @@
_SQLITE3_VERSION_PATTERN = re.compile(r"(?P^\d+(?:\.\d+)*)\D?.*$")
+ConfigType = Union[str, int, float, bool]
+ConfigOptionsDictType = Dict[str, ConfigType]
+ConfigSectionSourcesType = Dict[str, Union[str, Tuple[str, str]]]
+ConfigSourcesType = Dict[str, ConfigSectionSourcesType]
+
+ENV_VAR_PREFIX = 'AIRFLOW__'
+
def _parse_sqlite_version(s: str) -> Tuple[int, ...]:
match = _SQLITE3_VERSION_PATTERN.match(s)
@@ -58,7 +69,17 @@ def _parse_sqlite_version(s: str) -> Tuple[int, ...]:
return tuple(int(p) for p in match.group("version").split("."))
-def expand_env_var(env_var):
+@overload
+def expand_env_var(env_var: None) -> None:
+ ...
+
+
+@overload
+def expand_env_var(env_var: str) -> str:
+ ...
+
+
+def expand_env_var(env_var: Union[str, None]) -> Optional[Union[str, None]]:
"""
Expands (potentially nested) env vars by repeatedly applying
`expandvars` and `expanduser` until interpolation stops having
@@ -74,7 +95,7 @@ def expand_env_var(env_var):
env_var = interpolated
-def run_command(command):
+def run_command(command: str) -> str:
"""Runs command and returns stdout"""
process = subprocess.Popen(
shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True
@@ -90,7 +111,7 @@ def run_command(command):
return output
-def _get_config_value_from_secret_backend(config_key):
+def _get_config_value_from_secret_backend(config_key: str) -> Optional[str]:
"""Get Config option values from Secret Backend"""
try:
secrets_client = get_custom_secret_backend()
@@ -106,7 +127,7 @@ def _get_config_value_from_secret_backend(config_key):
)
-def _default_config_file_path(file_name: str):
+def _default_config_file_path(file_name: str) -> str:
templates_dir = os.path.join(os.path.dirname(__file__), 'config_templates')
return os.path.join(templates_dir, file_name)
@@ -125,12 +146,12 @@ class AirflowConfigParser(ConfigParser):
"""Custom Airflow Configparser supporting defaults and deprecated options"""
# These configuration elements can be fetched as the stdout of commands
- # following the "{section}__{name}__cmd" pattern, the idea behind this
+ # following the "{section}__{name}_cmd" pattern, the idea behind this
# is to not store password on boxes in text files.
# These configs can also be fetched from Secrets backend
# following the "{section}__{name}__secret" pattern
- sensitive_config_values = {
- ('core', 'sql_alchemy_conn'),
+ sensitive_config_values: Set[Tuple[str, str]] = {
+ ('database', 'sql_alchemy_conn'),
('core', 'fernet_key'),
('celery', 'broker_url'),
('celery', 'flower_basic_auth'),
@@ -138,12 +159,14 @@ class AirflowConfigParser(ConfigParser):
('atlas', 'password'),
('smtp', 'smtp_password'),
('webserver', 'secret_key'),
+ # The following options are deprecated
+ ('core', 'sql_alchemy_conn'),
}
# A mapping of (new section, new option) -> (old section, old option, since_version).
# When reading new option, the old option will be checked to see if it exists. If it does a
# DeprecationWarning will be issued and the old option will be used instead
- deprecated_options = {
+ deprecated_options: Dict[Tuple[str, str], Tuple[str, str, str]] = {
('celery', 'worker_precheck'): ('core', 'worker_precheck', '2.0.0'),
('logging', 'base_log_folder'): ('core', 'base_log_folder', '2.0.0'),
('logging', 'remote_logging'): ('core', 'remote_logging', '2.0.0'),
@@ -185,12 +208,24 @@ class AirflowConfigParser(ConfigParser):
('core', 'max_active_tasks_per_dag'): ('core', 'dag_concurrency', '2.2.0'),
('logging', 'worker_log_server_port'): ('celery', 'worker_log_server_port', '2.2.0'),
('api', 'access_control_allow_origins'): ('api', 'access_control_allow_origin', '2.2.0'),
- ('api', 'auth_backends'): ('api', 'auth_backend', '2.3'),
+ ('api', 'auth_backends'): ('api', 'auth_backend', '2.3.0'),
+ ('database', 'sql_alchemy_conn'): ('core', 'sql_alchemy_conn', '2.3.0'),
+ ('database', 'sql_engine_encoding'): ('core', 'sql_engine_encoding', '2.3.0'),
+ ('database', 'sql_engine_collation_for_ids'): ('core', 'sql_engine_collation_for_ids', '2.3.0'),
+ ('database', 'sql_alchemy_pool_enabled'): ('core', 'sql_alchemy_pool_enabled', '2.3.0'),
+ ('database', 'sql_alchemy_pool_size'): ('core', 'sql_alchemy_pool_size', '2.3.0'),
+ ('database', 'sql_alchemy_max_overflow'): ('core', 'sql_alchemy_max_overflow', '2.3.0'),
+ ('database', 'sql_alchemy_pool_recycle'): ('core', 'sql_alchemy_pool_recycle', '2.3.0'),
+ ('database', 'sql_alchemy_pool_pre_ping'): ('core', 'sql_alchemy_pool_pre_ping', '2.3.0'),
+ ('database', 'sql_alchemy_schema'): ('core', 'sql_alchemy_schema', '2.3.0'),
+ ('database', 'sql_alchemy_connect_args'): ('core', 'sql_alchemy_connect_args', '2.3.0'),
+ ('database', 'load_default_connections'): ('core', 'load_default_connections', '2.3.0'),
+ ('database', 'max_db_retries'): ('core', 'max_db_retries', '2.3.0'),
}
# A mapping of old default values that we want to change and warn the user
# about. Mapping of section -> setting -> { old, replace, by_version }
- deprecated_values = {
+ deprecated_values: Dict[str, Dict[str, Tuple[Pattern, str, str]]] = {
'core': {
'hostname_callable': (re.compile(r':'), r'.', '2.1'),
},
@@ -209,7 +244,7 @@ class AirflowConfigParser(ConfigParser):
'log_filename_template': (
re.compile(re.escape("{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log")),
"XX-set-after-default-config-loaded-XX",
- 3.0,
+ '3.0',
),
},
'api': {
@@ -223,7 +258,7 @@ class AirflowConfigParser(ConfigParser):
'log_id_template': (
re.compile('^' + re.escape('{dag_id}-{task_id}-{run_id}-{try_number}') + '$'),
'{dag_id}-{task_id}-{run_id}-{map_index}-{try_number}',
- 3.0,
+ '3.0',
)
},
}
@@ -231,24 +266,26 @@ class AirflowConfigParser(ConfigParser):
_available_logging_levels = ['CRITICAL', 'FATAL', 'ERROR', 'WARN', 'WARNING', 'INFO', 'DEBUG']
enums_options = {
("core", "default_task_weight_rule"): sorted(WeightRule.all_weight_rules()),
+ ("core", "dag_ignore_file_syntax"): ["regexp", "glob"],
('core', 'mp_start_method'): multiprocessing.get_all_start_methods(),
("scheduler", "file_parsing_sort_mode"): ["modified_time", "random_seeded_by_host", "alphabetical"],
("logging", "logging_level"): _available_logging_levels,
("logging", "fab_logging_level"): _available_logging_levels,
# celery_logging_level can be empty, which uses logging_level as fallback
("logging", "celery_logging_level"): _available_logging_levels + [''],
+ ("webserver", "analytical_tool"): ['google_analytics', 'metarouter', 'segment', ''],
}
upgraded_values: Dict[Tuple[str, str], str]
"""Mapping of (section,option) to the old value that was upgraded"""
# This method transforms option names on every read, get, or set operation.
- # This changes from the default behaviour of ConfigParser from lowercasing
+ # This changes from the default behaviour of ConfigParser from lower-casing
# to instead be case-preserving
def optionxform(self, optionstr: str) -> str:
return optionstr
- def __init__(self, default_config=None, *args, **kwargs):
+ def __init__(self, default_config: Optional[str] = None, *args, **kwargs):
super().__init__(*args, **kwargs)
self.upgraded_values = {}
@@ -256,7 +293,7 @@ def __init__(self, default_config=None, *args, **kwargs):
if default_config is not None:
self.airflow_defaults.read_string(default_config)
# Set the upgrade value based on the current loaded default
- default = self.airflow_defaults.get('logging', 'log_filename_template', fallback=None, raw=True)
+ default = self.airflow_defaults.get('logging', 'log_filename_template', fallback=None)
if default:
replacement = self.deprecated_values['logging']['log_filename_template']
self.deprecated_values['logging']['log_filename_template'] = (
@@ -275,9 +312,7 @@ def __init__(self, default_config=None, *args, **kwargs):
self.is_validated = False
def validate(self):
-
self._validate_config_dependencies()
-
self._validate_enums()
for section, replacement in self.deprecated_values.items():
@@ -297,6 +332,7 @@ def validate(self):
)
self._upgrade_auth_backends()
+ self._upgrade_postgres_metastore_conn()
self.is_validated = True
def _upgrade_auth_backends(self):
@@ -309,8 +345,15 @@ def _upgrade_auth_backends(self):
# handled by deprecated_values
pass
elif old_value.find('airflow.api.auth.backend.session') == -1:
- new_value = old_value + "\nairflow.api.auth.backend.session"
+ new_value = old_value + ",airflow.api.auth.backend.session"
self._update_env_var(section="api", name="auth_backends", new_value=new_value)
+ self.upgraded_values[("api", "auth_backends")] = old_value
+
+ # if the old value is set via env var, we need to wipe it
+ # otherwise, it'll "win" over our adjusted value
+ old_env_var = self._env_var_name("api", "auth_backend")
+ os.environ.pop(old_env_var, None)
+
warnings.warn(
'The auth_backends setting in [api] has had airflow.api.auth.backend.session added '
'in the running config, which is needed by the UI. Please update your config before '
@@ -318,6 +361,29 @@ def _upgrade_auth_backends(self):
FutureWarning,
)
+ def _upgrade_postgres_metastore_conn(self):
+ """As of sqlalchemy 1.4, scheme `postgres+psycopg2` must be replaced with `postgresql`"""
+ section, key = 'database', 'sql_alchemy_conn'
+ old_value = self.get(section, key)
+ bad_scheme = 'postgres+psycopg2'
+ good_scheme = 'postgresql'
+ parsed = urlparse(old_value)
+ if parsed.scheme == bad_scheme:
+ warnings.warn(
+ f"Bad scheme in Airflow configuration core > sql_alchemy_conn: `{bad_scheme}`. "
+ "As of SqlAlchemy 1.4 (adopted in Airflow 2.3) this is no longer supported. You must "
+ f"change to `{good_scheme}` before the next Airflow release.",
+ FutureWarning,
+ )
+ self.upgraded_values[(section, key)] = old_value
+ new_value = re.sub('^' + re.escape(f"{bad_scheme}://"), f"{good_scheme}://", old_value)
+ self._update_env_var(section=section, name=key, new_value=new_value)
+
+ # if the old value is set via env var, we need to wipe it
+ # otherwise, it'll "win" over our adjusted value
+ old_env_var = self._env_var_name("core", key)
+ os.environ.pop(old_env_var, None)
+
def _validate_enums(self):
"""Validate that enum type config has an accepted value"""
for (section_key, option_key), enum_options in self.enums_options.items():
@@ -326,7 +392,7 @@ def _validate_enums(self):
if value not in enum_options:
raise AirflowConfigException(
f"`[{section_key}] {option_key}` should not be "
- + f"{value!r}. Possible values: {', '.join(enum_options)}."
+ f"{value!r}. Possible values: {', '.join(enum_options)}."
)
def _validate_config_dependencies(self):
@@ -338,7 +404,7 @@ def _validate_config_dependencies(self):
'DebugExecutor',
'SequentialExecutor',
)
- is_sqlite = "sqlite" in self.get('core', 'sql_alchemy_conn')
+ is_sqlite = "sqlite" in self.get('database', 'sql_alchemy_conn')
if is_sqlite and is_executor_without_sqlite_support:
raise AirflowConfigException(f"error: cannot use sqlite with the {self.get('core', 'executor')}")
if is_sqlite:
@@ -346,7 +412,7 @@ def _validate_config_dependencies(self):
from airflow.utils.docs import get_docs_url
- # Some of the features in storing rendered fields require sqlite version >= 3.15.0
+ # Some features in storing rendered fields require sqlite version >= 3.15.0
min_sqlite_version = (3, 15, 0)
if _parse_sqlite_version(sqlite3.sqlite_version) < min_sqlite_version:
min_sqlite_version_str = ".".join(str(s) for s in min_sqlite_version)
@@ -355,21 +421,16 @@ def _validate_config_dependencies(self):
f"See {get_docs_url('howto/set-up-database.html#setting-up-a-sqlite-database')}"
)
- def _using_old_value(self, old, current_value):
+ def _using_old_value(self, old: Pattern, current_value: str) -> bool:
return old.search(current_value) is not None
- def _update_env_var(self, section, name, new_value):
+ def _update_env_var(self, section: str, name: str, new_value: Union[str]):
env_var = self._env_var_name(section, name)
- # If the config comes from environment, set it there so that any subprocesses keep the same override!
- if env_var in os.environ:
- os.environ[env_var] = new_value
- return
- if not self.has_section(section):
- self.add_section(section)
- self.set(section, name, new_value)
+ # Set it as an env var so that any subprocesses keep the same override!
+ os.environ[env_var] = new_value
@staticmethod
- def _create_future_warning(name, section, current_value, new_value, version):
+ def _create_future_warning(name: str, section: str, current_value: Any, new_value: Any, version: str):
warnings.warn(
f'The {name!r} setting in [{section}] has the old default value of {current_value!r}. '
f'This value has been changed to {new_value!r} in the running config, but '
@@ -377,12 +438,10 @@ def _create_future_warning(name, section, current_value, new_value, version):
FutureWarning,
)
- ENV_VAR_PREFIX = 'AIRFLOW__'
-
def _env_var_name(self, section: str, key: str) -> str:
- return f'{self.ENV_VAR_PREFIX}{section.upper()}__{key.upper()}'
+ return f'{ENV_VAR_PREFIX}{section.upper()}__{key.upper()}'
- def _get_env_var_option(self, section, key):
+ def _get_env_var_option(self, section: str, key: str):
# must have format AIRFLOW__{SECTION}__{KEY} (note double underscore)
env_var = self._env_var_name(section, key)
if env_var in os.environ:
@@ -401,26 +460,62 @@ def _get_env_var_option(self, section, key):
return _get_config_value_from_secret_backend(os.environ[env_var_secret_path])
return None
- def _get_cmd_option(self, section, key):
+ def _get_cmd_option(self, section: str, key: str):
fallback_key = key + '_cmd'
- # if this is a valid command key...
if (section, key) in self.sensitive_config_values:
if super().has_option(section, fallback_key):
command = super().get(section, fallback_key)
return run_command(command)
return None
- def _get_secret_option(self, section, key):
+ def _get_cmd_option_from_config_sources(
+ self, config_sources: ConfigSourcesType, section: str, key: str
+ ) -> Optional[str]:
+ fallback_key = key + '_cmd'
+ if (section, key) in self.sensitive_config_values:
+ section_dict = config_sources.get(section)
+ if section_dict is not None:
+ command_value = section_dict.get(fallback_key)
+ if command_value is not None:
+ if isinstance(command_value, str):
+ command = command_value
+ else:
+ command = command_value[0]
+ return run_command(command)
+ return None
+
+ def _get_secret_option(self, section: str, key: str) -> Optional[str]:
"""Get Config option values from Secret Backend"""
fallback_key = key + '_secret'
- # if this is a valid secret key...
if (section, key) in self.sensitive_config_values:
if super().has_option(section, fallback_key):
secrets_path = super().get(section, fallback_key)
return _get_config_value_from_secret_backend(secrets_path)
return None
- def get(self, section, key, **kwargs):
+ def _get_secret_option_from_config_sources(
+ self, config_sources: ConfigSourcesType, section: str, key: str
+ ) -> Optional[str]:
+ fallback_key = key + '_secret'
+ if (section, key) in self.sensitive_config_values:
+ section_dict = config_sources.get(section)
+ if section_dict is not None:
+ secrets_path_value = section_dict.get(fallback_key)
+ if secrets_path_value is not None:
+ if isinstance(secrets_path_value, str):
+ secrets_path = secrets_path_value
+ else:
+ secrets_path = secrets_path_value[0]
+ return _get_config_value_from_secret_backend(secrets_path)
+ return None
+
+ def get_mandatory_value(self, section: str, key: str, **kwargs) -> str:
+ value = self.get(section, key, **kwargs)
+ if value is None:
+ raise ValueError(f"The value {section}/{key} should be set!")
+ return value
+
+ def get(self, section: str, key: str, **kwargs) -> Optional[str]: # type: ignore[override]
section = str(section).lower()
key = str(key).lower()
@@ -446,7 +541,7 @@ def get(self, section, key, **kwargs):
return self._get_option_from_default_config(section, key, **kwargs)
- def _get_option_from_default_config(self, section, key, **kwargs):
+ def _get_option_from_default_config(self, section: str, key: str, **kwargs) -> Optional[str]:
# ...then the default config
if self.airflow_defaults.has_option(section, key) or 'fallback' in kwargs:
return expand_env_var(self.airflow_defaults.get(section, key, **kwargs))
@@ -456,55 +551,68 @@ def _get_option_from_default_config(self, section, key, **kwargs):
raise AirflowConfigException(f"section/key [{section}/{key}] not found in config")
- def _get_option_from_secrets(self, deprecated_key, deprecated_section, key, section):
+ def _get_option_from_secrets(
+ self, deprecated_key: Optional[str], deprecated_section: Optional[str], key: str, section: str
+ ) -> Optional[str]:
# ...then from secret backends
option = self._get_secret_option(section, key)
if option:
return option
- if deprecated_section:
+ if deprecated_section and deprecated_key:
option = self._get_secret_option(deprecated_section, deprecated_key)
if option:
self._warn_deprecate(section, key, deprecated_section, deprecated_key)
return option
return None
- def _get_option_from_commands(self, deprecated_key, deprecated_section, key, section):
+ def _get_option_from_commands(
+ self, deprecated_key: Optional[str], deprecated_section: Optional[str], key: str, section: str
+ ) -> Optional[str]:
# ...then commands
option = self._get_cmd_option(section, key)
if option:
return option
- if deprecated_section:
+ if deprecated_section and deprecated_key:
option = self._get_cmd_option(deprecated_section, deprecated_key)
if option:
self._warn_deprecate(section, key, deprecated_section, deprecated_key)
return option
return None
- def _get_option_from_config_file(self, deprecated_key, deprecated_section, key, kwargs, section):
+ def _get_option_from_config_file(
+ self,
+ deprecated_key: Optional[str],
+ deprecated_section: Optional[str],
+ key: str,
+ kwargs: Dict[str, Any],
+ section: str,
+ ) -> Optional[str]:
# ...then the config file
if super().has_option(section, key):
# Use the parent's methods to get the actual config here to be able to
# separate the config from default config.
return expand_env_var(super().get(section, key, **kwargs))
- if deprecated_section:
+ if deprecated_section and deprecated_key:
if super().has_option(deprecated_section, deprecated_key):
self._warn_deprecate(section, key, deprecated_section, deprecated_key)
return expand_env_var(super().get(deprecated_section, deprecated_key, **kwargs))
return None
- def _get_environment_variables(self, deprecated_key, deprecated_section, key, section):
+ def _get_environment_variables(
+ self, deprecated_key: Optional[str], deprecated_section: Optional[str], key: str, section: str
+ ) -> Optional[str]:
# first check environment variables
option = self._get_env_var_option(section, key)
if option is not None:
return option
- if deprecated_section:
+ if deprecated_section and deprecated_key:
option = self._get_env_var_option(deprecated_section, deprecated_key)
if option is not None:
self._warn_deprecate(section, key, deprecated_section, deprecated_key)
return option
return None
- def getboolean(self, section, key, **kwargs):
+ def getboolean(self, section: str, key: str, **kwargs) -> bool: # type: ignore[override]
val = str(self.get(section, key, **kwargs)).lower().strip()
if '#' in val:
val = val.split('#')[0].strip()
@@ -518,9 +626,13 @@ def getboolean(self, section, key, **kwargs):
f'Current value: "{val}".'
)
- def getint(self, section, key, **kwargs):
+ def getint(self, section: str, key: str, **kwargs) -> int: # type: ignore[override]
val = self.get(section, key, **kwargs)
-
+ if val is None:
+ raise AirflowConfigException(
+ f'Failed to convert value None to int. '
+ f'Please check "{key}" key in "{section}" section is set.'
+ )
try:
return int(val)
except ValueError:
@@ -529,9 +641,13 @@ def getint(self, section, key, **kwargs):
f'Current value: "{val}".'
)
- def getfloat(self, section, key, **kwargs):
+ def getfloat(self, section: str, key: str, **kwargs) -> float: # type: ignore[override]
val = self.get(section, key, **kwargs)
-
+ if val is None:
+ raise AirflowConfigException(
+ f'Failed to convert value None to float. '
+ f'Please check "{key}" key in "{section}" section is set.'
+ )
try:
return float(val)
except ValueError:
@@ -540,7 +656,7 @@ def getfloat(self, section, key, **kwargs):
f'Current value: "{val}".'
)
- def getimport(self, section, key, **kwargs):
+ def getimport(self, section: str, key: str, **kwargs) -> Any:
"""
Reads options, imports the full qualified name, and returns the object.
@@ -561,7 +677,9 @@ def getimport(self, section, key, **kwargs):
f'Current value: "{full_qualified_path}".'
)
- def getjson(self, section, key, fallback=_UNSET, **kwargs) -> Union[dict, list, str, int, float, None]:
+ def getjson(
+ self, section: str, key: str, fallback=_UNSET, **kwargs
+ ) -> Union[dict, list, str, int, float, None]:
"""
Return a config value parsed from a JSON string.
@@ -579,7 +697,7 @@ def getjson(self, section, key, fallback=_UNSET, **kwargs) -> Union[dict, list,
except (NoSectionError, NoOptionError):
return default
- if len(data) == 0:
+ if not data:
return default if default is not _UNSET else None
try:
@@ -587,13 +705,62 @@ def getjson(self, section, key, fallback=_UNSET, **kwargs) -> Union[dict, list,
except JSONDecodeError as e:
raise AirflowConfigException(f'Unable to parse [{section}] {key!r} as valid json') from e
- def read(self, filenames, encoding=None):
+ def gettimedelta(
+ self, section: str, key: str, fallback: Any = None, **kwargs
+ ) -> Optional[datetime.timedelta]:
+ """
+ Gets the config value for the given section and key, and converts it into datetime.timedelta object.
+ If the key is missing, then it is considered as `None`.
+
+ :param section: the section from the config
+ :param key: the key defined in the given section
+ :param fallback: fallback value when no config value is given, defaults to None
+ :raises AirflowConfigException: raised because ValueError or OverflowError
+ :return: datetime.timedelta(seconds=) or None
+ """
+ val = self.get(section, key, fallback=fallback, **kwargs)
+
+ if val:
+ # the given value must be convertible to integer
+ try:
+ int_val = int(val)
+ except ValueError:
+ raise AirflowConfigException(
+ f'Failed to convert value to int. Please check "{key}" key in "{section}" section. '
+ f'Current value: "{val}".'
+ )
+
+ try:
+ return datetime.timedelta(seconds=int_val)
+ except OverflowError as err:
+ raise AirflowConfigException(
+ f'Failed to convert value to timedelta in `seconds`. '
+ f'{err}. '
+ f'Please check "{key}" key in "{section}" section. Current value: "{val}".'
+ )
+
+ return fallback
+
+ def read(
+ self,
+ filenames: Union[
+ str,
+ bytes,
+ os.PathLike,
+ Iterable[Union[str, bytes, os.PathLike]],
+ ],
+ encoding=None,
+ ):
super().read(filenames=filenames, encoding=encoding)
- def read_dict(self, dictionary, source=''):
+ # The RawConfigParser defines "Mapping" from abc.collections is not subscriptable - so we have
+ # to use Dict here.
+ def read_dict( # type: ignore[override]
+ self, dictionary: Dict[str, Dict[str, Any]], source: str = ''
+ ):
super().read_dict(dictionary=dictionary, source=source)
- def has_option(self, section, option):
+ def has_option(self, section: str, option: str) -> bool:
try:
# Using self.get() to avoid reimplementing the priority order
# of config variables (env, config, cmd, defaults)
@@ -603,7 +770,7 @@ def has_option(self, section, option):
except (NoOptionError, NoSectionError):
return False
- def remove_option(self, section, option, remove_default=True):
+ def remove_option(self, section: str, option: str, remove_default: bool = True):
"""
Remove an option if it exists in config from a file or
default config. If both of config have the same option, this removes
@@ -615,7 +782,7 @@ def remove_option(self, section, option, remove_default=True):
if self.airflow_defaults.has_option(section, option) and remove_default:
self.airflow_defaults.remove_option(section, option)
- def getsection(self, section: str) -> Optional[Dict[str, Union[str, int, float, bool]]]:
+ def getsection(self, section: str) -> Optional[ConfigOptionsDictType]:
"""
Returns the section as a dict. Values are converted to int, float, bool
as required.
@@ -625,9 +792,8 @@ def getsection(self, section: str) -> Optional[Dict[str, Union[str, int, float,
"""
if not self.has_section(section) and not self.airflow_defaults.has_section(section):
return None
-
if self.airflow_defaults.has_section(section):
- _section = OrderedDict(self.airflow_defaults.items(section))
+ _section: ConfigOptionsDictType = OrderedDict(self.airflow_defaults.items(section))
else:
_section = OrderedDict()
@@ -644,40 +810,48 @@ def getsection(self, section: str) -> Optional[Dict[str, Union[str, int, float,
_section[key] = self._get_env_var_option(section, key)
for key, val in _section.items():
+ if val is None:
+ raise AirflowConfigException(
+ f'Failed to convert value automatically. '
+ f'Please check "{key}" key in "{section}" section is set.'
+ )
try:
- val = int(val)
+ _section[key] = int(val)
except ValueError:
try:
- val = float(val)
+ _section[key] = float(val)
except ValueError:
- if val.lower() in ('t', 'true'):
- val = True
- elif val.lower() in ('f', 'false'):
- val = False
- _section[key] = val
+ if isinstance(val, str) and val.lower() in ('t', 'true'):
+ _section[key] = True
+ elif isinstance(val, str) and val.lower() in ('f', 'false'):
+ _section[key] = False
return _section
- def write(self, fp, space_around_delimiters=True):
+ def write(self, fp: IO, space_around_delimiters: bool = True): # type: ignore[override]
# This is based on the configparser.RawConfigParser.write method code to add support for
# reading options from environment variables.
+ # Various type ignores below deal with less-than-perfect RawConfigParser superclass typing
if space_around_delimiters:
- delimiter = f" {self._delimiters[0]} "
+ delimiter = f" {self._delimiters[0]} " # type: ignore[attr-defined]
else:
- delimiter = self._delimiters[0]
- if self._defaults:
- self._write_section(fp, self.default_section, self._defaults.items(), delimiter)
- for section in self._sections:
- self._write_section(fp, section, self.getsection(section).items(), delimiter)
+ delimiter = self._delimiters[0] # type: ignore[attr-defined]
+ if self._defaults: # type: ignore
+ self._write_section( # type: ignore[attr-defined]
+ fp, self.default_section, self._defaults.items(), delimiter # type: ignore[attr-defined]
+ )
+ for section in self._sections: # type: ignore[attr-defined]
+ item_section: ConfigOptionsDictType = self.getsection(section) # type: ignore[assignment]
+ self._write_section(fp, section, item_section.items(), delimiter) # type: ignore[attr-defined]
def as_dict(
self,
- display_source=False,
- display_sensitive=False,
- raw=False,
- include_env=True,
- include_cmds=True,
- include_secret=True,
- ) -> Dict[str, Dict[str, str]]:
+ display_source: bool = False,
+ display_sensitive: bool = False,
+ raw: bool = False,
+ include_env: bool = True,
+ include_cmds: bool = True,
+ include_secret: bool = True,
+ ) -> ConfigSourcesType:
"""
Returns the current configuration as an OrderedDict of OrderedDicts.
@@ -710,13 +884,22 @@ def as_dict(
:return: Dictionary, where the key is the name of the section and the content is
the dictionary with the name of the parameter and its value.
"""
- config_sources: Dict[str, Dict[str, str]] = {}
+ config_sources: ConfigSourcesType = {}
configs = [
('default', self.airflow_defaults),
('airflow.cfg', self),
]
- self._replace_config_with_display_sources(config_sources, configs, display_source, raw)
+ self._replace_config_with_display_sources(
+ config_sources,
+ configs,
+ display_source,
+ raw,
+ self.deprecated_options,
+ include_cmds=include_cmds,
+ include_env=include_env,
+ include_secret=include_secret,
+ )
# add env vars and overwrite because they have priority
if include_env:
@@ -738,36 +921,59 @@ def as_dict(
return config_sources
- def _include_secrets(self, config_sources, display_sensitive, display_source, raw):
+ def _include_secrets(
+ self,
+ config_sources: ConfigSourcesType,
+ display_sensitive: bool,
+ display_source: bool,
+ raw: bool,
+ ):
for (section, key) in self.sensitive_config_values:
- opt = self._get_secret_option(section, key)
- if opt:
+ value: Optional[str] = self._get_secret_option_from_config_sources(config_sources, section, key)
+ if value:
if not display_sensitive:
- opt = '< hidden >'
+ value = '< hidden >'
if display_source:
- opt = (opt, 'secret')
+ opt: Union[str, Tuple[str, str]] = (value, 'secret')
elif raw:
- opt = opt.replace('%', '%%')
+ opt = value.replace('%', '%%')
+ else:
+ opt = value
config_sources.setdefault(section, OrderedDict()).update({key: opt})
del config_sources[section][key + '_secret']
- def _include_commands(self, config_sources, display_sensitive, display_source, raw):
+ def _include_commands(
+ self,
+ config_sources: ConfigSourcesType,
+ display_sensitive: bool,
+ display_source: bool,
+ raw: bool,
+ ):
for (section, key) in self.sensitive_config_values:
- opt = self._get_cmd_option(section, key)
+ opt = self._get_cmd_option_from_config_sources(config_sources, section, key)
if not opt:
continue
+ opt_to_set: Union[str, Tuple[str, str], None] = opt
if not display_sensitive:
- opt = '< hidden >'
+ opt_to_set = '< hidden >'
if display_source:
- opt = (opt, 'cmd')
+ opt_to_set = (str(opt_to_set), 'cmd')
elif raw:
- opt = opt.replace('%', '%%')
- config_sources.setdefault(section, OrderedDict()).update({key: opt})
- del config_sources[section][key + '_cmd']
+ opt_to_set = str(opt_to_set).replace('%', '%%')
+ if opt_to_set is not None:
+ dict_to_update: Dict[str, Union[str, Tuple[str, str]]] = {key: opt_to_set}
+ config_sources.setdefault(section, OrderedDict()).update(dict_to_update)
+ del config_sources[section][key + '_cmd']
- def _include_envs(self, config_sources, display_sensitive, display_source, raw):
+ def _include_envs(
+ self,
+ config_sources: ConfigSourcesType,
+ display_sensitive: bool,
+ display_source: bool,
+ raw: bool,
+ ):
for env_var in [
- os_environment for os_environment in os.environ if os_environment.startswith(self.ENV_VAR_PREFIX)
+ os_environment for os_environment in os.environ if os_environment.startswith(ENV_VAR_PREFIX)
]:
try:
_, section, key = env_var.split('__', 2)
@@ -793,7 +999,12 @@ def _include_envs(self, config_sources, display_sensitive, display_source, raw):
key = key.lower()
config_sources.setdefault(section, OrderedDict()).update({key: opt})
- def _filter_by_source(self, config_sources, display_source, getter_func):
+ def _filter_by_source(
+ self,
+ config_sources: ConfigSourcesType,
+ display_source: bool,
+ getter_func,
+ ):
"""
Deletes default configs from current configuration (an OrderedDict of
OrderedDicts) if it would conflict with special sensitive_config_values.
@@ -829,29 +1040,147 @@ def _filter_by_source(self, config_sources, display_source, getter_func):
continue
# Check to see if bare setting is the same as defaults
if display_source:
- opt, source = config_sources[section][key]
+ # when display_source = true, we know that the config_sources contains tuple
+ opt, source = config_sources[section][key] # type: ignore
else:
opt = config_sources[section][key]
if opt == self.airflow_defaults.get(section, key):
del config_sources[section][key]
@staticmethod
- def _replace_config_with_display_sources(config_sources, configs, display_source, raw):
+ def _replace_config_with_display_sources(
+ config_sources: ConfigSourcesType,
+ configs: Iterable[Tuple[str, ConfigParser]],
+ display_source: bool,
+ raw: bool,
+ deprecated_options: Dict[Tuple[str, str], Tuple[str, str, str]],
+ include_env: bool,
+ include_cmds: bool,
+ include_secret: bool,
+ ):
for (source_name, config) in configs:
for section in config.sections():
AirflowConfigParser._replace_section_config_with_display_sources(
- config, config_sources, display_source, raw, section, source_name
+ config,
+ config_sources,
+ display_source,
+ raw,
+ section,
+ source_name,
+ deprecated_options,
+ configs,
+ include_env=include_env,
+ include_cmds=include_cmds,
+ include_secret=include_secret,
)
+ @staticmethod
+ def _deprecated_value_is_set_in_config(
+ deprecated_section: str,
+ deprecated_key: str,
+ configs: Iterable[Tuple[str, ConfigParser]],
+ ) -> bool:
+ for config_type, config in configs:
+ if config_type == 'default':
+ continue
+ try:
+ deprecated_section_array = config.items(section=deprecated_section, raw=True)
+ for (key_candidate, _) in deprecated_section_array:
+ if key_candidate == deprecated_key:
+ return True
+ except NoSectionError:
+ pass
+ return False
+
+ @staticmethod
+ def _deprecated_variable_is_set(deprecated_section: str, deprecated_key: str) -> bool:
+ return (
+ os.environ.get(f'{ENV_VAR_PREFIX}{deprecated_section.upper()}__{deprecated_key.upper()}')
+ is not None
+ )
+
+ @staticmethod
+ def _deprecated_command_is_set_in_config(
+ deprecated_section: str, deprecated_key: str, configs: Iterable[Tuple[str, ConfigParser]]
+ ) -> bool:
+ return AirflowConfigParser._deprecated_value_is_set_in_config(
+ deprecated_section=deprecated_section, deprecated_key=deprecated_key + "_cmd", configs=configs
+ )
+
+ @staticmethod
+ def _deprecated_variable_command_is_set(deprecated_section: str, deprecated_key: str) -> bool:
+ return (
+ os.environ.get(f'{ENV_VAR_PREFIX}{deprecated_section.upper()}__{deprecated_key.upper()}_CMD')
+ is not None
+ )
+
+ @staticmethod
+ def _deprecated_secret_is_set_in_config(
+ deprecated_section: str, deprecated_key: str, configs: Iterable[Tuple[str, ConfigParser]]
+ ) -> bool:
+ return AirflowConfigParser._deprecated_value_is_set_in_config(
+ deprecated_section=deprecated_section, deprecated_key=deprecated_key + "_secret", configs=configs
+ )
+
+ @staticmethod
+ def _deprecated_variable_secret_is_set(deprecated_section: str, deprecated_key: str) -> bool:
+ return (
+ os.environ.get(f'{ENV_VAR_PREFIX}{deprecated_section.upper()}__{deprecated_key.upper()}_SECRET')
+ is not None
+ )
+
@staticmethod
def _replace_section_config_with_display_sources(
- config, config_sources, display_source, raw, section, source_name
+ config: ConfigParser,
+ config_sources: ConfigSourcesType,
+ display_source: bool,
+ raw: bool,
+ section: str,
+ source_name: str,
+ deprecated_options: Dict[Tuple[str, str], Tuple[str, str, str]],
+ configs: Iterable[Tuple[str, ConfigParser]],
+ include_env: bool,
+ include_cmds: bool,
+ include_secret: bool,
):
sect = config_sources.setdefault(section, OrderedDict())
for (k, val) in config.items(section=section, raw=raw):
+ deprecated_section, deprecated_key, _ = deprecated_options.get((section, k), (None, None, None))
+ if deprecated_section and deprecated_key:
+ if source_name == 'default':
+ # If deprecated entry has some non-default value set for any of the sources requested,
+ # We should NOT set default for the new entry (because it will override anything
+ # coming from the deprecated ones)
+ if AirflowConfigParser._deprecated_value_is_set_in_config(
+ deprecated_section, deprecated_key, configs
+ ):
+ continue
+ if include_env and AirflowConfigParser._deprecated_variable_is_set(
+ deprecated_section, deprecated_key
+ ):
+ continue
+ if include_cmds and (
+ AirflowConfigParser._deprecated_variable_command_is_set(
+ deprecated_section, deprecated_key
+ )
+ or AirflowConfigParser._deprecated_command_is_set_in_config(
+ deprecated_section, deprecated_key, configs
+ )
+ ):
+ continue
+ if include_secret and (
+ AirflowConfigParser._deprecated_variable_secret_is_set(
+ deprecated_section, deprecated_key
+ )
+ or AirflowConfigParser._deprecated_secret_is_set_in_config(
+ deprecated_section, deprecated_key, configs
+ )
+ ):
+ continue
if display_source:
- val = (val, source_name)
- sect[k] = val
+ sect[k] = (val, source_name)
+ else:
+ sect[k] = val
def load_test_config(self):
"""
@@ -873,7 +1202,7 @@ def load_test_config(self):
self.read(TEST_CONFIG_FILE)
@staticmethod
- def _warn_deprecate(section, key, deprecated_section, deprecated_name):
+ def _warn_deprecate(section: str, key: str, deprecated_section: str, deprecated_name: str):
if section == deprecated_section:
warnings.warn(
f'The {deprecated_name} option in [{section}] has been renamed to {key} - '
@@ -906,16 +1235,17 @@ def __setstate__(self, state):
self.__dict__.update(state)
-def get_airflow_home():
+def get_airflow_home() -> str:
"""Get path to Airflow Home"""
return expand_env_var(os.environ.get('AIRFLOW_HOME', '~/airflow'))
-def get_airflow_config(airflow_home):
+def get_airflow_config(airflow_home) -> str:
"""Get Path to airflow.cfg path"""
- if 'AIRFLOW_CONFIG' not in os.environ:
+ airflow_config_var = os.environ.get('AIRFLOW_CONFIG')
+ if airflow_config_var is None:
return os.path.join(airflow_home, 'airflow.cfg')
- return expand_env_var(os.environ['AIRFLOW_CONFIG'])
+ return expand_env_var(airflow_config_var)
def _parameterized_config_from_template(filename) -> str:
@@ -930,7 +1260,7 @@ def _parameterized_config_from_template(filename) -> str:
raise RuntimeError(f"Template marker not found in {path!r}")
-def parameterized_config(template):
+def parameterized_config(template) -> str:
"""
Generates a configuration from the provided template + variables defined in
current scope
@@ -941,20 +1271,21 @@ def parameterized_config(template):
return template.format(**all_vars)
-def get_airflow_test_config(airflow_home):
+def get_airflow_test_config(airflow_home) -> str:
"""Get path to unittests.cfg"""
if 'AIRFLOW_TEST_CONFIG' not in os.environ:
return os.path.join(airflow_home, 'unittests.cfg')
- return expand_env_var(os.environ['AIRFLOW_TEST_CONFIG'])
+ # It will never return None
+ return expand_env_var(os.environ['AIRFLOW_TEST_CONFIG']) # type: ignore[return-value]
-def _generate_fernet_key():
+def _generate_fernet_key() -> str:
from cryptography.fernet import Fernet
return Fernet.generate_key().decode()
-def initialize_config():
+def initialize_config() -> AirflowConfigParser:
"""
Load the Airflow config files.
@@ -964,9 +1295,9 @@ def initialize_config():
default_config = _parameterized_config_from_template('default_airflow.cfg')
- conf = AirflowConfigParser(default_config=default_config)
+ local_conf = AirflowConfigParser(default_config=default_config)
- if conf.getboolean('core', 'unit_test_mode'):
+ if local_conf.getboolean('core', 'unit_test_mode'):
# Load test config only
if not os.path.isfile(TEST_CONFIG_FILE):
from cryptography.fernet import Fernet
@@ -980,7 +1311,7 @@ def initialize_config():
cfg = _parameterized_config_from_template('default_test.cfg')
file.write(cfg)
- conf.load_test_config()
+ local_conf.load_test_config()
else:
# Load normal config
if not os.path.isfile(AIRFLOW_CONFIG):
@@ -996,9 +1327,9 @@ def initialize_config():
log.info("Reading the config from %s", AIRFLOW_CONFIG)
- conf.read(AIRFLOW_CONFIG)
+ local_conf.read(AIRFLOW_CONFIG)
- if conf.has_option('core', 'AIRFLOW_HOME'):
+ if local_conf.has_option('core', 'AIRFLOW_HOME'):
msg = (
'Specifying both AIRFLOW_HOME environment variable and airflow_home '
'in the config file is deprecated. Please use only the AIRFLOW_HOME '
@@ -1006,7 +1337,7 @@ def initialize_config():
)
if 'AIRFLOW_HOME' in os.environ:
warnings.warn(msg, category=DeprecationWarning)
- elif conf.get('core', 'airflow_home') == AIRFLOW_HOME:
+ elif local_conf.get('core', 'airflow_home') == AIRFLOW_HOME:
warnings.warn(
'Specifying airflow_home in the config file is deprecated. As you '
'have left it at the default value you should remove the setting '
@@ -1014,13 +1345,14 @@ def initialize_config():
category=DeprecationWarning,
)
else:
- AIRFLOW_HOME = conf.get('core', 'airflow_home')
+ # there
+ AIRFLOW_HOME = local_conf.get('core', 'airflow_home') # type: ignore[assignment]
warnings.warn(msg, category=DeprecationWarning)
# They _might_ have set unit_test_mode in the airflow.cfg, we still
# want to respect that and then load the unittests.cfg
- if conf.getboolean('core', 'unit_test_mode'):
- conf.load_test_config()
+ if local_conf.getboolean('core', 'unit_test_mode'):
+ local_conf.load_test_config()
# Make it no longer a proxy variable, just set it to an actual string
global WEBSERVER_CONFIG
@@ -1031,7 +1363,7 @@ def initialize_config():
log.info('Creating new FAB webserver config file in: %s', WEBSERVER_CONFIG)
shutil.copy(_default_config_file_path('default_webserver_config.py'), WEBSERVER_CONFIG)
- return conf
+ return local_conf
# Historical convenience functions to access config entries
@@ -1047,7 +1379,7 @@ def load_test_config():
conf.load_test_config()
-def get(*args, **kwargs):
+def get(*args, **kwargs) -> Optional[ConfigType]:
"""Historical get"""
warnings.warn(
"Accessing configuration method 'get' directly from the configuration module is "
@@ -1059,7 +1391,7 @@ def get(*args, **kwargs):
return conf.get(*args, **kwargs)
-def getboolean(*args, **kwargs):
+def getboolean(*args, **kwargs) -> bool:
"""Historical getboolean"""
warnings.warn(
"Accessing configuration method 'getboolean' directly from the configuration module is "
@@ -1071,7 +1403,7 @@ def getboolean(*args, **kwargs):
return conf.getboolean(*args, **kwargs)
-def getfloat(*args, **kwargs):
+def getfloat(*args, **kwargs) -> float:
"""Historical getfloat"""
warnings.warn(
"Accessing configuration method 'getfloat' directly from the configuration module is "
@@ -1083,7 +1415,7 @@ def getfloat(*args, **kwargs):
return conf.getfloat(*args, **kwargs)
-def getint(*args, **kwargs):
+def getint(*args, **kwargs) -> int:
"""Historical getint"""
warnings.warn(
"Accessing configuration method 'getint' directly from the configuration module is "
@@ -1095,7 +1427,7 @@ def getint(*args, **kwargs):
return conf.getint(*args, **kwargs)
-def getsection(*args, **kwargs):
+def getsection(*args, **kwargs) -> Optional[ConfigOptionsDictType]:
"""Historical getsection"""
warnings.warn(
"Accessing configuration method 'getsection' directly from the configuration module is "
@@ -1107,7 +1439,7 @@ def getsection(*args, **kwargs):
return conf.getsection(*args, **kwargs)
-def has_option(*args, **kwargs):
+def has_option(*args, **kwargs) -> bool:
"""Historical has_option"""
warnings.warn(
"Accessing configuration method 'has_option' directly from the configuration module is "
@@ -1119,7 +1451,7 @@ def has_option(*args, **kwargs):
return conf.has_option(*args, **kwargs)
-def remove_option(*args, **kwargs):
+def remove_option(*args, **kwargs) -> bool:
"""Historical remove_option"""
warnings.warn(
"Accessing configuration method 'remove_option' directly from the configuration module is "
@@ -1131,7 +1463,7 @@ def remove_option(*args, **kwargs):
return conf.remove_option(*args, **kwargs)
-def as_dict(*args, **kwargs):
+def as_dict(*args, **kwargs) -> ConfigSourcesType:
"""Historical as_dict"""
warnings.warn(
"Accessing configuration method 'as_dict' directly from the configuration module is "
@@ -1143,7 +1475,7 @@ def as_dict(*args, **kwargs):
return conf.as_dict(*args, **kwargs)
-def set(*args, **kwargs):
+def set(*args, **kwargs) -> None:
"""Historical set"""
warnings.warn(
"Accessing configuration method 'set' directly from the configuration module is "
@@ -1152,7 +1484,7 @@ def set(*args, **kwargs):
DeprecationWarning,
stacklevel=2,
)
- return conf.set(*args, **kwargs)
+ conf.set(*args, **kwargs)
def ensure_secrets_loaded() -> List[BaseSecretsBackend]:
@@ -1172,9 +1504,8 @@ def get_custom_secret_backend() -> Optional[BaseSecretsBackend]:
if secrets_backend_cls:
try:
- alternative_secrets_config_dict = json.loads(
- conf.get(section='secrets', key='backend_kwargs', fallback='{}')
- )
+ backends: Any = conf.get(section='secrets', key='backend_kwargs', fallback='{}')
+ alternative_secrets_config_dict = json.loads(backends)
except JSONDecodeError:
alternative_secrets_config_dict = {}
@@ -1202,14 +1533,14 @@ def initialize_secrets_backends() -> List[BaseSecretsBackend]:
@functools.lru_cache(maxsize=None)
-def _DEFAULT_CONFIG():
+def _DEFAULT_CONFIG() -> str:
path = _default_config_file_path('default_airflow.cfg')
with open(path) as fh:
return fh.read()
@functools.lru_cache(maxsize=None)
-def _TEST_CONFIG():
+def _TEST_CONFIG() -> str:
path = _default_config_file_path('default_test.cfg')
with open(path) as fh:
return fh.read()
@@ -1236,7 +1567,6 @@ def __getattr__(name):
# Setting AIRFLOW_HOME and AIRFLOW_CONFIG from environment variables, using
# "~/airflow" and "$AIRFLOW_HOME/airflow.cfg" respectively as defaults.
-
AIRFLOW_HOME = get_airflow_home()
AIRFLOW_CONFIG = get_airflow_config(AIRFLOW_HOME)
@@ -1270,10 +1600,3 @@ def __getattr__(name):
conf = initialize_config()
secrets_backend_list = initialize_secrets_backends()
conf.validate()
-
-
-PY37 = sys.version_info >= (3, 7)
-if not PY37:
- from pep562 import Pep562
-
- Pep562(__name__)
diff --git a/airflow/dag_processing/manager.py b/airflow/dag_processing/manager.py
index 6d5e1fbbaa5cc..38f5ffdbbf73a 100644
--- a/airflow/dag_processing/manager.py
+++ b/airflow/dag_processing/manager.py
@@ -49,7 +49,11 @@
from airflow.utils.log.logging_mixin import LoggingMixin
from airflow.utils.mixins import MultiprocessingStartMethodMixin
from airflow.utils.net import get_hostname
-from airflow.utils.process_utils import kill_child_processes_by_pids, reap_process_group
+from airflow.utils.process_utils import (
+ kill_child_processes_by_pids,
+ reap_process_group,
+ set_new_process_group,
+)
from airflow.utils.session import NEW_SESSION, provide_session
from airflow.utils.sqlalchemy import prohibit_commit, skip_locked, with_row_locks
@@ -70,7 +74,7 @@ class DagFileStat(NamedTuple):
num_dags: int
import_errors: int
last_finish_time: Optional[datetime]
- last_duration: Optional[float]
+ last_duration: Optional[timedelta]
run_count: int
@@ -239,13 +243,13 @@ def _run_processor_manager(
airflow.settings.initialize()
del os.environ['CONFIG_PROCESSOR_MANAGER_LOGGER']
processor_manager = DagFileProcessorManager(
- dag_directory,
- max_runs,
- processor_timeout,
- signal_conn,
- dag_ids,
- pickle_dags,
- async_mode,
+ dag_directory=dag_directory,
+ max_runs=max_runs,
+ processor_timeout=processor_timeout,
+ dag_ids=dag_ids,
+ pickle_dags=pickle_dags,
+ signal_conn=signal_conn,
+ async_mode=async_mode,
)
processor_manager.start()
@@ -368,9 +372,9 @@ def __init__(
dag_directory: Union[str, "pathlib.Path"],
max_runs: int,
processor_timeout: timedelta,
- signal_conn: MultiprocessingConnection,
dag_ids: Optional[List[str]],
pickle_dags: bool,
+ signal_conn: Optional[MultiprocessingConnection] = None,
async_mode: bool = True,
):
super().__init__()
@@ -378,7 +382,8 @@ def __init__(
self._file_path_queue: List[str] = []
self._dag_directory = dag_directory
self._max_runs = max_runs
- self._signal_conn = signal_conn
+ # signal_conn is None for dag_processor_standalone mode.
+ self._direct_scheduler_conn = signal_conn
self._pickle_dags = pickle_dags
self._dag_ids = dag_ids
self._async_mode = async_mode
@@ -390,11 +395,14 @@ def __init__(
#
# Don't do this in sync_mode, as we _need_ the DagParsingStat sent to
# continue the scheduler
- if self._async_mode:
- os.set_blocking(self._signal_conn.fileno(), False)
+ if self._async_mode and self._direct_scheduler_conn is not None:
+ os.set_blocking(self._direct_scheduler_conn.fileno(), False)
self._parallelism = conf.getint('scheduler', 'parsing_processes')
- if conf.get('core', 'sql_alchemy_conn').startswith('sqlite') and self._parallelism > 1:
+ if (
+ conf.get_mandatory_value('database', 'sql_alchemy_conn').startswith('sqlite')
+ and self._parallelism > 1
+ ):
self.log.warning(
"Because we cannot use more than 1 thread (parsing_processes = "
"%d) when using sqlite. So we set parallelism to 1.",
@@ -420,9 +428,12 @@ def __init__(
self.last_dag_dir_refresh_time = timezone.make_aware(datetime.fromtimestamp(0))
# Last time stats were printed
self.last_stat_print_time = 0
+ # Last time we cleaned up DAGs which are no longer in files
+ self.last_deactivate_stale_dags_time = timezone.make_aware(datetime.fromtimestamp(0))
+ # How often to check for DAGs which are no longer in files
+ self.deactivate_stale_dags_interval = conf.getint('scheduler', 'deactivate_stale_dags_interval')
# How long to wait before timing out a process to parse a DAG file
self._processor_timeout = processor_timeout
-
# How often to scan the DAGs directory for new files. Default to 5 minutes.
self.dag_dir_list_interval = conf.getint('scheduler', 'dag_dir_list_interval')
@@ -431,9 +442,13 @@ def __init__(
self._log = logging.getLogger('airflow.processor_manager')
- self.waitables: Dict[Any, Union[MultiprocessingConnection, DagFileProcessorProcess]] = {
- self._signal_conn: self._signal_conn,
- }
+ self.waitables: Dict[Any, Union[MultiprocessingConnection, DagFileProcessorProcess]] = (
+ {
+ self._direct_scheduler_conn: self._direct_scheduler_conn,
+ }
+ if self._direct_scheduler_conn is not None
+ else {}
+ )
def register_exit_signals(self):
"""Register signals that stop child processes"""
@@ -460,8 +475,7 @@ def start(self):
"""
self.register_exit_signals()
- # Start a new process group
- os.setpgid(0, 0)
+ set_new_process_group()
self.log.info("Processing files using up to %s processes at a time ", self._parallelism)
self.log.info("Process each file at most once every %s seconds", self._file_process_interval)
@@ -471,6 +485,51 @@ def start(self):
return self._run_parsing_loop()
+ @provide_session
+ def _deactivate_stale_dags(self, session=None):
+ """
+ Detects DAGs which are no longer present in files
+
+ Deactivate them and remove them in the serialized_dag table
+ """
+ now = timezone.utcnow()
+ elapsed_time_since_refresh = (now - self.last_deactivate_stale_dags_time).total_seconds()
+ if elapsed_time_since_refresh > self.deactivate_stale_dags_interval:
+ last_parsed = {
+ fp: self.get_last_finish_time(fp) for fp in self.file_paths if self.get_last_finish_time(fp)
+ }
+ to_deactivate = set()
+ dags_parsed = (
+ session.query(DagModel.dag_id, DagModel.fileloc, DagModel.last_parsed_time)
+ .filter(DagModel.is_active)
+ .all()
+ )
+ for dag in dags_parsed:
+ # The largest valid difference between a DagFileStat's last_finished_time and a DAG's
+ # last_parsed_time is _processor_timeout. Longer than that indicates that the DAG is
+ # no longer present in the file.
+ if (
+ dag.fileloc in last_parsed
+ and (dag.last_parsed_time + self._processor_timeout) < last_parsed[dag.fileloc]
+ ):
+ self.log.info("DAG %s is missing and will be deactivated.", dag.dag_id)
+ to_deactivate.add(dag.dag_id)
+
+ if to_deactivate:
+ deactivated = (
+ session.query(DagModel)
+ .filter(DagModel.dag_id.in_(to_deactivate))
+ .update({DagModel.is_active: False}, synchronize_session="fetch")
+ )
+ if deactivated:
+ self.log.info("Deactivated %i DAGs which are no longer present in file.", deactivated)
+
+ for dag_id in to_deactivate:
+ SerializedDagModel.remove_dag(dag_id)
+ self.log.info("Deleted DAG %s in serialized_dag table", dag_id)
+
+ self.last_deactivate_stale_dags_time = timezone.utcnow()
+
def _run_parsing_loop(self):
# In sync mode we want timeout=None -- wait forever until a message is received
@@ -481,8 +540,8 @@ def _run_parsing_loop(self):
self._refresh_dag_dir()
self.prepare_file_path_queue()
- maxCallbacksPerLoop = conf.getint("scheduler", "max_callbacks_per_loop")
-
+ max_callbacks_per_loop = conf.getint("scheduler", "max_callbacks_per_loop")
+ standalone_dag_processor = conf.getboolean("scheduler", "standalone_dag_processor")
if self._async_mode:
# If we're in async mode, we can start up straight away. If we're
# in sync mode we need to be told to start a "loop"
@@ -490,11 +549,8 @@ def _run_parsing_loop(self):
while True:
loop_start_time = time.monotonic()
ready = multiprocessing.connection.wait(self.waitables.keys(), timeout=poll_time)
- if self._signal_conn in ready:
- if conf.getboolean("scheduler", "standalone_dag_processor"):
- # Nothing to do if callbacks are not stored in the database
- self._fetch_callbacks(maxCallbacksPerLoop)
- agent_signal = self._signal_conn.recv()
+ if self._direct_scheduler_conn is not None and self._direct_scheduler_conn in ready:
+ agent_signal = self._direct_scheduler_conn.recv()
self.log.debug("Received %s signal from DagFileProcessorAgent", agent_signal)
if agent_signal == DagParsingSignal.TERMINATE_MANAGER:
@@ -525,7 +581,7 @@ def _run_parsing_loop(self):
continue
for sentinel in ready:
- if sentinel is self._signal_conn:
+ if sentinel is self._direct_scheduler_conn:
continue
processor = self.waitables.get(sentinel)
@@ -536,6 +592,9 @@ def _run_parsing_loop(self):
self.waitables.pop(sentinel)
self._processors.pop(processor.file_path)
+ if standalone_dag_processor:
+ self._fetch_callbacks(max_callbacks_per_loop)
+ self._deactivate_stale_dags()
self._refresh_dag_dir()
self._kill_timed_out_processors()
@@ -568,12 +627,13 @@ def _run_parsing_loop(self):
max_runs_reached = self.max_runs_reached()
try:
- self._signal_conn.send(
- DagParsingStat(
- max_runs_reached,
- all_files_processed,
+ if self._direct_scheduler_conn:
+ self._direct_scheduler_conn.send(
+ DagParsingStat(
+ max_runs_reached,
+ all_files_processed,
+ )
)
- )
except BlockingIOError:
# Try again next time around the loop!
@@ -598,7 +658,7 @@ def _run_parsing_loop(self):
@provide_session
def _fetch_callbacks(self, max_callbacks: int, session: Session = NEW_SESSION):
- """Fetches callbacks from database and add them to the internal pipe for execution."""
+ """Fetches callbacks from database and add them to the internal queue for execution."""
self.log.debug("Fetching callbacks from the database.")
with prohibit_commit(session) as guard:
query = (
@@ -611,7 +671,7 @@ def _fetch_callbacks(self, max_callbacks: int, session: Session = NEW_SESSION):
).all()
for callback in callbacks:
try:
- self._signal_conn.send(callback.get_callback_request())
+ self._add_callback_to_queue(callback.get_callback_request())
session.delete(callback)
except Exception as e:
self.log.warning("Error adding callback for execution: %s, %s", callback, e)
@@ -782,7 +842,7 @@ def get_last_runtime(self, file_path):
:rtype: float
"""
stat = self._file_stats.get(file_path)
- return stat.last_duration if stat else None
+ return stat.last_duration.total_seconds() if stat and stat.last_duration else None
def get_last_dag_count(self, file_path):
"""
@@ -875,7 +935,7 @@ def _collect_results_from_processor(self, processor) -> None:
count_import_errors = -1
num_dags = 0
- last_duration = (last_finish_time - processor.start_time).total_seconds()
+ last_duration = last_finish_time - processor.start_time
stat = DagFileStat(
num_dags=num_dags,
import_errors=count_import_errors,
@@ -890,10 +950,12 @@ def _collect_results_from_processor(self, processor) -> None:
def collect_results(self) -> None:
"""Collect the result from any finished DAG processors"""
- ready = multiprocessing.connection.wait(self.waitables.keys() - [self._signal_conn], timeout=0)
+ ready = multiprocessing.connection.wait(
+ self.waitables.keys() - [self._direct_scheduler_conn], timeout=0
+ )
for sentinel in ready:
- if sentinel is self._signal_conn:
+ if sentinel is self._direct_scheduler_conn:
continue
processor = cast(DagFileProcessorProcess, self.waitables[sentinel])
self.waitables.pop(processor.waitable_handle)
@@ -1017,6 +1079,7 @@ def prepare_file_path_queue(self):
def _kill_timed_out_processors(self):
"""Kill any file processors that timeout to defend against process hangs."""
now = timezone.utcnow()
+ processors_to_remove = []
for file_path, processor in self._processors.items():
duration = now - processor.start_time
if duration > self._processor_timeout:
@@ -1032,6 +1095,14 @@ def _kill_timed_out_processors(self):
Stats.incr('dag_file_processor_timeouts')
processor.kill()
+ # Clean up processor references
+ self.waitables.pop(processor.waitable_handle)
+ processors_to_remove.append(file_path)
+
+ # Clean up `self._processors` after iterating over it
+ for proc in processors_to_remove:
+ self._processors.pop(proc)
+
def max_runs_reached(self):
""":return: whether all file paths have been processed max_runs times"""
if self._max_runs == -1: # Unlimited runs.
diff --git a/airflow/dag_processing/processor.py b/airflow/dag_processing/processor.py
index c60c6ab8eeb53..34821ffb92010 100644
--- a/airflow/dag_processing/processor.py
+++ b/airflow/dag_processing/processor.py
@@ -21,6 +21,7 @@
import os
import signal
import threading
+import time
from contextlib import redirect_stderr, redirect_stdout, suppress
from datetime import timedelta
from multiprocessing.connection import Connection as MultiprocessingConnection
@@ -231,6 +232,12 @@ def _kill_process(self) -> None:
if self._process.is_alive() and self._process.pid:
self.log.warning("Killing DAGFileProcessorProcess (PID=%d)", self._process.pid)
os.kill(self._process.pid, signal.SIGKILL)
+
+ # Reap the spawned zombie. We active wait, because in Python 3.9 `waitpid` might lead to an
+ # exception, due to change in Python standard library and possibility of race condition
+ # see https://bugs.python.org/issue42558
+ while self._process._popen.poll() is None: # type: ignore
+ time.sleep(0.001)
if self._parent_channel:
self._parent_channel.close()
@@ -521,23 +528,35 @@ def update_import_errors(session: Session, dagbag: DagBag) -> None:
:param session: session for ORM operations
:param dagbag: DagBag containing DAGs with import errors
"""
+ files_without_error = dagbag.file_last_changed - dagbag.import_errors.keys()
+
# Clear the errors of the processed files
- for dagbag_file in dagbag.file_last_changed:
+ # that no longer have errors
+ for dagbag_file in files_without_error:
session.query(errors.ImportError).filter(
errors.ImportError.filename.startswith(dagbag_file)
).delete(synchronize_session="fetch")
+ # files that still have errors
+ existing_import_error_files = [x.filename for x in session.query(errors.ImportError.filename).all()]
+
# Add the errors of the processed files
for filename, stacktrace in dagbag.import_errors.items():
+ if filename in existing_import_error_files:
+ session.query(errors.ImportError).filter(errors.ImportError.filename == filename).update(
+ dict(filename=filename, timestamp=timezone.utcnow(), stacktrace=stacktrace),
+ synchronize_session='fetch',
+ )
+ else:
+ session.add(
+ errors.ImportError(filename=filename, timestamp=timezone.utcnow(), stacktrace=stacktrace)
+ )
(
session.query(DagModel)
.filter(DagModel.fileloc == filename)
.update({'has_import_errors': True}, synchronize_session='fetch')
)
- session.add(
- errors.ImportError(filename=filename, timestamp=timezone.utcnow(), stacktrace=stacktrace)
- )
session.commit()
@provide_session
@@ -585,10 +604,10 @@ def _execute_task_callbacks(self, dagbag: DagBag, request: TaskCallbackRequest):
if simple_ti.task_id in dag.task_ids:
task = dag.get_task(simple_ti.task_id)
if request.is_failure_callback:
- ti = TI(task, run_id=simple_ti.run_id)
+ ti = TI(task, run_id=simple_ti.run_id, map_index=simple_ti.map_index)
# TODO: Use simple_ti to improve performance here in the future
ti.refresh_from_db()
- ti.handle_failure_with_callback(error=request.msg, test_mode=self.UNIT_TEST_MODE)
+ ti.handle_failure(error=request.msg, test_mode=self.UNIT_TEST_MODE)
self.log.info('Executed failure callback for %s in state %s', ti, ti.state)
@provide_session
@@ -628,8 +647,6 @@ def process_file(
Stats.incr('dag_file_refresh_error', 1, 1)
return 0, 0
- self._deactivate_missing_dags(session, dagbag, file_path)
-
if len(dagbag.dags) > 0:
self.log.info("DAG(s) %s retrieved from %s", dagbag.dags.keys(), file_path)
else:
@@ -659,12 +676,3 @@ def process_file(
self.log.exception("Error logging import errors!")
return len(dagbag.dags), len(dagbag.import_errors)
-
- def _deactivate_missing_dags(self, session: Session, dagbag: DagBag, file_path: str) -> None:
- deactivated = (
- session.query(DagModel)
- .filter(DagModel.fileloc == file_path, DagModel.is_active, ~DagModel.dag_id.in_(dagbag.dag_ids))
- .update({DagModel.is_active: False}, synchronize_session="fetch")
- )
- if deactivated:
- self.log.info("Deactivated %i DAGs which are no longer present in %s", deactivated, file_path)
diff --git a/airflow/decorators/__init__.pyi b/airflow/decorators/__init__.pyi
index de3f067236218..e970f61657c64 100644
--- a/airflow/decorators/__init__.pyi
+++ b/airflow/decorators/__init__.pyi
@@ -55,10 +55,8 @@ class TaskDecoratorCollection:
) -> TaskDecorator:
"""Create a decorator to convert the decorated callable to a task.
- :param multiple_outputs: if set, function return value will be
- unrolled to multiple XCom values. List/Tuples will unroll to xcom values
- with index as key. Dict will unroll to xcom values with keys as XCom keys.
- Defaults to False.
+ :param multiple_outputs: If set, function return value will be unrolled to multiple XCom values.
+ Dict will unroll to XCom values with keys as XCom keys. Defaults to False.
:param templates_dict: a dictionary where the values are templates that
will get templated by the Airflow engine sometime between
``__init__`` and ``execute`` takes place and are made available
@@ -102,10 +100,8 @@ class TaskDecoratorCollection:
) -> TaskDecorator:
"""Create a decorator to convert the decorated callable to a virtual environment task.
- :param multiple_outputs: if set, function return value will be
- unrolled to multiple XCom values. List/Tuples will unroll to xcom values
- with index as key. Dict will unroll to xcom values with keys as XCom keys.
- Defaults to False.
+ :param multiple_outputs: If set, function return value will be unrolled to multiple XCom values.
+ Dict will unroll to XCom values with keys as XCom keys. Defaults to False.
:param requirements: Either a list of requirement strings, or a (templated)
"requirements file" as specified by pip.
:param python_version: The Python version to run the virtualenv with. Note that
@@ -188,10 +184,8 @@ class TaskDecoratorCollection:
) -> TaskDecorator:
"""Create a decorator to convert the decorated callable to a Docker task.
- :param multiple_outputs: if set, function return value will be
- unrolled to multiple XCom values. List/Tuples will unroll to xcom values
- with index as key. Dict will unroll to xcom values with keys as XCom keys.
- Defaults to False.
+ :param multiple_outputs: If set, function return value will be unrolled to multiple XCom values.
+ Dict will unroll to XCom values with keys as XCom keys. Defaults to False.
:param use_dill: Whether to use dill or pickle for serialization
:param python_command: Python command for executing functions, Default: python3
:param image: Docker image from which to create the container.
diff --git a/airflow/decorators/base.py b/airflow/decorators/base.py
index f3642f0ef084b..1b5b5b760bf77 100644
--- a/airflow/decorators/base.py
+++ b/airflow/decorators/base.py
@@ -15,7 +15,6 @@
# specific language governing permissions and limitations
# under the License.
-import collections.abc
import functools
import inspect
import re
@@ -39,12 +38,17 @@
import attr
import typing_extensions
-from sqlalchemy.orm import Session
from airflow.compat.functools import cache, cached_property
from airflow.exceptions import AirflowException
from airflow.models.abstractoperator import DEFAULT_RETRIES, DEFAULT_RETRY_DELAY
-from airflow.models.baseoperator import BaseOperator, coerce_resources, coerce_retry_delay, parse_retries
+from airflow.models.baseoperator import (
+ BaseOperator,
+ coerce_resources,
+ coerce_timedelta,
+ get_merged_defaults,
+ parse_retries,
+)
from airflow.models.dag import DAG, DagContext
from airflow.models.mappedoperator import (
MappedOperator,
@@ -62,6 +66,9 @@
from airflow.utils.types import NOTSET
if TYPE_CHECKING:
+ import jinja2 # Slow import.
+ from sqlalchemy.orm import Session
+
from airflow.models.mappedoperator import Mappable
@@ -260,6 +267,10 @@ def __call__(self, *args, **kwargs) -> XComArg:
op.doc_md = self.function.__doc__
return XComArg(op)
+ @property
+ def __wrapped__(self) -> Function:
+ return self.function
+
@cached_property
def function_signature(self):
return inspect.signature(self.function)
@@ -301,16 +312,27 @@ def _validate_arg_names(self, func: ValidationSource, kwargs: Dict[str, Any]):
raise TypeError(f"{func}() got unexpected keyword arguments {names}")
def expand(self, **map_kwargs: "Mappable") -> XComArg:
+ if not map_kwargs:
+ raise TypeError("no arguments to expand against")
+
self._validate_arg_names("expand", map_kwargs)
prevent_duplicates(self.kwargs, map_kwargs, fail_reason="mapping already partial")
ensure_xcomarg_return_value(map_kwargs)
- partial_kwargs = self.kwargs.copy()
+ task_kwargs = self.kwargs.copy()
+ dag = task_kwargs.pop("dag", None) or DagContext.get_current_dag()
+ task_group = task_kwargs.pop("task_group", None) or TaskGroupContext.get_current_task_group(dag)
+
+ partial_kwargs, default_params = get_merged_defaults(
+ dag=dag,
+ task_group=task_group,
+ task_params=task_kwargs.pop("params", None),
+ task_default_args=task_kwargs.pop("default_args", None),
+ )
+ partial_kwargs.update(task_kwargs)
- dag = partial_kwargs.pop("dag", DagContext.get_current_dag())
- task_group = partial_kwargs.pop("task_group", TaskGroupContext.get_current_task_group(dag))
task_id = get_unique_task_id(partial_kwargs.pop("task_id"), dag, task_group)
- params = partial_kwargs.pop("params", None)
+ params = partial_kwargs.pop("params", None) or default_params
# Logic here should be kept in sync with BaseOperatorMeta.partial().
if "task_concurrency" in partial_kwargs:
@@ -322,8 +344,15 @@ def expand(self, **map_kwargs: "Mappable") -> XComArg:
if partial_kwargs.get("pool") is None:
partial_kwargs["pool"] = Pool.DEFAULT_POOL_NAME
partial_kwargs["retries"] = parse_retries(partial_kwargs.get("retries", DEFAULT_RETRIES))
- partial_kwargs["retry_delay"] = coerce_retry_delay(
+ partial_kwargs["retry_delay"] = coerce_timedelta(
partial_kwargs.get("retry_delay", DEFAULT_RETRY_DELAY),
+ key="retry_delay",
+ )
+ max_retry_delay = partial_kwargs.get("max_retry_delay")
+ partial_kwargs["max_retry_delay"] = (
+ max_retry_delay
+ if max_retry_delay is None
+ else coerce_timedelta(max_retry_delay, key="max_retry_delay")
)
partial_kwargs["resources"] = coerce_resources(partial_kwargs.get("resources"))
partial_kwargs.setdefault("executor_config", {})
@@ -342,9 +371,10 @@ def expand(self, **map_kwargs: "Mappable") -> XComArg:
operator_extra_links=self.operator_class.operator_extra_links,
template_ext=self.operator_class.template_ext,
template_fields=self.operator_class.template_fields,
+ template_fields_renderers=self.operator_class.template_fields_renderers,
ui_color=self.operator_class.ui_color,
ui_fgcolor=self.operator_class.ui_fgcolor,
- is_dummy=False,
+ is_empty=False,
task_module=self.operator_class.__module__,
task_type=self.operator_class.__name__,
dag=dag,
@@ -354,6 +384,10 @@ def expand(self, **map_kwargs: "Mappable") -> XComArg:
multiple_outputs=self.multiple_outputs,
python_callable=self.function,
mapped_op_kwargs=map_kwargs,
+ # Different from classic operators, kwargs passed to a taskflow
+ # task's expand() contribute to the op_kwargs operator argument, not
+ # the operator arguments themselves, and should expand against it.
+ expansion_kwargs_attr="mapped_op_kwargs",
)
return XComArg(operator=operator)
@@ -365,6 +399,9 @@ def partial(self, **kwargs) -> "_TaskDecorator[Function, OperatorSubclass]":
return attr.evolve(self, kwargs={**self.kwargs, "op_kwargs": op_kwargs})
+ def override(self, **kwargs) -> "_TaskDecorator[Function, OperatorSubclass]":
+ return attr.evolve(self, kwargs={**self.kwargs, **kwargs})
+
def _merge_kwargs(kwargs1: Dict[str, Any], kwargs2: Dict[str, Any], *, fail_reason: str) -> Dict[str, Any]:
duplicated_keys = set(kwargs1).intersection(kwargs2)
@@ -387,6 +424,9 @@ class DecoratedMappedOperator(MappedOperator):
# in partial_kwargs, and MappedOperator prevents duplication.
mapped_op_kwargs: Dict[str, "Mappable"]
+ def __hash__(self):
+ return id(self)
+
@classmethod
@cache
def get_serialized_fields(cls):
@@ -401,15 +441,6 @@ def __attrs_post_init__(self):
super(DecoratedMappedOperator, DecoratedMappedOperator).__attrs_post_init__(self)
XComArg.apply_upstream_relationship(self, self.mapped_op_kwargs)
- def _get_expansion_kwargs(self) -> Dict[str, "Mappable"]:
- """The kwargs to calculate expansion length against.
-
- Different from classic operators, a decorated (taskflow) operator's
- ``map()`` contributes to the ``op_kwargs`` operator argument (not the
- operator arguments themselves), and should therefore expand against it.
- """
- return self.mapped_op_kwargs
-
def _get_unmap_kwargs(self) -> Dict[str, Any]:
partial_kwargs = self.partial_kwargs.copy()
op_kwargs = _merge_kwargs(
@@ -417,6 +448,7 @@ def _get_unmap_kwargs(self) -> Dict[str, Any]:
self.mapped_op_kwargs,
fail_reason="mapping already partial",
)
+ self._combined_op_kwargs = op_kwargs
return {
"dag": self.dag,
"task_group": self.task_group,
@@ -428,13 +460,38 @@ def _get_unmap_kwargs(self) -> Dict[str, Any]:
**self.mapped_kwargs,
}
- def _expand_mapped_field(self, key: str, content: Any, context: Context, *, session: Session) -> Any:
- if key != "op_kwargs" or not isinstance(content, collections.abc.Mapping):
- return content
- # The magic super() doesn't work here, so we use the explicit form.
- # Not using super(..., self) to work around pyupgrade bug.
- sup: Any = super(DecoratedMappedOperator, DecoratedMappedOperator)
- return {k: sup._expand_mapped_field(self, k, v, context, session=session) for k, v in content.items()}
+ def _resolve_expansion_kwargs(
+ self, kwargs: Dict[str, Any], template_fields: Set[str], context: Context, session: "Session"
+ ) -> None:
+ expansion_kwargs = self._get_expansion_kwargs()
+
+ self._already_resolved_op_kwargs = set()
+ for k, v in expansion_kwargs.items():
+ if isinstance(v, XComArg):
+ self._already_resolved_op_kwargs.add(k)
+ v = v.resolve(context, session=session)
+ v = self._expand_mapped_field(k, v, context, session=session)
+ kwargs['op_kwargs'][k] = v
+ template_fields.discard(k)
+
+ def render_template(
+ self,
+ value: Any,
+ context: Context,
+ jinja_env: Optional["jinja2.Environment"] = None,
+ seen_oids: Optional[Set] = None,
+ ) -> Any:
+ if hasattr(self, '_combined_op_kwargs') and value is self._combined_op_kwargs:
+ # Avoid rendering values that came out of resolved XComArgs
+ return {
+ k: v
+ if k in self._already_resolved_op_kwargs
+ else super(DecoratedMappedOperator, DecoratedMappedOperator).render_template(
+ self, v, context, jinja_env=jinja_env, seen_oids=seen_oids
+ )
+ for k, v in value.items()
+ }
+ return super().render_template(value, context, jinja_env=jinja_env, seen_oids=seen_oids)
class Task(Generic[Function]):
@@ -452,6 +509,10 @@ class Task(Generic[Function]):
function: Function
+ @property
+ def __wrapped__(self) -> Function:
+ ...
+
def expand(self, **kwargs: "Mappable") -> XComArg:
...
@@ -484,7 +545,7 @@ def task_decorator_factory(
**kwargs,
) -> TaskDecorator:
"""
- A factory that generates a wrapper that raps a function into an Airflow operator.
+ A factory that generates a wrapper that wraps a function into an Airflow operator.
Accepts kwargs for operator kwarg. Can be reused in a single DAG.
:param python_callable: Function to decorate
diff --git a/airflow/decorators/task_group.py b/airflow/decorators/task_group.py
index 8ce8612d863d2..674474947036e 100644
--- a/airflow/decorators/task_group.py
+++ b/airflow/decorators/task_group.py
@@ -20,13 +20,12 @@
together when the DAG is displayed graphically.
"""
import functools
-import warnings
from inspect import signature
from typing import TYPE_CHECKING, Any, Callable, Dict, Generic, Optional, TypeVar, Union, cast, overload
import attr
-from airflow.utils.task_group import MappedTaskGroup, TaskGroup
+from airflow.utils.task_group import TaskGroup
if TYPE_CHECKING:
from airflow.models.dag import DAG
@@ -81,58 +80,6 @@ def __call__(self, *args, **kwargs) -> Union[R, TaskGroup]:
# start >> tg >> end
return task_group
- def partial(self, **kwargs) -> "MappedTaskGroupDecorator[R]":
- return MappedTaskGroupDecorator(function=self.function, kwargs=self.kwargs).partial(**kwargs)
-
- def expand(self, **kwargs) -> Union[R, TaskGroup]:
- return MappedTaskGroupDecorator(function=self.function, kwargs=self.kwargs).expand(**kwargs)
-
-
-@attr.define
-class MappedTaskGroupDecorator(TaskGroupDecorator[R]):
- """:meta private:"""
-
- partial_kwargs: Dict[str, Any] = attr.ib(factory=dict)
- """static kwargs for the decorated function"""
- mapped_kwargs: Dict[str, Any] = attr.ib(factory=dict)
- """kwargs for the decorated function"""
-
- def __call__(self, *args, **kwargs):
- raise RuntimeError("A mapped @task_group cannot be called. Use `.map` and `.partial` instead")
-
- def _make_task_group(self, **kwargs) -> MappedTaskGroup:
- tg = MappedTaskGroup(**kwargs)
- tg.partial_kwargs = self.partial_kwargs
- tg.mapped_kwargs = self.mapped_kwargs
- return tg
-
- def partial(self, **kwargs) -> "MappedTaskGroupDecorator[R]":
- duplicated_keys = [k for k in kwargs if k in self.partial_kwargs]
- if len(duplicated_keys) == 1:
- raise ValueError(f"Cannot overwrite partial argument: {duplicated_keys[0]!r}")
- elif duplicated_keys:
- joined = ", ".join(repr(k) for k in duplicated_keys)
- raise ValueError(f"Cannot overwrite partial arguments: {joined}")
- self.partial_kwargs.update(kwargs)
- return self
-
- def expand(self, **kwargs) -> Union[R, TaskGroup]:
- if self.mapped_kwargs:
- raise RuntimeError("Already a mapped task group")
- self.mapped_kwargs = kwargs
-
- call_kwargs = self.partial_kwargs.copy()
- duplicated_keys = set(call_kwargs).intersection(kwargs)
- if duplicated_keys:
- raise RuntimeError(f"Cannot map partial arguments: {', '.join(sorted(duplicated_keys))}")
- call_kwargs.update({k: object() for k in kwargs})
-
- return super().__call__(**call_kwargs)
-
- def __del__(self):
- if not self.mapped_kwargs:
- warnings.warn(f"Partial task group {self.function.__name__} was never mapped!")
-
class Group(Generic[F]):
"""Declaration of a @task_group-decorated callable for type-checking.
diff --git a/airflow/example_dags/example_bash_operator.py b/airflow/example_dags/example_bash_operator.py
index 8204592220350..335f3ad961d3b 100644
--- a/airflow/example_dags/example_bash_operator.py
+++ b/airflow/example_dags/example_bash_operator.py
@@ -24,7 +24,7 @@
from airflow import DAG
from airflow.operators.bash import BashOperator
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
with DAG(
dag_id='example_bash_operator',
@@ -35,7 +35,7 @@
tags=['example', 'example2'],
params={"example_key": "example_value"},
) as dag:
- run_this_last = DummyOperator(
+ run_this_last = EmptyOperator(
task_id='run_this_last',
)
diff --git a/airflow/example_dags/example_branch_datetime_operator.py b/airflow/example_dags/example_branch_datetime_operator.py
index 76b109fb688e7..3c86e40402aef 100644
--- a/airflow/example_dags/example_branch_datetime_operator.py
+++ b/airflow/example_dags/example_branch_datetime_operator.py
@@ -24,7 +24,7 @@
from airflow import DAG
from airflow.operators.datetime import BranchDateTimeOperator
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
dag = DAG(
dag_id="example_branch_datetime_operator",
@@ -35,8 +35,8 @@
)
# [START howto_branch_datetime_operator]
-dummy_task_1 = DummyOperator(task_id='date_in_range', dag=dag)
-dummy_task_2 = DummyOperator(task_id='date_outside_range', dag=dag)
+empty_task_1 = EmptyOperator(task_id='date_in_range', dag=dag)
+empty_task_2 = EmptyOperator(task_id='date_outside_range', dag=dag)
cond1 = BranchDateTimeOperator(
task_id='datetime_branch',
@@ -47,8 +47,8 @@
dag=dag,
)
-# Run dummy_task_1 if cond1 executes between 2020-10-10 14:00:00 and 2020-10-10 15:00:00
-cond1 >> [dummy_task_1, dummy_task_2]
+# Run empty_task_1 if cond1 executes between 2020-10-10 14:00:00 and 2020-10-10 15:00:00
+cond1 >> [empty_task_1, empty_task_2]
# [END howto_branch_datetime_operator]
@@ -60,8 +60,8 @@
schedule_interval="@daily",
)
# [START howto_branch_datetime_operator_next_day]
-dummy_task_1 = DummyOperator(task_id='date_in_range', dag=dag)
-dummy_task_2 = DummyOperator(task_id='date_outside_range', dag=dag)
+empty_task_1 = EmptyOperator(task_id='date_in_range', dag=dag)
+empty_task_2 = EmptyOperator(task_id='date_outside_range', dag=dag)
cond2 = BranchDateTimeOperator(
task_id='datetime_branch',
@@ -73,6 +73,6 @@
)
# Since target_lower happens after target_upper, target_upper will be moved to the following day
-# Run dummy_task_1 if cond2 executes between 15:00:00, and 00:00:00 of the following day
-cond2 >> [dummy_task_1, dummy_task_2]
+# Run empty_task_1 if cond2 executes between 15:00:00, and 00:00:00 of the following day
+cond2 >> [empty_task_1, empty_task_2]
# [END howto_branch_datetime_operator_next_day]
diff --git a/airflow/example_dags/example_branch_day_of_week_operator.py b/airflow/example_dags/example_branch_day_of_week_operator.py
index dae303a9035fb..879824ab1c876 100644
--- a/airflow/example_dags/example_branch_day_of_week_operator.py
+++ b/airflow/example_dags/example_branch_day_of_week_operator.py
@@ -22,7 +22,7 @@
import pendulum
from airflow import DAG
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.operators.weekday import BranchDayOfWeekOperator
with DAG(
@@ -33,8 +33,8 @@
schedule_interval="@daily",
) as dag:
# [START howto_operator_day_of_week_branch]
- dummy_task_1 = DummyOperator(task_id='branch_true', dag=dag)
- dummy_task_2 = DummyOperator(task_id='branch_false', dag=dag)
+ empty_task_1 = EmptyOperator(task_id='branch_true')
+ empty_task_2 = EmptyOperator(task_id='branch_false')
branch = BranchDayOfWeekOperator(
task_id="make_choice",
@@ -43,6 +43,6 @@
week_day="Monday",
)
- # Run dummy_task_1 if branch executes on Monday
- branch >> [dummy_task_1, dummy_task_2]
+ # Run empty_task_1 if branch executes on Monday
+ branch >> [empty_task_1, empty_task_2]
# [END howto_operator_day_of_week_branch]
diff --git a/airflow/example_dags/example_branch_labels.py b/airflow/example_dags/example_branch_labels.py
index 2215bcfe19c41..72337eb45c873 100644
--- a/airflow/example_dags/example_branch_labels.py
+++ b/airflow/example_dags/example_branch_labels.py
@@ -22,7 +22,7 @@
import pendulum
from airflow import DAG
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.utils.edgemodifier import Label
with DAG(
@@ -31,13 +31,13 @@
start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
catchup=False,
) as dag:
- ingest = DummyOperator(task_id="ingest")
- analyse = DummyOperator(task_id="analyze")
- check = DummyOperator(task_id="check_integrity")
- describe = DummyOperator(task_id="describe_integrity")
- error = DummyOperator(task_id="email_error")
- save = DummyOperator(task_id="save")
- report = DummyOperator(task_id="report")
+ ingest = EmptyOperator(task_id="ingest")
+ analyse = EmptyOperator(task_id="analyze")
+ check = EmptyOperator(task_id="check_integrity")
+ describe = EmptyOperator(task_id="describe_integrity")
+ error = EmptyOperator(task_id="email_error")
+ save = EmptyOperator(task_id="save")
+ report = EmptyOperator(task_id="report")
ingest >> analyse >> check
check >> Label("No errors") >> save >> report
diff --git a/airflow/example_dags/example_branch_operator.py b/airflow/example_dags/example_branch_operator.py
index eaa1532eeef81..8721c78bcbc8e 100644
--- a/airflow/example_dags/example_branch_operator.py
+++ b/airflow/example_dags/example_branch_operator.py
@@ -23,7 +23,7 @@
import pendulum
from airflow import DAG
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.operators.python import BranchPythonOperator
from airflow.utils.edgemodifier import Label
from airflow.utils.trigger_rule import TriggerRule
@@ -35,7 +35,7 @@
schedule_interval="@daily",
tags=['example', 'example2'],
) as dag:
- run_this_first = DummyOperator(
+ run_this_first = EmptyOperator(
task_id='run_this_first',
)
@@ -47,19 +47,19 @@
)
run_this_first >> branching
- join = DummyOperator(
+ join = EmptyOperator(
task_id='join',
trigger_rule=TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS,
)
for option in options:
- t = DummyOperator(
+ t = EmptyOperator(
task_id=option,
)
- dummy_follow = DummyOperator(
+ empty_follow = EmptyOperator(
task_id='follow_' + option,
)
# Label is optional here, but it can help identify more complex branches
- branching >> Label(option) >> t >> dummy_follow >> join
+ branching >> Label(option) >> t >> empty_follow >> join
diff --git a/airflow/example_dags/example_branch_operator_decorator.py b/airflow/example_dags/example_branch_operator_decorator.py
index d0978322c1a3c..0ab4f76cafa2a 100644
--- a/airflow/example_dags/example_branch_operator_decorator.py
+++ b/airflow/example_dags/example_branch_operator_decorator.py
@@ -23,7 +23,7 @@
from airflow import DAG
from airflow.decorators import task
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.utils.edgemodifier import Label
from airflow.utils.trigger_rule import TriggerRule
@@ -34,7 +34,7 @@
schedule_interval="@daily",
tags=['example', 'example2'],
) as dag:
- run_this_first = DummyOperator(
+ run_this_first = EmptyOperator(
task_id='run_this_first',
)
@@ -48,19 +48,19 @@ def random_choice():
run_this_first >> random_choice_instance
- join = DummyOperator(
+ join = EmptyOperator(
task_id='join',
trigger_rule=TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS,
)
for option in options:
- t = DummyOperator(
+ t = EmptyOperator(
task_id=option,
)
- dummy_follow = DummyOperator(
+ empty_follow = EmptyOperator(
task_id='follow_' + option,
)
# Label is optional here, but it can help identify more complex branches
- random_choice_instance >> Label(option) >> t >> dummy_follow >> join
+ random_choice_instance >> Label(option) >> t >> empty_follow >> join
diff --git a/airflow/example_dags/example_branch_python_dop_operator_3.py b/airflow/example_dags/example_branch_python_dop_operator_3.py
index d85eda140aedc..a8e0ce2c1c8af 100644
--- a/airflow/example_dags/example_branch_python_dop_operator_3.py
+++ b/airflow/example_dags/example_branch_python_dop_operator_3.py
@@ -23,27 +23,25 @@
import pendulum
from airflow import DAG
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.operators.python import BranchPythonOperator
def should_run(**kwargs):
"""
- Determine which dummy_task should be run based on if the execution date minute is even or odd.
+ Determine which empty_task should be run based on if the execution date minute is even or odd.
:param dict kwargs: Context
:return: Id of the task to run
:rtype: str
"""
print(
- '------------- exec dttm = {} and minute = {}'.format(
- kwargs['execution_date'], kwargs['execution_date'].minute
- )
+ f"------------- exec dttm = {kwargs['execution_date']} and minute = {kwargs['execution_date'].minute}"
)
if kwargs['execution_date'].minute % 2 == 0:
- return "dummy_task_1"
+ return "empty_task_1"
else:
- return "dummy_task_2"
+ return "empty_task_2"
with DAG(
@@ -59,6 +57,6 @@ def should_run(**kwargs):
python_callable=should_run,
)
- dummy_task_1 = DummyOperator(task_id='dummy_task_1')
- dummy_task_2 = DummyOperator(task_id='dummy_task_2')
- cond >> [dummy_task_1, dummy_task_2]
+ empty_task_1 = EmptyOperator(task_id='empty_task_1')
+ empty_task_2 = EmptyOperator(task_id='empty_task_2')
+ cond >> [empty_task_1, empty_task_2]
diff --git a/airflow/example_dags/example_external_task_marker_dag.py b/airflow/example_dags/example_external_task_marker_dag.py
index a39ebf5c40439..0c4479a0d66f0 100644
--- a/airflow/example_dags/example_external_task_marker_dag.py
+++ b/airflow/example_dags/example_external_task_marker_dag.py
@@ -40,7 +40,7 @@
import pendulum
from airflow import DAG
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.sensors.external_task import ExternalTaskMarker, ExternalTaskSensor
start_date = pendulum.datetime(2021, 1, 1, tz="UTC")
@@ -78,5 +78,5 @@
mode="reschedule",
)
# [END howto_operator_external_task_sensor]
- child_task2 = DummyOperator(task_id="child_task2")
+ child_task2 = EmptyOperator(task_id="child_task2")
child_task1 >> child_task2
diff --git a/airflow/example_dags/example_kubernetes_executor.py b/airflow/example_dags/example_kubernetes_executor.py
index d5d42f87d92fb..f278eb379b1e8 100644
--- a/airflow/example_dags/example_kubernetes_executor.py
+++ b/airflow/example_dags/example_kubernetes_executor.py
@@ -42,6 +42,7 @@
)
k8s = None
+
if k8s:
with DAG(
dag_id='example_kubernetes_executor',
diff --git a/airflow/example_dags/example_latest_only.py b/airflow/example_dags/example_latest_only.py
index d320ad7d06e73..92ec1436a6951 100644
--- a/airflow/example_dags/example_latest_only.py
+++ b/airflow/example_dags/example_latest_only.py
@@ -21,7 +21,7 @@
import datetime as dt
from airflow import DAG
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.operators.latest_only import LatestOnlyOperator
with DAG(
@@ -32,6 +32,6 @@
tags=['example2', 'example3'],
) as dag:
latest_only = LatestOnlyOperator(task_id='latest_only')
- task1 = DummyOperator(task_id='task1')
+ task1 = EmptyOperator(task_id='task1')
latest_only >> task1
diff --git a/airflow/example_dags/example_latest_only_with_trigger.py b/airflow/example_dags/example_latest_only_with_trigger.py
index 67f004aef38f7..56d6a24462217 100644
--- a/airflow/example_dags/example_latest_only_with_trigger.py
+++ b/airflow/example_dags/example_latest_only_with_trigger.py
@@ -25,7 +25,7 @@
import pendulum
from airflow import DAG
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.operators.latest_only import LatestOnlyOperator
from airflow.utils.trigger_rule import TriggerRule
@@ -37,10 +37,10 @@
tags=['example3'],
) as dag:
latest_only = LatestOnlyOperator(task_id='latest_only')
- task1 = DummyOperator(task_id='task1')
- task2 = DummyOperator(task_id='task2')
- task3 = DummyOperator(task_id='task3')
- task4 = DummyOperator(task_id='task4', trigger_rule=TriggerRule.ALL_DONE)
+ task1 = EmptyOperator(task_id='task1')
+ task2 = EmptyOperator(task_id='task2')
+ task3 = EmptyOperator(task_id='task3')
+ task4 = EmptyOperator(task_id='task4', trigger_rule=TriggerRule.ALL_DONE)
latest_only >> task1 >> [task3, task4]
task2 >> [task3, task4]
diff --git a/airflow/example_dags/example_local_kubernetes_executor.py b/airflow/example_dags/example_local_kubernetes_executor.py
index 03fbd06c53b4f..e586cafda9416 100644
--- a/airflow/example_dags/example_local_kubernetes_executor.py
+++ b/airflow/example_dags/example_local_kubernetes_executor.py
@@ -36,32 +36,34 @@
except ImportError:
log.warning("Could not import DAGs in example_local_kubernetes_executor.py", exc_info=True)
log.warning("Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]")
+ k8s = None
-with DAG(
- dag_id='example_local_kubernetes_executor',
- schedule_interval=None,
- start_date=datetime(2021, 1, 1),
- catchup=False,
- tags=['example3'],
-) as dag:
- # You can use annotations on your kubernetes pods!
- start_task_executor_config = {
- "pod_override": k8s.V1Pod(metadata=k8s.V1ObjectMeta(annotations={"test": "annotation"}))
- }
+if k8s:
+ with DAG(
+ dag_id='example_local_kubernetes_executor',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=['example3'],
+ ) as dag:
+ # You can use annotations on your kubernetes pods!
+ start_task_executor_config = {
+ "pod_override": k8s.V1Pod(metadata=k8s.V1ObjectMeta(annotations={"test": "annotation"}))
+ }
- @task(
- executor_config=start_task_executor_config,
- queue='kubernetes',
- task_id='task_with_kubernetes_executor',
- )
- def task_with_template():
- print_stuff()
+ @task(
+ executor_config=start_task_executor_config,
+ queue='kubernetes',
+ task_id='task_with_kubernetes_executor',
+ )
+ def task_with_template():
+ print_stuff()
- @task(task_id='task_with_local_executor')
- def task_with_local(ds=None, **kwargs):
- """Print the Airflow context and ds variable from the context."""
- print(kwargs)
- print(ds)
- return 'Whatever you return gets printed in the logs'
+ @task(task_id='task_with_local_executor')
+ def task_with_local(ds=None, **kwargs):
+ """Print the Airflow context and ds variable from the context."""
+ print(kwargs)
+ print(ds)
+ return 'Whatever you return gets printed in the logs'
- task_with_local() >> task_with_template()
+ task_with_local() >> task_with_template()
diff --git a/airflow/example_dags/example_nested_branch_dag.py b/airflow/example_dags/example_nested_branch_dag.py
index 27e71054d1176..14ac0a43ce245 100644
--- a/airflow/example_dags/example_nested_branch_dag.py
+++ b/airflow/example_dags/example_nested_branch_dag.py
@@ -24,7 +24,7 @@
import pendulum
from airflow.models import DAG
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.operators.python import BranchPythonOperator
from airflow.utils.trigger_rule import TriggerRule
@@ -36,14 +36,14 @@
tags=["example"],
) as dag:
branch_1 = BranchPythonOperator(task_id="branch_1", python_callable=lambda: "true_1")
- join_1 = DummyOperator(task_id="join_1", trigger_rule=TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS)
- true_1 = DummyOperator(task_id="true_1")
- false_1 = DummyOperator(task_id="false_1")
+ join_1 = EmptyOperator(task_id="join_1", trigger_rule=TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS)
+ true_1 = EmptyOperator(task_id="true_1")
+ false_1 = EmptyOperator(task_id="false_1")
branch_2 = BranchPythonOperator(task_id="branch_2", python_callable=lambda: "true_2")
- join_2 = DummyOperator(task_id="join_2", trigger_rule=TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS)
- true_2 = DummyOperator(task_id="true_2")
- false_2 = DummyOperator(task_id="false_2")
- false_3 = DummyOperator(task_id="false_3")
+ join_2 = EmptyOperator(task_id="join_2", trigger_rule=TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS)
+ true_2 = EmptyOperator(task_id="true_2")
+ false_2 = EmptyOperator(task_id="false_2")
+ false_3 = EmptyOperator(task_id="false_3")
branch_1 >> true_1 >> join_1
branch_1 >> false_1 >> branch_2 >> [true_2, false_2] >> join_2 >> false_3 >> join_1
diff --git a/airflow/example_dags/example_short_circuit_operator.py b/airflow/example_dags/example_short_circuit_operator.py
index 619785731ac1e..2278de30e6294 100644
--- a/airflow/example_dags/example_short_circuit_operator.py
+++ b/airflow/example_dags/example_short_circuit_operator.py
@@ -21,7 +21,7 @@
from airflow import DAG
from airflow.models.baseoperator import chain
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.operators.python import ShortCircuitOperator
from airflow.utils.trigger_rule import TriggerRule
@@ -42,8 +42,8 @@
python_callable=lambda: False,
)
- ds_true = [DummyOperator(task_id='true_' + str(i)) for i in [1, 2]]
- ds_false = [DummyOperator(task_id='false_' + str(i)) for i in [1, 2]]
+ ds_true = [EmptyOperator(task_id='true_' + str(i)) for i in [1, 2]]
+ ds_false = [EmptyOperator(task_id='false_' + str(i)) for i in [1, 2]]
chain(cond_true, *ds_true)
chain(cond_false, *ds_false)
@@ -51,10 +51,10 @@
# [START howto_operator_short_circuit_trigger_rules]
[task_1, task_2, task_3, task_4, task_5, task_6] = [
- DummyOperator(task_id=f"task_{i}") for i in range(1, 7)
+ EmptyOperator(task_id=f"task_{i}") for i in range(1, 7)
]
- task_7 = DummyOperator(task_id="task_7", trigger_rule=TriggerRule.ALL_DONE)
+ task_7 = EmptyOperator(task_id="task_7", trigger_rule=TriggerRule.ALL_DONE)
short_circuit = ShortCircuitOperator(
task_id="short_circuit", ignore_downstream_trigger_rules=False, python_callable=lambda: False
diff --git a/airflow/example_dags/example_skip_dag.py b/airflow/example_dags/example_skip_dag.py
index fd220eb7a61eb..00d3a3d91e11b 100644
--- a/airflow/example_dags/example_skip_dag.py
+++ b/airflow/example_dags/example_skip_dag.py
@@ -16,20 +16,20 @@
# specific language governing permissions and limitations
# under the License.
-"""Example DAG demonstrating the DummyOperator and a custom DummySkipOperator which skips by default."""
+"""Example DAG demonstrating the EmptyOperator and a custom EmptySkipOperator which skips by default."""
import pendulum
from airflow import DAG
from airflow.exceptions import AirflowSkipException
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.utils.context import Context
from airflow.utils.trigger_rule import TriggerRule
# Create some placeholder operators
-class DummySkipOperator(DummyOperator):
- """Dummy operator which always skips the task."""
+class EmptySkipOperator(EmptyOperator):
+ """Empty operator which always skips the task."""
ui_color = '#e8b7e4'
@@ -45,10 +45,10 @@ def create_test_pipeline(suffix, trigger_rule):
:param str trigger_rule: TriggerRule for the join task
:param DAG dag_: The DAG to run the operators on
"""
- skip_operator = DummySkipOperator(task_id=f'skip_operator_{suffix}')
- always_true = DummyOperator(task_id=f'always_true_{suffix}')
- join = DummyOperator(task_id=trigger_rule, trigger_rule=trigger_rule)
- final = DummyOperator(task_id=f'final_{suffix}')
+ skip_operator = EmptySkipOperator(task_id=f'skip_operator_{suffix}')
+ always_true = EmptyOperator(task_id=f'always_true_{suffix}')
+ join = EmptyOperator(task_id=trigger_rule, trigger_rule=trigger_rule)
+ final = EmptyOperator(task_id=f'final_{suffix}')
skip_operator >> join
always_true >> join
diff --git a/airflow/example_dags/example_subdag_operator.py b/airflow/example_dags/example_subdag_operator.py
index 424dc7f873d7e..79d369d638d6a 100644
--- a/airflow/example_dags/example_subdag_operator.py
+++ b/airflow/example_dags/example_subdag_operator.py
@@ -19,23 +19,24 @@
"""Example DAG demonstrating the usage of the SubDagOperator."""
# [START example_subdag_operator]
+import datetime
+
from airflow import DAG
from airflow.example_dags.subdags.subdag import subdag
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.operators.subdag import SubDagOperator
-from airflow.utils.dates import days_ago
DAG_NAME = 'example_subdag_operator'
with DAG(
dag_id=DAG_NAME,
default_args={"retries": 2},
- start_date=days_ago(2),
+ start_date=datetime.datetime(2022, 1, 1),
schedule_interval="@once",
tags=['example'],
) as dag:
- start = DummyOperator(
+ start = EmptyOperator(
task_id='start',
)
@@ -44,7 +45,7 @@
subdag=subdag(DAG_NAME, 'section-1', dag.default_args),
)
- some_other_task = DummyOperator(
+ some_other_task = EmptyOperator(
task_id='some-other-task',
)
@@ -53,7 +54,7 @@
subdag=subdag(DAG_NAME, 'section-2', dag.default_args),
)
- end = DummyOperator(
+ end = EmptyOperator(
task_id='end',
)
diff --git a/airflow/example_dags/example_task_group.py b/airflow/example_dags/example_task_group.py
index 46f709eaf873b..7bc319f98b569 100644
--- a/airflow/example_dags/example_task_group.py
+++ b/airflow/example_dags/example_task_group.py
@@ -21,7 +21,7 @@
from airflow.models.dag import DAG
from airflow.operators.bash import BashOperator
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.utils.task_group import TaskGroup
# [START howto_task_group]
@@ -31,33 +31,33 @@
catchup=False,
tags=["example"],
) as dag:
- start = DummyOperator(task_id="start")
+ start = EmptyOperator(task_id="start")
# [START howto_task_group_section_1]
with TaskGroup("section_1", tooltip="Tasks for section_1") as section_1:
- task_1 = DummyOperator(task_id="task_1")
+ task_1 = EmptyOperator(task_id="task_1")
task_2 = BashOperator(task_id="task_2", bash_command='echo 1')
- task_3 = DummyOperator(task_id="task_3")
+ task_3 = EmptyOperator(task_id="task_3")
task_1 >> [task_2, task_3]
# [END howto_task_group_section_1]
# [START howto_task_group_section_2]
with TaskGroup("section_2", tooltip="Tasks for section_2") as section_2:
- task_1 = DummyOperator(task_id="task_1")
+ task_1 = EmptyOperator(task_id="task_1")
# [START howto_task_group_inner_section_2]
with TaskGroup("inner_section_2", tooltip="Tasks for inner_section2") as inner_section_2:
task_2 = BashOperator(task_id="task_2", bash_command='echo 1')
- task_3 = DummyOperator(task_id="task_3")
- task_4 = DummyOperator(task_id="task_4")
+ task_3 = EmptyOperator(task_id="task_3")
+ task_4 = EmptyOperator(task_id="task_4")
[task_2, task_3] >> task_4
# [END howto_task_group_inner_section_2]
# [END howto_task_group_section_2]
- end = DummyOperator(task_id='end')
+ end = EmptyOperator(task_id='end')
start >> section_1 >> section_2 >> end
# [END howto_task_group]
diff --git a/airflow/example_dags/example_task_group_decorator.py b/airflow/example_dags/example_task_group_decorator.py
index 2e207fbd81fb2..637c721886bea 100644
--- a/airflow/example_dags/example_task_group_decorator.py
+++ b/airflow/example_dags/example_task_group_decorator.py
@@ -28,31 +28,31 @@
# Creating Tasks
@task
def task_start():
- """Dummy Task which is First Task of Dag"""
+ """Empty Task which is First Task of Dag"""
return '[Task_start]'
@task
def task_1(value: int) -> str:
- """Dummy Task1"""
+ """Empty Task1"""
return f'[ Task1 {value} ]'
@task
def task_2(value: str) -> str:
- """Dummy Task2"""
+ """Empty Task2"""
return f'[ Task2 {value} ]'
@task
def task_3(value: str) -> None:
- """Dummy Task3"""
+ """Empty Task3"""
print(f'[ Task3 {value} ]')
@task
def task_end() -> None:
- """Dummy Task which is Last Task of Dag"""
+ """Empty Task which is Last Task of Dag"""
print('[ Task_End ]')
diff --git a/airflow/example_dags/example_time_delta_sensor_async.py b/airflow/example_dags/example_time_delta_sensor_async.py
index 1a7126a22627a..a2aa3cb66edee 100644
--- a/airflow/example_dags/example_time_delta_sensor_async.py
+++ b/airflow/example_dags/example_time_delta_sensor_async.py
@@ -26,7 +26,7 @@
import pendulum
from airflow import DAG
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
from airflow.sensors.time_delta import TimeDeltaSensorAsync
with DAG(
@@ -37,5 +37,5 @@
tags=["example"],
) as dag:
wait = TimeDeltaSensorAsync(task_id="wait", delta=datetime.timedelta(seconds=10))
- finish = DummyOperator(task_id="finish")
+ finish = EmptyOperator(task_id="finish")
wait >> finish
diff --git a/airflow/example_dags/example_xcomargs.py b/airflow/example_dags/example_xcomargs.py
index 00af4725c240f..8312aca8c9d25 100644
--- a/airflow/example_dags/example_xcomargs.py
+++ b/airflow/example_dags/example_xcomargs.py
@@ -30,13 +30,13 @@
@task
def generate_value():
- """Dummy function"""
+ """Empty function"""
return "Bring me a shrubbery!"
@task
def print_value(value, ts=None):
- """Dummy function"""
+ """Empty function"""
log.info("The knights of Ni say: %s (at %s)", value, ts)
diff --git a/airflow/example_dags/subdags/subdag.py b/airflow/example_dags/subdags/subdag.py
index 7c913099b3107..2fcab731092fb 100644
--- a/airflow/example_dags/subdags/subdag.py
+++ b/airflow/example_dags/subdags/subdag.py
@@ -22,7 +22,7 @@
import pendulum
from airflow import DAG
-from airflow.operators.dummy import DummyOperator
+from airflow.operators.empty import EmptyOperator
def subdag(parent_dag_name, child_dag_name, args):
@@ -44,7 +44,7 @@ def subdag(parent_dag_name, child_dag_name, args):
)
for i in range(5):
- DummyOperator(
+ EmptyOperator(
task_id=f'{child_dag_name}-task-{i + 1}',
default_args=args,
dag=dag_subdag,
diff --git a/airflow/exceptions.py b/airflow/exceptions.py
index 630268b01f2d1..2f1e53e182a10 100644
--- a/airflow/exceptions.py
+++ b/airflow/exceptions.py
@@ -21,6 +21,7 @@
"""Exceptions used by Airflow"""
import datetime
import warnings
+from http import HTTPStatus
from typing import Any, Dict, List, NamedTuple, Optional, Sized
@@ -31,19 +32,19 @@ class AirflowException(Exception):
Each custom exception should be derived from this class.
"""
- status_code = 500
+ status_code = HTTPStatus.INTERNAL_SERVER_ERROR
class AirflowBadRequest(AirflowException):
"""Raise when the application or server cannot handle the request."""
- status_code = 400
+ status_code = HTTPStatus.BAD_REQUEST
class AirflowNotFoundException(AirflowException):
"""Raise when the requested object/resource is not available in the system."""
- status_code = 404
+ status_code = HTTPStatus.NOT_FOUND
class AirflowConfigException(AirflowException):
@@ -102,6 +103,13 @@ class UnmappableOperator(AirflowException):
"""Raise when an operator is not implemented to be mappable."""
+class XComForMappingNotPushed(AirflowException):
+ """Raise when a mapped downstream's dependency fails to push XCom for task mapping."""
+
+ def __str__(self) -> str:
+ return "did not push XCom for task mapping"
+
+
class UnmappableXComTypePushed(AirflowException):
"""Raise when an unmappable type is pushed as a mapped downstream's dependency."""
diff --git a/airflow/executors/base_executor.py b/airflow/executors/base_executor.py
index 73f7f57edd1f6..7fcbd0642e14d 100644
--- a/airflow/executors/base_executor.py
+++ b/airflow/executors/base_executor.py
@@ -49,6 +49,9 @@
# Tuple of: state, info
EventBufferValueType = Tuple[Optional[str], Any]
+# Task tuple to send to be executed
+TaskTuple = Tuple[TaskInstanceKey, CommandType, Optional[str], Optional[Any]]
+
class BaseExecutor(LoggingMixin):
"""
@@ -186,6 +189,7 @@ def trigger_tasks(self, open_slots: int) -> None:
:param open_slots: Number of open slots
"""
sorted_queue = self.order_queued_tasks_by_priority()
+ task_tuples = []
for _ in range(min((open_slots, len(self.queued_tasks)))):
key, (command, _, queue, ti) = sorted_queue.pop(0)
@@ -212,9 +216,16 @@ def trigger_tasks(self, open_slots: int) -> None:
del self.attempts[key]
del self.queued_tasks[key]
else:
- del self.queued_tasks[key]
- self.running.add(key)
- self.execute_async(key=key, command=command, queue=queue, executor_config=ti.executor_config)
+ task_tuples.append((key, command, queue, ti.executor_config))
+
+ if task_tuples:
+ self._process_tasks(task_tuples)
+
+ def _process_tasks(self, task_tuples: List[TaskTuple]) -> None:
+ for key, command, queue, executor_config in task_tuples:
+ del self.queued_tasks[key]
+ self.execute_async(key=key, command=command, queue=queue, executor_config=executor_config)
+ self.running.add(key)
def change_state(self, key: TaskInstanceKey, state: str, info=None) -> None:
"""
diff --git a/airflow/executors/celery_executor.py b/airflow/executors/celery_executor.py
index 04983c4a9c78d..7b4c04e225a75 100644
--- a/airflow/executors/celery_executor.py
+++ b/airflow/executors/celery_executor.py
@@ -29,8 +29,9 @@
import subprocess
import time
import traceback
-from collections import OrderedDict
+from collections import Counter
from concurrent.futures import ProcessPoolExecutor
+from enum import Enum
from multiprocessing import cpu_count
from typing import Any, Dict, List, Mapping, MutableMapping, Optional, Set, Tuple, Union
@@ -46,7 +47,7 @@
from airflow.config_templates.default_celery import DEFAULT_CELERY_CONFIG
from airflow.configuration import conf
from airflow.exceptions import AirflowException, AirflowTaskTimeout
-from airflow.executors.base_executor import BaseExecutor, CommandType, EventBufferValueType
+from airflow.executors.base_executor import BaseExecutor, CommandType, EventBufferValueType, TaskTuple
from airflow.models.taskinstance import TaskInstance, TaskInstanceKey
from airflow.stats import Stats
from airflow.utils.log.logging_mixin import LoggingMixin
@@ -209,6 +210,11 @@ def on_celery_import_modules(*args, **kwargs):
pass
+class _CeleryPendingTaskTimeoutType(Enum):
+ ADOPTED = 1
+ STALLED = 2
+
+
class CeleryExecutor(BaseExecutor):
"""
CeleryExecutor is recommended for production use of Airflow. It allows
@@ -232,16 +238,16 @@ def __init__(self):
self._sync_parallelism = max(1, cpu_count() - 1)
self.bulk_state_fetcher = BulkStateFetcher(self._sync_parallelism)
self.tasks = {}
- # Mapping of tasks we've adopted, ordered by the earliest date they timeout
- self.adopted_task_timeouts: Dict[TaskInstanceKey, datetime.datetime] = OrderedDict()
- self.task_adoption_timeout = datetime.timedelta(
- seconds=conf.getint('celery', 'task_adoption_timeout', fallback=600)
+ self.stalled_task_timeouts: Dict[TaskInstanceKey, datetime.datetime] = {}
+ self.stalled_task_timeout = datetime.timedelta(
+ seconds=conf.getint('celery', 'stalled_task_timeout', fallback=0)
)
- self.stuck_tasks_last_check_time: int = time.time()
- self.stuck_queued_task_check_interval = conf.getint(
- 'celery', 'stuck_queued_task_check_interval', fallback=300
+ self.adopted_task_timeouts: Dict[TaskInstanceKey, datetime.datetime] = {}
+ self.task_adoption_timeout = (
+ datetime.timedelta(seconds=conf.getint('celery', 'task_adoption_timeout', fallback=600))
+ or self.stalled_task_timeout
)
- self.task_publish_retries: Dict[TaskInstanceKey, int] = OrderedDict()
+ self.task_publish_retries: Counter[TaskInstanceKey] = Counter()
self.task_publish_max_retries = conf.getint('celery', 'task_publish_max_retries', fallback=3)
def start(self) -> None:
@@ -256,28 +262,8 @@ def _num_tasks_per_send_process(self, to_send_count: int) -> int:
"""
return max(1, int(math.ceil(1.0 * to_send_count / self._sync_parallelism)))
- def trigger_tasks(self, open_slots: int) -> None:
- """
- Overwrite trigger_tasks function from BaseExecutor
-
- :param open_slots: Number of open slots
- :return:
- """
- sorted_queue = self.order_queued_tasks_by_priority()
-
- task_tuples_to_send: List[TaskInstanceInCelery] = []
-
- for _ in range(min(open_slots, len(self.queued_tasks))):
- key, (command, _, queue, _) = sorted_queue.pop(0)
- task_tuple = (key, command, queue, execute_command)
- task_tuples_to_send.append(task_tuple)
- if key not in self.task_publish_retries:
- self.task_publish_retries[key] = 1
-
- if task_tuples_to_send:
- self._process_tasks(task_tuples_to_send)
-
- def _process_tasks(self, task_tuples_to_send: List[TaskInstanceInCelery]) -> None:
+ def _process_tasks(self, task_tuples: List[TaskTuple]) -> None:
+ task_tuples_to_send = [task_tuple[:3] + (execute_command,) for task_tuple in task_tuples]
first_task = next(t[3] for t in task_tuples_to_send)
# Celery state queries will stuck if we do not use one same backend
@@ -291,20 +277,19 @@ def _process_tasks(self, task_tuples_to_send: List[TaskInstanceInCelery]) -> Non
if isinstance(result, ExceptionWithTraceback) and isinstance(
result.exception, AirflowTaskTimeout
):
- if key in self.task_publish_retries and (
- self.task_publish_retries.get(key) <= self.task_publish_max_retries
- ):
+ retries = self.task_publish_retries[key]
+ if retries < self.task_publish_max_retries:
Stats.incr("celery.task_timeout_error")
self.log.info(
"[Try %s of %s] Task Timeout Error for Task: (%s).",
- self.task_publish_retries[key],
+ self.task_publish_retries[key] + 1,
self.task_publish_max_retries,
key,
)
- self.task_publish_retries[key] += 1
+ self.task_publish_retries[key] = retries + 1
continue
self.queued_tasks.pop(key)
- self.task_publish_retries.pop(key)
+ self.task_publish_retries.pop(key, None)
if isinstance(result, ExceptionWithTraceback):
self.log.error(CELERY_SEND_ERR_MSG_HEADER + ": %s\n%s\n", result.exception, result.traceback)
self.event_buffer[key] = (State.FAILED, None)
@@ -312,6 +297,7 @@ def _process_tasks(self, task_tuples_to_send: List[TaskInstanceInCelery]) -> Non
result.backend = cached_celery_backend
self.running.add(key)
self.tasks[key] = result
+ self._set_celery_pending_task_timeout(key, _CeleryPendingTaskTimeoutType.STALLED)
# Store the Celery task_id in the event buffer. This will get "overwritten" if the task
# has another event, but that is fine, because the only other events are success/failed at
@@ -342,27 +328,47 @@ def sync(self) -> None:
self.log.debug("No task to query celery, skipping sync")
return
self.update_all_task_states()
+ self._check_for_timedout_adopted_tasks()
+ self._check_for_stalled_tasks()
- if self.adopted_task_timeouts:
- self._check_for_stalled_adopted_tasks()
- if time.time() - self.stuck_tasks_last_check_time > self.stuck_queued_task_check_interval:
- self._clear_stuck_queued_tasks()
+ def _check_for_timedout_adopted_tasks(self) -> None:
+ timedout_keys = self._get_timedout_ti_keys(self.adopted_task_timeouts)
+ if timedout_keys:
+ self.log.error(
+ "Adopted tasks were still pending after %s, assuming they never made it to celery "
+ "and sending back to the scheduler:\n\t%s",
+ self.task_adoption_timeout,
+ "\n\t".join(repr(x) for x in timedout_keys),
+ )
+ self._send_stalled_tis_back_to_scheduler(timedout_keys)
- def _check_for_stalled_adopted_tasks(self):
- """
- See if any of the tasks we adopted from another Executor run have not
- progressed after the configured timeout.
+ def _check_for_stalled_tasks(self) -> None:
+ timedout_keys = self._get_timedout_ti_keys(self.stalled_task_timeouts)
+ if timedout_keys:
+ self.log.error(
+ "Tasks were still pending after %s, assuming they never made it to celery "
+ "and sending back to the scheduler:\n\t%s",
+ self.stalled_task_timeout,
+ "\n\t".join(repr(x) for x in timedout_keys),
+ )
+ self._send_stalled_tis_back_to_scheduler(timedout_keys)
- If they haven't, they likely never made it to Celery, and we should
- just resend them. We do that by clearing the state and letting the
- normal scheduler loop deal with that
+ def _get_timedout_ti_keys(
+ self, task_timeouts: Dict[TaskInstanceKey, datetime.datetime]
+ ) -> List[TaskInstanceKey]:
+ """
+ These timeouts exist to check to see if any of our tasks have not progressed
+ in the expected time. This can happen for few different reasons, usually related
+ to race conditions while shutting down schedulers and celery workers.
+
+ It is, of course, always possible that these tasks are not actually
+ stalled - they could just be waiting in a long celery queue.
+ Unfortunately there's no way for us to know for sure, so we'll just
+ reschedule them and let the normal scheduler loop requeue them.
"""
now = utcnow()
-
- sorted_adopted_task_timeouts = sorted(self.adopted_task_timeouts.items(), key=lambda k: k[1])
-
timedout_keys = []
- for key, stalled_after in sorted_adopted_task_timeouts:
+ for key, stalled_after in task_timeouts.items():
if stalled_after > now:
# Since items are stored sorted, if we get to a stalled_after
# in the future then we can stop
@@ -372,69 +378,46 @@ def _check_for_stalled_adopted_tasks(self):
# already finished, then it will be removed from this list -- so
# the only time it's still in this list is when it a) never made it
# to celery in the first place (i.e. race condition somewhere in
- # the dying executor) or b) a really long celery queue and it just
+ # the dying executor), b) celery lost the task before execution
+ # started, or c) a really long celery queue and it just
# hasn't started yet -- better cancel it and let the scheduler
# re-queue rather than have this task risk stalling for ever
timedout_keys.append(key)
-
- if timedout_keys:
- self.log.error(
- "Adopted tasks were still pending after %s, assuming they never made it to celery and "
- "clearing:\n\t%s",
- self.task_adoption_timeout,
- "\n\t".join(repr(x) for x in timedout_keys),
- )
- for key in timedout_keys:
- self.change_state(key, State.FAILED)
+ return timedout_keys
@provide_session
- def _clear_stuck_queued_tasks(self, session: Session = NEW_SESSION) -> None:
- """
- Tasks can get stuck in queued state in DB while still not in
- worker. This happens when the worker is autoscaled down and
- the task is queued but has not been picked up by any worker prior to the scaling.
-
- In such situation, we update the task instance state to scheduled so that
- it can be queued again. We chose to use task_adoption_timeout to decide when
- a queued task is considered stuck and should be reschelduled.
- """
- if not isinstance(app.backend, DatabaseBackend):
- # We only want to do this for database backends where
- # this case has been spotted
+ def _send_stalled_tis_back_to_scheduler(
+ self, keys: List[TaskInstanceKey], session: Session = NEW_SESSION
+ ) -> None:
+ try:
+ session.query(TaskInstance).filter(
+ TaskInstance.filter_for_tis(keys),
+ TaskInstance.state == State.QUEUED,
+ TaskInstance.queued_by_job_id == self.job_id,
+ ).update(
+ {
+ TaskInstance.state: State.SCHEDULED,
+ TaskInstance.queued_dttm: None,
+ TaskInstance.queued_by_job_id: None,
+ TaskInstance.external_executor_id: None,
+ },
+ synchronize_session=False,
+ )
+ session.commit()
+ except Exception:
+ self.log.exception("Error sending tasks back to scheduler")
+ session.rollback()
return
- # We use this instead of using bulk_state_fetcher because we
- # may not have the stuck task in self.tasks and we don't want
- # to clear task in self.tasks too
- session_ = app.backend.ResultSession()
- task_cls = getattr(app.backend, "task_cls", TaskDb)
- with session_cleanup(session_):
- celery_task_ids = [
- t.task_id
- for t in session_.query(task_cls.task_id)
- .filter(~task_cls.status.in_([celery_states.SUCCESS, celery_states.FAILURE]))
- .all()
- ]
- self.log.debug("Checking for stuck queued tasks")
-
- max_allowed_time = utcnow() - self.task_adoption_timeout
-
- for task in session.query(TaskInstance).filter(
- TaskInstance.state == State.QUEUED, TaskInstance.queued_dttm < max_allowed_time
- ):
- if task.key in self.queued_tasks or task.key in self.running:
- continue
-
- if task.external_executor_id in celery_task_ids:
- # The task is still running in the worker
- continue
- self.log.info(
- 'TaskInstance: %s found in queued state for more than %s seconds, rescheduling',
- task,
- self.task_adoption_timeout.total_seconds(),
- )
- task.state = State.SCHEDULED
- session.merge(task)
+ for key in keys:
+ self._set_celery_pending_task_timeout(key, None)
+ self.running.discard(key)
+ celery_async_result = self.tasks.pop(key, None)
+ if celery_async_result:
+ try:
+ app.control.revoke(celery_async_result.task_id)
+ except Exception as ex:
+ self.log.error("Error revoking task instance %s from celery: %s", key, ex)
def debug_dump(self) -> None:
"""Called in response to SIGUSR2 by the scheduler"""
@@ -447,6 +430,11 @@ def debug_dump(self) -> None:
len(self.adopted_task_timeouts),
"\n\t".join(map(repr, self.adopted_task_timeouts.items())),
)
+ self.log.info(
+ "executor.stalled_task_timeouts (%d)\n\t%s",
+ len(self.stalled_task_timeouts),
+ "\n\t".join(map(repr, self.stalled_task_timeouts.items())),
+ )
def update_all_task_states(self) -> None:
"""Updates states of the tasks."""
@@ -462,7 +450,7 @@ def update_all_task_states(self) -> None:
def change_state(self, key: TaskInstanceKey, state: str, info=None) -> None:
super().change_state(key, state, info)
self.tasks.pop(key, None)
- self.adopted_task_timeouts.pop(key, None)
+ self._set_celery_pending_task_timeout(key, None)
def update_task_state(self, key: TaskInstanceKey, state: str, info: Any) -> None:
"""Updates state of a single task."""
@@ -472,8 +460,8 @@ def update_task_state(self, key: TaskInstanceKey, state: str, info: Any) -> None
elif state in (celery_states.FAILURE, celery_states.REVOKED):
self.fail(key, info)
elif state == celery_states.STARTED:
- # It's now actually running, so know it made it to celery okay!
- self.adopted_task_timeouts.pop(key, None)
+ # It's now actually running, so we know it made it to celery okay!
+ self._set_celery_pending_task_timeout(key, None)
elif state == celery_states.PENDING:
pass
else:
@@ -487,16 +475,6 @@ def end(self, synchronous: bool = False) -> None:
time.sleep(5)
self.sync()
- def execute_async(
- self,
- key: TaskInstanceKey,
- command: CommandType,
- queue: Optional[str] = None,
- executor_config: Optional[Any] = None,
- ):
- """Do not allow async execution for Celery executor."""
- raise AirflowException("No Async execution for Celery executor.")
-
def terminate(self):
pass
@@ -543,7 +521,7 @@ def try_adopt_task_instances(self, tis: List[TaskInstance]) -> List[TaskInstance
# Set the correct elements of the state dicts, then update this
# like we just queried it.
- self.adopted_task_timeouts[ti.key] = ti.queued_dttm + self.task_adoption_timeout
+ self._set_celery_pending_task_timeout(ti.key, _CeleryPendingTaskTimeoutType.ADOPTED)
self.tasks[ti.key] = result
self.running.add(ti.key)
self.update_task_state(ti.key, state, info)
@@ -557,6 +535,21 @@ def try_adopt_task_instances(self, tis: List[TaskInstance]) -> List[TaskInstance
return not_adopted_tis
+ def _set_celery_pending_task_timeout(
+ self, key: TaskInstanceKey, timeout_type: Optional[_CeleryPendingTaskTimeoutType]
+ ) -> None:
+ """
+ We use the fact that dicts maintain insertion order, and the the timeout for a
+ task is always "now + delta" to maintain the property that oldest item = first to
+ time out.
+ """
+ self.adopted_task_timeouts.pop(key, None)
+ self.stalled_task_timeouts.pop(key, None)
+ if timeout_type == _CeleryPendingTaskTimeoutType.ADOPTED and self.task_adoption_timeout:
+ self.adopted_task_timeouts[key] = utcnow() + self.task_adoption_timeout
+ elif timeout_type == _CeleryPendingTaskTimeoutType.STALLED and self.stalled_task_timeout:
+ self.stalled_task_timeouts[key] = utcnow() + self.stalled_task_timeout
+
def fetch_celery_task_state(async_result: AsyncResult) -> Tuple[str, Union[str, ExceptionWithTraceback], Any]:
"""
diff --git a/airflow/executors/celery_kubernetes_executor.py b/airflow/executors/celery_kubernetes_executor.py
index 2802b6954e38c..b1edc32235727 100644
--- a/airflow/executors/celery_kubernetes_executor.py
+++ b/airflow/executors/celery_kubernetes_executor.py
@@ -17,6 +17,8 @@
# under the License.
from typing import Dict, List, Optional, Set, Union
+from airflow.callbacks.base_callback_sink import BaseCallbackSink
+from airflow.callbacks.callback_requests import CallbackRequest
from airflow.configuration import conf
from airflow.executors.base_executor import CommandType, EventBufferValueType, QueuedTaskInstanceType
from airflow.executors.celery_executor import CeleryExecutor
@@ -35,6 +37,7 @@ class CeleryKubernetesExecutor(LoggingMixin):
"""
supports_ad_hoc_ti_run: bool = True
+ callback_sink: Optional[BaseCallbackSink] = None
KUBERNETES_QUEUE = conf.get('celery_kubernetes_executor', 'kubernetes_queue')
@@ -43,6 +46,7 @@ def __init__(self, celery_executor: CeleryExecutor, kubernetes_executor: Kuberne
self._job_id: Optional[int] = None
self.celery_executor = celery_executor
self.kubernetes_executor = kubernetes_executor
+ self.kubernetes_executor.kubernetes_queue = self.KUBERNETES_QUEUE
@property
def queued_tasks(self) -> Dict[TaskInstanceKey, QueuedTaskInstanceType]:
@@ -203,3 +207,12 @@ def debug_dump(self) -> None:
self.celery_executor.debug_dump()
self.log.info("Dumping KubernetesExecutor state")
self.kubernetes_executor.debug_dump()
+
+ def send_callback(self, request: CallbackRequest) -> None:
+ """Sends callback for execution.
+
+ :param request: Callback request to be executed.
+ """
+ if not self.callback_sink:
+ raise ValueError("Callback sink is not ready.")
+ self.callback_sink.send(request)
diff --git a/airflow/executors/debug_executor.py b/airflow/executors/debug_executor.py
index 865186dd18ec7..7a2dddec79358 100644
--- a/airflow/executors/debug_executor.py
+++ b/airflow/executors/debug_executor.py
@@ -66,7 +66,6 @@ def sync(self) -> None:
self.log.info("Executor is terminated! Stopping %s to %s", ti.key, State.FAILED)
ti.set_state(State.FAILED)
self.change_state(ti.key, State.FAILED)
- ti._run_finished_callback()
continue
task_succeeded = self._run_task(ti)
@@ -78,12 +77,10 @@ def _run_task(self, ti: TaskInstance) -> bool:
params = self.tasks_params.pop(ti.key, {})
ti._run_raw_task(job_id=ti.job_id, **params)
self.change_state(key, State.SUCCESS)
- ti._run_finished_callback()
return True
except Exception as e:
ti.set_state(State.FAILED)
self.change_state(key, State.FAILED)
- ti._run_finished_callback(error=e)
self.log.exception("Failed to execute task: %s.", str(e))
return False
diff --git a/airflow/executors/executor_loader.py b/airflow/executors/executor_loader.py
index b98d2a80005dc..723060db0f179 100644
--- a/airflow/executors/executor_loader.py
+++ b/airflow/executors/executor_loader.py
@@ -71,8 +71,7 @@ def get_default_executor(cls) -> "BaseExecutor":
from airflow.configuration import conf
- executor_name = conf.get('core', 'EXECUTOR')
-
+ executor_name = conf.get_mandatory_value('core', 'EXECUTOR')
cls._default_executor = cls.load_executor(executor_name)
return cls._default_executor
diff --git a/airflow/executors/kubernetes_executor.py b/airflow/executors/kubernetes_executor.py
index bdac0889f5b2f..c76cf58f418d4 100644
--- a/airflow/executors/kubernetes_executor.py
+++ b/airflow/executors/kubernetes_executor.py
@@ -109,6 +109,8 @@ def run(self) -> None:
time.sleep(1)
except Exception:
self.log.exception('Unknown error in KubernetesJobWatcher. Failing')
+ self.resource_version = "0"
+ ResourceVersion().resource_version = "0"
raise
else:
self.log.warning(
@@ -288,6 +290,7 @@ def _health_check_kube_watcher(self):
self.log.error(
'Error while health checking kube watcher process. Process died for unknown reasons'
)
+ ResourceVersion().resource_version = "0"
self.kube_watcher = self._make_kube_watcher()
def run_next(self, next_job: KubernetesJobType) -> None:
@@ -438,6 +441,7 @@ def __init__(self):
self.scheduler_job_id: Optional[str] = None
self.event_scheduler: Optional[EventScheduler] = None
self.last_handled: Dict[TaskInstanceKey, float] = {}
+ self.kubernetes_queue: Optional[str] = None
super().__init__(parallelism=self.kube_config.parallelism)
@provide_session
@@ -456,9 +460,11 @@ def clear_not_launched_queued_tasks(self, session=None) -> None:
self.log.debug("Clearing tasks that have not been launched")
if not self.kube_client:
raise AirflowException(NOT_STARTED_MESSAGE)
- queued_tis: List[TaskInstance] = (
- session.query(TaskInstance).filter(TaskInstance.state == State.QUEUED).all()
- )
+
+ query = session.query(TaskInstance).filter(TaskInstance.state == State.QUEUED)
+ if self.kubernetes_queue:
+ query = query.filter(TaskInstance.queue == self.kubernetes_queue)
+ queued_tis: List[TaskInstance] = query.all()
self.log.info('Found %s queued task instances', len(queued_tis))
# Go through the "last seen" dictionary and clean out old entries
@@ -505,6 +511,7 @@ def clear_not_launched_queued_tasks(self, session=None) -> None:
TaskInstance.dag_id == ti.dag_id,
TaskInstance.task_id == ti.task_id,
TaskInstance.run_id == ti.run_id,
+ TaskInstance.map_index == ti.map_index,
).update({TaskInstance.state: State.SCHEDULED})
def start(self) -> None:
diff --git a/airflow/executors/local_executor.py b/airflow/executors/local_executor.py
index fc662fcd59320..431add69bcfbb 100644
--- a/airflow/executors/local_executor.py
+++ b/airflow/executors/local_executor.py
@@ -203,6 +203,8 @@ class LocalExecutor(BaseExecutor):
def __init__(self, parallelism: int = PARALLELISM):
super().__init__(parallelism=parallelism)
+ if self.parallelism < 0:
+ raise AirflowException("parallelism must be bigger than or equal to 0")
self.manager: Optional[SyncManager] = None
self.result_queue: Optional['Queue[TaskInstanceStateType]'] = None
self.workers: List[QueuedLocalWorker] = []
diff --git a/airflow/executors/local_kubernetes_executor.py b/airflow/executors/local_kubernetes_executor.py
index befbc325f659b..9944cfe1ef1fa 100644
--- a/airflow/executors/local_kubernetes_executor.py
+++ b/airflow/executors/local_kubernetes_executor.py
@@ -17,6 +17,8 @@
# under the License.
from typing import Dict, List, Optional, Set, Union
+from airflow.callbacks.base_callback_sink import BaseCallbackSink
+from airflow.callbacks.callback_requests import CallbackRequest
from airflow.configuration import conf
from airflow.executors.base_executor import CommandType, EventBufferValueType, QueuedTaskInstanceType
from airflow.executors.kubernetes_executor import KubernetesExecutor
@@ -35,6 +37,7 @@ class LocalKubernetesExecutor(LoggingMixin):
"""
supports_ad_hoc_ti_run: bool = True
+ callback_sink: Optional[BaseCallbackSink] = None
KUBERNETES_QUEUE = conf.get('local_kubernetes_executor', 'kubernetes_queue')
@@ -43,6 +46,7 @@ def __init__(self, local_executor: LocalExecutor, kubernetes_executor: Kubernete
self._job_id: Optional[str] = None
self.local_executor = local_executor
self.kubernetes_executor = kubernetes_executor
+ self.kubernetes_executor.kubernetes_queue = self.KUBERNETES_QUEUE
@property
def queued_tasks(self) -> Dict[TaskInstanceKey, QueuedTaskInstanceType]:
@@ -202,3 +206,12 @@ def debug_dump(self) -> None:
self.local_executor.debug_dump()
self.log.info("Dumping KubernetesExecutor state")
self.kubernetes_executor.debug_dump()
+
+ def send_callback(self, request: CallbackRequest) -> None:
+ """Sends callback for execution.
+
+ :param request: Callback request to be executed.
+ """
+ if not self.callback_sink:
+ raise ValueError("Callback sink is not ready.")
+ self.callback_sink.send(request)
diff --git a/airflow/hooks/base.py b/airflow/hooks/base.py
index b3a46e27a8b14..aa506cf62de28 100644
--- a/airflow/hooks/base.py
+++ b/airflow/hooks/base.py
@@ -22,7 +22,6 @@
from airflow.typing_compat import Protocol
from airflow.utils.log.logging_mixin import LoggingMixin
-from airflow.utils.log.secrets_masker import redact
if TYPE_CHECKING:
from airflow.models.connection import Connection # Avoid circular imports.
@@ -67,17 +66,6 @@ def get_connection(cls, conn_id: str) -> "Connection":
conn = Connection.get_connection_from_secrets(conn_id)
log.info("Using connection ID '%s' for task execution.", conn.conn_id)
- log.debug(
- "Connection details for '%s':: Host: %s, Port: %s, Schema: %s, Login: %s, Password: %s, "
- "Extra: %s",
- conn.conn_id,
- conn.host,
- conn.port,
- conn.schema,
- conn.login,
- redact(conn.password),
- redact(conn.extra_dejson),
- )
return conn
@classmethod
diff --git a/airflow/hooks/dbapi.py b/airflow/hooks/dbapi.py
index 1f898706a5e91..0b9ce4377be23 100644
--- a/airflow/hooks/dbapi.py
+++ b/airflow/hooks/dbapi.py
@@ -128,6 +128,24 @@ def get_pandas_df(self, sql, parameters=None, **kwargs):
with closing(self.get_conn()) as conn:
return psql.read_sql(sql, con=conn, params=parameters, **kwargs)
+ def get_pandas_df_by_chunks(self, sql, parameters=None, *, chunksize, **kwargs):
+ """
+ Executes the sql and returns a generator
+
+ :param sql: the sql statement to be executed (str) or a list of
+ sql statements to execute
+ :param parameters: The parameters to render the SQL query with
+ :param chunksize: number of rows to include in each chunk
+ :param kwargs: (optional) passed into pandas.io.sql.read_sql method
+ """
+ try:
+ from pandas.io import sql as psql
+ except ImportError:
+ raise Exception("pandas library not installed, run: pip install 'apache-airflow[pandas]'.")
+
+ with closing(self.get_conn()) as conn:
+ yield from psql.read_sql(sql, con=conn, params=parameters, chunksize=chunksize, **kwargs)
+
def get_records(self, sql, parameters=None):
"""
Executes the sql and returns a set of records.
@@ -178,6 +196,11 @@ def run(self, sql, autocommit=False, parameters=None, handler=None):
if scalar:
sql = [sql]
+ if sql:
+ self.log.debug("Executing %d statements", len(sql))
+ else:
+ raise ValueError("List of SQL statements is empty")
+
with closing(self.get_conn()) as conn:
if self.supports_autocommit:
self.set_autocommit(conn, autocommit)
diff --git a/airflow/hooks/subprocess.py b/airflow/hooks/subprocess.py
index c814b0528d25a..fa8c706c6963d 100644
--- a/airflow/hooks/subprocess.py
+++ b/airflow/hooks/subprocess.py
@@ -88,7 +88,7 @@ def pre_exec():
raise RuntimeError("The subprocess should be created here and is None!")
if self.sub_process.stdout is not None:
for raw_line in iter(self.sub_process.stdout.readline, b''):
- line = raw_line.decode(output_encoding).rstrip()
+ line = raw_line.decode(output_encoding, errors='backslashreplace').rstrip()
self.log.info("%s", line)
self.sub_process.wait()
diff --git a/airflow/jobs/backfill_job.py b/airflow/jobs/backfill_job.py
index 1b464575bcd41..c5b98c2a8df2a 100644
--- a/airflow/jobs/backfill_job.py
+++ b/airflow/jobs/backfill_job.py
@@ -22,6 +22,7 @@
import attr
import pendulum
+from sqlalchemy.exc import OperationalError
from sqlalchemy.orm.session import Session, make_transient
from tabulate import tabulate
@@ -101,7 +102,7 @@ class _DagRunTaskStatus:
def __init__(
self,
- dag,
+ dag: DAG,
start_date=None,
end_date=None,
mark_success=False,
@@ -115,6 +116,7 @@ def __init__(
rerun_failed_tasks=False,
run_backwards=False,
run_at_least_once=False,
+ continue_on_failures=False,
*args,
**kwargs,
):
@@ -153,6 +155,7 @@ def __init__(
self.rerun_failed_tasks = rerun_failed_tasks
self.run_backwards = run_backwards
self.run_at_least_once = run_at_least_once
+ self.continue_on_failures = continue_on_failures
super().__init__(*args, **kwargs)
def _update_counters(self, ti_status, session=None):
@@ -225,7 +228,7 @@ def _update_counters(self, ti_status, session=None):
def _manage_executor_state(
self, running, session
- ) -> Iterator[Tuple["MappedOperator", str, Sequence[TaskInstance]]]:
+ ) -> Iterator[Tuple["MappedOperator", str, Sequence[TaskInstance], int]]:
"""
Checks if the executor agrees with the state of task instances
that are running.
@@ -235,8 +238,6 @@ def _manage_executor_state(
:param running: dict of key, task to verify
:return: An iterable of expanded TaskInstance per MappedTask
"""
- from airflow.models.mappedoperator import MappedOperator
-
executor = self.executor
# TODO: query all instead of refresh from db
@@ -260,12 +261,14 @@ def _manage_executor_state(
f"{ti.state}. Was the task killed externally? Info: {info}"
)
self.log.error(msg)
- ti.handle_failure_with_callback(error=msg)
+ ti.handle_failure(error=msg)
continue
if ti.state not in self.STATES_COUNT_AS_RUNNING:
- for node in ti.task.mapped_dependants():
- assert isinstance(node, MappedOperator)
- yield node, ti.run_id, node.expand_mapped_task(ti.run_id, session=session)
+ # Don't use ti.task; if this task is mapped, that attribute
+ # would hold the unmapped task. We need to original task here.
+ for node in self.dag.get_task(ti.task_id, include_subdags=True).iter_mapped_dependants():
+ new_tis, num_mapped_tis = node.expand_mapped_task(ti.run_id, session=session)
+ yield node, ti.run_id, new_tis, num_mapped_tis
@provide_session
def _get_dag_run(self, dagrun_info: DagRunInfo, dag: DAG, session: Session = None):
@@ -294,6 +297,8 @@ def _get_dag_run(self, dagrun_info: DagRunInfo, dag: DAG, session: Session = Non
run = runs[0]
if run.state == DagRunState.RUNNING:
respect_dag_max_active_limit = False
+ # Fixes --conf overwrite for backfills with already existing DagRuns
+ run.conf = self.conf or {}
else:
run = None
@@ -486,6 +491,13 @@ def _per_task_process(key, ti: TaskInstance, session=None):
ti.queued_by_job_id = self.id
ti.queued_dttm = timezone.utcnow()
session.merge(ti)
+ try:
+ session.commit()
+ except OperationalError:
+ self.log.exception("Failed to commit task state change due to operational error")
+ session.rollback()
+ # early exit so the outer loop can retry
+ return
cfg_path = None
if self.executor_class in (
@@ -505,7 +517,6 @@ def _per_task_process(key, ti: TaskInstance, session=None):
)
ti_status.running[key] = ti
ti_status.to_run.pop(key)
- session.commit()
return
if ti.state == TaskInstanceState.UPSTREAM_FAILED:
@@ -598,18 +609,23 @@ def _per_task_process(key, ti: TaskInstance, session=None):
ti_status.to_run.clear()
# check executor state -- and expand any mapped TIs
- for node, run_id, mapped_tis in self._manage_executor_state(ti_status.running, session):
+ for node, run_id, new_mapped_tis, max_map_index in self._manage_executor_state(
+ ti_status.running, session
+ ):
def to_keep(key: TaskInstanceKey) -> bool:
if key.dag_id != node.dag_id or key.task_id != node.task_id or key.run_id != run_id:
# For another Dag/Task/Run -- don't remove
return True
- return False
+ return 0 <= key.map_index <= max_map_index
# remove the old unmapped TIs for node -- they have been replaced with the mapped TIs
ti_status.to_run = {key: ti for (key, ti) in ti_status.to_run.items() if to_keep(key)}
- ti_status.to_run.update({ti.key: ti for ti in mapped_tis})
+ ti_status.to_run.update({ti.key: ti for ti in new_mapped_tis})
+
+ for new_ti in new_mapped_tis:
+ new_ti.set_state(TaskInstanceState.SCHEDULED, session=session)
# update the task counters
self._update_counters(ti_status=ti_status, session=session)
@@ -691,6 +707,9 @@ def tabulate_ti_keys_set(ti_keys: Iterable[TaskInstanceKey]) -> str:
return err
+ def _get_dag_with_subdags(self) -> List[DAG]:
+ return [self.dag] + self.dag.subdags
+
@provide_session
def _execute_dagruns(self, dagrun_infos, ti_status, executor, pickle_id, start_date, session=None):
"""
@@ -706,7 +725,7 @@ def _execute_dagruns(self, dagrun_infos, ti_status, executor, pickle_id, start_d
:param session: the current session object
"""
for dagrun_info in dagrun_infos:
- for dag in [self.dag] + self.dag.subdags:
+ for dag in self._get_dag_with_subdags():
dag_run = self._get_dag_run(dagrun_info, dag, session=session)
tis_map = self._task_instances_for_dag_run(dag_run, session=session)
if dag_run is None:
@@ -773,6 +792,31 @@ def _execute(self, session=None):
return
dagrun_infos = [DagRunInfo.interval(dagrun_start_date, dagrun_end_date)]
+ dag_with_subdags_ids = [d.dag_id for d in self._get_dag_with_subdags()]
+ running_dagruns = DagRun.find(
+ dag_id=dag_with_subdags_ids,
+ execution_start_date=self.bf_start_date,
+ execution_end_date=self.bf_end_date,
+ no_backfills=True,
+ state=DagRunState.RUNNING,
+ )
+
+ if running_dagruns:
+ for run in running_dagruns:
+ self.log.error(
+ "Backfill cannot be created for DagRun %s in %s, as there's already %s in a RUNNING "
+ "state.",
+ run.run_id,
+ run.execution_date.strftime("%Y-%m-%dT%H:%M:%S"),
+ run.run_type,
+ )
+ self.log.error(
+ "Changing DagRun into BACKFILL would cause scheduler to lose track of executing "
+ "tasks. Not changing DagRun type into BACKFILL, and trying insert another DagRun into "
+ "database would cause database constraint violation for dag_id + execution_date "
+ "combination. Please adjust backfill dates or wait for this DagRun to finish.",
+ )
+ return
# picklin'
pickle_id = None
@@ -812,7 +856,8 @@ def _execute(self, session=None):
remaining_dates = ti_status.total_runs - len(ti_status.executed_dag_run_dates)
err = self._collect_errors(ti_status=ti_status, session=session)
if err:
- raise BackfillUnfinished(err, ti_status)
+ if not self.continue_on_failures or ti_status.deadlocked:
+ raise BackfillUnfinished(err, ti_status)
if remaining_dates > 0:
self.log.info(
diff --git a/airflow/jobs/local_task_job.py b/airflow/jobs/local_task_job.py
index 0334799d3e1f2..5711342e04d97 100644
--- a/airflow/jobs/local_task_job.py
+++ b/airflow/jobs/local_task_job.py
@@ -73,6 +73,8 @@ def __init__(
# terminate multiple times
self.terminating = False
+ self._state_change_checks = 0
+
super().__init__(*args, **kwargs)
def _execute(self):
@@ -84,7 +86,6 @@ def signal_handler(signum, frame):
self.log.error("Received SIGTERM. Terminating subprocesses")
self.task_runner.terminate()
self.handle_task_exit(128 + signum)
- return
signal.signal(signal.SIGTERM, signal_handler)
@@ -104,20 +105,17 @@ def signal_handler(signum, frame):
try:
self.task_runner.start()
- # Unmap the task _after_ it has forked/execed. (This is a bit of a kludge, but if we unmap before
- # fork, then the "run_raw_task" command will see the mapping index and an Non-mapped task and
- # fail)
- self.task_instance.task = self.task_instance.task.unmap()
-
heartbeat_time_limit = conf.getint('scheduler', 'scheduler_zombie_task_threshold')
- # task callback invocation happens either here or in
- # self.heartbeat() instead of taskinstance._run_raw_task to
- # avoid race conditions
- #
- # When self.terminating is set to True by heartbeat_callback, this
- # loop should not be restarted. Otherwise self.handle_task_exit
- # will be invoked and we will end up with duplicated callbacks
+ # LocalTaskJob should not run callbacks, which are handled by TaskInstance._run_raw_task
+ # 1, LocalTaskJob does not parse DAG, thus cannot run callbacks
+ # 2, The run_as_user of LocalTaskJob is likely not same as the TaskInstance._run_raw_task.
+ # When run_as_user is specified, the process owner of the LocalTaskJob must be sudoable.
+ # It is not secure to run callbacks with sudoable users.
+
+ # If _run_raw_task receives SIGKILL, scheduler will mark it as zombie and invoke callbacks
+ # If LocalTaskJob receives SIGTERM, LocalTaskJob passes SIGTERM to _run_raw_task
+ # If the state of task_instance is changed, LocalTaskJob sends SIGTERM to _run_raw_task
while not self.terminating:
# Monitor the task to see if it's done. Wait in a syscall
# (`os.wait`) for as long as possible so we notice the
@@ -155,26 +153,18 @@ def signal_handler(signum, frame):
self.on_kill()
def handle_task_exit(self, return_code: int) -> None:
- """Handle case where self.task_runner exits by itself or is externally killed"""
+ """
+ Handle case where self.task_runner exits by itself or is externally killed
+
+ Dont run any callbacks
+ """
# Without setting this, heartbeat may get us
self.terminating = True
self.log.info("Task exited with return code %s", return_code)
- self.task_instance.refresh_from_db()
- if self.task_instance.state == State.RUNNING:
- # This is for a case where the task received a SIGKILL
- # while running or the task runner received a sigterm
- self.task_instance.handle_failure(error=None)
- # We need to check for error file
- # in case it failed due to runtime exception/error
- error = None
- if self.task_instance.state != State.SUCCESS:
- error = self.task_runner.deserialize_run_error()
- self.task_instance._run_finished_callback(error=error)
if not self.task_instance.test_mode:
if conf.getboolean('scheduler', 'schedule_after_task_execution', fallback=True):
self._run_mini_scheduler_on_child_tasks()
- self._update_dagrun_state_for_paused_dag()
def on_kill(self):
self.task_runner.terminate()
@@ -195,7 +185,7 @@ def heartbeat_callback(self, session=None):
fqdn = get_hostname()
same_hostname = fqdn == ti.hostname
if not same_hostname:
- self.log.warning(
+ self.log.error(
"The recorded hostname %s does not match this instance's hostname %s",
ti.hostname,
fqdn,
@@ -222,19 +212,16 @@ def heartbeat_callback(self, session=None):
dagrun_timeout = ti.task.dag.dagrun_timeout
if dagrun_timeout and execution_time > dagrun_timeout:
self.log.warning("DagRun timed out after %s.", str(execution_time))
- self.log.warning(
- "State of this instance has been externally set to %s. Terminating instance.", ti.state
- )
- self.task_runner.terminate()
- if ti.state == State.SUCCESS:
- error = None
- else:
- # if ti.state is not set by taskinstance.handle_failure, then
- # error file will not be populated and it must be updated by
- # external source such as web UI
- error = self.task_runner.deserialize_run_error() or "task marked as failed externally"
- ti._run_finished_callback(error=error)
- self.terminating = True
+
+ # potential race condition, the _run_raw_task commits `success` or other state
+ # but task_runner does not exit right away due to slow process shutdown or any other reasons
+ # let's do a throttle here, if the above case is true, the handle_task_exit will handle it
+ if self._state_change_checks >= 1: # defer to next round of heartbeat
+ self.log.warning(
+ "State of this instance has been externally set to %s. Terminating instance.", ti.state
+ )
+ self.terminating = True
+ self._state_change_checks += 1
@provide_session
@Sentry.enrich_errors
@@ -287,19 +274,6 @@ def _run_mini_scheduler_on_child_tasks(self, session=None) -> None:
)
session.rollback()
- @provide_session
- def _update_dagrun_state_for_paused_dag(self, session=None):
- """
- Checks for paused dags with DagRuns in the running state and
- update the DagRun state if possible
- """
- dag = self.task_instance.task.dag
- if dag.get_is_paused():
- dag_run = self.task_instance.get_dagrun(session=session)
- if dag_run:
- dag_run.dag = dag
- dag_run.update_state(session=session, execute_callbacks=True)
-
@staticmethod
def _enable_task_listeners():
"""
diff --git a/airflow/jobs/scheduler_job.py b/airflow/jobs/scheduler_job.py
index a09b39af2304a..22ba5decb4111 100644
--- a/airflow/jobs/scheduler_job.py
+++ b/airflow/jobs/scheduler_job.py
@@ -26,9 +26,9 @@
import warnings
from collections import defaultdict
from datetime import timedelta
-from typing import Collection, DefaultDict, Dict, Iterator, List, Optional, Tuple
+from typing import Collection, DefaultDict, Dict, Iterator, List, Optional, Set, Tuple
-from sqlalchemy import and_, func, not_, or_, text, tuple_
+from sqlalchemy import func, not_, or_, text
from sqlalchemy.exc import OperationalError
from sqlalchemy.orm import load_only, selectinload
from sqlalchemy.orm.session import Session, make_transient
@@ -55,7 +55,13 @@
from airflow.utils.event_scheduler import EventScheduler
from airflow.utils.retries import MAX_DB_RETRIES, retry_db_transaction, run_with_db_retries
from airflow.utils.session import create_session, provide_session
-from airflow.utils.sqlalchemy import is_lock_not_available_error, prohibit_commit, skip_locked, with_row_locks
+from airflow.utils.sqlalchemy import (
+ is_lock_not_available_error,
+ prohibit_commit,
+ skip_locked,
+ tuple_in_condition,
+ with_row_locks,
+)
from airflow.utils.state import DagRunState, State, TaskInstanceState
from airflow.utils.types import DagRunType
@@ -128,7 +134,7 @@ def __init__(
self._scheduler_idle_sleep_time = scheduler_idle_sleep_time
# How many seconds do we wait for tasks to heartbeat before mark them as zombies.
self._zombie_threshold_secs = conf.getint('scheduler', 'scheduler_zombie_task_threshold')
-
+ self._standalone_dag_processor = conf.getboolean("scheduler", "standalone_dag_processor")
self.do_pickle = do_pickle
super().__init__(*args, **kwargs)
@@ -136,17 +142,21 @@ def __init__(
self._log = log
# Check what SQL backend we use
- sql_conn: str = conf.get('core', 'sql_alchemy_conn').lower()
+ sql_conn: str = conf.get_mandatory_value('database', 'sql_alchemy_conn').lower()
self.using_sqlite = sql_conn.startswith('sqlite')
self.using_mysql = sql_conn.startswith('mysql')
-
+ # Dag Processor agent - not used in Dag Processor standalone mode.
self.processor_agent: Optional[DagFileProcessorAgent] = None
self.dagbag = DagBag(dag_folder=self.subdir, read_dags_from_db=True, load_op_links=False)
+ self._paused_dag_without_running_dagruns: Set = set()
if conf.getboolean('smart_sensor', 'use_smart_sensor'):
compatible_sensors = set(
- map(lambda l: l.strip(), conf.get('smart_sensor', 'sensors_enabled').split(','))
+ map(
+ lambda l: l.strip(),
+ conf.get_mandatory_value('smart_sensor', 'sensors_enabled').split(','),
+ )
)
docs_url = get_docs_url('concepts/smart-sensors.html#migrating-to-deferrable-operators')
warnings.warn(
@@ -267,58 +277,20 @@ def _executable_task_instances_to_queued(self, max_tis: int, session: Session =
# If the pools are full, there is no point doing anything!
# If _somehow_ the pool is overfull, don't let the limit go negative - it breaks SQL
- pool_slots_free = max(0, sum(pool['open'] for pool in pools.values()))
+ pool_slots_free = sum(max(0, pool['open']) for pool in pools.values())
if pool_slots_free == 0:
self.log.debug("All pools are full!")
- return executable_tis
+ return []
max_tis = min(max_tis, pool_slots_free)
- # Get all task instances associated with scheduled
- # DagRuns which are not backfilled, in the given states,
- # and the dag is not paused
- query = (
- session.query(TI)
- .join(TI.dag_run)
- .filter(DR.run_type != DagRunType.BACKFILL_JOB, DR.state == DagRunState.RUNNING)
- .join(TI.dag_model)
- .filter(not_(DM.is_paused))
- .filter(TI.state == TaskInstanceState.SCHEDULED)
- .options(selectinload('dag_model'))
- .order_by(-TI.priority_weight, DR.execution_date)
- )
- starved_pools = [pool_name for pool_name, stats in pools.items() if stats['open'] <= 0]
- if starved_pools:
- query = query.filter(not_(TI.pool.in_(starved_pools)))
-
- query = query.limit(max_tis)
-
- task_instances_to_examine: List[TI] = with_row_locks(
- query,
- of=TI,
- session=session,
- **skip_locked(session=session),
- ).all()
- # TODO[HA]: This was wrong before anyway, as it only looked at a sub-set of dags, not everything.
- # Stats.gauge('scheduler.tasks.pending', len(task_instances_to_examine))
-
- if len(task_instances_to_examine) == 0:
- self.log.debug("No tasks to consider for execution.")
- return executable_tis
-
- # Put one task instance on each line
- task_instance_str = "\n\t".join(repr(x) for x in task_instances_to_examine)
- self.log.info("%s tasks up for execution:\n\t%s", len(task_instances_to_examine), task_instance_str)
-
- pool_to_task_instances: DefaultDict[str, List[models.Pool]] = defaultdict(list)
- for task_instance in task_instances_to_examine:
- pool_to_task_instances[task_instance.pool].append(task_instance)
+ starved_pools = {pool_name for pool_name, stats in pools.items() if stats['open'] <= 0}
# dag_id to # of running tasks and (dag_id, task_id) to # of running tasks.
- dag_max_active_tasks_map: DefaultDict[str, int]
+ dag_active_tasks_map: DefaultDict[str, int]
task_concurrency_map: DefaultDict[Tuple[str, str], int]
- dag_max_active_tasks_map, task_concurrency_map = self.__get_concurrency_maps(
+ dag_active_tasks_map, task_concurrency_map = self.__get_concurrency_maps(
states=list(EXECUTION_STATES), session=session
)
@@ -326,17 +298,88 @@ def _executable_task_instances_to_queued(self, max_tis: int, session: Session =
# Number of tasks that cannot be scheduled because of no open slot in pool
num_starving_tasks_total = 0
- # Go through each pool, and queue up a task for execution if there are
- # any open slots in the pool.
+ # dag and task ids that can't be queued because of concurrency limits
+ starved_dags: Set[str] = set()
+ starved_tasks: Set[Tuple[str, str]] = set()
- for pool, task_instances in pool_to_task_instances.items():
- pool_name = pool
- if pool not in pools:
- self.log.warning("Tasks using non-existent pool '%s' will not be scheduled", pool)
- continue
+ pool_num_starving_tasks: DefaultDict[str, int] = defaultdict(int)
+
+ for loop_count in itertools.count(start=1):
+
+ num_starved_pools = len(starved_pools)
+ num_starved_dags = len(starved_dags)
+ num_starved_tasks = len(starved_tasks)
+
+ # Get task instances associated with scheduled
+ # DagRuns which are not backfilled, in the given states,
+ # and the dag is not paused
+ query = (
+ session.query(TI)
+ .join(TI.dag_run)
+ .filter(DR.run_type != DagRunType.BACKFILL_JOB, DR.state == DagRunState.RUNNING)
+ .join(TI.dag_model)
+ .filter(not_(DM.is_paused))
+ .filter(TI.state == TaskInstanceState.SCHEDULED)
+ .options(selectinload('dag_model'))
+ .order_by(-TI.priority_weight, DR.execution_date)
+ )
+
+ if starved_pools:
+ query = query.filter(not_(TI.pool.in_(starved_pools)))
+
+ if starved_dags:
+ query = query.filter(not_(TI.dag_id.in_(starved_dags)))
+
+ if starved_tasks:
+ task_filter = tuple_in_condition((TaskInstance.dag_id, TaskInstance.task_id), starved_tasks)
+ query = query.filter(not_(task_filter))
+
+ query = query.limit(max_tis)
+
+ task_instances_to_examine: List[TI] = with_row_locks(
+ query,
+ of=TI,
+ session=session,
+ **skip_locked(session=session),
+ ).all()
+ # TODO[HA]: This was wrong before anyway, as it only looked at a sub-set of dags, not everything.
+ # Stats.gauge('scheduler.tasks.pending', len(task_instances_to_examine))
+
+ if len(task_instances_to_examine) == 0:
+ self.log.debug("No tasks to consider for execution.")
+ break
+
+ # Put one task instance on each line
+ task_instance_str = "\n\t".join(repr(x) for x in task_instances_to_examine)
+ self.log.info(
+ "%s tasks up for execution:\n\t%s", len(task_instances_to_examine), task_instance_str
+ )
+
+ for task_instance in task_instances_to_examine:
+ pool_name = task_instance.pool
+
+ pool_stats = pools.get(pool_name)
+ if not pool_stats:
+ self.log.warning("Tasks using non-existent pool '%s' will not be scheduled", pool_name)
+ starved_pools.add(pool_name)
+ continue
+
+ # Make sure to emit metrics if pool has no starving tasks
+ pool_num_starving_tasks.setdefault(pool_name, 0)
+
+ pool_total = pool_stats["total"]
+ open_slots = pool_stats["open"]
+
+ if open_slots <= 0:
+ self.log.info(
+ "Not scheduling since there are %s open slots in pool %s", open_slots, pool_name
+ )
+ # Can't schedule any more since there are no more open slots.
+ pool_num_starving_tasks[pool_name] += 1
+ num_starving_tasks_total += 1
+ starved_pools.add(pool_name)
+ continue
- pool_total = pools[pool]["total"]
- for task_instance in task_instances:
if task_instance.pool_slots > pool_total:
self.log.warning(
"Not executing %s. Requested pool slots (%s) are greater than "
@@ -344,48 +387,42 @@ def _executable_task_instances_to_queued(self, max_tis: int, session: Session =
task_instance,
task_instance.pool_slots,
pool_total,
- pool,
+ pool_name,
)
- task_instances.remove(task_instance)
-
- open_slots = pools[pool]["open"]
- num_ready = len(task_instances)
- self.log.info(
- "Figuring out tasks to run in Pool(name=%s) with %s open slots "
- "and %s task instances ready to be queued",
- pool,
- open_slots,
- num_ready,
- )
-
- priority_sorted_task_instances = sorted(
- task_instances, key=lambda ti: (-ti.priority_weight, ti.execution_date)
- )
+ pool_num_starving_tasks[pool_name] += 1
+ num_starving_tasks_total += 1
+ starved_tasks.add((task_instance.dag_id, task_instance.task_id))
+ continue
- num_starving_tasks = 0
- for current_index, task_instance in enumerate(priority_sorted_task_instances):
- if open_slots <= 0:
- self.log.info("Not scheduling since there are %s open slots in pool %s", open_slots, pool)
- # Can't schedule any more since there are no more open slots.
- num_unhandled = len(priority_sorted_task_instances) - current_index
- num_starving_tasks += num_unhandled
- num_starving_tasks_total += num_unhandled
- break
+ if task_instance.pool_slots > open_slots:
+ self.log.info(
+ "Not executing %s since it requires %s slots "
+ "but there are %s open slots in the pool %s.",
+ task_instance,
+ task_instance.pool_slots,
+ open_slots,
+ pool_name,
+ )
+ pool_num_starving_tasks[pool_name] += 1
+ num_starving_tasks_total += 1
+ starved_tasks.add((task_instance.dag_id, task_instance.task_id))
+ # Though we can execute tasks with lower priority if there's enough room
+ continue
# Check to make sure that the task max_active_tasks of the DAG hasn't been
# reached.
dag_id = task_instance.dag_id
- current_max_active_tasks_per_dag = dag_max_active_tasks_map[dag_id]
+ current_active_tasks_per_dag = dag_active_tasks_map[dag_id]
max_active_tasks_per_dag_limit = task_instance.dag_model.max_active_tasks
self.log.info(
"DAG %s has %s/%s running and queued tasks",
dag_id,
- current_max_active_tasks_per_dag,
+ current_active_tasks_per_dag,
max_active_tasks_per_dag_limit,
)
- if current_max_active_tasks_per_dag >= max_active_tasks_per_dag_limit:
+ if current_active_tasks_per_dag >= max_active_tasks_per_dag_limit:
self.log.info(
"Not executing %s since the number of tasks running or queued "
"from DAG %s is >= to the DAG's max_active_tasks limit of %s",
@@ -393,9 +430,9 @@ def _executable_task_instances_to_queued(self, max_tis: int, session: Session =
dag_id,
max_active_tasks_per_dag_limit,
)
+ starved_dags.add(dag_id)
continue
- task_concurrency_limit: Optional[int] = None
if task_instance.dag_model.has_task_concurrency_limits:
# Many dags don't have a task_concurrency, so where we can avoid loading the full
# serialized DAG the better.
@@ -411,6 +448,8 @@ def _executable_task_instances_to_queued(self, max_tis: int, session: Session =
{TI.state: State.FAILED}, synchronize_session='fetch'
)
continue
+
+ task_concurrency_limit: Optional[int] = None
if serialized_dag.has_task(task_instance.task_id):
task_concurrency_limit = serialized_dag.get_task(
task_instance.task_id
@@ -427,36 +466,44 @@ def _executable_task_instances_to_queued(self, max_tis: int, session: Session =
" this task has been reached.",
task_instance,
)
+ starved_tasks.add((task_instance.dag_id, task_instance.task_id))
continue
- if task_instance.pool_slots > open_slots:
- self.log.info(
- "Not executing %s since it requires %s slots "
- "but there are %s open slots in the pool %s.",
- task_instance,
- task_instance.pool_slots,
- open_slots,
- pool,
- )
- num_starving_tasks += 1
- num_starving_tasks_total += 1
- # Though we can execute tasks with lower priority if there's enough room
- continue
-
executable_tis.append(task_instance)
open_slots -= task_instance.pool_slots
- dag_max_active_tasks_map[dag_id] += 1
+ dag_active_tasks_map[dag_id] += 1
task_concurrency_map[(task_instance.dag_id, task_instance.task_id)] += 1
+ pool_stats["open"] = open_slots
+
+ is_done = executable_tis or len(task_instances_to_examine) < max_tis
+ # Check this to avoid accidental infinite loops
+ found_new_filters = (
+ len(starved_pools) > num_starved_pools
+ or len(starved_dags) > num_starved_dags
+ or len(starved_tasks) > num_starved_tasks
+ )
+
+ if is_done or not found_new_filters:
+ break
+
+ self.log.debug(
+ "Found no task instances to queue on the %s. iteration "
+ "but there could be more candidate task instances to check.",
+ loop_count,
+ )
+
+ for pool_name, num_starving_tasks in pool_num_starving_tasks.items():
Stats.gauge(f'pool.starving_tasks.{pool_name}', num_starving_tasks)
Stats.gauge('scheduler.tasks.starving', num_starving_tasks_total)
Stats.gauge('scheduler.tasks.running', num_tasks_in_executor)
Stats.gauge('scheduler.tasks.executable', len(executable_tis))
- task_instance_str = "\n\t".join(repr(x) for x in executable_tis)
- self.log.info("Setting the following tasks to queued state:\n\t%s", task_instance_str)
if len(executable_tis) > 0:
+ task_instance_str = "\n\t".join(repr(x) for x in executable_tis)
+ self.log.info("Setting the following tasks to queued state:\n\t%s", task_instance_str)
+
# set TIs to queued state
filter_for_tis = TI.filter_for_tis(executable_tis)
session.query(TI).filter(filter_for_tis).update(
@@ -506,9 +553,9 @@ def _enqueue_task_instances_with_queued_state(
queue=queue,
)
- def _critical_section_execute_task_instances(self, session: Session) -> int:
+ def _critical_section_enqueue_task_instances(self, session: Session) -> int:
"""
- Attempts to execute TaskInstances that should be executed by the scheduler.
+ Enqueues TaskInstances for execution.
There are three steps:
1. Pick TIs by priority with the constraint that they are in the expected states
@@ -537,7 +584,7 @@ def _critical_section_execute_task_instances(self, session: Session) -> int:
@provide_session
def _process_executor_events(self, session: Session = None) -> int:
"""Respond to executor events."""
- if not self.processor_agent:
+ if not self._standalone_dag_processor and not self.processor_agent:
raise ValueError("Processor agent is not started.")
ti_primary_key_to_try_number_map: Dict[Tuple[str, str, str, int], int] = {}
event_buffer = self.executor.get_event_buffer()
@@ -660,47 +707,50 @@ def _execute(self) -> None:
processor_timeout_seconds: int = conf.getint('core', 'dag_file_processor_timeout')
processor_timeout = timedelta(seconds=processor_timeout_seconds)
- self.processor_agent = DagFileProcessorAgent(
- dag_directory=self.subdir,
- max_runs=self.num_times_parse_dags,
- processor_timeout=processor_timeout,
- dag_ids=[],
- pickle_dags=pickle_dags,
- async_mode=async_mode,
- )
+ if not self._standalone_dag_processor:
+ self.processor_agent = DagFileProcessorAgent(
+ dag_directory=self.subdir,
+ max_runs=self.num_times_parse_dags,
+ processor_timeout=processor_timeout,
+ dag_ids=[],
+ pickle_dags=pickle_dags,
+ async_mode=async_mode,
+ )
try:
self.executor.job_id = self.id
- if conf.getboolean("scheduler", "standalone_dag_processor"):
- self.log.debug("Using DatabaseCallbackSink as callback sink.")
- self.executor.callback_sink = DatabaseCallbackSink()
- else:
+ if self.processor_agent:
self.log.debug("Using PipeCallbackSink as callback sink.")
self.executor.callback_sink = PipeCallbackSink(
get_sink_pipe=self.processor_agent.get_callbacks_pipe
)
+ else:
+ self.log.debug("Using DatabaseCallbackSink as callback sink.")
+ self.executor.callback_sink = DatabaseCallbackSink()
self.executor.start()
self.register_signals()
- self.processor_agent.start()
+ if self.processor_agent:
+ self.processor_agent.start()
execute_start_time = timezone.utcnow()
self._run_scheduler_loop()
- # Stop any processors
- self.processor_agent.terminate()
+ if self.processor_agent:
+ # Stop any processors
+ self.processor_agent.terminate()
- # Verify that all files were processed, and if so, deactivate DAGs that
- # haven't been touched by the scheduler as they likely have been
- # deleted.
- if self.processor_agent.all_files_processed:
- self.log.info(
- "Deactivating DAGs that haven't been touched since %s", execute_start_time.isoformat()
- )
- models.DAG.deactivate_stale_dags(execute_start_time)
+ # Verify that all files were processed, and if so, deactivate DAGs that
+ # haven't been touched by the scheduler as they likely have been
+ # deleted.
+ if self.processor_agent.all_files_processed:
+ self.log.info(
+ "Deactivating DAGs that haven't been touched since %s", execute_start_time.isoformat()
+ )
+ models.DAG.deactivate_stale_dags(execute_start_time)
settings.Session.remove() # type: ignore
except Exception:
@@ -711,12 +761,33 @@ def _execute(self) -> None:
self.executor.end()
except Exception:
self.log.exception("Exception when executing Executor.end")
- try:
- self.processor_agent.end()
- except Exception:
- self.log.exception("Exception when executing DagFileProcessorAgent.end")
+ if self.processor_agent:
+ try:
+ self.processor_agent.end()
+ except Exception:
+ self.log.exception("Exception when executing DagFileProcessorAgent.end")
self.log.info("Exited execute loop")
+ def _update_dag_run_state_for_paused_dags(self):
+ try:
+ paused_dag_ids = DagModel.get_all_paused_dag_ids()
+ for dag_id in paused_dag_ids:
+ if dag_id in self._paused_dag_without_running_dagruns:
+ continue
+
+ dag = SerializedDagModel.get_dag(dag_id)
+ if dag is None:
+ continue
+ dag_runs = DagRun.find(dag_id=dag_id, state=State.RUNNING)
+ for dag_run in dag_runs:
+ dag_run.dag = dag
+ _, callback_to_run = dag_run.update_state(execute_callbacks=False)
+ if callback_to_run:
+ self._send_dag_callbacks_to_processor(dag, callback_to_run)
+ self._paused_dag_without_running_dagruns.add(dag_id)
+ except Exception as e: # should not fail the scheduler
+ self.log.exception('Failed to update dag run state for paused dags due to %s', str(e))
+
def _run_scheduler_loop(self) -> None:
"""
The actual scheduler loop. The main steps in the loop are:
@@ -734,7 +805,7 @@ def _run_scheduler_loop(self) -> None:
:rtype: None
"""
- if not self.processor_agent:
+ if not self.processor_agent and not self._standalone_dag_processor:
raise ValueError("Processor agent is not started.")
is_unit_test: bool = conf.getboolean('core', 'unit_test_mode')
@@ -762,11 +833,12 @@ def _run_scheduler_loop(self) -> None:
conf.getfloat('scheduler', 'zombie_detection_interval', fallback=10.0),
self._find_zombies,
)
+ timers.call_regular_interval(60.0, self._update_dag_run_state_for_paused_dags)
for loop_count in itertools.count(start=1):
with Stats.timer() as timer:
- if self.using_sqlite:
+ if self.using_sqlite and self.processor_agent:
self.processor_agent.run_single_parsing_loop()
# For the sqlite case w/ 1 thread, wait until the processor
# is finished to avoid concurrent access to the DB.
@@ -779,8 +851,8 @@ def _run_scheduler_loop(self) -> None:
self.executor.heartbeat()
session.expunge_all()
num_finished_events = self._process_executor_events(session=session)
-
- self.processor_agent.heartbeat()
+ if self.processor_agent:
+ self.processor_agent.heartbeat()
# Heartbeat the scheduler periodically
self.heartbeat(only_if_necessary=True)
@@ -804,7 +876,7 @@ def _run_scheduler_loop(self) -> None:
loop_count,
)
break
- if self.processor_agent.done:
+ if self.processor_agent and self.processor_agent.done:
self.log.info(
"Exiting scheduler loop as requested DAG parse count (%d) has been reached after %d"
" scheduler loops",
@@ -830,7 +902,7 @@ def _do_scheduling(self, session) -> int:
By "next oldest", we mean hasn't been examined/scheduled in the most time.
- The reason we don't select all dagruns at once because the rows are selected with row locks, meaning
+ We don't select all dagruns at once, because the rows are selected with row locks, meaning
that only one scheduler can "process them", even it is waiting behind other dags. Increasing this
limit will allow more throughput for smaller DAGs but will likely slow down throughput for larger
(>500 tasks.) DAGs
@@ -838,7 +910,7 @@ def _do_scheduling(self, session) -> int:
- Then, via a Critical Section (locking the rows of the Pool model) we queue tasks, and then send them
to the executor.
- See docs of _critical_section_execute_task_instances for more.
+ See docs of _critical_section_enqueue_task_instances for more.
:return: Number of TIs enqueued in this iteration
:rtype: int
@@ -886,7 +958,7 @@ def _do_scheduling(self, session) -> int:
timer.start()
# Find anything TIs in state SCHEDULED, try to QUEUE it (send it to the executor)
- num_queued_tis = self._critical_section_execute_task_instances(session=session)
+ num_queued_tis = self._critical_section_enqueue_task_instances(session=session)
# Make sure we only sent this metric if we obtained the lock, otherwise we'll skew the
# metric, way down
@@ -929,24 +1001,15 @@ def _create_dag_runs(self, dag_models: Collection[DagModel], session: Session) -
# as DagModel.dag_id and DagModel.next_dagrun
# This list is used to verify if the DagRun already exist so that we don't attempt to create
# duplicate dag runs
-
- if session.bind.dialect.name == 'mssql':
- existing_dagruns_filter = or_(
- *(
- and_(
- DagRun.dag_id == dm.dag_id,
- DagRun.execution_date == dm.next_dagrun,
- )
- for dm in dag_models
- )
- )
- else:
- existing_dagruns_filter = tuple_(DagRun.dag_id, DagRun.execution_date).in_(
- [(dm.dag_id, dm.next_dagrun) for dm in dag_models]
- )
-
existing_dagruns = (
- session.query(DagRun.dag_id, DagRun.execution_date).filter(existing_dagruns_filter).all()
+ session.query(DagRun.dag_id, DagRun.execution_date)
+ .filter(
+ tuple_in_condition(
+ (DagRun.dag_id, DagRun.execution_date),
+ ((dm.dag_id, dm.next_dagrun) for dm in dag_models),
+ ),
+ )
+ .all()
)
active_runs_of_dags = defaultdict(
diff --git a/airflow/jobs/triggerer_job.py b/airflow/jobs/triggerer_job.py
index 2c0a0bd5bffee..ac7d22a6b1da9 100644
--- a/airflow/jobs/triggerer_job.py
+++ b/airflow/jobs/triggerer_job.py
@@ -26,7 +26,6 @@
from sqlalchemy import func
-from airflow.compat.asyncio import create_task
from airflow.configuration import conf
from airflow.jobs.base_job import BaseJob
from airflow.models.trigger import Trigger
@@ -236,7 +235,7 @@ async def arun(self):
The loop in here runs trigger addition/deletion/cleanup. Actual
triggers run in their own separate coroutines.
"""
- watchdog = create_task(self.block_watchdog())
+ watchdog = asyncio.create_task(self.block_watchdog())
last_status = time.time()
while not self.stop:
# Run core logic
@@ -263,7 +262,7 @@ async def create_triggers(self):
trigger_id, trigger_instance = self.to_create.popleft()
if trigger_id not in self.triggers:
self.triggers[trigger_id] = {
- "task": create_task(self.run_trigger(trigger_id, trigger_instance)),
+ "task": asyncio.create_task(self.run_trigger(trigger_id, trigger_instance)),
"name": f"{trigger_instance!r} (ID {trigger_id})",
"events": 0,
}
diff --git a/airflow/kubernetes/pod.py b/airflow/kubernetes/pod.py
index 733f18477779e..a5b6cde0e335b 100644
--- a/airflow/kubernetes/pod.py
+++ b/airflow/kubernetes/pod.py
@@ -17,17 +17,18 @@
# under the License.
"""
This module is deprecated.
-Please use :mod:`kubernetes.client.models for V1ResourceRequirements and Port.
+Please use :mod:`kubernetes.client.models` for `V1ResourceRequirements` and `Port`.
"""
# flake8: noqa
import warnings
with warnings.catch_warnings():
+ warnings.simplefilter("ignore", DeprecationWarning)
from airflow.providers.cncf.kubernetes.backcompat.pod import Port, Resources # noqa: autoflake
warnings.warn(
- "This module is deprecated. Please use `kubernetes.client.models for V1ResourceRequirements and Port.",
+ "This module is deprecated. Please use `kubernetes.client.models` for `V1ResourceRequirements` and `Port`.",
DeprecationWarning,
stacklevel=2,
)
diff --git a/airflow/kubernetes/pod_launcher_deprecated.py b/airflow/kubernetes/pod_launcher_deprecated.py
index 3367aa0cc23ef..97845dad51d5a 100644
--- a/airflow/kubernetes/pod_launcher_deprecated.py
+++ b/airflow/kubernetes/pod_launcher_deprecated.py
@@ -149,7 +149,8 @@ def monitor_pod(self, pod: V1Pod, get_logs: bool) -> Tuple[State, Optional[str]]
logs = self.read_pod_logs(pod, timestamps=True, since_seconds=read_logs_since_sec)
for line in logs:
timestamp, message = self.parse_log_line(line.decode('utf-8'))
- last_log_time = pendulum.parse(timestamp)
+ if timestamp:
+ last_log_time = pendulum.parse(timestamp)
self.log.info(message)
time.sleep(1)
@@ -174,7 +175,7 @@ def monitor_pod(self, pod: V1Pod, get_logs: bool) -> Tuple[State, Optional[str]]
time.sleep(2)
return self._task_status(self.read_pod(pod)), result
- def parse_log_line(self, line: str) -> Tuple[str, str]:
+ def parse_log_line(self, line: str) -> Tuple[Optional[str], str]:
"""
Parse K8s log line and returns the final state
@@ -184,7 +185,12 @@ def parse_log_line(self, line: str) -> Tuple[str, str]:
"""
split_at = line.find(' ')
if split_at == -1:
- raise Exception(f'Log not in "{{timestamp}} {{log}}" format. Got: {line}')
+ self.log.error(
+ "Error parsing timestamp (no timestamp in message: %r). "
+ "Will continue execution but won't update timestamp",
+ line,
+ )
+ return None, line
timestamp = line[:split_at]
message = line[split_at + 1 :].rstrip()
return timestamp, message
diff --git a/airflow/kubernetes/pod_runtime_info_env.py b/airflow/kubernetes/pod_runtime_info_env.py
index 5dbbd4249d211..a51f3b96fc39a 100644
--- a/airflow/kubernetes/pod_runtime_info_env.py
+++ b/airflow/kubernetes/pod_runtime_info_env.py
@@ -19,6 +19,7 @@
import warnings
with warnings.catch_warnings():
+ warnings.simplefilter("ignore", DeprecationWarning)
from airflow.providers.cncf.kubernetes.backcompat.pod_runtime_info_env import PodRuntimeInfoEnv # noqa
warnings.warn(
diff --git a/airflow/kubernetes/pod_template_file_examples/dags_in_image_template.yaml b/airflow/kubernetes/pod_template_file_examples/dags_in_image_template.yaml
index 103d5cf48cb8f..79a4e470564cc 100644
--- a/airflow/kubernetes/pod_template_file_examples/dags_in_image_template.yaml
+++ b/airflow/kubernetes/pod_template_file_examples/dags_in_image_template.yaml
@@ -34,7 +34,7 @@ spec:
secretKeyRef:
name: RELEASE-NAME-fernet-key
key: fernet-key
- - name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
+ - name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
name: RELEASE-NAME-airflow-metadata
diff --git a/airflow/kubernetes/pod_template_file_examples/dags_in_volume_template.yaml b/airflow/kubernetes/pod_template_file_examples/dags_in_volume_template.yaml
index cc4614996c760..237f14ced0820 100644
--- a/airflow/kubernetes/pod_template_file_examples/dags_in_volume_template.yaml
+++ b/airflow/kubernetes/pod_template_file_examples/dags_in_volume_template.yaml
@@ -34,7 +34,7 @@ spec:
secretKeyRef:
name: RELEASE-NAME-fernet-key
key: fernet-key
- - name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
+ - name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
name: RELEASE-NAME-airflow-metadata
diff --git a/airflow/kubernetes/pod_template_file_examples/git_sync_template.yaml b/airflow/kubernetes/pod_template_file_examples/git_sync_template.yaml
index 6d99761a1f4e1..021c113832adf 100644
--- a/airflow/kubernetes/pod_template_file_examples/git_sync_template.yaml
+++ b/airflow/kubernetes/pod_template_file_examples/git_sync_template.yaml
@@ -55,7 +55,7 @@ spec:
secretKeyRef:
name: RELEASE-NAME-fernet-key
key: fernet-key
- - name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
+ - name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
name: RELEASE-NAME-airflow-metadata
diff --git a/airflow/kubernetes/volume.py b/airflow/kubernetes/volume.py
index 90bd4c23bbd25..81b4fda3a15da 100644
--- a/airflow/kubernetes/volume.py
+++ b/airflow/kubernetes/volume.py
@@ -19,6 +19,7 @@
import warnings
with warnings.catch_warnings():
+ warnings.simplefilter("ignore", DeprecationWarning)
from airflow.providers.cncf.kubernetes.backcompat.volume import Volume # noqa: autoflake
warnings.warn(
diff --git a/airflow/kubernetes/volume_mount.py b/airflow/kubernetes/volume_mount.py
index aff5f30d5840e..f558425881752 100644
--- a/airflow/kubernetes/volume_mount.py
+++ b/airflow/kubernetes/volume_mount.py
@@ -19,6 +19,7 @@
import warnings
with warnings.catch_warnings():
+ warnings.simplefilter("ignore", DeprecationWarning)
from airflow.providers.cncf.kubernetes.backcompat.volume_mount import VolumeMount # noqa: autoflake
warnings.warn(
diff --git a/airflow/kubernetes_executor_templates/basic_template.yaml b/airflow/kubernetes_executor_templates/basic_template.yaml
index a6eb83f8ad8eb..579857a2443c1 100644
--- a/airflow/kubernetes_executor_templates/basic_template.yaml
+++ b/airflow/kubernetes_executor_templates/basic_template.yaml
@@ -38,7 +38,7 @@ spec:
secretKeyRef:
name: airflow-fernet-key
key: fernet-key
- - name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
+ - name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
name: airflow-airflow-metadata
diff --git a/airflow/logging_config.py b/airflow/logging_config.py
index 3b318c55e415c..645e53eb3efaa 100644
--- a/airflow/logging_config.py
+++ b/airflow/logging_config.py
@@ -17,7 +17,6 @@
# under the License.
#
import logging
-import sys
import warnings
from logging.config import dictConfig
@@ -103,22 +102,3 @@ def _get_handler(name):
f"Configured task_log_reader {task_log_reader!r} was not a handler of "
f"the 'airflow.task' logger."
)
-
-
-if sys.version_info < (3, 7):
- # Python 3.7 added this via https://bugs.python.org/issue30520 -- but Python 3.6 doesn't have this
- # support.
- import copyreg
-
- def _reduce_Logger(logger):
- if logging.getLogger(logger.name) is not logger:
- import pickle
-
- raise pickle.PicklingError('logger cannot be pickled')
- return logging.getLogger, (logger.name,)
-
- def _reduce_RootLogger(logger):
- return logging.getLogger, ()
-
- copyreg.pickle(logging.Logger, _reduce_Logger)
- copyreg.pickle(logging.RootLogger, _reduce_RootLogger)
diff --git a/airflow/migrations/db_types.py b/airflow/migrations/db_types.py
index 02f0542df321a..9b8f3e974ffd9 100644
--- a/airflow/migrations/db_types.py
+++ b/airflow/migrations/db_types.py
@@ -16,8 +16,6 @@
# specific language governing permissions and limitations
# under the License.
#
-import sys
-
import sqlalchemy as sa
from alembic import context
from lazy_object_proxy import Proxy
@@ -31,31 +29,14 @@
######################################
-def _mssql_use_date_time2():
- conn = context.get_bind()
- result = conn.execute(
- """SELECT CASE WHEN CONVERT(VARCHAR(128), SERVERPROPERTY ('productversion'))
- like '8%' THEN '2000' WHEN CONVERT(VARCHAR(128), SERVERPROPERTY ('productversion'))
- like '9%' THEN '2005' ELSE '2005Plus' END AS MajorVersion"""
- ).fetchone()
- mssql_version = result[0]
- return mssql_version not in ("2000", "2005")
-
-
-MSSQL_USE_DATE_TIME2 = Proxy(_mssql_use_date_time2)
-
-
def _mssql_TIMESTAMP():
from sqlalchemy.dialects import mssql
- if MSSQL_USE_DATE_TIME2:
+ class DATETIME2(mssql.DATETIME2):
+ def __init__(self, *args, precision=6, **kwargs):
+ super().__init__(*args, precision=precision, **kwargs)
- class DATETIME2(mssql.DATETIME2):
- def __init__(self, *args, precision=6, **kwargs):
- super().__init__(*args, precision=precision, **kwargs)
-
- return DATETIME2
- return mssql.DATETIME
+ return DATETIME2
def _mysql_TIMESTAMP():
@@ -100,9 +81,3 @@ def lazy_load():
return Proxy(lazy_load)
raise AttributeError(f"module {__name__} has no attribute {name}")
-
-
-if sys.version_info < (3, 7):
- from pep562 import Pep562
-
- Pep562(__name__)
diff --git a/airflow/migrations/db_types.pyi b/airflow/migrations/db_types.pyi
index 0a56b5f408174..bdde6f96924f9 100644
--- a/airflow/migrations/db_types.pyi
+++ b/airflow/migrations/db_types.pyi
@@ -24,5 +24,3 @@ TIMESTAMP = sa.TIMESTAMP
StringID = sa.String
"""String column type with correct DB collation applied"""
-
-MSSQL_USE_DATE_TIME2: bool
diff --git a/airflow/migrations/env.py b/airflow/migrations/env.py
index 60e67e8ebaa6b..58a8f7f4a0465 100644
--- a/airflow/migrations/env.py
+++ b/airflow/migrations/env.py
@@ -25,8 +25,8 @@
def include_object(_, name, type_, *args):
"""Filter objects for autogenerating revisions"""
- # Ignore _anything_ to do with Flask AppBuilder, Celery, or FlaskSession's tables
- if type_ == "table" and (name.startswith("ab_") or name.startswith("celery_") or name == "session"):
+ # Ignore _anything_ to do with Celery, or FlaskSession's tables
+ if type_ == "table" and (name.startswith("celery_") or name == "session"):
return False
else:
return True
diff --git a/airflow/migrations/utils.py b/airflow/migrations/utils.py
index 3294ced3887f1..5737fa950774d 100644
--- a/airflow/migrations/utils.py
+++ b/airflow/migrations/utils.py
@@ -16,6 +16,7 @@
# under the License.
from collections import defaultdict
+from contextlib import contextmanager
def get_mssql_table_constraints(conn, table_name):
@@ -34,10 +35,21 @@ def get_mssql_table_constraints(conn, table_name):
FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS AS tc
JOIN INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE AS ccu ON ccu.CONSTRAINT_NAME = tc.CONSTRAINT_NAME
WHERE tc.TABLE_NAME = '{table_name}' AND
- (tc.CONSTRAINT_TYPE = 'PRIMARY KEY' or UPPER(tc.CONSTRAINT_TYPE) = 'UNIQUE')
+ (tc.CONSTRAINT_TYPE = 'PRIMARY KEY' or UPPER(tc.CONSTRAINT_TYPE) = 'UNIQUE'
+ or UPPER(tc.CONSTRAINT_TYPE) = 'FOREIGN KEY')
"""
result = conn.execute(query).fetchall()
constraint_dict = defaultdict(lambda: defaultdict(list))
for constraint, constraint_type, col_name in result:
constraint_dict[constraint_type][constraint].append(col_name)
return constraint_dict
+
+
+@contextmanager
+def disable_sqlite_fkeys(op):
+ if op.get_bind().dialect.name == 'sqlite':
+ op.execute("PRAGMA foreign_keys=off")
+ yield op
+ op.execute("PRAGMA foreign_keys=on")
+ else:
+ yield op
diff --git a/airflow/migrations/versions/0001_e3a246e0dc1_current_schema.py b/airflow/migrations/versions/0001_1_5_0_current_schema.py
similarity index 100%
rename from airflow/migrations/versions/0001_e3a246e0dc1_current_schema.py
rename to airflow/migrations/versions/0001_1_5_0_current_schema.py
diff --git a/airflow/migrations/versions/0002_1507a7289a2f_create_is_encrypted.py b/airflow/migrations/versions/0002_1_5_0_create_is_encrypted.py
similarity index 100%
rename from airflow/migrations/versions/0002_1507a7289a2f_create_is_encrypted.py
rename to airflow/migrations/versions/0002_1_5_0_create_is_encrypted.py
diff --git a/airflow/migrations/versions/0003_13eb55f81627_for_compatibility.py b/airflow/migrations/versions/0003_1_5_0_for_compatibility.py
similarity index 100%
rename from airflow/migrations/versions/0003_13eb55f81627_for_compatibility.py
rename to airflow/migrations/versions/0003_1_5_0_for_compatibility.py
diff --git a/airflow/migrations/versions/0004_338e90f54d61_more_logging_into_task_isntance.py b/airflow/migrations/versions/0004_1_5_0_more_logging_into_task_isntance.py
similarity index 100%
rename from airflow/migrations/versions/0004_338e90f54d61_more_logging_into_task_isntance.py
rename to airflow/migrations/versions/0004_1_5_0_more_logging_into_task_isntance.py
diff --git a/airflow/migrations/versions/0005_52d714495f0_job_id_indices.py b/airflow/migrations/versions/0005_1_5_2_job_id_indices.py
similarity index 100%
rename from airflow/migrations/versions/0005_52d714495f0_job_id_indices.py
rename to airflow/migrations/versions/0005_1_5_2_job_id_indices.py
diff --git a/airflow/migrations/versions/0006_502898887f84_adding_extra_to_log.py b/airflow/migrations/versions/0006_1_6_0_adding_extra_to_log.py
similarity index 100%
rename from airflow/migrations/versions/0006_502898887f84_adding_extra_to_log.py
rename to airflow/migrations/versions/0006_1_6_0_adding_extra_to_log.py
diff --git a/airflow/migrations/versions/0007_1b38cef5b76e_add_dagrun.py b/airflow/migrations/versions/0007_1_6_0_add_dagrun.py
similarity index 100%
rename from airflow/migrations/versions/0007_1b38cef5b76e_add_dagrun.py
rename to airflow/migrations/versions/0007_1_6_0_add_dagrun.py
diff --git a/airflow/migrations/versions/0008_2e541a1dcfed_task_duration.py b/airflow/migrations/versions/0008_1_6_0_task_duration.py
similarity index 100%
rename from airflow/migrations/versions/0008_2e541a1dcfed_task_duration.py
rename to airflow/migrations/versions/0008_1_6_0_task_duration.py
diff --git a/airflow/migrations/versions/0009_40e67319e3a9_dagrun_config.py b/airflow/migrations/versions/0009_1_6_0_dagrun_config.py
similarity index 100%
rename from airflow/migrations/versions/0009_40e67319e3a9_dagrun_config.py
rename to airflow/migrations/versions/0009_1_6_0_dagrun_config.py
diff --git a/airflow/migrations/versions/0010_561833c1c74b_add_password_column_to_user.py b/airflow/migrations/versions/0010_1_6_2_add_password_column_to_user.py
similarity index 100%
rename from airflow/migrations/versions/0010_561833c1c74b_add_password_column_to_user.py
rename to airflow/migrations/versions/0010_1_6_2_add_password_column_to_user.py
diff --git a/airflow/migrations/versions/0011_4446e08588_dagrun_start_end.py b/airflow/migrations/versions/0011_1_6_2_dagrun_start_end.py
similarity index 100%
rename from airflow/migrations/versions/0011_4446e08588_dagrun_start_end.py
rename to airflow/migrations/versions/0011_1_6_2_dagrun_start_end.py
diff --git a/airflow/migrations/versions/0012_bbc73705a13e_add_notification_sent_column_to_sla_miss.py b/airflow/migrations/versions/0012_1_7_0_add_notification_sent_column_to_sla_miss.py
similarity index 100%
rename from airflow/migrations/versions/0012_bbc73705a13e_add_notification_sent_column_to_sla_miss.py
rename to airflow/migrations/versions/0012_1_7_0_add_notification_sent_column_to_sla_miss.py
diff --git a/airflow/migrations/versions/0013_bba5a7cfc896_add_a_column_to_track_the_encryption_.py b/airflow/migrations/versions/0013_1_7_0_add_a_column_to_track_the_encryption_.py
similarity index 100%
rename from airflow/migrations/versions/0013_bba5a7cfc896_add_a_column_to_track_the_encryption_.py
rename to airflow/migrations/versions/0013_1_7_0_add_a_column_to_track_the_encryption_.py
diff --git a/airflow/migrations/versions/0014_1968acfc09e3_add_is_encrypted_column_to_variable_.py b/airflow/migrations/versions/0014_1_7_0_add_is_encrypted_column_to_variable_.py
similarity index 100%
rename from airflow/migrations/versions/0014_1968acfc09e3_add_is_encrypted_column_to_variable_.py
rename to airflow/migrations/versions/0014_1_7_0_add_is_encrypted_column_to_variable_.py
diff --git a/airflow/migrations/versions/0015_2e82aab8ef20_rename_user_table.py b/airflow/migrations/versions/0015_1_7_1_rename_user_table.py
similarity index 100%
rename from airflow/migrations/versions/0015_2e82aab8ef20_rename_user_table.py
rename to airflow/migrations/versions/0015_1_7_1_rename_user_table.py
diff --git a/airflow/migrations/versions/0016_211e584da130_add_ti_state_index.py b/airflow/migrations/versions/0016_1_7_1_add_ti_state_index.py
similarity index 100%
rename from airflow/migrations/versions/0016_211e584da130_add_ti_state_index.py
rename to airflow/migrations/versions/0016_1_7_1_add_ti_state_index.py
diff --git a/airflow/migrations/versions/0017_64de9cddf6c9_add_task_fails_journal_table.py b/airflow/migrations/versions/0017_1_7_1_add_task_fails_journal_table.py
similarity index 100%
rename from airflow/migrations/versions/0017_64de9cddf6c9_add_task_fails_journal_table.py
rename to airflow/migrations/versions/0017_1_7_1_add_task_fails_journal_table.py
diff --git a/airflow/migrations/versions/0018_f2ca10b85618_add_dag_stats_table.py b/airflow/migrations/versions/0018_1_7_1_add_dag_stats_table.py
similarity index 100%
rename from airflow/migrations/versions/0018_f2ca10b85618_add_dag_stats_table.py
rename to airflow/migrations/versions/0018_1_7_1_add_dag_stats_table.py
diff --git a/airflow/migrations/versions/0019_4addfa1236f1_add_fractional_seconds_to_mysql_tables.py b/airflow/migrations/versions/0019_1_7_1_add_fractional_seconds_to_mysql_tables.py
similarity index 100%
rename from airflow/migrations/versions/0019_4addfa1236f1_add_fractional_seconds_to_mysql_tables.py
rename to airflow/migrations/versions/0019_1_7_1_add_fractional_seconds_to_mysql_tables.py
diff --git a/airflow/migrations/versions/0020_8504051e801b_xcom_dag_task_indices.py b/airflow/migrations/versions/0020_1_7_1_xcom_dag_task_indices.py
similarity index 100%
rename from airflow/migrations/versions/0020_8504051e801b_xcom_dag_task_indices.py
rename to airflow/migrations/versions/0020_1_7_1_xcom_dag_task_indices.py
diff --git a/airflow/migrations/versions/0021_5e7d17757c7a_add_pid_field_to_taskinstance.py b/airflow/migrations/versions/0021_1_7_1_add_pid_field_to_taskinstance.py
similarity index 100%
rename from airflow/migrations/versions/0021_5e7d17757c7a_add_pid_field_to_taskinstance.py
rename to airflow/migrations/versions/0021_1_7_1_add_pid_field_to_taskinstance.py
diff --git a/airflow/migrations/versions/0022_127d2bf2dfa7_add_dag_id_state_index_on_dag_run_table.py b/airflow/migrations/versions/0022_1_7_1_add_dag_id_state_index_on_dag_run_table.py
similarity index 100%
rename from airflow/migrations/versions/0022_127d2bf2dfa7_add_dag_id_state_index_on_dag_run_table.py
rename to airflow/migrations/versions/0022_1_7_1_add_dag_id_state_index_on_dag_run_table.py
diff --git a/airflow/migrations/versions/0023_cc1e65623dc7_add_max_tries_column_to_task_instance.py b/airflow/migrations/versions/0023_1_8_2_add_max_tries_column_to_task_instance.py
similarity index 100%
rename from airflow/migrations/versions/0023_cc1e65623dc7_add_max_tries_column_to_task_instance.py
rename to airflow/migrations/versions/0023_1_8_2_add_max_tries_column_to_task_instance.py
diff --git a/airflow/migrations/versions/0024_bdaa763e6c56_make_xcom_value_column_a_large_binary.py b/airflow/migrations/versions/0024_1_8_2_make_xcom_value_column_a_large_binary.py
similarity index 100%
rename from airflow/migrations/versions/0024_bdaa763e6c56_make_xcom_value_column_a_large_binary.py
rename to airflow/migrations/versions/0024_1_8_2_make_xcom_value_column_a_large_binary.py
diff --git a/airflow/migrations/versions/0025_947454bf1dff_add_ti_job_id_index.py b/airflow/migrations/versions/0025_1_8_2_add_ti_job_id_index.py
similarity index 100%
rename from airflow/migrations/versions/0025_947454bf1dff_add_ti_job_id_index.py
rename to airflow/migrations/versions/0025_1_8_2_add_ti_job_id_index.py
diff --git a/airflow/migrations/versions/0026_d2ae31099d61_increase_text_size_for_mysql.py b/airflow/migrations/versions/0026_1_8_2_increase_text_size_for_mysql.py
similarity index 100%
rename from airflow/migrations/versions/0026_d2ae31099d61_increase_text_size_for_mysql.py
rename to airflow/migrations/versions/0026_1_8_2_increase_text_size_for_mysql.py
diff --git a/airflow/migrations/versions/0027_0e2a74e0fc9f_add_time_zone_awareness.py b/airflow/migrations/versions/0027_1_10_0_add_time_zone_awareness.py
similarity index 100%
rename from airflow/migrations/versions/0027_0e2a74e0fc9f_add_time_zone_awareness.py
rename to airflow/migrations/versions/0027_1_10_0_add_time_zone_awareness.py
diff --git a/airflow/migrations/versions/0028_33ae817a1ff4_add_kubernetes_resource_checkpointing.py b/airflow/migrations/versions/0028_1_10_0_add_kubernetes_resource_checkpointing.py
similarity index 100%
rename from airflow/migrations/versions/0028_33ae817a1ff4_add_kubernetes_resource_checkpointing.py
rename to airflow/migrations/versions/0028_1_10_0_add_kubernetes_resource_checkpointing.py
diff --git a/airflow/migrations/versions/0029_27c6a30d7c24_add_executor_config_to_task_instance.py b/airflow/migrations/versions/0029_1_10_0_add_executor_config_to_task_instance.py
similarity index 100%
rename from airflow/migrations/versions/0029_27c6a30d7c24_add_executor_config_to_task_instance.py
rename to airflow/migrations/versions/0029_1_10_0_add_executor_config_to_task_instance.py
diff --git a/airflow/migrations/versions/0030_86770d1215c0_add_kubernetes_scheduler_uniqueness.py b/airflow/migrations/versions/0030_1_10_0_add_kubernetes_scheduler_uniqueness.py
similarity index 100%
rename from airflow/migrations/versions/0030_86770d1215c0_add_kubernetes_scheduler_uniqueness.py
rename to airflow/migrations/versions/0030_1_10_0_add_kubernetes_scheduler_uniqueness.py
diff --git a/airflow/migrations/versions/0031_05f30312d566_merge_heads.py b/airflow/migrations/versions/0031_1_10_0_merge_heads.py
similarity index 100%
rename from airflow/migrations/versions/0031_05f30312d566_merge_heads.py
rename to airflow/migrations/versions/0031_1_10_0_merge_heads.py
diff --git a/airflow/migrations/versions/0032_f23433877c24_fix_mysql_not_null_constraint.py b/airflow/migrations/versions/0032_1_10_0_fix_mysql_not_null_constraint.py
similarity index 100%
rename from airflow/migrations/versions/0032_f23433877c24_fix_mysql_not_null_constraint.py
rename to airflow/migrations/versions/0032_1_10_0_fix_mysql_not_null_constraint.py
diff --git a/airflow/migrations/versions/0033_856955da8476_fix_sqlite_foreign_key.py b/airflow/migrations/versions/0033_1_10_0_fix_sqlite_foreign_key.py
similarity index 100%
rename from airflow/migrations/versions/0033_856955da8476_fix_sqlite_foreign_key.py
rename to airflow/migrations/versions/0033_1_10_0_fix_sqlite_foreign_key.py
diff --git a/airflow/migrations/versions/0034_9635ae0956e7_index_taskfail.py b/airflow/migrations/versions/0034_1_10_0_index_taskfail.py
similarity index 100%
rename from airflow/migrations/versions/0034_9635ae0956e7_index_taskfail.py
rename to airflow/migrations/versions/0034_1_10_0_index_taskfail.py
diff --git a/airflow/migrations/versions/0035_dd25f486b8ea_add_idx_log_dag.py b/airflow/migrations/versions/0035_1_10_2_add_idx_log_dag.py
similarity index 100%
rename from airflow/migrations/versions/0035_dd25f486b8ea_add_idx_log_dag.py
rename to airflow/migrations/versions/0035_1_10_2_add_idx_log_dag.py
diff --git a/airflow/migrations/versions/0036_bf00311e1990_add_index_to_taskinstance.py b/airflow/migrations/versions/0036_1_10_2_add_index_to_taskinstance.py
similarity index 100%
rename from airflow/migrations/versions/0036_bf00311e1990_add_index_to_taskinstance.py
rename to airflow/migrations/versions/0036_1_10_2_add_index_to_taskinstance.py
diff --git a/airflow/migrations/versions/0037_0a2a5b66e19d_add_task_reschedule_table.py b/airflow/migrations/versions/0037_1_10_2_add_task_reschedule_table.py
similarity index 100%
rename from airflow/migrations/versions/0037_0a2a5b66e19d_add_task_reschedule_table.py
rename to airflow/migrations/versions/0037_1_10_2_add_task_reschedule_table.py
diff --git a/airflow/migrations/versions/0038_03bc53e68815_add_sm_dag_index.py b/airflow/migrations/versions/0038_1_10_2_add_sm_dag_index.py
similarity index 100%
rename from airflow/migrations/versions/0038_03bc53e68815_add_sm_dag_index.py
rename to airflow/migrations/versions/0038_1_10_2_add_sm_dag_index.py
diff --git a/airflow/migrations/versions/0039_41f5f12752f8_add_superuser_field.py b/airflow/migrations/versions/0039_1_10_2_add_superuser_field.py
similarity index 100%
rename from airflow/migrations/versions/0039_41f5f12752f8_add_superuser_field.py
rename to airflow/migrations/versions/0039_1_10_2_add_superuser_field.py
diff --git a/airflow/migrations/versions/0040_c8ffec048a3b_add_fields_to_dag.py b/airflow/migrations/versions/0040_1_10_3_add_fields_to_dag.py
similarity index 100%
rename from airflow/migrations/versions/0040_c8ffec048a3b_add_fields_to_dag.py
rename to airflow/migrations/versions/0040_1_10_3_add_fields_to_dag.py
diff --git a/airflow/migrations/versions/0041_dd4ecb8fbee3_add_schedule_interval_to_dag.py b/airflow/migrations/versions/0041_1_10_3_add_schedule_interval_to_dag.py
similarity index 100%
rename from airflow/migrations/versions/0041_dd4ecb8fbee3_add_schedule_interval_to_dag.py
rename to airflow/migrations/versions/0041_1_10_3_add_schedule_interval_to_dag.py
diff --git a/airflow/migrations/versions/0042_939bb1e647c8_task_reschedule_fk_on_cascade_delete.py b/airflow/migrations/versions/0042_1_10_3_task_reschedule_fk_on_cascade_delete.py
similarity index 100%
rename from airflow/migrations/versions/0042_939bb1e647c8_task_reschedule_fk_on_cascade_delete.py
rename to airflow/migrations/versions/0042_1_10_3_task_reschedule_fk_on_cascade_delete.py
diff --git a/airflow/migrations/versions/0043_6e96a59344a4_make_taskinstance_pool_not_nullable.py b/airflow/migrations/versions/0043_1_10_4_make_taskinstance_pool_not_nullable.py
similarity index 100%
rename from airflow/migrations/versions/0043_6e96a59344a4_make_taskinstance_pool_not_nullable.py
rename to airflow/migrations/versions/0043_1_10_4_make_taskinstance_pool_not_nullable.py
diff --git a/airflow/migrations/versions/0044_d38e04c12aa2_add_serialized_dag_table.py b/airflow/migrations/versions/0044_1_10_7_add_serialized_dag_table.py
similarity index 100%
rename from airflow/migrations/versions/0044_d38e04c12aa2_add_serialized_dag_table.py
rename to airflow/migrations/versions/0044_1_10_7_add_serialized_dag_table.py
diff --git a/airflow/migrations/versions/0045_b3b105409875_add_root_dag_id_to_dag.py b/airflow/migrations/versions/0045_1_10_7_add_root_dag_id_to_dag.py
similarity index 91%
rename from airflow/migrations/versions/0045_b3b105409875_add_root_dag_id_to_dag.py
rename to airflow/migrations/versions/0045_1_10_7_add_root_dag_id_to_dag.py
index 3d184e6b9bd1d..f879450369ab8 100644
--- a/airflow/migrations/versions/0045_b3b105409875_add_root_dag_id_to_dag.py
+++ b/airflow/migrations/versions/0045_1_10_7_add_root_dag_id_to_dag.py
@@ -27,6 +27,8 @@
import sqlalchemy as sa
from alembic import op
+from airflow.migrations.db_types import StringID
+
# revision identifiers, used by Alembic.
revision = 'b3b105409875'
down_revision = 'd38e04c12aa2'
@@ -37,7 +39,7 @@
def upgrade():
"""Apply Add ``root_dag_id`` to ``DAG``"""
- op.add_column('dag', sa.Column('root_dag_id', sa.String(length=250), nullable=True))
+ op.add_column('dag', sa.Column('root_dag_id', StringID(), nullable=True))
op.create_index('idx_root_dag_id', 'dag', ['root_dag_id'], unique=False)
diff --git a/airflow/migrations/versions/0046_74effc47d867_change_datetime_to_datetime2_6_on_mssql_.py b/airflow/migrations/versions/0046_1_10_5_change_datetime_to_datetime2_6_on_mssql_.py
similarity index 100%
rename from airflow/migrations/versions/0046_74effc47d867_change_datetime_to_datetime2_6_on_mssql_.py
rename to airflow/migrations/versions/0046_1_10_5_change_datetime_to_datetime2_6_on_mssql_.py
diff --git a/airflow/migrations/versions/0047_004c1210f153_increase_queue_name_size_limit.py b/airflow/migrations/versions/0047_1_10_4_increase_queue_name_size_limit.py
similarity index 100%
rename from airflow/migrations/versions/0047_004c1210f153_increase_queue_name_size_limit.py
rename to airflow/migrations/versions/0047_1_10_4_increase_queue_name_size_limit.py
diff --git a/airflow/migrations/versions/0048_a56c9515abdc_remove_dag_stat_table.py b/airflow/migrations/versions/0048_1_10_3_remove_dag_stat_table.py
similarity index 100%
rename from airflow/migrations/versions/0048_a56c9515abdc_remove_dag_stat_table.py
rename to airflow/migrations/versions/0048_1_10_3_remove_dag_stat_table.py
diff --git a/airflow/migrations/versions/0049_08364691d074_merge_heads.py b/airflow/migrations/versions/0049_1_10_7_merge_heads.py
similarity index 100%
rename from airflow/migrations/versions/0049_08364691d074_merge_heads.py
rename to airflow/migrations/versions/0049_1_10_7_merge_heads.py
diff --git a/airflow/migrations/versions/0050_fe461863935f_increase_length_for_connection_password.py b/airflow/migrations/versions/0050_1_10_7_increase_length_for_connection_password.py
similarity index 100%
rename from airflow/migrations/versions/0050_fe461863935f_increase_length_for_connection_password.py
rename to airflow/migrations/versions/0050_1_10_7_increase_length_for_connection_password.py
diff --git a/airflow/migrations/versions/0051_7939bcff74ba_add_dagtags_table.py b/airflow/migrations/versions/0051_1_10_8_add_dagtags_table.py
similarity index 100%
rename from airflow/migrations/versions/0051_7939bcff74ba_add_dagtags_table.py
rename to airflow/migrations/versions/0051_1_10_8_add_dagtags_table.py
diff --git a/airflow/migrations/versions/0052_a4c2fd67d16b_add_pool_slots_field_to_task_instance.py b/airflow/migrations/versions/0052_1_10_10_add_pool_slots_field_to_task_instance.py
similarity index 100%
rename from airflow/migrations/versions/0052_a4c2fd67d16b_add_pool_slots_field_to_task_instance.py
rename to airflow/migrations/versions/0052_1_10_10_add_pool_slots_field_to_task_instance.py
diff --git a/airflow/migrations/versions/0053_852ae6c715af_add_rendered_task_instance_fields_table.py b/airflow/migrations/versions/0053_1_10_10_add_rendered_task_instance_fields_table.py
similarity index 100%
rename from airflow/migrations/versions/0053_852ae6c715af_add_rendered_task_instance_fields_table.py
rename to airflow/migrations/versions/0053_1_10_10_add_rendered_task_instance_fields_table.py
diff --git a/airflow/migrations/versions/0054_952da73b5eff_add_dag_code_table.py b/airflow/migrations/versions/0054_1_10_10_add_dag_code_table.py
similarity index 100%
rename from airflow/migrations/versions/0054_952da73b5eff_add_dag_code_table.py
rename to airflow/migrations/versions/0054_1_10_10_add_dag_code_table.py
diff --git a/airflow/migrations/versions/0055_a66efa278eea_add_precision_to_execution_date_in_mysql.py b/airflow/migrations/versions/0055_1_10_11_add_precision_to_execution_date_in_mysql.py
similarity index 100%
rename from airflow/migrations/versions/0055_a66efa278eea_add_precision_to_execution_date_in_mysql.py
rename to airflow/migrations/versions/0055_1_10_11_add_precision_to_execution_date_in_mysql.py
diff --git a/airflow/migrations/versions/0056_da3f683c3a5a_add_dag_hash_column_to_serialized_dag_.py b/airflow/migrations/versions/0056_1_10_12_add_dag_hash_column_to_serialized_dag_.py
similarity index 100%
rename from airflow/migrations/versions/0056_da3f683c3a5a_add_dag_hash_column_to_serialized_dag_.py
rename to airflow/migrations/versions/0056_1_10_12_add_dag_hash_column_to_serialized_dag_.py
diff --git a/airflow/migrations/versions/0057_92c57b58940d_add_fab_tables.py b/airflow/migrations/versions/0057_1_10_13_add_fab_tables.py
similarity index 100%
rename from airflow/migrations/versions/0057_92c57b58940d_add_fab_tables.py
rename to airflow/migrations/versions/0057_1_10_13_add_fab_tables.py
diff --git a/airflow/migrations/versions/0058_03afc6b6f902_increase_length_of_fab_ab_view_menu_.py b/airflow/migrations/versions/0058_1_10_13_increase_length_of_fab_ab_view_menu_.py
similarity index 100%
rename from airflow/migrations/versions/0058_03afc6b6f902_increase_length_of_fab_ab_view_menu_.py
rename to airflow/migrations/versions/0058_1_10_13_increase_length_of_fab_ab_view_menu_.py
diff --git a/airflow/migrations/versions/0059_cf5dc11e79ad_drop_user_and_chart.py b/airflow/migrations/versions/0059_2_0_0_drop_user_and_chart.py
similarity index 100%
rename from airflow/migrations/versions/0059_cf5dc11e79ad_drop_user_and_chart.py
rename to airflow/migrations/versions/0059_2_0_0_drop_user_and_chart.py
diff --git a/airflow/migrations/versions/0060_bbf4a7ad0465_remove_id_column_from_xcom.py b/airflow/migrations/versions/0060_2_0_0_remove_id_column_from_xcom.py
similarity index 100%
rename from airflow/migrations/versions/0060_bbf4a7ad0465_remove_id_column_from_xcom.py
rename to airflow/migrations/versions/0060_2_0_0_remove_id_column_from_xcom.py
diff --git a/airflow/migrations/versions/0061_b25a55525161_increase_length_of_pool_name.py b/airflow/migrations/versions/0061_2_0_0_increase_length_of_pool_name.py
similarity index 100%
rename from airflow/migrations/versions/0061_b25a55525161_increase_length_of_pool_name.py
rename to airflow/migrations/versions/0061_2_0_0_increase_length_of_pool_name.py
diff --git a/airflow/migrations/versions/0062_3c20cacc0044_add_dagrun_run_type.py b/airflow/migrations/versions/0062_2_0_0_add_dagrun_run_type.py
similarity index 100%
rename from airflow/migrations/versions/0062_3c20cacc0044_add_dagrun_run_type.py
rename to airflow/migrations/versions/0062_2_0_0_add_dagrun_run_type.py
diff --git a/airflow/migrations/versions/0063_8f966b9c467a_set_conn_type_as_non_nullable.py b/airflow/migrations/versions/0063_2_0_0_set_conn_type_as_non_nullable.py
similarity index 100%
rename from airflow/migrations/versions/0063_8f966b9c467a_set_conn_type_as_non_nullable.py
rename to airflow/migrations/versions/0063_2_0_0_set_conn_type_as_non_nullable.py
diff --git a/airflow/migrations/versions/0064_8d48763f6d53_add_unique_constraint_to_conn_id.py b/airflow/migrations/versions/0064_2_0_0_add_unique_constraint_to_conn_id.py
similarity index 100%
rename from airflow/migrations/versions/0064_8d48763f6d53_add_unique_constraint_to_conn_id.py
rename to airflow/migrations/versions/0064_2_0_0_add_unique_constraint_to_conn_id.py
diff --git a/airflow/migrations/versions/0065_e38be357a868_update_schema_for_smart_sensor.py b/airflow/migrations/versions/0065_2_0_0_update_schema_for_smart_sensor.py
similarity index 100%
rename from airflow/migrations/versions/0065_e38be357a868_update_schema_for_smart_sensor.py
rename to airflow/migrations/versions/0065_2_0_0_update_schema_for_smart_sensor.py
diff --git a/airflow/migrations/versions/0066_b247b1e3d1ed_add_queued_by_job_id_to_ti.py b/airflow/migrations/versions/0066_2_0_0_add_queued_by_job_id_to_ti.py
similarity index 100%
rename from airflow/migrations/versions/0066_b247b1e3d1ed_add_queued_by_job_id_to_ti.py
rename to airflow/migrations/versions/0066_2_0_0_add_queued_by_job_id_to_ti.py
diff --git a/airflow/migrations/versions/0067_e1a11ece99cc_add_external_executor_id_to_ti.py b/airflow/migrations/versions/0067_2_0_0_add_external_executor_id_to_ti.py
similarity index 100%
rename from airflow/migrations/versions/0067_e1a11ece99cc_add_external_executor_id_to_ti.py
rename to airflow/migrations/versions/0067_2_0_0_add_external_executor_id_to_ti.py
diff --git a/airflow/migrations/versions/0068_bef4f3d11e8b_drop_kuberesourceversion_and_.py b/airflow/migrations/versions/0068_2_0_0_drop_kuberesourceversion_and_.py
similarity index 100%
rename from airflow/migrations/versions/0068_bef4f3d11e8b_drop_kuberesourceversion_and_.py
rename to airflow/migrations/versions/0068_2_0_0_drop_kuberesourceversion_and_.py
diff --git a/airflow/migrations/versions/0069_98271e7606e2_add_scheduling_decision_to_dagrun_and_.py b/airflow/migrations/versions/0069_2_0_0_add_scheduling_decision_to_dagrun_and_.py
similarity index 100%
rename from airflow/migrations/versions/0069_98271e7606e2_add_scheduling_decision_to_dagrun_and_.py
rename to airflow/migrations/versions/0069_2_0_0_add_scheduling_decision_to_dagrun_and_.py
diff --git a/airflow/migrations/versions/0070_52d53670a240_fix_mssql_exec_date_rendered_task_instance.py b/airflow/migrations/versions/0070_2_0_0_fix_mssql_exec_date_rendered_task_instance.py
similarity index 100%
rename from airflow/migrations/versions/0070_52d53670a240_fix_mssql_exec_date_rendered_task_instance.py
rename to airflow/migrations/versions/0070_2_0_0_fix_mssql_exec_date_rendered_task_instance.py
diff --git a/airflow/migrations/versions/0071_364159666cbd_add_job_id_to_dagrun_table.py b/airflow/migrations/versions/0071_2_0_0_add_job_id_to_dagrun_table.py
similarity index 100%
rename from airflow/migrations/versions/0071_364159666cbd_add_job_id_to_dagrun_table.py
rename to airflow/migrations/versions/0071_2_0_0_add_job_id_to_dagrun_table.py
diff --git a/airflow/migrations/versions/0072_45ba3f1493b9_add_k8s_yaml_to_rendered_templates.py b/airflow/migrations/versions/0072_2_0_0_add_k8s_yaml_to_rendered_templates.py
similarity index 100%
rename from airflow/migrations/versions/0072_45ba3f1493b9_add_k8s_yaml_to_rendered_templates.py
rename to airflow/migrations/versions/0072_2_0_0_add_k8s_yaml_to_rendered_templates.py
diff --git a/airflow/migrations/versions/0073_849da589634d_prefix_dag_permissions.py b/airflow/migrations/versions/0073_2_0_0_prefix_dag_permissions.py
similarity index 100%
rename from airflow/migrations/versions/0073_849da589634d_prefix_dag_permissions.py
rename to airflow/migrations/versions/0073_2_0_0_prefix_dag_permissions.py
diff --git a/airflow/migrations/versions/0074_2c6edca13270_resource_based_permissions.py b/airflow/migrations/versions/0074_2_0_0_resource_based_permissions.py
similarity index 100%
rename from airflow/migrations/versions/0074_2c6edca13270_resource_based_permissions.py
rename to airflow/migrations/versions/0074_2_0_0_resource_based_permissions.py
diff --git a/airflow/migrations/versions/0075_61ec73d9401f_add_description_field_to_connection.py b/airflow/migrations/versions/0075_2_0_0_add_description_field_to_connection.py
similarity index 100%
rename from airflow/migrations/versions/0075_61ec73d9401f_add_description_field_to_connection.py
rename to airflow/migrations/versions/0075_2_0_0_add_description_field_to_connection.py
diff --git a/airflow/migrations/versions/0076_64a7d6477aae_fix_description_field_in_connection_to_.py b/airflow/migrations/versions/0076_2_0_0_fix_description_field_in_connection_to_.py
similarity index 100%
rename from airflow/migrations/versions/0076_64a7d6477aae_fix_description_field_in_connection_to_.py
rename to airflow/migrations/versions/0076_2_0_0_fix_description_field_in_connection_to_.py
diff --git a/airflow/migrations/versions/0077_e959f08ac86c_change_field_in_dagcode_to_mediumtext_.py b/airflow/migrations/versions/0077_2_0_0_change_field_in_dagcode_to_mediumtext_.py
similarity index 100%
rename from airflow/migrations/versions/0077_e959f08ac86c_change_field_in_dagcode_to_mediumtext_.py
rename to airflow/migrations/versions/0077_2_0_0_change_field_in_dagcode_to_mediumtext_.py
diff --git a/airflow/migrations/versions/0078_82b7c48c147f_remove_can_read_permission_on_config_.py b/airflow/migrations/versions/0078_2_0_1_remove_can_read_permission_on_config_.py
similarity index 100%
rename from airflow/migrations/versions/0078_82b7c48c147f_remove_can_read_permission_on_config_.py
rename to airflow/migrations/versions/0078_2_0_1_remove_can_read_permission_on_config_.py
diff --git a/airflow/migrations/versions/0079_449b4072c2da_increase_size_of_connection_extra_field_.py b/airflow/migrations/versions/0079_2_0_2_increase_size_of_connection_extra_field_.py
similarity index 100%
rename from airflow/migrations/versions/0079_449b4072c2da_increase_size_of_connection_extra_field_.py
rename to airflow/migrations/versions/0079_2_0_2_increase_size_of_connection_extra_field_.py
diff --git a/airflow/migrations/versions/0080_8646922c8a04_change_default_pool_slots_to_1.py b/airflow/migrations/versions/0080_2_0_2_change_default_pool_slots_to_1.py
similarity index 100%
rename from airflow/migrations/versions/0080_8646922c8a04_change_default_pool_slots_to_1.py
rename to airflow/migrations/versions/0080_2_0_2_change_default_pool_slots_to_1.py
diff --git a/airflow/migrations/versions/0081_2e42bb497a22_rename_last_scheduler_run_column.py b/airflow/migrations/versions/0081_2_0_2_rename_last_scheduler_run_column.py
similarity index 100%
rename from airflow/migrations/versions/0081_2e42bb497a22_rename_last_scheduler_run_column.py
rename to airflow/migrations/versions/0081_2_0_2_rename_last_scheduler_run_column.py
diff --git a/airflow/migrations/versions/0082_90d1635d7b86_increase_pool_name_size_in_taskinstance.py b/airflow/migrations/versions/0082_2_1_0_increase_pool_name_size_in_taskinstance.py
similarity index 100%
rename from airflow/migrations/versions/0082_90d1635d7b86_increase_pool_name_size_in_taskinstance.py
rename to airflow/migrations/versions/0082_2_1_0_increase_pool_name_size_in_taskinstance.py
diff --git a/airflow/migrations/versions/0083_e165e7455d70_add_description_field_to_variable.py b/airflow/migrations/versions/0083_2_1_0_add_description_field_to_variable.py
similarity index 100%
rename from airflow/migrations/versions/0083_e165e7455d70_add_description_field_to_variable.py
rename to airflow/migrations/versions/0083_2_1_0_add_description_field_to_variable.py
diff --git a/airflow/migrations/versions/0084_a13f7613ad25_resource_based_permissions_for_default_.py b/airflow/migrations/versions/0084_2_1_0_resource_based_permissions_for_default_.py
similarity index 100%
rename from airflow/migrations/versions/0084_a13f7613ad25_resource_based_permissions_for_default_.py
rename to airflow/migrations/versions/0084_2_1_0_resource_based_permissions_for_default_.py
diff --git a/airflow/migrations/versions/0085_97cdd93827b8_add_queued_at_column_to_dagrun_table.py b/airflow/migrations/versions/0085_2_1_3_add_queued_at_column_to_dagrun_table.py
similarity index 100%
rename from airflow/migrations/versions/0085_97cdd93827b8_add_queued_at_column_to_dagrun_table.py
rename to airflow/migrations/versions/0085_2_1_3_add_queued_at_column_to_dagrun_table.py
diff --git a/airflow/migrations/versions/0086_092435bf5d12_add_max_active_runs_column_to_dagmodel_.py b/airflow/migrations/versions/0086_2_1_4_add_max_active_runs_column_to_dagmodel_.py
similarity index 100%
rename from airflow/migrations/versions/0086_092435bf5d12_add_max_active_runs_column_to_dagmodel_.py
rename to airflow/migrations/versions/0086_2_1_4_add_max_active_runs_column_to_dagmodel_.py
diff --git a/airflow/migrations/versions/0087_ccde3e26fe78_add_index_on_state_dag_id_for_queued_.py b/airflow/migrations/versions/0087_2_1_4_add_index_on_state_dag_id_for_queued_.py
similarity index 100%
rename from airflow/migrations/versions/0087_ccde3e26fe78_add_index_on_state_dag_id_for_queued_.py
rename to airflow/migrations/versions/0087_2_1_4_add_index_on_state_dag_id_for_queued_.py
diff --git a/airflow/migrations/versions/0088_83f031fd9f1c_improve_mssql_compatibility.py b/airflow/migrations/versions/0088_2_2_0_improve_mssql_compatibility.py
similarity index 93%
rename from airflow/migrations/versions/0088_83f031fd9f1c_improve_mssql_compatibility.py
rename to airflow/migrations/versions/0088_2_2_0_improve_mssql_compatibility.py
index 66fe513713fc3..9fdb56ac9bf16 100644
--- a/airflow/migrations/versions/0088_83f031fd9f1c_improve_mssql_compatibility.py
+++ b/airflow/migrations/versions/0088_2_2_0_improve_mssql_compatibility.py
@@ -30,7 +30,7 @@
from alembic import op
from sqlalchemy.dialects import mssql
-from airflow.migrations.db_types import MSSQL_USE_DATE_TIME2, TIMESTAMP
+from airflow.migrations.db_types import TIMESTAMP
# revision identifiers, used by Alembic.
revision = '83f031fd9f1c'
@@ -135,21 +135,12 @@ def recreate_mssql_ts_column(conn, op, table_name, column_name):
def alter_mssql_datetime_column(conn, op, table_name, column_name, nullable):
"""Update the datetime column to datetime2(6)"""
- if MSSQL_USE_DATE_TIME2:
- op.alter_column(
- table_name=table_name,
- column_name=column_name,
- type_=mssql.DATETIME2(precision=6),
- nullable=nullable,
- )
-
-
-def alter_mssql_datetime2_column(conn, op, table_name, column_name, nullable):
- """Update the datetime2(6) column to datetime"""
- if MSSQL_USE_DATE_TIME2:
- op.alter_column(
- table_name=table_name, column_name=column_name, type_=mssql.DATETIME, nullable=nullable
- )
+ op.alter_column(
+ table_name=table_name,
+ column_name=column_name,
+ type_=mssql.DATETIME2(precision=6),
+ nullable=nullable,
+ )
def upgrade():
@@ -199,7 +190,6 @@ def downgrade():
conn = op.get_bind()
if conn.dialect.name != 'mssql':
return
- alter_mssql_datetime2_column(conn, op, 'serialized_dag', 'last_updated', False)
op.alter_column(table_name="xcom", column_name="timestamp", type_=TIMESTAMP, nullable=True)
with op.batch_alter_table('task_reschedule') as task_reschedule_batch_op:
task_reschedule_batch_op.alter_column(column_name='end_date', type_=TIMESTAMP, nullable=True)
diff --git a/airflow/migrations/versions/0089_e9304a3141f0_make_xcom_pkey_columns_non_nullable.py b/airflow/migrations/versions/0089_2_2_0_make_xcom_pkey_columns_non_nullable.py
similarity index 100%
rename from airflow/migrations/versions/0089_e9304a3141f0_make_xcom_pkey_columns_non_nullable.py
rename to airflow/migrations/versions/0089_2_2_0_make_xcom_pkey_columns_non_nullable.py
diff --git a/airflow/migrations/versions/0090_30867afad44a_rename_concurrency_column_in_dag_table_.py b/airflow/migrations/versions/0090_2_2_0_rename_concurrency_column_in_dag_table_.py
similarity index 100%
rename from airflow/migrations/versions/0090_30867afad44a_rename_concurrency_column_in_dag_table_.py
rename to airflow/migrations/versions/0090_2_2_0_rename_concurrency_column_in_dag_table_.py
diff --git a/airflow/migrations/versions/0091_54bebd308c5f_add_trigger_table_and_task_info.py b/airflow/migrations/versions/0091_2_2_0_add_trigger_table_and_task_info.py
similarity index 100%
rename from airflow/migrations/versions/0091_54bebd308c5f_add_trigger_table_and_task_info.py
rename to airflow/migrations/versions/0091_2_2_0_add_trigger_table_and_task_info.py
diff --git a/airflow/migrations/versions/0092_142555e44c17_add_data_interval_start_end_to_dagmodel_and_dagrun.py b/airflow/migrations/versions/0092_2_2_0_add_data_interval_start_end_to_dagmodel_and_dagrun.py
similarity index 100%
rename from airflow/migrations/versions/0092_142555e44c17_add_data_interval_start_end_to_dagmodel_and_dagrun.py
rename to airflow/migrations/versions/0092_2_2_0_add_data_interval_start_end_to_dagmodel_and_dagrun.py
diff --git a/airflow/migrations/versions/0093_7b2661a43ba3_taskinstance_keyed_to_dagrun.py b/airflow/migrations/versions/0093_2_2_0_taskinstance_keyed_to_dagrun.py
similarity index 100%
rename from airflow/migrations/versions/0093_7b2661a43ba3_taskinstance_keyed_to_dagrun.py
rename to airflow/migrations/versions/0093_2_2_0_taskinstance_keyed_to_dagrun.py
diff --git a/airflow/migrations/versions/0094_be2bfac3da23_add_has_import_errors_column_to_dagmodel.py b/airflow/migrations/versions/0094_2_2_3_add_has_import_errors_column_to_dagmodel.py
similarity index 100%
rename from airflow/migrations/versions/0094_be2bfac3da23_add_has_import_errors_column_to_dagmodel.py
rename to airflow/migrations/versions/0094_2_2_3_add_has_import_errors_column_to_dagmodel.py
diff --git a/airflow/migrations/versions/0095_c381b21cb7e4_add_session_table_to_db.py b/airflow/migrations/versions/0095_2_2_4_add_session_table_to_db.py
similarity index 100%
rename from airflow/migrations/versions/0095_c381b21cb7e4_add_session_table_to_db.py
rename to airflow/migrations/versions/0095_2_2_4_add_session_table_to_db.py
diff --git a/airflow/migrations/versions/0096_587bdf053233_adding_index_for_dag_id_in_job.py b/airflow/migrations/versions/0096_2_2_4_adding_index_for_dag_id_in_job.py
similarity index 98%
rename from airflow/migrations/versions/0096_587bdf053233_adding_index_for_dag_id_in_job.py
rename to airflow/migrations/versions/0096_2_2_4_adding_index_for_dag_id_in_job.py
index 693984b6edbfc..2d08d7f922a58 100644
--- a/airflow/migrations/versions/0096_587bdf053233_adding_index_for_dag_id_in_job.py
+++ b/airflow/migrations/versions/0096_2_2_4_adding_index_for_dag_id_in_job.py
@@ -31,7 +31,7 @@
down_revision = 'c381b21cb7e4'
branch_labels = None
depends_on = None
-airflow_version = '2.3.0'
+airflow_version = '2.2.4'
def upgrade():
diff --git a/airflow/migrations/versions/0097_5e3ec427fdd3_increase_length_of_email_and_username.py b/airflow/migrations/versions/0097_2_3_0_increase_length_of_email_and_username.py
similarity index 98%
rename from airflow/migrations/versions/0097_5e3ec427fdd3_increase_length_of_email_and_username.py
rename to airflow/migrations/versions/0097_2_3_0_increase_length_of_email_and_username.py
index 968b5f0162368..be028e83378fe 100644
--- a/airflow/migrations/versions/0097_5e3ec427fdd3_increase_length_of_email_and_username.py
+++ b/airflow/migrations/versions/0097_2_3_0_increase_length_of_email_and_username.py
@@ -80,4 +80,3 @@ def downgrade():
batch_op.alter_column('username', type_=sa.String(64))
batch_op.create_unique_constraint(None, ['username'])
batch_op.alter_column('email', type_=sa.String(64))
- batch_op.create_unique_constraint(None, ['email'])
diff --git a/airflow/migrations/versions/0098_786e3737b18f_added_timetable_description_column.py b/airflow/migrations/versions/0098_2_3_0_added_timetable_description_column.py
similarity index 100%
rename from airflow/migrations/versions/0098_786e3737b18f_added_timetable_description_column.py
rename to airflow/migrations/versions/0098_2_3_0_added_timetable_description_column.py
diff --git a/airflow/migrations/versions/0099_f9da662e7089_add_task_log_filename_template_model.py b/airflow/migrations/versions/0099_2_3_0_add_task_log_filename_template_model.py
similarity index 94%
rename from airflow/migrations/versions/0099_f9da662e7089_add_task_log_filename_template_model.py
rename to airflow/migrations/versions/0099_2_3_0_add_task_log_filename_template_model.py
index a2511a398da08..05fc2767fa9fe 100644
--- a/airflow/migrations/versions/0099_f9da662e7089_add_task_log_filename_template_model.py
+++ b/airflow/migrations/versions/0099_2_3_0_add_task_log_filename_template_model.py
@@ -26,6 +26,7 @@
from alembic import op
from sqlalchemy import Column, ForeignKey, Integer, Text
+from airflow.migrations.utils import disable_sqlite_fkeys
from airflow.utils.sqlalchemy import UtcDateTime
# Revision identifiers, used by Alembic.
@@ -50,7 +51,8 @@ def upgrade():
Integer,
ForeignKey("log_template.id", name="task_instance_log_template_id_fkey", ondelete="NO ACTION"),
)
- with op.batch_alter_table("dag_run") as batch_op:
+
+ with disable_sqlite_fkeys(op), op.batch_alter_table("dag_run") as batch_op:
batch_op.add_column(dag_run_log_filename_id)
diff --git a/airflow/migrations/versions/0100_e655c0453f75_add_taskmap_and_map_id_on_taskinstance.py b/airflow/migrations/versions/0100_2_3_0_add_taskmap_and_map_id_on_taskinstance.py
similarity index 100%
rename from airflow/migrations/versions/0100_e655c0453f75_add_taskmap_and_map_id_on_taskinstance.py
rename to airflow/migrations/versions/0100_2_3_0_add_taskmap_and_map_id_on_taskinstance.py
diff --git a/airflow/migrations/versions/0101_a3bcd0914482_add_data_compressed_to_serialized_dag.py b/airflow/migrations/versions/0101_2_3_0_add_data_compressed_to_serialized_dag.py
similarity index 100%
rename from airflow/migrations/versions/0101_a3bcd0914482_add_data_compressed_to_serialized_dag.py
rename to airflow/migrations/versions/0101_2_3_0_add_data_compressed_to_serialized_dag.py
diff --git a/airflow/migrations/versions/0102_c306b5b5ae4a_switch_xcom_table_to_use_run_id.py b/airflow/migrations/versions/0102_2_3_0_switch_xcom_table_to_use_run_id.py
similarity index 95%
rename from airflow/migrations/versions/0102_c306b5b5ae4a_switch_xcom_table_to_use_run_id.py
rename to airflow/migrations/versions/0102_2_3_0_switch_xcom_table_to_use_run_id.py
index 633f545c6e51c..f987a1fe12606 100644
--- a/airflow/migrations/versions/0102_c306b5b5ae4a_switch_xcom_table_to_use_run_id.py
+++ b/airflow/migrations/versions/0102_2_3_0_switch_xcom_table_to_use_run_id.py
@@ -122,7 +122,13 @@ def upgrade():
with op.batch_alter_table("xcom") as batch_op:
batch_op.create_primary_key("xcom_pkey", ["dag_run_id", "task_id", "map_index", "key"])
batch_op.create_index("idx_xcom_key", ["key"])
- batch_op.create_index("idx_xcom_ti_id", ["dag_id", "run_id", "task_id", "map_index"])
+ batch_op.create_foreign_key(
+ "xcom_task_instance_fkey",
+ "task_instance",
+ ["dag_id", "task_id", "run_id", "map_index"],
+ ["dag_id", "task_id", "run_id", "map_index"],
+ ondelete="CASCADE",
+ )
def downgrade():
diff --git a/airflow/migrations/versions/0103_c97c2ab6aa23_add_callback_request_table.py b/airflow/migrations/versions/0103_2_3_0_add_callback_request_table.py
similarity index 100%
rename from airflow/migrations/versions/0103_c97c2ab6aa23_add_callback_request_table.py
rename to airflow/migrations/versions/0103_2_3_0_add_callback_request_table.py
diff --git a/airflow/migrations/versions/0104_4eaab2fe6582_migrate_rtif_to_use_run_id_and_map_index.py b/airflow/migrations/versions/0104_2_3_0_migrate_rtif_to_use_run_id_and_map_index.py
similarity index 100%
rename from airflow/migrations/versions/0104_4eaab2fe6582_migrate_rtif_to_use_run_id_and_map_index.py
rename to airflow/migrations/versions/0104_2_3_0_migrate_rtif_to_use_run_id_and_map_index.py
diff --git a/airflow/migrations/versions/0105_2_3_0_add_map_index_to_taskfail.py b/airflow/migrations/versions/0105_2_3_0_add_map_index_to_taskfail.py
new file mode 100644
index 0000000000000..303e1eda58241
--- /dev/null
+++ b/airflow/migrations/versions/0105_2_3_0_add_map_index_to_taskfail.py
@@ -0,0 +1,158 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Add map_index to TaskFail
+
+Drop index idx_task_fail_dag_task_date
+Add run_id and map_index
+Drop execution_date
+Add FK `task_fail_ti_fkey`: TF -> TI ([dag_id, task_id, run_id, map_index])
+
+
+Revision ID: 48925b2719cb
+Revises: 4eaab2fe6582
+Create Date: 2022-03-14 10:31:11.220720
+"""
+
+from typing import List
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.sql import ColumnElement, Update, and_, select
+
+from airflow.migrations.db_types import TIMESTAMP, StringID
+
+# revision identifiers, used by Alembic.
+revision = '48925b2719cb'
+down_revision = '4eaab2fe6582'
+branch_labels = None
+depends_on = None
+airflow_version = '2.3.0'
+
+ID_LEN = 250
+
+
+def tables():
+ global task_instance, task_fail, dag_run
+ metadata = sa.MetaData()
+ task_instance = sa.Table(
+ 'task_instance',
+ metadata,
+ sa.Column('task_id', StringID()),
+ sa.Column('dag_id', StringID()),
+ sa.Column('run_id', StringID()),
+ sa.Column('map_index', sa.Integer(), server_default='-1'),
+ sa.Column('execution_date', TIMESTAMP),
+ )
+ task_fail = sa.Table(
+ 'task_fail',
+ metadata,
+ sa.Column('dag_id', StringID()),
+ sa.Column('task_id', StringID()),
+ sa.Column('run_id', StringID()),
+ sa.Column('map_index', StringID()),
+ sa.Column('execution_date', TIMESTAMP),
+ )
+ dag_run = sa.Table(
+ 'dag_run',
+ metadata,
+ sa.Column('dag_id', StringID()),
+ sa.Column('run_id', StringID()),
+ sa.Column('execution_date', TIMESTAMP),
+ )
+
+
+def _update_value_from_dag_run(
+ dialect_name: str,
+ target_table: sa.Table,
+ target_column: ColumnElement,
+ join_columns: List[str],
+) -> Update:
+ """
+ Grabs a value from the source table ``dag_run`` and updates target with this value.
+ :param dialect_name: dialect in use
+ :param target_table: the table to update
+ :param target_column: the column to update
+ """
+ # for run_id: dag_id, execution_date
+ # otherwise: dag_id, run_id
+ condition_list = [getattr(dag_run.c, x) == getattr(target_table.c, x) for x in join_columns]
+ condition = and_(*condition_list)
+
+ if dialect_name == "sqlite":
+ # Most SQLite versions don't support multi table update (and SQLA doesn't know about it anyway), so we
+ # need to do a Correlated subquery update
+ sub_q = select([dag_run.c[target_column.name]]).where(condition)
+
+ return target_table.update().values({target_column: sub_q})
+ else:
+ return target_table.update().where(condition).values({target_column: dag_run.c[target_column.name]})
+
+
+def upgrade():
+ tables()
+ dialect_name = op.get_bind().dialect.name
+
+ op.drop_index('idx_task_fail_dag_task_date', table_name='task_fail')
+
+ with op.batch_alter_table('task_fail') as batch_op:
+ batch_op.add_column(sa.Column('map_index', sa.Integer(), server_default='-1', nullable=False))
+ batch_op.add_column(sa.Column('run_id', type_=StringID(), nullable=True))
+
+ update_query = _update_value_from_dag_run(
+ dialect_name=dialect_name,
+ target_table=task_fail,
+ target_column=task_fail.c.run_id,
+ join_columns=['dag_id', 'execution_date'],
+ )
+ op.execute(update_query)
+ with op.batch_alter_table('task_fail') as batch_op:
+ batch_op.alter_column('run_id', existing_type=StringID(), existing_nullable=True, nullable=False)
+ batch_op.drop_column('execution_date')
+ batch_op.create_foreign_key(
+ 'task_fail_ti_fkey',
+ 'task_instance',
+ ['dag_id', 'task_id', 'run_id', 'map_index'],
+ ['dag_id', 'task_id', 'run_id', 'map_index'],
+ ondelete='CASCADE',
+ )
+
+
+def downgrade():
+ tables()
+ dialect_name = op.get_bind().dialect.name
+ op.add_column('task_fail', sa.Column('execution_date', TIMESTAMP, nullable=True))
+ update_query = _update_value_from_dag_run(
+ dialect_name=dialect_name,
+ target_table=task_fail,
+ target_column=task_fail.c.execution_date,
+ join_columns=['dag_id', 'run_id'],
+ )
+ op.execute(update_query)
+ with op.batch_alter_table('task_fail', copy_from=task_fail) as batch_op:
+ batch_op.alter_column('execution_date', existing_type=TIMESTAMP, nullable=False)
+ if dialect_name != 'sqlite':
+ batch_op.drop_constraint('task_fail_ti_fkey', type_='foreignkey')
+ batch_op.drop_column('map_index', mssql_drop_default=True)
+ batch_op.drop_column('run_id')
+ op.create_index(
+ index_name='idx_task_fail_dag_task_date',
+ table_name='task_fail',
+ columns=['dag_id', 'task_id', 'execution_date'],
+ unique=False,
+ )
diff --git a/airflow/migrations/versions/0106_2_3_0_update_migration_for_fab_tables_to_add_missing_constraints.py b/airflow/migrations/versions/0106_2_3_0_update_migration_for_fab_tables_to_add_missing_constraints.py
new file mode 100644
index 0000000000000..81f6f1f34492b
--- /dev/null
+++ b/airflow/migrations/versions/0106_2_3_0_update_migration_for_fab_tables_to_add_missing_constraints.py
@@ -0,0 +1,108 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Update migration for FAB tables to add missing constraints
+
+Revision ID: 909884dea523
+Revises: 48925b2719cb
+Create Date: 2022-03-21 08:33:01.635688
+
+"""
+
+
+import sqlalchemy as sa
+from alembic import op
+
+from airflow.migrations.utils import get_mssql_table_constraints
+
+# revision identifiers, used by Alembic.
+revision = '909884dea523'
+down_revision = '48925b2719cb'
+branch_labels = None
+depends_on = None
+airflow_version = '2.3.0'
+
+
+def upgrade():
+ """Apply Update migration for FAB tables to add missing constraints"""
+ conn = op.get_bind()
+ if conn.dialect.name == 'sqlite':
+ op.execute('PRAGMA foreign_keys=OFF')
+ with op.batch_alter_table('ab_view_menu', schema=None) as batch_op:
+ batch_op.create_unique_constraint(batch_op.f('ab_view_menu_name_uq'), ['name'])
+ op.execute('PRAGMA foreign_keys=ON')
+ elif conn.dialect.name == 'mysql':
+ with op.batch_alter_table('ab_register_user', schema=None) as batch_op:
+ batch_op.alter_column('username', existing_type=sa.String(256), nullable=False)
+ batch_op.alter_column('email', existing_type=sa.String(256), nullable=False)
+ with op.batch_alter_table('ab_user', schema=None) as batch_op:
+ batch_op.alter_column('username', existing_type=sa.String(256), nullable=False)
+ batch_op.alter_column('email', existing_type=sa.String(256), nullable=False)
+ elif conn.dialect.name == 'mssql':
+ with op.batch_alter_table('ab_register_user') as batch_op:
+ # Drop the unique constraint on username and email
+ constraints = get_mssql_table_constraints(conn, 'ab_register_user')
+ for k, _ in constraints.get('UNIQUE').items():
+ batch_op.drop_constraint(k, type_='unique')
+ batch_op.alter_column('username', existing_type=sa.String(256), nullable=False)
+ batch_op.create_unique_constraint(None, ['username'])
+ batch_op.alter_column('email', existing_type=sa.String(256), nullable=False)
+ with op.batch_alter_table('ab_user') as batch_op:
+ # Drop the unique constraint on username and email
+ constraints = get_mssql_table_constraints(conn, 'ab_user')
+ for k, _ in constraints.get('UNIQUE').items():
+ batch_op.drop_constraint(k, type_='unique')
+ batch_op.alter_column('username', existing_type=sa.String(256), nullable=False)
+ batch_op.create_unique_constraint(None, ['username'])
+ batch_op.alter_column('email', existing_type=sa.String(256), nullable=False)
+ batch_op.create_unique_constraint(None, ['email'])
+
+
+def downgrade():
+ """Unapply Update migration for FAB tables to add missing constraints"""
+ conn = op.get_bind()
+ if conn.dialect.name == 'sqlite':
+ op.execute('PRAGMA foreign_keys=OFF')
+ with op.batch_alter_table('ab_view_menu', schema=None) as batch_op:
+ batch_op.drop_constraint('ab_view_menu_name_uq', type_='unique')
+ op.execute('PRAGMA foreign_keys=ON')
+ elif conn.dialect.name == 'mysql':
+ with op.batch_alter_table('ab_user', schema=None) as batch_op:
+ batch_op.alter_column('email', existing_type=sa.String(256), nullable=True)
+ batch_op.alter_column('username', existing_type=sa.String(256), nullable=True, unique=True)
+ with op.batch_alter_table('ab_register_user', schema=None) as batch_op:
+ batch_op.alter_column('email', existing_type=sa.String(256), nullable=True)
+ batch_op.alter_column('username', existing_type=sa.String(256), nullable=True, unique=True)
+ elif conn.dialect.name == 'mssql':
+ with op.batch_alter_table('ab_register_user') as batch_op:
+ # Drop the unique constraint on username and email
+ constraints = get_mssql_table_constraints(conn, 'ab_register_user')
+ for k, _ in constraints.get('UNIQUE').items():
+ batch_op.drop_constraint(k, type_='unique')
+ batch_op.alter_column('username', existing_type=sa.String(256), nullable=False, unique=True)
+ batch_op.create_unique_constraint(None, ['username'])
+ batch_op.alter_column('email', existing_type=sa.String(256), nullable=False, unique=True)
+ with op.batch_alter_table('ab_user') as batch_op:
+ # Drop the unique constraint on username and email
+ constraints = get_mssql_table_constraints(conn, 'ab_user')
+ for k, _ in constraints.get('UNIQUE').items():
+ batch_op.drop_constraint(k, type_='unique')
+ batch_op.alter_column('username', existing_type=sa.String(256), nullable=True)
+ batch_op.create_unique_constraint(None, ['username'])
+ batch_op.alter_column('email', existing_type=sa.String(256), nullable=True, unique=True)
+ batch_op.create_unique_constraint(None, ['email'])
diff --git a/tests/providers/google/cloud/operators/test_stackdriver_system.py b/airflow/migrations/versions/0107_2_3_0_add_map_index_to_log.py
similarity index 57%
rename from tests/providers/google/cloud/operators/test_stackdriver_system.py
rename to airflow/migrations/versions/0107_2_3_0_add_map_index_to_log.py
index f49aa4101cfc2..f48cd90cfa1a3 100644
--- a/tests/providers/google/cloud/operators/test_stackdriver_system.py
+++ b/airflow/migrations/versions/0107_2_3_0_add_map_index_to_log.py
@@ -16,21 +16,29 @@
# specific language governing permissions and limitations
# under the License.
-import pytest
+"""Add map_index to Log.
-from tests.providers.google.cloud.utils.gcp_authenticator import GCP_STACKDRIVER
-from tests.test_utils.gcp_system_helpers import CLOUD_DAG_FOLDER, GoogleSystemTest, provide_gcp_context
+Revision ID: 75d5ed6c2b43
+Revises: 909884dea523
+Create Date: 2022-03-15 16:35:54.816863
+"""
+from alembic import op
+from sqlalchemy import Column, Integer
+# Revision identifiers, used by Alembic.
+revision = "75d5ed6c2b43"
+down_revision = "909884dea523"
+branch_labels = None
+depends_on = None
+airflow_version = '2.3.0'
-@pytest.mark.backend("mysql", "postgres")
-@pytest.mark.credential_file(GCP_STACKDRIVER)
-class GCPStackdriverExampleDagSystemTest(GoogleSystemTest):
- def setUp(self):
- super().setUp()
- @provide_gcp_context(GCP_STACKDRIVER)
- def test_run_example_dag(self):
- self.run_dag("example_stackdriver", CLOUD_DAG_FOLDER)
+def upgrade():
+ """Add map_index to Log."""
+ op.add_column("log", Column("map_index", Integer))
- def tearDown(self):
- super().tearDown()
+
+def downgrade():
+ """Remove map_index from Log."""
+ with op.batch_alter_table("log") as batch_op:
+ batch_op.drop_column("map_index")
diff --git a/airflow/migrations/versions/0108_2_3_0_default_dag_view_grid.py b/airflow/migrations/versions/0108_2_3_0_default_dag_view_grid.py
new file mode 100644
index 0000000000000..03a9f43c3ad62
--- /dev/null
+++ b/airflow/migrations/versions/0108_2_3_0_default_dag_view_grid.py
@@ -0,0 +1,55 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Update dag.default_view to grid
+
+Revision ID: b1b348e02d07
+Revises: 75d5ed6c2b43
+Create Date: 2022-04-19 17:25:00.872220
+
+"""
+
+from alembic import op
+from sqlalchemy import String
+from sqlalchemy.sql import column, table
+
+# revision identifiers, used by Alembic.
+revision = 'b1b348e02d07'
+down_revision = '75d5ed6c2b43'
+branch_labels = None
+depends_on = '75d5ed6c2b43'
+airflow_version = '2.3.0'
+
+
+dag = table('dag', column('default_view', String))
+
+
+def upgrade():
+ op.execute(
+ dag.update()
+ .where(dag.c.default_view == op.inline_literal('tree'))
+ .values({'default_view': op.inline_literal('grid')})
+ )
+
+
+def downgrade():
+ op.execute(
+ dag.update()
+ .where(dag.c.default_view == op.inline_literal('grid'))
+ .values({'default_view': op.inline_literal('tree')})
+ )
diff --git a/tests/providers/google/cloud/operators/test_kubernetes_engine_system.py b/airflow/migrations/versions/0109_2_3_1_add_index_for_event_in_log.py
similarity index 56%
rename from tests/providers/google/cloud/operators/test_kubernetes_engine_system.py
rename to airflow/migrations/versions/0109_2_3_1_add_index_for_event_in_log.py
index 5f1cfada23dcf..2023a3c294fee 100644
--- a/tests/providers/google/cloud/operators/test_kubernetes_engine_system.py
+++ b/airflow/migrations/versions/0109_2_3_1_add_index_for_event_in_log.py
@@ -15,21 +15,30 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-import pytest
-from tests.providers.google.cloud.utils.gcp_authenticator import GCP_GKE_KEY
-from tests.test_utils.gcp_system_helpers import CLOUD_DAG_FOLDER, GoogleSystemTest, provide_gcp_context
+"""Add index for ``event`` column in ``log`` table.
+Revision ID: 1de7bc13c950
+Revises: b1b348e02d07
+Create Date: 2022-05-10 18:18:43.484829
-@pytest.mark.backend("mysql", "postgres")
-@pytest.mark.credential_file(GCP_GKE_KEY)
-class KubernetesEngineExampleDagTest(GoogleSystemTest):
- def setUp(self):
- super().setUp()
+"""
- @provide_gcp_context(GCP_GKE_KEY)
- def test_run_example_gcp_gke(self):
- self.run_dag('example_gcp_gke', CLOUD_DAG_FOLDER)
+from alembic import op
- def tearDown(self):
- super().tearDown()
+# revision identifiers, used by Alembic.
+revision = '1de7bc13c950'
+down_revision = 'b1b348e02d07'
+branch_labels = None
+depends_on = None
+airflow_version = '2.3.1'
+
+
+def upgrade():
+ """Apply Add index for ``event`` column in ``log`` table."""
+ op.create_index('idx_log_event', 'log', ['event'], unique=False)
+
+
+def downgrade():
+ """Unapply Add index for ``event`` column in ``log`` table."""
+ op.drop_index('idx_log_event', table_name='log')
diff --git a/airflow/migrations/versions/0110_2_3_2_add_cascade_to_dag_tag_foreignkey.py b/airflow/migrations/versions/0110_2_3_2_add_cascade_to_dag_tag_foreignkey.py
new file mode 100644
index 0000000000000..55d9e9754e532
--- /dev/null
+++ b/airflow/migrations/versions/0110_2_3_2_add_cascade_to_dag_tag_foreignkey.py
@@ -0,0 +1,84 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Add cascade to dag_tag foreign key
+
+Revision ID: 3c94c427fdf6
+Revises: 1de7bc13c950
+Create Date: 2022-05-03 09:47:41.957710
+
+"""
+
+from alembic import op
+
+from airflow.migrations.utils import get_mssql_table_constraints
+
+# revision identifiers, used by Alembic.
+revision = '3c94c427fdf6'
+down_revision = '1de7bc13c950'
+branch_labels = None
+depends_on = None
+airflow_version = '2.3.2'
+
+
+def upgrade():
+ """Apply Add cascade to dag_tag foreignkey"""
+ conn = op.get_bind()
+ if conn.dialect.name == 'sqlite':
+ naming_convention = {
+ "fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
+ }
+ with op.batch_alter_table(
+ 'dag_tag', naming_convention=naming_convention, recreate='always'
+ ) as batch_op:
+ batch_op.drop_constraint('fk_dag_tag_dag_id_dag', type_='foreignkey')
+ batch_op.create_foreign_key(
+ "dag_tag_dag_id_fkey", 'dag', ['dag_id'], ['dag_id'], ondelete='CASCADE'
+ )
+ else:
+ with op.batch_alter_table('dag_tag') as batch_op:
+ if conn.dialect.name == 'mssql':
+ constraints = get_mssql_table_constraints(conn, 'dag_tag')
+ Fk, _ = constraints['FOREIGN KEY'].popitem()
+ batch_op.drop_constraint(Fk, type_='foreignkey')
+ if conn.dialect.name == 'postgresql':
+ batch_op.drop_constraint('dag_tag_dag_id_fkey', type_='foreignkey')
+ if conn.dialect.name == 'mysql':
+ batch_op.drop_constraint('dag_tag_ibfk_1', type_='foreignkey')
+
+ batch_op.create_foreign_key(
+ "dag_tag_dag_id_fkey", 'dag', ['dag_id'], ['dag_id'], ondelete='CASCADE'
+ )
+
+
+def downgrade():
+ """Unapply Add cascade to dag_tag foreignkey"""
+ conn = op.get_bind()
+ if conn.dialect.name == 'sqlite':
+ with op.batch_alter_table('dag_tag') as batch_op:
+ batch_op.drop_constraint('dag_tag_dag_id_fkey', type_='foreignkey')
+ batch_op.create_foreign_key("fk_dag_tag_dag_id_dag", 'dag', ['dag_id'], ['dag_id'])
+ else:
+ with op.batch_alter_table('dag_tag') as batch_op:
+ batch_op.drop_constraint('dag_tag_dag_id_fkey', type_='foreignkey')
+ batch_op.create_foreign_key(
+ None,
+ 'dag',
+ ['dag_id'],
+ ['dag_id'],
+ )
diff --git a/airflow/models/abstractoperator.py b/airflow/models/abstractoperator.py
index 18e22a8d2cc30..cb566ea6f9557 100644
--- a/airflow/models/abstractoperator.py
+++ b/airflow/models/abstractoperator.py
@@ -18,9 +18,22 @@
import datetime
import inspect
-from typing import TYPE_CHECKING, Any, Callable, Collection, Dict, Iterable, List, Optional, Set, Type, Union
-
-from sqlalchemy.orm import Session
+from typing import (
+ TYPE_CHECKING,
+ Any,
+ Callable,
+ ClassVar,
+ Collection,
+ Dict,
+ FrozenSet,
+ Iterable,
+ List,
+ Optional,
+ Sequence,
+ Set,
+ Type,
+ Union,
+)
from airflow.compat.functools import cached_property
from airflow.configuration import conf
@@ -37,22 +50,31 @@
if TYPE_CHECKING:
import jinja2 # Slow import.
+ from sqlalchemy.orm import Session
from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
from airflow.models.dag import DAG
from airflow.models.operator import Operator
from airflow.models.taskinstance import TaskInstance
-DEFAULT_OWNER: str = conf.get("operators", "default_owner")
+DEFAULT_OWNER: str = conf.get_mandatory_value("operators", "default_owner")
DEFAULT_POOL_SLOTS: int = 1
DEFAULT_PRIORITY_WEIGHT: int = 1
-DEFAULT_QUEUE: str = conf.get("operators", "default_queue")
+DEFAULT_QUEUE: str = conf.get_mandatory_value("operators", "default_queue")
+DEFAULT_IGNORE_FIRST_DEPENDS_ON_PAST: bool = conf.getboolean(
+ "scheduler", "ignore_first_depends_on_past_by_default"
+)
DEFAULT_RETRIES: int = conf.getint("core", "default_task_retries", fallback=0)
-DEFAULT_RETRY_DELAY: datetime.timedelta = datetime.timedelta(seconds=300)
+DEFAULT_RETRY_DELAY: datetime.timedelta = datetime.timedelta(
+ seconds=conf.getint("core", "default_task_retry_delay", fallback=300)
+)
DEFAULT_WEIGHT_RULE: WeightRule = WeightRule(
conf.get("core", "default_task_weight_rule", fallback=WeightRule.DOWNSTREAM)
)
DEFAULT_TRIGGER_RULE: TriggerRule = TriggerRule.ALL_SUCCESS
+DEFAULT_TASK_EXECUTION_TIMEOUT: Optional[datetime.timedelta] = conf.gettimedelta(
+ "core", "default_task_execution_timeout"
+)
class AbstractOperator(LoggingMixin, DAGNode):
@@ -68,7 +90,7 @@ class AbstractOperator(LoggingMixin, DAGNode):
:meta private:
"""
- operator_class: Union[str, Type["BaseOperator"]]
+ operator_class: Union[Type["BaseOperator"], Dict[str, Any]]
weight_rule: str
priority_weight: int
@@ -78,11 +100,30 @@ class AbstractOperator(LoggingMixin, DAGNode):
# For derived classes to define which fields will get jinjaified.
template_fields: Collection[str]
# Defines which files extensions to look for in the templated fields.
- template_ext: Collection[str]
+ template_ext: Sequence[str]
owner: str
task_id: str
+ HIDE_ATTRS_FROM_UI: ClassVar[FrozenSet[str]] = frozenset(
+ (
+ 'log',
+ 'dag', # We show dag_id, don't need to show this too
+ 'node_id', # Duplicates task_id
+ 'task_group', # Doesn't have a useful repr, no point showing in UI
+ 'inherits_from_empty_operator', # impl detail
+ # For compatibility with TG, for operators these are just the current task, no point showing
+ 'roots',
+ 'leaves',
+ # These lists are already shown via *_task_ids
+ 'upstream_list',
+ 'downstream_list',
+ # Not useful, implementation detail, already shown elsewhere
+ 'global_operator_extra_link_dict',
+ 'operator_extra_link_dict',
+ )
+ )
+
def get_dag(self) -> "Optional[DAG]":
raise NotImplementedError()
@@ -91,7 +132,7 @@ def task_type(self) -> str:
raise NotImplementedError()
@property
- def inherits_from_dummy_operator(self) -> bool:
+ def inherits_from_empty_operator(self) -> bool:
raise NotImplementedError()
@property
@@ -115,7 +156,7 @@ def get_template_env(self) -> "jinja2.Environment":
dag = self.get_dag()
if dag:
- return dag.get_template_env()
+ return dag.get_template_env(force_sandboxed=False)
return SandboxedEnvironment(cache_size=0)
def prepare_template(self) -> None:
@@ -144,8 +185,8 @@ def resolve_template_files(self) -> None:
if isinstance(item, str) and any(item.endswith(ext) for ext in self.template_ext):
try:
content[i] = env.loader.get_source(env, item)[0] # type: ignore
- except Exception as e:
- self.log.exception(e)
+ except Exception:
+ self.log.exception("Failed to get source %s", item)
self.prepare_template()
def get_direct_relative_ids(self, upstream: bool = False) -> Set[str]:
@@ -293,7 +334,7 @@ def _do_render_template_fields(
jinja_env: "jinja2.Environment",
seen_oids: Set,
*,
- session: Session = NEW_SESSION,
+ session: "Session" = NEW_SESSION,
) -> None:
for attr_name in template_fields:
try:
@@ -305,29 +346,14 @@ def _do_render_template_fields(
)
if not value:
continue
- rendered_content = self._render_template_field(
- attr_name,
+ rendered_content = self.render_template(
value,
context,
jinja_env,
seen_oids,
- session=session,
)
setattr(parent, attr_name, rendered_content)
- def _render_template_field(
- self,
- key: str,
- value: Any,
- context: Context,
- jinja_env: Optional["jinja2.Environment"] = None,
- seen_oids: Optional[Set] = None,
- *,
- session: Session,
- ) -> Any:
- """Override point for MappedOperator to perform further resolution."""
- return self.render_template(value, context, jinja_env, seen_oids)
-
def render_template(
self,
content: Any,
diff --git a/airflow/models/base.py b/airflow/models/base.py
index c586f2019ca28..478bd904eb83c 100644
--- a/airflow/models/base.py
+++ b/airflow/models/base.py
@@ -23,7 +23,7 @@
from airflow.configuration import conf
-SQL_ALCHEMY_SCHEMA = conf.get("core", "SQL_ALCHEMY_SCHEMA")
+SQL_ALCHEMY_SCHEMA = conf.get("database", "SQL_ALCHEMY_SCHEMA")
metadata = (
None if not SQL_ALCHEMY_SCHEMA or SQL_ALCHEMY_SCHEMA.isspace() else MetaData(schema=SQL_ALCHEMY_SCHEMA)
@@ -35,7 +35,7 @@
def get_id_collation_args():
"""Get SQLAlchemy args to use for COLLATION"""
- collation = conf.get('core', 'sql_engine_collation_for_ids', fallback=None)
+ collation = conf.get('database', 'sql_engine_collation_for_ids', fallback=None)
if collation:
return {'collation': collation}
else:
@@ -49,7 +49,7 @@ def get_id_collation_args():
#
# We cannot use session/dialect as at this point we are trying to determine the right connection
# parameters, so we use the connection
- conn = conf.get('core', 'sql_alchemy_conn', fallback='')
+ conn = conf.get('database', 'sql_alchemy_conn', fallback='')
if conn.startswith('mysql') or conn.startswith("mariadb"):
return {'collation': 'utf8mb3_bin'}
return {}
diff --git a/airflow/models/baseoperator.py b/airflow/models/baseoperator.py
index 5208901c81e3b..3208c435769b3 100644
--- a/airflow/models/baseoperator.py
+++ b/airflow/models/baseoperator.py
@@ -18,6 +18,7 @@
"""Base operator for all operators."""
import abc
import collections
+import collections.abc
import contextlib
import copy
import functools
@@ -58,12 +59,14 @@
from airflow.exceptions import AirflowException, TaskDeferred
from airflow.lineage import apply_lineage, prepare_lineage
from airflow.models.abstractoperator import (
+ DEFAULT_IGNORE_FIRST_DEPENDS_ON_PAST,
DEFAULT_OWNER,
DEFAULT_POOL_SLOTS,
DEFAULT_PRIORITY_WEIGHT,
DEFAULT_QUEUE,
DEFAULT_RETRIES,
DEFAULT_RETRY_DELAY,
+ DEFAULT_TASK_EXECUTION_TIMEOUT,
DEFAULT_TRIGGER_RULE,
DEFAULT_WEIGHT_RULE,
AbstractOperator,
@@ -118,11 +121,11 @@ def parse_retries(retries: Any) -> Optional[int]:
return parsed_retries
-def coerce_retry_delay(retry_delay: Union[float, timedelta]) -> timedelta:
- if isinstance(retry_delay, timedelta):
- return retry_delay
- logger.debug("retry_delay isn't a timedelta object, assuming secs")
- return timedelta(seconds=retry_delay)
+def coerce_timedelta(value: Union[float, timedelta], *, key: str) -> timedelta:
+ if isinstance(value, timedelta):
+ return value
+ logger.debug("%s isn't a timedelta object, assuming secs", key)
+ return timedelta(seconds=value)
def coerce_resources(resources: Optional[Dict[str, Any]]) -> Optional[Resources]:
@@ -131,7 +134,7 @@ def coerce_resources(resources: Optional[Dict[str, Any]]) -> Optional[Resources]
return Resources(**resources)
-def _get_dag_defaults(dag: Optional["DAG"], task_group: Optional["TaskGroup"]) -> Tuple[dict, ParamsDict]:
+def _get_parent_defaults(dag: Optional["DAG"], task_group: Optional["TaskGroup"]) -> Tuple[dict, ParamsDict]:
if not dag:
return {}, ParamsDict()
dag_args = copy.copy(dag.default_args)
@@ -143,20 +146,24 @@ def _get_dag_defaults(dag: Optional["DAG"], task_group: Optional["TaskGroup"]) -
return dag_args, dag_params
-def _merge_defaults(
- dag_args: dict,
- dag_params: ParamsDict,
+def get_merged_defaults(
+ dag: Optional["DAG"],
+ task_group: Optional["TaskGroup"],
task_params: Optional[dict],
- task_default_args: dict,
+ task_default_args: Optional[dict],
) -> Tuple[dict, ParamsDict]:
+ args, params = _get_parent_defaults(dag, task_group)
if task_params:
- dag_params.update(task_params)
- if task_default_args and not isinstance(task_default_args, collections.abc.Mapping):
- raise TypeError("default_args must be a mapping")
- dag_args.update(task_default_args)
- with contextlib.suppress(KeyError):
- dag_params.update(task_default_args.pop("params"))
- return dag_args, dag_params
+ if not isinstance(task_params, collections.abc.Mapping):
+ raise TypeError("params must be a mapping")
+ params.update(task_params)
+ if task_default_args:
+ if not isinstance(task_default_args, collections.abc.Mapping):
+ raise TypeError("default_args must be a mapping")
+ args.update(task_default_args)
+ with contextlib.suppress(KeyError):
+ params.update(task_default_args["params"] or {})
+ return args, params
class _PartialDescriptor:
@@ -191,12 +198,14 @@ def partial(
resources: Optional[Dict[str, Any]] = None,
trigger_rule: str = DEFAULT_TRIGGER_RULE,
depends_on_past: bool = False,
+ ignore_first_depends_on_past: bool = DEFAULT_IGNORE_FIRST_DEPENDS_ON_PAST,
wait_for_downstream: bool = False,
retries: Optional[int] = DEFAULT_RETRIES,
queue: str = DEFAULT_QUEUE,
pool: Optional[str] = None,
pool_slots: int = DEFAULT_POOL_SLOTS,
- execution_timeout: Optional[timedelta] = None,
+ execution_timeout: Optional[timedelta] = DEFAULT_TASK_EXECUTION_TIMEOUT,
+ max_retry_delay: Union[None, timedelta, float] = None,
retry_delay: Union[timedelta, float] = DEFAULT_RETRY_DELAY,
retry_exponential_backoff: bool = False,
priority_weight: int = DEFAULT_PRIORITY_WEIGHT,
@@ -211,6 +220,11 @@ def partial(
executor_config: Optional[Dict] = None,
inlets: Optional[Any] = None,
outlets: Optional[Any] = None,
+ doc: Optional[str] = None,
+ doc_md: Optional[str] = None,
+ doc_json: Optional[str] = None,
+ doc_yaml: Optional[str] = None,
+ doc_rst: Optional[str] = None,
**kwargs,
) -> OperatorPartial:
from airflow.models.dag import DagContext
@@ -225,12 +239,11 @@ def partial(
task_id = task_group.child_id(task_id)
# Merge DAG and task group level defaults into user-supplied values.
- dag_args, dag_params = _get_dag_defaults(dag, task_group)
- partial_kwargs, default_params = _merge_defaults(
- dag_args=dag_args,
- dag_params=dag_params,
+ partial_kwargs, default_params = get_merged_defaults(
+ dag=dag,
+ task_group=task_group,
task_params=params,
- task_default_args=kwargs.pop("default_args", {}),
+ task_default_args=kwargs.pop("default_args", None),
)
partial_kwargs.update(kwargs)
@@ -245,12 +258,14 @@ def partial(
partial_kwargs.setdefault("params", default_params)
partial_kwargs.setdefault("trigger_rule", trigger_rule)
partial_kwargs.setdefault("depends_on_past", depends_on_past)
+ partial_kwargs.setdefault("ignore_first_depends_on_past", ignore_first_depends_on_past)
partial_kwargs.setdefault("wait_for_downstream", wait_for_downstream)
partial_kwargs.setdefault("retries", retries)
partial_kwargs.setdefault("queue", queue)
partial_kwargs.setdefault("pool", pool)
partial_kwargs.setdefault("pool_slots", pool_slots)
partial_kwargs.setdefault("execution_timeout", execution_timeout)
+ partial_kwargs.setdefault("max_retry_delay", max_retry_delay)
partial_kwargs.setdefault("retry_delay", retry_delay)
partial_kwargs.setdefault("retry_exponential_backoff", retry_exponential_backoff)
partial_kwargs.setdefault("priority_weight", priority_weight)
@@ -266,6 +281,11 @@ def partial(
partial_kwargs.setdefault("inlets", inlets)
partial_kwargs.setdefault("outlets", outlets)
partial_kwargs.setdefault("resources", resources)
+ partial_kwargs.setdefault("doc", doc)
+ partial_kwargs.setdefault("doc_json", doc_json)
+ partial_kwargs.setdefault("doc_md", doc_md)
+ partial_kwargs.setdefault("doc_rst", doc_rst)
+ partial_kwargs.setdefault("doc_yaml", doc_yaml)
# Post-process arguments. Should be kept in sync with _TaskDecorator.expand().
if "task_concurrency" in kwargs: # Reject deprecated option.
@@ -277,7 +297,12 @@ def partial(
if partial_kwargs["pool"] is None:
partial_kwargs["pool"] = Pool.DEFAULT_POOL_NAME
partial_kwargs["retries"] = parse_retries(partial_kwargs["retries"])
- partial_kwargs["retry_delay"] = coerce_retry_delay(partial_kwargs["retry_delay"])
+ partial_kwargs["retry_delay"] = coerce_timedelta(partial_kwargs["retry_delay"], key="retry_delay")
+ if partial_kwargs["max_retry_delay"] is not None:
+ partial_kwargs["max_retry_delay"] = coerce_timedelta(
+ partial_kwargs["max_retry_delay"],
+ key="max_retry_delay",
+ )
partial_kwargs["executor_config"] = partial_kwargs["executor_config"] or {}
partial_kwargs["resources"] = coerce_resources(partial_kwargs["resources"])
@@ -342,9 +367,9 @@ def apply_defaults(self: "BaseOperator", *args: Any, **kwargs: Any) -> Any:
if len(args) > 0:
raise AirflowException("Use keyword arguments when initializing operators")
- mapped_validation_only = kwargs.pop(
- "_airflow_mapped_validation_only",
- getattr(self, "_BaseOperator__mapped_validation", False),
+ instantiated_from_mapped = kwargs.pop(
+ "_airflow_from_mapped",
+ getattr(self, "_BaseOperator__from_mapped", False),
)
dag: Optional[DAG] = kwargs.get('dag') or DagContext.get_current_dag()
@@ -352,12 +377,11 @@ def apply_defaults(self: "BaseOperator", *args: Any, **kwargs: Any) -> Any:
if dag and not task_group:
task_group = TaskGroupContext.get_current_task_group(dag)
- dag_args, dag_params = _get_dag_defaults(dag, task_group)
- default_args, merged_params = _merge_defaults(
- dag_args=dag_args,
- dag_params=dag_params,
+ default_args, merged_params = get_merged_defaults(
+ dag=dag,
+ task_group=task_group,
task_params=kwargs.pop("params", None),
- task_default_args=kwargs.pop("default_args", {}),
+ task_default_args=kwargs.pop("default_args", None),
)
for arg in sig_cache.parameters:
@@ -381,14 +405,14 @@ def apply_defaults(self: "BaseOperator", *args: Any, **kwargs: Any) -> Any:
if not hasattr(self, '_BaseOperator__init_kwargs'):
self._BaseOperator__init_kwargs = {}
- self._BaseOperator__mapped_validation = mapped_validation_only
+ self._BaseOperator__from_mapped = instantiated_from_mapped
result = func(self, **kwargs, default_args=default_args)
# Store the args passed to init -- we need them to support task.map serialzation!
self._BaseOperator__init_kwargs.update(kwargs) # type: ignore
- if not mapped_validation_only:
+ if not instantiated_from_mapped:
# Set upstream task defined by XComArgs passed to template fields of the operator.
self.set_xcomargs_dependencies()
# Mark instance as instantiated.
@@ -593,8 +617,8 @@ class derived from this one results in the creation of a task object,
"""
# Implementing Operator.
- template_fields: Collection[str] = ()
- template_ext: Collection[str] = ()
+ template_fields: Sequence[str] = ()
+ template_ext: Sequence[str] = ()
template_fields_renderers: Dict[str, str] = {}
@@ -673,27 +697,8 @@ class derived from this one results in the creation of a task object,
# logic (default implementation is 'pass' i.e. no validation whatsoever).
mapped_arguments_validated_by_init: ClassVar[bool] = False
- # Set to True for an operator instantiated only for mapping validation.
- __mapped_validation = False
-
- def __new__(
- cls,
- dag: Optional['DAG'] = None,
- task_group: Optional["TaskGroup"] = None,
- _airflow_mapped_validation_only: bool = False, # Whether called to validate a MappedOperator.
- **kwargs,
- ):
- # If we are creating a new Task _and_ we are in the context of a MappedTaskGroup, then we should only
- # create mapped operators.
- from airflow.models.dag import DagContext
- from airflow.utils.task_group import MappedTaskGroup, TaskGroupContext
-
- dag = dag or DagContext.get_current_dag()
- task_group = task_group or TaskGroupContext.get_current_task_group(dag)
-
- if not _airflow_mapped_validation_only and isinstance(task_group, MappedTaskGroup):
- return cls.partial(dag=dag, task_group=task_group, **kwargs).expand()
- return super().__new__(cls)
+ # Set to True for an operator instantiated by a mapped operator.
+ __from_mapped = False
def __init__(
self,
@@ -709,6 +714,7 @@ def __init__(
start_date: Optional[datetime] = None,
end_date: Optional[datetime] = None,
depends_on_past: bool = False,
+ ignore_first_depends_on_past: bool = DEFAULT_IGNORE_FIRST_DEPENDS_ON_PAST,
wait_for_downstream: bool = False,
dag: Optional['DAG'] = None,
params: Optional[Dict] = None,
@@ -719,7 +725,7 @@ def __init__(
pool: Optional[str] = None,
pool_slots: int = DEFAULT_POOL_SLOTS,
sla: Optional[timedelta] = None,
- execution_timeout: Optional[timedelta] = None,
+ execution_timeout: Optional[timedelta] = DEFAULT_TASK_EXECUTION_TIMEOUT,
on_execute_callback: Optional[TaskStateChangeCallback] = None,
on_failure_callback: Optional[TaskStateChangeCallback] = None,
on_success_callback: Optional[TaskStateChangeCallback] = None,
@@ -768,18 +774,21 @@ def __init__(
dag = dag or DagContext.get_current_dag()
task_group = task_group or TaskGroupContext.get_current_task_group(dag)
- if task_group:
- self.task_id = task_group.child_id(task_id)
- else:
- self.task_id = task_id
- if not self.__mapped_validation and task_group:
+ self.task_id = task_group.child_id(task_id) if task_group else task_id
+ if not self.__from_mapped and task_group:
task_group.add(self)
self.owner = owner
self.email = email
self.email_on_retry = email_on_retry
self.email_on_failure = email_on_failure
+
+ if execution_timeout is not None and not isinstance(execution_timeout, timedelta):
+ raise ValueError(
+ f'execution_timeout must be timedelta object but passed as type: {type(execution_timeout)}'
+ )
self.execution_timeout = execution_timeout
+
self.on_execute_callback = on_execute_callback
self.on_failure_callback = on_failure_callback
self.on_success_callback = on_success_callback
@@ -829,21 +838,20 @@ def __init__(
f"'{dag.dag_id if dag else ''}.{task_id}'; received '{trigger_rule}'."
)
- self.trigger_rule = trigger_rule
+ self.trigger_rule = TriggerRule(trigger_rule)
self.depends_on_past: bool = depends_on_past
- self.wait_for_downstream = wait_for_downstream
+ self.ignore_first_depends_on_past: bool = ignore_first_depends_on_past
+ self.wait_for_downstream: bool = wait_for_downstream
if wait_for_downstream:
self.depends_on_past = True
- self.retry_delay = coerce_retry_delay(retry_delay)
+ self.retry_delay = coerce_timedelta(retry_delay, key="retry_delay")
self.retry_exponential_backoff = retry_exponential_backoff
- self.max_retry_delay = max_retry_delay
- if max_retry_delay:
- if isinstance(max_retry_delay, timedelta):
- self.max_retry_delay = max_retry_delay
- else:
- self.log.debug("max_retry_delay isn't a timedelta object, assuming secs")
- self.max_retry_delay = timedelta(seconds=max_retry_delay)
+ self.max_retry_delay = (
+ max_retry_delay
+ if max_retry_delay is None
+ else coerce_timedelta(max_retry_delay, key="max_retry_delay")
+ )
# At execution_time this becomes a normal dict
self.params: Union[ParamsDict, dict] = ParamsDict(params)
@@ -993,7 +1001,7 @@ def __lt__(self, other):
def __setattr__(self, key, value):
super().__setattr__(key, value)
- if self.__mapped_validation or self._lock_for_execution:
+ if self.__from_mapped or self._lock_for_execution:
return # Skip any custom behavior for validation and during execute.
if key in self.__init_kwargs:
self.__init_kwargs[key] = value
@@ -1047,8 +1055,8 @@ def dag(self, dag: Optional['DAG']):
elif self.has_dag() and self.dag is not dag:
raise AirflowException(f"The DAG assigned to {self} can not be changed.")
- if self.__mapped_validation:
- pass # Don't add task to DAG for validation.
+ if self.__from_mapped:
+ pass # Don't add to DAG -- the mapped task takes the place.
elif self.task_id not in dag.task_dict:
dag.add_task(self)
elif self.task_id in dag.task_dict and dag.task_dict[self.task_id] is not self:
@@ -1440,7 +1448,7 @@ def get_serialized_fields(cls):
'label',
'_BaseOperator__instantiated',
'_BaseOperator__init_kwargs',
- '_BaseOperator__mapped_validation',
+ '_BaseOperator__from_mapped',
}
| { # Class level defaults need to be added to this list
'start_date',
@@ -1470,12 +1478,12 @@ def is_smart_sensor_compatible(self):
is_mapped: ClassVar[bool] = False
@property
- def inherits_from_dummy_operator(self):
- """Used to determine if an Operator is inherited from DummyOperator"""
- # This looks like `isinstance(self, DummyOperator) would work, but this also
- # needs to cope when `self` is a Serialized instance of a DummyOperator or one
+ def inherits_from_empty_operator(self):
+ """Used to determine if an Operator is inherited from EmptyOperator"""
+ # This looks like `isinstance(self, EmptyOperator) would work, but this also
+ # needs to cope when `self` is a Serialized instance of a EmptyOperator or one
# of its sub-classes (which don't inherit from anything but BaseOperator).
- return getattr(self, '_is_dummy', False)
+ return getattr(self, '_is_empty', False)
def defer(
self,
@@ -1498,7 +1506,7 @@ def defer(
def validate_mapped_arguments(cls, **kwargs: Any) -> None:
"""Validate arguments when this operator is being mapped."""
if cls.mapped_arguments_validated_by_init:
- cls(**kwargs, _airflow_mapped_validation_only=True)
+ cls(**kwargs, _airflow_from_mapped=True)
def unmap(self) -> "BaseOperator":
""":meta private:"""
diff --git a/airflow/models/connection.py b/airflow/models/connection.py
index 6b8f1fa4b5550..b21e68b73b356 100644
--- a/airflow/models/connection.py
+++ b/airflow/models/connection.py
@@ -208,8 +208,8 @@ def get_uri(self) -> str:
"""Return connection in URI format"""
if '_' in self.conn_type:
self.log.warning(
- f"Connection schemes (type: {str(self.conn_type)}) "
- f"shall not contain '_' according to RFC3986."
+ "Connection schemes (type: %s) shall not contain '_' according to RFC3986.",
+ self.conn_type,
)
uri = f"{str(self.conn_type).lower().replace('_', '-')}://"
@@ -297,9 +297,9 @@ def get_extra(self) -> Dict:
def set_extra(self, value: str):
"""Encrypt extra-data and save in object attribute to object."""
if value:
+ self._validate_extra(value, self.conn_id)
fernet = get_fernet()
self._extra = fernet.encrypt(bytes(value, 'utf-8')).decode()
- self._validate_extra(self._extra, self.conn_id)
self.is_extra_encrypted = fernet.is_encrypted
else:
self._extra = value
diff --git a/airflow/models/dag.py b/airflow/models/dag.py
index b6ede6fec3897..ece03bafe8c92 100644
--- a/airflow/models/dag.py
+++ b/airflow/models/dag.py
@@ -18,6 +18,7 @@
import copy
import functools
+import itertools
import logging
import os
import pathlib
@@ -26,7 +27,7 @@
import sys
import traceback
import warnings
-from collections import OrderedDict
+import weakref
from datetime import datetime, timedelta
from inspect import signature
from typing import (
@@ -39,6 +40,7 @@
Iterable,
List,
Optional,
+ Sequence,
Set,
Tuple,
Type,
@@ -51,7 +53,7 @@
import pendulum
from dateutil.relativedelta import relativedelta
from pendulum.tz.timezone import Timezone
-from sqlalchemy import Boolean, Column, ForeignKey, Index, Integer, String, Text, func, or_
+from sqlalchemy import Boolean, Column, ForeignKey, Index, Integer, String, Text, func, not_, or_
from sqlalchemy.orm import backref, joinedload, relationship
from sqlalchemy.orm.query import Query
from sqlalchemy.orm.session import Session
@@ -76,7 +78,7 @@
from airflow.timetables.base import DagRunInfo, DataInterval, TimeRestriction, Timetable
from airflow.timetables.interval import CronDataIntervalTimetable, DeltaDataIntervalTimetable
from airflow.timetables.simple import NullTimetable, OnceTimetable
-from airflow.typing_compat import Literal, RePatternType
+from airflow.typing_compat import Literal
from airflow.utils import timezone
from airflow.utils.dag_cycle_tester import check_cycle
from airflow.utils.dates import cron_presets, date_range as utils_date_range
@@ -84,7 +86,7 @@
from airflow.utils.helpers import exactly_one, validate_key
from airflow.utils.log.logging_mixin import LoggingMixin
from airflow.utils.session import NEW_SESSION, provide_session
-from airflow.utils.sqlalchemy import Interval, UtcDateTime, skip_locked, with_row_locks
+from airflow.utils.sqlalchemy import Interval, UtcDateTime, skip_locked, tuple_in_condition, with_row_locks
from airflow.utils.state import DagRunState, State, TaskInstanceState
from airflow.utils.types import NOTSET, ArgNotSet, DagRunType, EdgeInfoType
@@ -324,8 +326,8 @@ def __init__(
sla_miss_callback: Optional[
Callable[["DAG", str, str, List["SlaMiss"], List[TaskInstance]], None]
] = None,
- default_view: str = conf.get('webserver', 'dag_default_view').lower(),
- orientation: str = conf.get('webserver', 'dag_orientation'),
+ default_view: str = conf.get_mandatory_value('webserver', 'dag_default_view').lower(),
+ orientation: str = conf.get_mandatory_value('webserver', 'dag_orientation'),
catchup: bool = conf.getboolean('scheduler', 'catchup_by_default'),
on_success_callback: Optional[DagStateChangeCallback] = None,
on_failure_callback: Optional[DagStateChangeCallback] = None,
@@ -763,12 +765,13 @@ def _time_restriction(self) -> TimeRestriction:
earliest = None
if start_dates:
earliest = timezone.coerce_datetime(min(start_dates))
+ latest = self.end_date
end_dates = [t.end_date for t in self.tasks if t.end_date]
- if self.end_date is not None:
- end_dates.append(self.end_date)
- latest = None
- if end_dates:
- latest = timezone.coerce_datetime(max(end_dates))
+ if len(end_dates) == len(self.tasks): # not exists null end_date
+ if self.end_date is not None:
+ end_dates.append(self.end_date)
+ if end_dates:
+ latest = timezone.coerce_datetime(max(end_dates))
return TimeRestriction(earliest, latest, self.catchup)
def iter_dagrun_infos_between(
@@ -985,7 +988,7 @@ def pickle_id(self) -> Optional[int]:
def pickle_id(self, value: int) -> None:
self._pickle_id = value
- def param(self, name: str, default=None) -> DagParam:
+ def param(self, name: str, default: Any = NOTSET) -> DagParam:
"""
Return a DagParam object for current dag.
@@ -1074,14 +1077,12 @@ def concurrency_reached(self):
@provide_session
def get_is_active(self, session=NEW_SESSION) -> Optional[None]:
"""Returns a boolean indicating whether this DAG is active"""
- qry = session.query(DagModel).filter(DagModel.dag_id == self.dag_id)
- return qry.value(DagModel.is_active)
+ return session.query(DagModel.is_active).filter(DagModel.dag_id == self.dag_id).scalar()
@provide_session
def get_is_paused(self, session=NEW_SESSION) -> Optional[None]:
"""Returns a boolean indicating whether this DAG is paused"""
- qry = session.query(DagModel).filter(DagModel.dag_id == self.dag_id)
- return qry.value(DagModel.is_paused)
+ return session.query(DagModel.is_paused).filter(DagModel.dag_id == self.dag_id).scalar()
@property
def is_paused(self):
@@ -1260,7 +1261,7 @@ def resolve_template_files(self):
for t in self.tasks:
t.resolve_template_files()
- def get_template_env(self) -> jinja2.Environment:
+ def get_template_env(self, *, force_sandboxed: bool = False) -> jinja2.Environment:
"""Build a Jinja2 environment."""
# Collect directories to search for template files
searchpath = [self.folder]
@@ -1277,7 +1278,7 @@ def get_template_env(self) -> jinja2.Environment:
if self.jinja_environment_kwargs:
jinja_env_options.update(self.jinja_environment_kwargs)
env: jinja2.Environment
- if self.render_template_as_native_obj:
+ if self.render_template_as_native_obj and not force_sandboxed:
env = airflow.templates.NativeEnvironment(**jinja_env_options)
else:
env = airflow.templates.SandboxedEnvironment(**jinja_env_options)
@@ -1341,43 +1342,33 @@ def get_task_instances(
start_date = (timezone.utcnow() - timedelta(30)).replace(
hour=0, minute=0, second=0, microsecond=0
)
-
- if state is None:
- state = []
-
- return (
- cast(
- Query,
- self._get_task_instances(
- task_ids=None,
- start_date=start_date,
- end_date=end_date,
- run_id=None,
- state=state,
- include_subdags=False,
- include_parentdag=False,
- include_dependent_dags=False,
- exclude_task_ids=cast(List[str], []),
- session=session,
- ),
- )
- .order_by(DagRun.execution_date)
- .all()
+ query = self._get_task_instances(
+ task_ids=None,
+ start_date=start_date,
+ end_date=end_date,
+ run_id=None,
+ state=state or (),
+ include_subdags=False,
+ include_parentdag=False,
+ include_dependent_dags=False,
+ exclude_task_ids=(),
+ session=session,
)
+ return cast(Query, query).order_by(DagRun.execution_date).all()
@overload
def _get_task_instances(
self,
*,
- task_ids,
+ task_ids: Optional[Collection[Union[str, Tuple[str, int]]]],
start_date: Optional[datetime],
end_date: Optional[datetime],
run_id: Optional[str],
- state: Union[TaskInstanceState, List[TaskInstanceState]],
+ state: Union[TaskInstanceState, Sequence[TaskInstanceState]],
include_subdags: bool,
include_parentdag: bool,
include_dependent_dags: bool,
- exclude_task_ids: Collection[str],
+ exclude_task_ids: Optional[Collection[Union[str, Tuple[str, int]]]],
session: Session,
dag_bag: Optional["DagBag"] = ...,
) -> Iterable[TaskInstance]:
@@ -1387,16 +1378,16 @@ def _get_task_instances(
def _get_task_instances(
self,
*,
- task_ids,
+ task_ids: Optional[Collection[Union[str, Tuple[str, int]]]],
as_pk_tuple: Literal[True],
start_date: Optional[datetime],
end_date: Optional[datetime],
run_id: Optional[str],
- state: Union[TaskInstanceState, List[TaskInstanceState]],
+ state: Union[TaskInstanceState, Sequence[TaskInstanceState]],
include_subdags: bool,
include_parentdag: bool,
include_dependent_dags: bool,
- exclude_task_ids: Collection[str],
+ exclude_task_ids: Optional[Collection[Union[str, Tuple[str, int]]]],
session: Session,
dag_bag: Optional["DagBag"] = ...,
recursion_depth: int = ...,
@@ -1408,16 +1399,16 @@ def _get_task_instances(
def _get_task_instances(
self,
*,
- task_ids,
+ task_ids: Optional[Collection[Union[str, Tuple[str, int]]]],
as_pk_tuple: Literal[True, None] = None,
start_date: Optional[datetime],
end_date: Optional[datetime],
run_id: Optional[str],
- state: Union[TaskInstanceState, List[TaskInstanceState]],
+ state: Union[TaskInstanceState, Sequence[TaskInstanceState]],
include_subdags: bool,
include_parentdag: bool,
include_dependent_dags: bool,
- exclude_task_ids: Collection[str],
+ exclude_task_ids: Optional[Collection[Union[str, Tuple[str, int]]]],
session: Session,
dag_bag: Optional["DagBag"] = None,
recursion_depth: int = 0,
@@ -1436,7 +1427,7 @@ def _get_task_instances(
# Do we want full objects, or just the primary columns?
if as_pk_tuple:
- tis = session.query(TI.dag_id, TI.task_id, TI.run_id)
+ tis = session.query(TI.dag_id, TI.task_id, TI.run_id, TI.map_index)
else:
tis = session.query(TaskInstance)
tis = tis.join(TaskInstance.dag_run)
@@ -1449,14 +1440,16 @@ def _get_task_instances(
(TaskInstance.dag_id == dag.dag_id) & TaskInstance.task_id.in_(dag.task_ids)
)
tis = tis.filter(or_(*conditions))
- else:
+ elif self.partial:
tis = tis.filter(TaskInstance.dag_id == self.dag_id, TaskInstance.task_id.in_(self.task_ids))
+ else:
+ tis = tis.filter(TaskInstance.dag_id == self.dag_id)
if run_id:
tis = tis.filter(TaskInstance.run_id == run_id)
if start_date:
tis = tis.filter(DagRun.execution_date >= start_date)
- if task_ids:
- tis = tis.filter(TaskInstance.task_id.in_(task_ids))
+ if task_ids is not None:
+ tis = tis.filter(TaskInstance.ti_selector_condition(task_ids))
# This allows allow_trigger_in_future config to take affect, rather than mandating exec_date <= UTC
if end_date or not self.allow_future_exec_dates:
@@ -1594,25 +1587,29 @@ def _get_task_instances(
if as_pk_tuple:
result.update(TaskInstanceKey(*cols) for cols in tis.all())
else:
- result.update(ti.key for ti in tis.all())
-
- if exclude_task_ids:
- result = set(
- filter(
- lambda key: key.task_id not in exclude_task_ids,
- result,
- )
- )
+ result.update(ti.key for ti in tis)
+
+ if exclude_task_ids is not None:
+ result = {
+ task
+ for task in result
+ if task.task_id not in exclude_task_ids
+ and (task.task_id, task.map_index) not in exclude_task_ids
+ }
if as_pk_tuple:
return result
- elif result:
+ if result:
# We've been asked for objects, lets combine it all back in to a result set
- tis = tis.with_entities(TI.dag_id, TI.task_id, TI.run_id)
-
- tis = session.query(TI).filter(TI.filter_for_tis(result))
- elif exclude_task_ids:
- tis = tis.filter(TI.task_id.notin_(list(exclude_task_ids)))
+ ti_filters = TI.filter_for_tis(result)
+ if ti_filters is not None:
+ tis = session.query(TI).filter(ti_filters)
+ elif exclude_task_ids is None:
+ pass # Disable filter if not set.
+ elif isinstance(next(iter(exclude_task_ids), None), str):
+ tis = tis.filter(TI.task_id.notin_(exclude_task_ids))
+ else:
+ tis = tis.filter(not_(tuple_in_condition((TI.task_id, TI.map_index), exclude_task_ids)))
return tis
@@ -1621,6 +1618,7 @@ def set_task_instance_state(
self,
*,
task_id: str,
+ map_indexes: Optional[Collection[int]] = None,
execution_date: Optional[datetime] = None,
run_id: Optional[str] = None,
state: TaskInstanceState,
@@ -1636,6 +1634,8 @@ def set_task_instance_state(
in failed or upstream_failed state.
:param task_id: Task ID of the TaskInstance
+ :param map_indexes: Only set TaskInstance if its map_index matches.
+ If None (default), all mapped TaskInstances of the task are set.
:param execution_date: Execution date of the TaskInstance
:param run_id: The run_id of the TaskInstance
:param state: State to set the TaskInstance to
@@ -1650,19 +1650,17 @@ def set_task_instance_state(
if not exactly_one(execution_date, run_id):
raise ValueError("Exactly one of execution_date or run_id must be provided")
- if execution_date is None:
- dag_run = (
- session.query(DagRun).filter(DagRun.run_id == run_id, DagRun.dag_id == self.dag_id).one()
- ) # Raises an error if not found
- resolve_execution_date = dag_run.execution_date
- else:
- resolve_execution_date = execution_date
-
task = self.get_task(task_id)
task.dag = self
+ tasks_to_set_state: List[Union[Operator, Tuple[Operator, int]]]
+ if map_indexes is None:
+ tasks_to_set_state = [task]
+ else:
+ tasks_to_set_state = [(task, map_index) for map_index in map_indexes]
+
altered = set_state(
- tasks=[task],
+ tasks=tasks_to_set_state,
execution_date=execution_date,
run_id=run_id,
upstream=upstream,
@@ -1686,6 +1684,14 @@ def set_task_instance_state(
include_upstream=False,
)
+ if execution_date is None:
+ dag_run = (
+ session.query(DagRun).filter(DagRun.run_id == run_id, DagRun.dag_id == self.dag_id).one()
+ ) # Raises an error if not found
+ resolve_execution_date = dag_run.execution_date
+ else:
+ resolve_execution_date = execution_date
+
end_date = resolve_execution_date if not future else None
start_date = resolve_execution_date if not past else None
@@ -1717,54 +1723,18 @@ def topological_sort(self, include_subdag_tasks: bool = False):
Sorts tasks in topographical order, such that a task comes after any of its
upstream dependencies.
- Heavily inspired by:
- http://blog.jupo.org/2012/04/06/topological-sorting-acyclic-directed-graphs/
-
- :param include_subdag_tasks: whether to include tasks in subdags, default to False
- :return: list of tasks in topological order
- """
- from airflow.operators.subdag import SubDagOperator # Avoid circular import
-
- # convert into an OrderedDict to speedup lookup while keeping order the same
- graph_unsorted = OrderedDict((task.task_id, task) for task in self.tasks)
-
- graph_sorted: List[Operator] = []
-
- # special case
- if len(self.tasks) == 0:
- return tuple(graph_sorted)
-
- # Run until the unsorted graph is empty.
- while graph_unsorted:
- # Go through each of the node/edges pairs in the unsorted
- # graph. If a set of edges doesn't contain any nodes that
- # haven't been resolved, that is, that are still in the
- # unsorted graph, remove the pair from the unsorted graph,
- # and append it to the sorted graph. Note here that by using
- # using the items() method for iterating, a copy of the
- # unsorted graph is used, allowing us to modify the unsorted
- # graph as we move through it. We also keep a flag for
- # checking that graph is acyclic, which is true if any
- # nodes are resolved during each pass through the graph. If
- # not, we need to exit as the graph therefore can't be
- # sorted.
- acyclic = False
- for node in list(graph_unsorted.values()):
- for edge in node.upstream_list:
- if edge.node_id in graph_unsorted:
- break
- # no edges in upstream tasks
- else:
- acyclic = True
- del graph_unsorted[node.task_id]
- graph_sorted.append(node)
- if include_subdag_tasks and isinstance(node, SubDagOperator):
- graph_sorted.extend(node.subdag.topological_sort(include_subdag_tasks=True))
+ Deprecated in place of ``task_group.topological_sort``
+ """
+ from airflow.utils.task_group import TaskGroup
- if not acyclic:
- raise AirflowException(f"A cyclic dependency occurred in dag: {self.dag_id}")
+ def nested_topo(group):
+ for node in group.topological_sort(_include_subdag_tasks=include_subdag_tasks):
+ if isinstance(node, TaskGroup):
+ yield from nested_topo(node)
+ else:
+ yield node
- return tuple(graph_sorted)
+ return tuple(nested_topo(self.task_group))
@provide_session
def set_dag_runs_state(
@@ -1791,7 +1761,7 @@ def set_dag_runs_state(
@provide_session
def clear(
self,
- task_ids=None,
+ task_ids: Union[Collection[str], Collection[Tuple[str, int]], None] = None,
start_date: Optional[datetime] = None,
end_date: Optional[datetime] = None,
only_failed: bool = False,
@@ -1806,13 +1776,13 @@ def clear(
recursion_depth: int = 0,
max_recursion_depth: Optional[int] = None,
dag_bag: Optional["DagBag"] = None,
- exclude_task_ids: FrozenSet[str] = frozenset({}),
+ exclude_task_ids: Union[FrozenSet[str], FrozenSet[Tuple[str, int]], None] = frozenset(),
) -> Union[int, Iterable[TaskInstance]]:
"""
Clears a set of task instances associated with the current dag for
a specified date range.
- :param task_ids: List of task ids to clear
+ :param task_ids: List of task ids or (``task_id``, ``map_index``) tuples to clear
:param start_date: The minimum execution_date to clear
:param end_date: The maximum execution_date to clear
:param only_failed: Only clear failed tasks
@@ -1826,7 +1796,8 @@ def clear(
:param dry_run: Find the tasks to clear but don't clear them.
:param session: The sqlalchemy session to use
:param dag_bag: The DagBag used to find the dags subdags (Optional)
- :param exclude_task_ids: A set of ``task_id`` that should not be cleared
+ :param exclude_task_ids: A set of ``task_id`` or (``task_id``, ``map_index``)
+ tuples that should not be cleared
"""
if get_tis:
warnings.warn(
@@ -1986,7 +1957,7 @@ def sub_dag(self, *args, **kwargs):
def partial_subset(
self,
- task_ids_or_regex: Union[str, RePatternType, Iterable[str]],
+ task_ids_or_regex: Union[str, re.Pattern, Iterable[str]],
include_downstream=False,
include_upstream=True,
include_direct_upstream=False,
@@ -2002,7 +1973,10 @@ def partial_subset(
tasks, in addition to matched tasks.
:param include_upstream: Include all upstream tasks of matched tasks,
in addition to matched tasks.
+ :param include_direct_upstream: Include all tasks directly upstream of matched
+ and downstream (if include_downstream = True) tasks
"""
+
from airflow.models.baseoperator import BaseOperator
from airflow.models.mappedoperator import MappedOperator
@@ -2011,7 +1985,7 @@ def partial_subset(
memo = {id(self.task_dict): None, id(self._task_group): None}
dag = copy.deepcopy(self, memo) # type: ignore
- if isinstance(task_ids_or_regex, (str, RePatternType)):
+ if isinstance(task_ids_or_regex, (str, re.Pattern)):
matched_tasks = [t for t in self.tasks if re.findall(task_ids_or_regex, t.task_id)]
else:
matched_tasks = [t for t in self.tasks if t.task_id in task_ids_or_regex]
@@ -2022,15 +1996,23 @@ def partial_subset(
also_include.extend(t.get_flat_relatives(upstream=False))
if include_upstream:
also_include.extend(t.get_flat_relatives(upstream=True))
- elif include_direct_upstream:
+
+ direct_upstreams: List[Operator] = []
+ if include_direct_upstream:
+ for t in itertools.chain(matched_tasks, also_include):
upstream = (u for u in t.upstream_list if isinstance(u, (BaseOperator, MappedOperator)))
- also_include.extend(upstream)
+ direct_upstreams.extend(upstream)
# Compiling the unique list of tasks that made the cut
- # Make sure to not recursively deepcopy the dag while copying the task
+ # Make sure to not recursively deepcopy the dag or task_group while copying the task.
+ # task_group is reset later
+ def _deepcopy_task(t) -> "Operator":
+ memo.setdefault(id(t.task_group), None)
+ return copy.deepcopy(t, memo)
+
dag.task_dict = {
- t.task_id: copy.deepcopy(t, {id(t.dag): dag}) # type: ignore
- for t in matched_tasks + also_include
+ t.task_id: _deepcopy_task(t)
+ for t in itertools.chain(matched_tasks, also_include, direct_upstreams)
}
def filter_task_group(group, parent_group):
@@ -2044,7 +2026,8 @@ def filter_task_group(group, parent_group):
for child in group.children.values():
if isinstance(child, AbstractOperator):
if child.task_id in dag.task_dict:
- copied.children[child.task_id] = dag.task_dict[child.task_id]
+ task = copied.children[child.task_id] = dag.task_dict[child.task_id]
+ task.task_group = weakref.proxy(copied)
else:
copied.used_group_ids.discard(child.task_id)
else:
@@ -2209,6 +2192,7 @@ def run(
rerun_failed_tasks=False,
run_backwards=False,
run_at_least_once=False,
+ continue_on_failures=False,
):
"""
Runs the DAG.
@@ -2258,6 +2242,7 @@ def run(
rerun_failed_tasks=rerun_failed_tasks,
run_backwards=run_backwards,
run_at_least_once=run_at_least_once,
+ continue_on_failures=continue_on_failures,
)
job.run()
@@ -2304,18 +2289,26 @@ def create_dagrun(
"""
if run_id: # Infer run_type from run_id if needed.
if not isinstance(run_id, str):
- raise ValueError(f"`run_id` expected to be a str is {type(run_id)}")
+ raise ValueError(f"`run_id` should be a str, not {type(run_id)}")
if not run_type:
run_type = DagRunType.from_run_id(run_id)
elif run_type and execution_date is not None: # Generate run_id from run_type and execution_date.
if not isinstance(run_type, DagRunType):
- raise ValueError(f"`run_type` expected to be a DagRunType is {type(run_type)}")
+ raise ValueError(f"`run_type` should be a DagRunType, not {type(run_type)}")
run_id = DagRun.generate_run_id(run_type, execution_date)
else:
raise AirflowException(
"Creating DagRun needs either `run_id` or both `run_type` and `execution_date`"
)
+ if run_id and "/" in run_id:
+ warnings.warn(
+ "Using forward slash ('/') in a DAG run ID is deprecated. Note that this character "
+ "also makes the run impossible to retrieve via Airflow's REST API.",
+ DeprecationWarning,
+ stacklevel=3,
+ )
+
logical_date = timezone.coerce_datetime(execution_date)
if data_interval is None and logical_date is not None:
warnings.warn(
@@ -2645,7 +2638,11 @@ class DagTag(Base):
__tablename__ = "dag_tag"
name = Column(String(100), primary_key=True)
- dag_id = Column(String(ID_LEN), ForeignKey('dag.dag_id'), primary_key=True)
+ dag_id = Column(
+ String(ID_LEN),
+ ForeignKey('dag.dag_id', name='dag_tag_dag_id_fkey', ondelete='CASCADE'),
+ primary_key=True,
+ )
def __repr__(self):
return self.name
@@ -2688,7 +2685,7 @@ class DagModel(Base):
owners = Column(String(2000))
# Description of the dag
description = Column(Text)
- # Default view of the inside the webserver
+ # Default view of the DAG inside the webserver
default_view = Column(String(25))
# Schedule interval
schedule_interval = Column(Interval)
@@ -2696,7 +2693,7 @@ class DagModel(Base):
timetable_description = Column(String(1000), nullable=True)
# Tags for view filter
- tags = relationship('DagTag', cascade='all,delete-orphan', backref=backref('dag'))
+ tags = relationship('DagTag', cascade='all, delete, delete-orphan', backref=backref('dag'))
max_active_tasks = Column(Integer, nullable=False)
max_active_runs = Column(Integer, nullable=True)
@@ -2777,6 +2774,15 @@ def get_dagmodel(dag_id, session=NEW_SESSION):
def get_current(cls, dag_id, session=NEW_SESSION):
return session.query(cls).filter(cls.dag_id == dag_id).first()
+ @staticmethod
+ @provide_session
+ def get_all_paused_dag_ids(session: Session = NEW_SESSION) -> Set[str]:
+ """Get a set of paused DAG ids"""
+ paused_dag_ids = session.query(DagModel.dag_id).filter(DagModel.is_paused == expression.true()).all()
+
+ paused_dag_ids = {paused_dag_id for paused_dag_id, in paused_dag_ids}
+ return paused_dag_ids
+
@provide_session
def get_last_dagrun(self, session=NEW_SESSION, include_externally_triggered=False):
return get_last_dagrun(
@@ -2813,7 +2819,7 @@ def get_default_view(self) -> str:
have a value
"""
# This is for backwards-compatibility with old dags that don't have None as default_view
- return self.default_view or conf.get('webserver', 'dag_default_view').lower()
+ return self.default_view or conf.get_mandatory_value('webserver', 'dag_default_view').lower()
@property
def safe_dag_id(self):
diff --git a/airflow/models/dagbag.py b/airflow/models/dagbag.py
index 2c82f13f801f4..3673ce095ea16 100644
--- a/airflow/models/dagbag.py
+++ b/airflow/models/dagbag.py
@@ -494,11 +494,12 @@ def collect_dags(
Note that if a ``.airflowignore`` file is found while processing
the directory, it will behave much like a ``.gitignore``,
- ignoring files that match any of the regex patterns specified
+ ignoring files that match any of the patterns specified
in the file.
- **Note**: The patterns in .airflowignore are treated as
- un-anchored regexes, not shell-like glob patterns.
+ **Note**: The patterns in ``.airflowignore`` are interpreted as either
+ un-anchored regexes or gitignore-like glob expressions, depending on
+ the ``DAG_IGNORE_FILE_SYNTAX`` configuration parameter.
"""
if self.read_dags_from_db:
return
diff --git a/airflow/models/dagrun.py b/airflow/models/dagrun.py
index 887ad20ed73b6..eeec4d5b099b2 100644
--- a/airflow/models/dagrun.py
+++ b/airflow/models/dagrun.py
@@ -20,7 +20,20 @@
import warnings
from collections import defaultdict
from datetime import datetime
-from typing import TYPE_CHECKING, Any, Dict, Iterable, List, NamedTuple, Optional, Tuple, Union
+from typing import (
+ TYPE_CHECKING,
+ Any,
+ Dict,
+ Generator,
+ Iterable,
+ List,
+ NamedTuple,
+ Optional,
+ Sequence,
+ Tuple,
+ Union,
+ cast,
+)
from sqlalchemy import (
Boolean,
@@ -57,7 +70,7 @@
from airflow.utils.helpers import is_container
from airflow.utils.log.logging_mixin import LoggingMixin
from airflow.utils.session import NEW_SESSION, provide_session
-from airflow.utils.sqlalchemy import UtcDateTime, nulls_first, skip_locked, with_row_locks
+from airflow.utils.sqlalchemy import UtcDateTime, nulls_first, skip_locked, tuple_in_condition, with_row_locks
from airflow.utils.state import DagRunState, State, TaskInstanceState
from airflow.utils.types import NOTSET, ArgNotSet, DagRunType
@@ -527,7 +540,7 @@ def update_state(
)
leaf_task_ids = {t.task_id for t in dag.leaves}
- leaf_tis = [ti for ti in tis if ti.task_id in leaf_task_ids]
+ leaf_tis = [ti for ti in tis if ti.task_id in leaf_task_ids if ti.state != TaskInstanceState.REMOVED]
# if all roots finished and at least one failed, the run failed
if not unfinished_tis and any(leaf_ti.state in State.failed_states for leaf_ti in leaf_tis):
@@ -642,7 +655,18 @@ def task_instance_scheduling_decisions(self, session: Session = NEW_SESSION) ->
if unfinished_tis:
schedulable_tis = [ut for ut in unfinished_tis if ut.state in SCHEDULEABLE_STATES]
self.log.debug("number of scheduleable tasks for %s: %s task(s)", self, len(schedulable_tis))
- schedulable_tis, changed_tis = self._get_ready_tis(schedulable_tis, finished_tis, session)
+ schedulable_tis, changed_tis, expansion_happened = self._get_ready_tis(
+ schedulable_tis,
+ finished_tis,
+ session=session,
+ )
+
+ # During expansion we may change some tis into non-schedulable
+ # states, so we need to re-compute.
+ if expansion_happened:
+ new_unfinished_tis = [t for t in unfinished_tis if t.state in State.unfinished]
+ finished_tis.extend(t for t in unfinished_tis if t.state in State.finished)
+ unfinished_tis = new_unfinished_tis
return TISchedulingDecision(
tis=tis,
@@ -657,56 +681,51 @@ def _get_ready_tis(
schedulable_tis: List[TI],
finished_tis: List[TI],
session: Session,
- ) -> Tuple[List[TI], bool]:
+ ) -> Tuple[List[TI], bool, bool]:
old_states = {}
ready_tis: List[TI] = []
changed_tis = False
if not schedulable_tis:
- return ready_tis, changed_tis
+ return ready_tis, changed_tis, False
# If we expand TIs, we need a new list so that we iterate over them too. (We can't alter
# `schedulable_tis` in place and have the `for` loop pick them up
- expanded_tis: List[TI] = []
+ additional_tis: List[TI] = []
dep_context = DepContext(
flag_upstream_failed=True,
ignore_unmapped_tasks=True, # Ignore this Dep, as we will expand it if we can.
finished_tis=finished_tis,
)
- # Check dependencies
- for schedulable in itertools.chain(schedulable_tis, expanded_tis):
-
+ # Check dependencies.
+ expansion_happened = False
+ for schedulable in itertools.chain(schedulable_tis, additional_tis):
old_state = schedulable.state
- if schedulable.are_dependencies_met(session=session, dep_context=dep_context):
- ready_tis.append(schedulable)
- else:
+ if not schedulable.are_dependencies_met(session=session, dep_context=dep_context):
old_states[schedulable.key] = old_state
continue
-
- # Expansion of last resort! This is ideally handled in the mini-scheduler in LocalTaskJob, but if
- # for any reason it wasn't, we need to expand it now
- if schedulable.map_index < 0 and schedulable.task.is_mapped:
- # HACK. This needs a better way, one that copes with multiple upstreams!
- for ti in finished_tis:
- if schedulable.task_id in ti.task.downstream_task_ids:
-
- assert isinstance(schedulable.task, MappedOperator)
- new_tis = schedulable.task.expand_mapped_task(self.run_id, session=session)
- if schedulable.state == TaskInstanceState.SKIPPED:
- # Task is now skipped (likely cos upstream returned 0 tasks
- continue
- assert new_tis[0] is schedulable
- expanded_tis.extend(new_tis[1:])
- break
+ # If schedulable is from a mapped task, but not yet expanded, do it
+ # now. This is called in two places: First and ideally in the mini
+ # scheduler at the end of LocalTaskJob, and then as an "expansion of
+ # last resort" in the scheduler to ensure that the mapped task is
+ # correctly expanded before executed.
+ if schedulable.map_index < 0 and isinstance(schedulable.task, MappedOperator):
+ expanded_tis, _ = schedulable.task.expand_mapped_task(self.run_id, session=session)
+ if expanded_tis:
+ assert expanded_tis[0] is schedulable
+ additional_tis.extend(expanded_tis[1:])
+ expansion_happened = True
+ if schedulable.state in SCHEDULEABLE_STATES:
+ ready_tis.append(schedulable)
# Check if any ti changed state
- tis_filter = TI.filter_for_tis(old_states.keys())
+ tis_filter = TI.filter_for_tis(old_states)
if tis_filter is not None:
fresh_tis = session.query(TI).filter(tis_filter).all()
changed_tis = any(ti.state != old_states[ti.key] for ti in fresh_tis)
- return ready_tis, changed_tis
+ return ready_tis, changed_tis, expansion_happened
def _are_premature_tis(
self,
@@ -770,8 +789,8 @@ def _emit_true_scheduling_delay_stats_for_finished_state(self, finished_tis: Lis
true_delay = first_start_date - data_interval_end
if true_delay.total_seconds() > 0:
Stats.timing(f'dagrun.{dag.dag_id}.first_task_scheduling_delay', true_delay)
- except Exception as e:
- self.log.warning(f'Failed to record first_task_scheduling_delay metric:\n{e}')
+ except Exception:
+ self.log.warning('Failed to record first_task_scheduling_delay metric:', exc_info=True)
def _emit_duration_stats_for_finished_state(self):
if self.state == State.RUNNING:
@@ -810,6 +829,12 @@ def verify_integrity(self, session: Session = NEW_SESSION):
task = None
try:
task = dag.get_task(ti.task_id)
+
+ should_restore_task = (task is not None) and ti.state == State.REMOVED
+ if should_restore_task:
+ self.log.info("Restoring task '%s' which was previously removed from DAG '%s'", ti, dag)
+ Stats.incr(f"task_restored_to_dag.{dag.dag_id}", 1, 1)
+ ti.state = State.NONE
except AirflowException:
if ti.state == State.REMOVED:
pass # ti has already been removed, just ignore it
@@ -817,17 +842,55 @@ def verify_integrity(self, session: Session = NEW_SESSION):
self.log.warning("Failed to get task '%s' for dag '%s'. Marking it as removed.", ti, dag)
Stats.incr(f"task_removed_from_dag.{dag.dag_id}", 1, 1)
ti.state = State.REMOVED
+ continue
- should_restore_task = (task is not None) and ti.state == State.REMOVED
- if should_restore_task:
- self.log.info("Restoring task '%s' which was previously removed from DAG '%s'", ti, dag)
- Stats.incr(f"task_restored_to_dag.{dag.dag_id}", 1, 1)
- ti.state = State.NONE
- session.merge(ti)
+ if not task.is_mapped:
+ continue
+ task = cast("MappedOperator", task)
+ num_mapped_tis = task.parse_time_mapped_ti_count
+ # Check if the number of mapped literals has changed and we need to mark this TI as removed
+ if num_mapped_tis is not None:
+ if ti.map_index >= num_mapped_tis:
+ self.log.debug(
+ "Removing task '%s' as the map_index is longer than the literal mapping list (%s)",
+ ti,
+ num_mapped_tis,
+ )
+ ti.state = State.REMOVED
+ elif ti.map_index < 0:
+ self.log.debug("Removing the unmapped TI '%s' as the mapping can now be performed", ti)
+ ti.state = State.REMOVED
+ else:
+ self.log.info("Restoring mapped task '%s'", ti)
+ Stats.incr(f"task_restored_to_dag.{dag.dag_id}", 1, 1)
+ ti.state = State.NONE
+ else:
+ # What if it is _now_ dynamically mapped, but wasn't before?
+ total_length = task.run_time_mapped_ti_count(self.run_id, session=session)
+
+ if total_length is None:
+ # Not all upstreams finished, so we can't tell what should be here. Remove everything.
+ if ti.map_index >= 0:
+ self.log.debug(
+ "Removing the unmapped TI '%s' as the mapping can't be resolved yet", ti
+ )
+ ti.state = State.REMOVED
+ continue
+ # Upstreams finished, check there aren't any extras
+ if ti.map_index >= total_length:
+ self.log.debug(
+ "Removing task '%s' as the map_index is longer than the resolved mapping list (%d)",
+ ti,
+ total_length,
+ )
+ ti.state = State.REMOVED
+ ...
def task_filter(task: "Operator") -> bool:
return task.task_id not in task_ids and (
- self.is_backfill or task.start_date <= self.execution_date
+ self.is_backfill
+ or task.start_date <= self.execution_date
+ and (task.end_date is None or self.execution_date <= task.end_date)
)
created_counts: Dict[str, int] = defaultdict(int)
@@ -837,25 +900,44 @@ def task_filter(task: "Operator") -> bool:
if hook_is_noop:
- def create_ti_mapping(task: "Operator") -> dict:
+ def create_ti_mapping(task: "Operator", indexes: Tuple[int, ...]) -> Generator:
created_counts[task.task_type] += 1
- return TI.insert_mapping(self.run_id, task, map_index=-1)
+ for map_index in indexes:
+ yield TI.insert_mapping(self.run_id, task, map_index=map_index)
+
+ creator = create_ti_mapping
else:
- def create_ti(task: "Operator") -> TI:
- ti = TI(task, run_id=self.run_id)
- task_instance_mutation_hook(ti)
- created_counts[ti.operator] += 1
- return ti
+ def create_ti(task: "Operator", indexes: Tuple[int, ...]) -> Generator:
+ for map_index in indexes:
+ ti = TI(task, run_id=self.run_id, map_index=map_index)
+ task_instance_mutation_hook(ti)
+ created_counts[ti.operator] += 1
+ yield ti
+
+ creator = create_ti
+
+ # Create missing tasks -- and expand any MappedOperator that _only_ have literals as input
+ def expand_mapped_literals(task: "Operator") -> Tuple["Operator", Sequence[int]]:
+ if not task.is_mapped:
+ return (task, (-1,))
+ task = cast("MappedOperator", task)
+ count = task.parse_time_mapped_ti_count or task.run_time_mapped_ti_count(
+ self.run_id, session=session
+ )
+ if not count:
+ return (task, (-1,))
+ return (task, range(count))
+
+ tasks_and_map_idxs = map(expand_mapped_literals, filter(task_filter, dag.task_dict.values()))
+ tasks = itertools.chain.from_iterable(itertools.starmap(creator, tasks_and_map_idxs))
- # Create missing tasks
- tasks = list(filter(task_filter, dag.task_dict.values()))
try:
if hook_is_noop:
- session.bulk_insert_mappings(TI, map(create_ti_mapping, tasks))
+ session.bulk_insert_mappings(TI, tasks)
else:
- session.bulk_save_objects(map(create_ti, tasks))
+ session.bulk_save_objects(tasks)
for task_type, count in created_counts.items():
Stats.incr(f"task_instance_created-{task_type}", count)
@@ -928,24 +1010,24 @@ def schedule_tis(self, schedulable_tis: Iterable[TI], session: Session = NEW_SES
Each element of ``schedulable_tis`` should have it's ``task`` attribute already set.
- Any DummyOperator without callbacks is instead set straight to the success state.
+ Any EmptyOperator without callbacks is instead set straight to the success state.
All the TIs should belong to this DagRun, but this code is in the hot-path, this is not checked -- it
is the caller's responsibility to call this function only with TIs from a single dag run.
"""
# Get list of TI IDs that do not need to executed, these are
- # tasks using DummyOperator and without on_execute_callback / on_success_callback
+ # tasks using EmptyOperator and without on_execute_callback / on_success_callback
dummy_ti_ids = []
schedulable_ti_ids = []
for ti in schedulable_tis:
if (
- ti.task.inherits_from_dummy_operator
+ ti.task.inherits_from_empty_operator
and not ti.task.on_execute_callback
and not ti.task.on_success_callback
):
dummy_ti_ids.append(ti.task_id)
else:
- schedulable_ti_ids.append(ti.task_id)
+ schedulable_ti_ids.append((ti.task_id, ti.map_index))
count = 0
@@ -955,12 +1037,12 @@ def schedule_tis(self, schedulable_tis: Iterable[TI], session: Session = NEW_SES
.filter(
TI.dag_id == self.dag_id,
TI.run_id == self.run_id,
- TI.task_id.in_(schedulable_ti_ids),
+ tuple_in_condition((TI.task_id, TI.map_index), schedulable_ti_ids),
)
.update({TI.state: State.SCHEDULED}, synchronize_session=False)
)
- # Tasks using DummyOperator should not be executed, mark them as success
+ # Tasks using EmptyOperator should not be executed, mark them as success
if dummy_ti_ids:
count += (
session.query(TI)
diff --git a/airflow/models/log.py b/airflow/models/log.py
index bc2aad66680db..b2a5639dcdd57 100644
--- a/airflow/models/log.py
+++ b/airflow/models/log.py
@@ -16,9 +16,9 @@
# specific language governing permissions and limitations
# under the License.
-from sqlalchemy import Column, Index, Integer, String, Text
+from sqlalchemy import Column, Index, Integer, String, Text, text
-from airflow.models.base import COLLATION_ARGS, ID_LEN, Base
+from airflow.models.base import Base, StringID
from airflow.utils import timezone
from airflow.utils.sqlalchemy import UtcDateTime
@@ -30,14 +30,18 @@ class Log(Base):
id = Column(Integer, primary_key=True)
dttm = Column(UtcDateTime)
- dag_id = Column(String(ID_LEN, **COLLATION_ARGS))
- task_id = Column(String(ID_LEN, **COLLATION_ARGS))
+ dag_id = Column(StringID())
+ task_id = Column(StringID())
+ map_index = Column(Integer, server_default=text('NULL'))
event = Column(String(30))
execution_date = Column(UtcDateTime)
owner = Column(String(500))
extra = Column(Text)
- __table_args__ = (Index('idx_log_dag', dag_id),)
+ __table_args__ = (
+ Index('idx_log_dag', dag_id),
+ Index('idx_log_event', event),
+ )
def __init__(self, event, task_instance=None, owner=None, extra=None, **kwargs):
self.dttm = timezone.utcnow()
@@ -50,14 +54,16 @@ def __init__(self, event, task_instance=None, owner=None, extra=None, **kwargs):
self.dag_id = task_instance.dag_id
self.task_id = task_instance.task_id
self.execution_date = task_instance.execution_date
+ self.map_index = task_instance.map_index
task_owner = task_instance.task.owner
if 'task_id' in kwargs:
self.task_id = kwargs['task_id']
if 'dag_id' in kwargs:
self.dag_id = kwargs['dag_id']
- if 'execution_date' in kwargs:
- if kwargs['execution_date']:
- self.execution_date = kwargs['execution_date']
+ if kwargs.get('execution_date'):
+ self.execution_date = kwargs['execution_date']
+ if 'map_index' in kwargs:
+ self.map_index = kwargs['map_index']
self.owner = owner or task_owner
diff --git a/airflow/models/mappedoperator.py b/airflow/models/mappedoperator.py
index c5cdeb2249b7d..fe18a97cc1414 100644
--- a/airflow/models/mappedoperator.py
+++ b/airflow/models/mappedoperator.py
@@ -30,6 +30,7 @@
Dict,
FrozenSet,
Iterable,
+ Iterator,
List,
Optional,
Sequence,
@@ -44,9 +45,11 @@
from sqlalchemy import func, or_
from sqlalchemy.orm.session import Session
-from airflow.compat.functools import cache
-from airflow.exceptions import UnmappableOperator
+from airflow import settings
+from airflow.compat.functools import cache, cached_property
+from airflow.exceptions import AirflowException, UnmappableOperator
from airflow.models.abstractoperator import (
+ DEFAULT_IGNORE_FIRST_DEPENDS_ON_PAST,
DEFAULT_OWNER,
DEFAULT_POOL_SLOTS,
DEFAULT_PRIORITY_WEIGHT,
@@ -75,6 +78,7 @@
from airflow.models.baseoperator import BaseOperator, BaseOperatorLink
from airflow.models.dag import DAG
+ from airflow.models.operator import Operator
from airflow.models.taskinstance import TaskInstance
from airflow.models.xcom_arg import XComArg
from airflow.utils.task_group import TaskGroup
@@ -83,7 +87,10 @@
# any mapping since we need the value to be ordered).
Mappable = Union[XComArg, Sequence, dict]
-ValidationSource = Union[Literal["map"], Literal["partial"]]
+ValidationSource = Union[Literal["expand"], Literal["partial"]]
+
+
+MAPPABLE_LITERAL_TYPES = (dict, list)
# For isinstance() check.
@@ -91,7 +98,7 @@
def get_mappable_types() -> Tuple[type, ...]:
from airflow.models.xcom_arg import XComArg
- return (XComArg, dict, list)
+ return (XComArg,) + MAPPABLE_LITERAL_TYPES
def validate_mapping_kwargs(op: Type["BaseOperator"], func: ValidationSource, value: Dict[str, Any]) -> None:
@@ -178,13 +185,24 @@ def __repr__(self) -> str:
def __del__(self):
if not self._expand_called:
- warnings.warn(f"{self!r} was never mapped!")
+ try:
+ task_id = repr(self.kwargs["task_id"])
+ except KeyError:
+ task_id = f"at {hex(id(self))}"
+ warnings.warn(f"Task {task_id} was never mapped!")
def expand(self, **mapped_kwargs: "Mappable") -> "MappedOperator":
- from airflow.operators.dummy import DummyOperator
+ if not mapped_kwargs:
+ raise TypeError("no arguments to expand against")
+ return self._expand(**mapped_kwargs)
+
+ def _expand(self, **mapped_kwargs: "Mappable") -> "MappedOperator":
+ self._expand_called = True
+
+ from airflow.operators.empty import EmptyOperator
validate_mapping_kwargs(self.operator_class, "expand", mapped_kwargs)
- prevent_duplicates(self.kwargs, mapped_kwargs, fail_reason="mapping already partial")
+ prevent_duplicates(self.kwargs, mapped_kwargs, fail_reason="unmappable or already specified")
ensure_xcomarg_return_value(mapped_kwargs)
partial_kwargs = self.kwargs.copy()
@@ -205,17 +223,20 @@ def expand(self, **mapped_kwargs: "Mappable") -> "MappedOperator":
operator_extra_links=self.operator_class.operator_extra_links,
template_ext=self.operator_class.template_ext,
template_fields=self.operator_class.template_fields,
+ template_fields_renderers=self.operator_class.template_fields_renderers,
ui_color=self.operator_class.ui_color,
ui_fgcolor=self.operator_class.ui_fgcolor,
- is_dummy=issubclass(self.operator_class, DummyOperator),
+ is_empty=issubclass(self.operator_class, EmptyOperator),
task_module=self.operator_class.__module__,
task_type=self.operator_class.__name__,
dag=dag,
task_group=task_group,
start_date=start_date,
end_date=end_date,
+ # For classic operators, this points to mapped_kwargs because kwargs
+ # to BaseOperator.expand() contribute to operator arguments.
+ expansion_kwargs_attr="mapped_kwargs",
)
- self._expand_called = True
return op
@@ -223,7 +244,13 @@ def expand(self, **mapped_kwargs: "Mappable") -> "MappedOperator":
class MappedOperator(AbstractOperator):
"""Object representing a mapped operator in a DAG."""
- operator_class: Union[Type["BaseOperator"], str]
+ # This attribute serves double purpose. For a "normal" operator instance
+ # loaded from DAG, this holds the underlying non-mapped operator class that
+ # can be used to create an unmapped operator for execution. For an operator
+ # recreated from a serialized DAG, however, this holds the serialized data
+ # that can be used to unmap this into a SerializedBaseOperator.
+ operator_class: Union[Type["BaseOperator"], Dict[str, Any]]
+
mapped_kwargs: Dict[str, "Mappable"]
partial_kwargs: Dict[str, Any]
@@ -232,11 +259,12 @@ class MappedOperator(AbstractOperator):
params: Optional[dict]
deps: FrozenSet[BaseTIDep]
operator_extra_links: Collection["BaseOperatorLink"]
- template_ext: Collection[str]
+ template_ext: Sequence[str]
template_fields: Collection[str]
+ template_fields_renderers: Dict[str, str]
ui_color: str
ui_fgcolor: str
- _is_dummy: bool
+ _is_empty: bool
_task_module: str
_task_type: str
@@ -247,9 +275,25 @@ class MappedOperator(AbstractOperator):
upstream_task_ids: Set[str] = attr.ib(factory=set, init=False)
downstream_task_ids: Set[str] = attr.ib(factory=set, init=False)
+ _expansion_kwargs_attr: str
+ """Where to get kwargs to calculate expansion length against.
+
+ This should be a name to call ``getattr()`` on.
+ """
+
is_mapped: ClassVar[bool] = True
subdag: None = None # Since we don't support SubDagOperator, this is always None.
+ HIDE_ATTRS_FROM_UI: ClassVar[FrozenSet[str]] = AbstractOperator.HIDE_ATTRS_FROM_UI | frozenset(
+ (
+ 'parse_time_mapped_ti_count',
+ 'operator_class',
+ )
+ )
+
+ def __hash__(self):
+ return id(self)
+
def __repr__(self):
return f""
@@ -262,11 +306,15 @@ def __attrs_post_init__(self):
if self.dag:
self.dag.add_task(self)
for k, v in self.mapped_kwargs.items():
- if k in self.template_fields:
- XComArg.apply_upstream_relationship(self, v)
+ XComArg.apply_upstream_relationship(self, v)
for k, v in self.partial_kwargs.items():
if k in self.template_fields:
XComArg.apply_upstream_relationship(self, v)
+ if self.partial_kwargs.get('sla') is not None:
+ raise AirflowException(
+ f"SLAs are unsupported with mapped tasks. Please set `sla=None` for task "
+ f"{self.task_id!r}."
+ )
@classmethod
@cache
@@ -300,7 +348,7 @@ def _validate_argument_count(self) -> None:
arguments are *valid* (that depends on the actual mapping values), but
makes sure there are *enough* of them.
"""
- if isinstance(self.operator_class, str):
+ if not isinstance(self.operator_class, type):
return # No need to validate deserialized operator.
self.operator_class.validate_mapped_arguments(**self._get_unmap_kwargs())
@@ -310,9 +358,9 @@ def task_type(self) -> str:
return self._task_type
@property
- def inherits_from_dummy_operator(self) -> bool:
+ def inherits_from_empty_operator(self) -> bool:
"""Implementing Operator."""
- return self._is_dummy
+ return self._is_empty
@property
def roots(self) -> Sequence[AbstractOperator]:
@@ -340,6 +388,11 @@ def trigger_rule(self) -> TriggerRule:
def depends_on_past(self) -> bool:
return bool(self.partial_kwargs.get("depends_on_past"))
+ @property
+ def ignore_first_depends_on_past(self) -> bool:
+ value = self.partial_kwargs.get("ignore_first_depends_on_past", DEFAULT_IGNORE_FIRST_DEPENDS_ON_PAST)
+ return bool(value)
+
@property
def wait_for_downstream(self) -> bool:
return bool(self.partial_kwargs.get("wait_for_downstream"))
@@ -364,6 +417,10 @@ def pool_slots(self) -> Optional[str]:
def execution_timeout(self) -> Optional[datetime.timedelta]:
return self.partial_kwargs.get("execution_timeout")
+ @property
+ def max_retry_delay(self) -> Optional[datetime.timedelta]:
+ return self.partial_kwargs.get("max_retry_delay")
+
@property
def retry_delay(self) -> datetime.timedelta:
return self.partial_kwargs.get("retry_delay", DEFAULT_RETRY_DELAY)
@@ -424,6 +481,26 @@ def inlets(self) -> Optional[Any]:
def outlets(self) -> Optional[Any]:
return self.partial_kwargs.get("outlets", None)
+ @property
+ def doc(self) -> Optional[str]:
+ return self.partial_kwargs.get("doc")
+
+ @property
+ def doc_md(self) -> Optional[str]:
+ return self.partial_kwargs.get("doc_md")
+
+ @property
+ def doc_json(self) -> Optional[str]:
+ return self.partial_kwargs.get("doc_json")
+
+ @property
+ def doc_yaml(self) -> Optional[str]:
+ return self.partial_kwargs.get("doc_yaml")
+
+ @property
+ def doc_rst(self) -> Optional[str]:
+ return self.partial_kwargs.get("doc_rst")
+
def get_dag(self) -> Optional["DAG"]:
"""Implementing Operator."""
return self.dag
@@ -444,27 +521,52 @@ def _get_unmap_kwargs(self) -> Dict[str, Any]:
**self.mapped_kwargs,
}
- def unmap(self) -> "BaseOperator":
- """Get the "normal" Operator after applying the current mapping."""
- dag = self.dag
- if not dag:
- raise RuntimeError("Cannot unmap a task without a DAG")
- if isinstance(self.operator_class, str):
- raise RuntimeError("Cannot unmap a deserialized operator")
- dag._remove_task(self.task_id)
- return self.operator_class(**self._get_unmap_kwargs())
+ def unmap(self, unmap_kwargs: Optional[Dict[str, Any]] = None) -> "BaseOperator":
+ """
+ Get the "normal" Operator after applying the current mapping.
- def _get_expansion_kwargs(self) -> Dict[str, "Mappable"]:
- """The kwargs to calculate expansion length against.
+ If ``operator_class`` is not a class (i.e. this DAG has been deserialized) then this will return a
+ SerializedBaseOperator that aims to "look like" the real operator.
- This is ``self.mapped_kwargs`` for classic operators because kwargs to
- ``BaseOperator.expand()`` contribute to operator arguments.
+ :param unmap_kwargs: Override the args to pass to the Operator constructor. Only used when
+ ``operator_class`` is still an actual class.
+
+ :meta private:
"""
- return self.mapped_kwargs
+ if isinstance(self.operator_class, type):
+ # We can't simply specify task_id here because BaseOperator further
+ # mangles the task_id based on the task hierarchy (namely, group_id
+ # is prepended, and '__N' appended to deduplicate). Instead of
+ # recreating the whole logic here, we just overwrite task_id later.
+ if unmap_kwargs is None:
+ unmap_kwargs = self._get_unmap_kwargs()
+ op = self.operator_class(**unmap_kwargs, _airflow_from_mapped=True)
+ op.task_id = self.task_id
+ return op
+
+ # After a mapped operator is serialized, there's no real way to actually
+ # unmap it since we've lost access to the underlying operator class.
+ # This tries its best to simply "forward" all the attributes on this
+ # mapped operator to a new SerializedBaseOperator instance.
+ from airflow.serialization.serialized_objects import SerializedBaseOperator
+
+ op = SerializedBaseOperator(task_id=self.task_id, _airflow_from_mapped=True)
+ SerializedBaseOperator.populate_operator(op, self.operator_class)
+ return op
+
+ def _get_expansion_kwargs(self) -> Dict[str, "Mappable"]:
+ """The kwargs to calculate expansion length against."""
+ return getattr(self, self._expansion_kwargs_attr)
def _get_map_lengths(self, run_id: str, *, session: Session) -> Dict[str, int]:
+ """Return dict of argument name to map length.
+
+ If any arguments are not known right now (upstream task not finished) they will not be present in the
+ dict.
+ """
# TODO: Find a way to cache this.
from airflow.models.taskmap import TaskMap
+ from airflow.models.xcom import XCom
from airflow.models.xcom_arg import XComArg
expansion_kwargs = self._get_expansion_kwargs()
@@ -474,36 +576,71 @@ def _get_map_lengths(self, run_id: str, *, session: Session) -> Dict[str, int]:
map_lengths.update((k, len(v)) for k, v in expansion_kwargs.items() if not isinstance(v, XComArg))
# Build a reverse mapping of what arguments each task contributes to.
- dep_keys: Dict[str, Set[str]] = collections.defaultdict(set)
+ mapped_dep_keys: Dict[str, Set[str]] = collections.defaultdict(set)
+ non_mapped_dep_keys: Dict[str, Set[str]] = collections.defaultdict(set)
for k, v in expansion_kwargs.items():
if not isinstance(v, XComArg):
continue
- dep_keys[v.operator.task_id].add(k)
-
+ if v.operator.is_mapped:
+ mapped_dep_keys[v.operator.task_id].add(k)
+ else:
+ non_mapped_dep_keys[v.operator.task_id].add(k)
+ # TODO: It's not possible now, but in the future (AIP-42 Phase 2)
+ # we will add support for depending on one single mapped task
+ # instance. When that happens, we need to further analyze the mapped
+ # case to contain only tasks we depend on "as a whole", and put
+ # those we only depend on individually to the non-mapped lookup.
+
+ # Collect lengths from unmapped upstreams.
taskmap_query = session.query(TaskMap.task_id, TaskMap.length).filter(
TaskMap.dag_id == self.dag_id,
TaskMap.run_id == run_id,
- TaskMap.task_id.in_(list(dep_keys)),
+ TaskMap.task_id.in_(non_mapped_dep_keys),
+ TaskMap.map_index < 0,
)
for task_id, length in taskmap_query:
- for mapped_arg_name in dep_keys[task_id]:
+ for mapped_arg_name in non_mapped_dep_keys[task_id]:
+ map_lengths[mapped_arg_name] += length
+
+ # Collect lengths from mapped upstreams.
+ xcom_query = (
+ session.query(XCom.task_id, func.count(XCom.map_index))
+ .group_by(XCom.task_id)
+ .filter(
+ XCom.dag_id == self.dag_id,
+ XCom.run_id == run_id,
+ XCom.task_id.in_(mapped_dep_keys),
+ XCom.map_index >= 0,
+ )
+ )
+ for task_id, length in xcom_query:
+ for mapped_arg_name in mapped_dep_keys[task_id]:
map_lengths[mapped_arg_name] += length
+ return map_lengths
+ @cache
+ def _resolve_map_lengths(self, run_id: str, *, session: Session) -> Dict[str, int]:
+ """Return dict of argument name to map length, or throw if some are not resolvable"""
+ expansion_kwargs = self._get_expansion_kwargs()
+ map_lengths = self._get_map_lengths(run_id, session=session)
if len(map_lengths) < len(expansion_kwargs):
keys = ", ".join(repr(k) for k in sorted(set(expansion_kwargs).difference(map_lengths)))
raise RuntimeError(f"Failed to populate all mapping metadata; missing: {keys}")
return map_lengths
- def expand_mapped_task(self, run_id: str, *, session: Session) -> Sequence["TaskInstance"]:
+ def expand_mapped_task(self, run_id: str, *, session: Session) -> Tuple[Sequence["TaskInstance"], int]:
"""Create the mapped task instances for mapped task.
- :return: The mapped task instances, in ascending order by map index.
+ :return: The newly created mapped TaskInstances (if any) in ascending order by map index, and the
+ maximum map_index.
"""
from airflow.models.taskinstance import TaskInstance
from airflow.settings import task_instance_mutation_hook
- total_length = functools.reduce(operator.mul, self._get_map_lengths(run_id, session=session).values())
+ total_length = functools.reduce(
+ operator.mul, self._resolve_map_lengths(run_id, session=session).values()
+ )
state: Optional[TaskInstanceState] = None
unmapped_ti: Optional[TaskInstance] = (
@@ -518,7 +655,7 @@ def expand_mapped_task(self, run_id: str, *, session: Session) -> Sequence["Task
.one_or_none()
)
- ret: List[TaskInstance] = []
+ all_expanded_tis: List[TaskInstance] = []
if unmapped_ti:
# The unmapped task instance still exists and is unfinished, i.e. we
@@ -532,14 +669,13 @@ def expand_mapped_task(self, run_id: str, *, session: Session) -> Sequence["Task
total_length,
)
unmapped_ti.state = TaskInstanceState.SKIPPED
- session.flush()
- return ret
- # Otherwise convert this into the first mapped index, and create
- # TaskInstance for other indexes.
- unmapped_ti.map_index = 0
+ else:
+ # Otherwise convert this into the first mapped index, and create
+ # TaskInstance for other indexes.
+ unmapped_ti.map_index = 0
+ self.log.debug("Updated in place to become %s", unmapped_ti)
+ all_expanded_tis.append(unmapped_ti)
state = unmapped_ti.state
- self.log.debug("Updated in place to become %s", unmapped_ti)
- ret.append(unmapped_ti)
indexes_to_map = range(1, total_length)
else:
# Only create "missing" ones.
@@ -556,13 +692,12 @@ def expand_mapped_task(self, run_id: str, *, session: Session) -> Sequence["Task
for index in indexes_to_map:
# TODO: Make more efficient with bulk_insert_mappings/bulk_save_mappings.
- # TODO: Change `TaskInstance` ctor to take Operator, not BaseOperator
- ti = TaskInstance(self, run_id=run_id, map_index=index, state=state) # type: ignore
+ ti = TaskInstance(self, run_id=run_id, map_index=index, state=state)
self.log.debug("Expanding TIs upserted %s", ti)
task_instance_mutation_hook(ti)
ti = session.merge(ti)
- ti.task = self
- ret.append(ti)
+ ti.refresh_from_task(self) # session.merge() loses task information.
+ all_expanded_tis.append(ti)
# Set to "REMOVED" any (old) TaskInstances with map indices greater
# than the current map value
@@ -574,8 +709,7 @@ def expand_mapped_task(self, run_id: str, *, session: Session) -> Sequence["Task
).update({TaskInstance.state: TaskInstanceState.REMOVED})
session.flush()
-
- return ret
+ return all_expanded_tis, total_length
def prepare_for_execution(self) -> "MappedOperator":
# Since a mapped operator cannot be used for execution, and an unmapped
@@ -599,40 +733,56 @@ def render_template_fields(
"""
if not jinja_env:
jinja_env = self.get_template_env()
- unmapped_task = self.unmap()
+ # Before we unmap we have to resolve the mapped arguments, otherwise the real operator constructor
+ # could be called with an XComArg, rather than the value it resolves to.
+ #
+ # We also need to resolve _all_ mapped arguments, even if they aren't marked as templated
+ kwargs = self._get_unmap_kwargs()
+
+ template_fields = set(self.template_fields)
+
+ # Ideally we'd like to pass in session as an argument to this function, but since operators _could_
+ # override this we can't easily change this function signature.
+ # We can't use @provide_session, as that closes and expunges everything, which we don't want to do
+ # when we are so "deep" in the weeds here.
+ #
+ # Nor do we want to close the session -- that would expunge all the things from the internal cache
+ # which we don't want to do either
+ session = settings.Session()
+ self._resolve_expansion_kwargs(kwargs, template_fields, context, session)
+
+ unmapped_task = self.unmap(unmap_kwargs=kwargs)
self._do_render_template_fields(
parent=unmapped_task,
- template_fields=unmapped_task.template_fields,
+ template_fields=template_fields,
context=context,
jinja_env=jinja_env,
seen_oids=set(),
+ session=session,
)
return unmapped_task
- def _render_template_field(
- self,
- key: str,
- value: Any,
- context: Context,
- jinja_env: Optional["jinja2.Environment"] = None,
- seen_oids: Optional[Set] = None,
- *,
- session: Session,
- ) -> Any:
- """Override the ordinary template rendering to add more logic.
-
- Specifically, if we're rendering a mapped argument, we need to "unmap"
- the value as well to assign it to the unmapped operator.
- """
- value = super()._render_template_field(key, value, context, jinja_env, seen_oids, session=session)
- return self._expand_mapped_field(key, value, context, session=session)
+ def _resolve_expansion_kwargs(
+ self, kwargs: Dict[str, Any], template_fields: Set[str], context: Context, session: Session
+ ) -> None:
+ """Update mapped fields in place in kwargs dict"""
+ from airflow.models.xcom_arg import XComArg
+
+ expansion_kwargs = self._get_expansion_kwargs()
+
+ for k, v in expansion_kwargs.items():
+ if isinstance(v, XComArg):
+ v = v.resolve(context, session=session)
+ v = self._expand_mapped_field(k, v, context, session=session)
+ template_fields.discard(k)
+ kwargs[k] = v
def _expand_mapped_field(self, key: str, value: Any, context: Context, *, session: Session) -> Any:
map_index = context["ti"].map_index
if map_index < 0:
return value
expansion_kwargs = self._get_expansion_kwargs()
- all_lengths = self._get_map_lengths(context["run_id"], session=session)
+ all_lengths = self._resolve_map_lengths(context["run_id"], session=session)
def _find_index_for_this_field(index: int) -> int:
# Need to use self.mapped_kwargs for the original argument order.
@@ -656,3 +806,55 @@ def _find_index_for_this_field(index: int) -> int:
if i == found_index:
return k, v
raise IndexError(f"index {map_index} is over mapped length")
+
+ def iter_mapped_dependencies(self) -> Iterator["Operator"]:
+ """Upstream dependencies that provide XComs used by this task for task mapping."""
+ from airflow.models.xcom_arg import XComArg
+
+ for ref in XComArg.iter_xcom_args(self._get_expansion_kwargs()):
+ yield ref.operator
+
+ @cached_property
+ def parse_time_mapped_ti_count(self) -> Optional[int]:
+ """
+ Number of mapped TaskInstances that can be created at DagRun create time.
+
+ :return: None if non-literal mapped arg encountered, or else total number of mapped TIs this task
+ should have
+ """
+ total = 0
+
+ for value in self._get_expansion_kwargs().values():
+ if not isinstance(value, MAPPABLE_LITERAL_TYPES):
+ # None literal type encountered, so give up
+ return None
+ if total == 0:
+ total = len(value)
+ else:
+ total *= len(value)
+ return total
+
+ @cache
+ def run_time_mapped_ti_count(self, run_id: str, *, session: Session) -> Optional[int]:
+ """
+ Number of mapped TaskInstances that can be created at run time, or None if upstream tasks are not
+ complete yet.
+
+ :return: None if upstream tasks are not complete yet, or else total number of mapped TIs this task
+ should have
+ """
+
+ lengths = self._get_map_lengths(run_id, session=session)
+ expansion_kwargs = self._get_expansion_kwargs()
+
+ if not lengths or not expansion_kwargs:
+ return None
+
+ total = 1
+ for name in expansion_kwargs:
+ val = lengths.get(name)
+ if val is None:
+ return None
+ total *= val
+
+ return total
diff --git a/airflow/models/param.py b/airflow/models/param.py
index 22d6a026399c0..fcbe7a0f931c6 100644
--- a/airflow/models/param.py
+++ b/airflow/models/param.py
@@ -14,15 +14,19 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+import contextlib
import copy
import json
import warnings
-from typing import Any, Dict, ItemsView, MutableMapping, Optional, ValuesView
+from typing import TYPE_CHECKING, Any, Dict, ItemsView, MutableMapping, Optional, ValuesView
from airflow.exceptions import AirflowException, ParamValidationError
from airflow.utils.context import Context
from airflow.utils.types import NOTSET, ArgNotSet
+if TYPE_CHECKING:
+ from airflow.models.dag import DAG
+
class Param:
"""
@@ -228,18 +232,18 @@ class DagParam:
:param default: Default value used if no parameter was set.
"""
- def __init__(self, current_dag, name: str, default: Optional[Any] = None):
- if default:
+ def __init__(self, current_dag: "DAG", name: str, default: Any = NOTSET):
+ if default is not NOTSET:
current_dag.params[name] = default
self._name = name
self._default = default
def resolve(self, context: Context) -> Any:
"""Pull DagParam value from DagRun context. This method is run during ``op.execute()``."""
- default = self._default
- if not self._default:
- default = context['params'][self._name] if self._name in context['params'] else None
- resolved = context['dag_run'].conf.get(self._name, default)
- if not resolved:
- raise AirflowException(f'No value could be resolved for parameter {self._name}')
- return resolved
+ with contextlib.suppress(KeyError):
+ return context['dag_run'].conf[self._name]
+ if self._default is not NOTSET:
+ return self._default
+ with contextlib.suppress(KeyError):
+ return context['params'][self._name]
+ raise AirflowException(f'No value could be resolved for parameter {self._name}')
diff --git a/airflow/models/pool.py b/airflow/models/pool.py
index 3a65e6c188aeb..f195b93d42623 100644
--- a/airflow/models/pool.py
+++ b/airflow/models/pool.py
@@ -191,11 +191,7 @@ def slots_stats(
# calculate open metric
for pool_name, stats_dict in pools.items():
- if stats_dict["total"] == -1:
- # -1 means infinite
- stats_dict["open"] = -1
- else:
- stats_dict["open"] = stats_dict["total"] - stats_dict["running"] - stats_dict["queued"]
+ stats_dict["open"] = stats_dict["total"] - stats_dict["running"] - stats_dict["queued"]
return pools
diff --git a/airflow/models/renderedtifields.py b/airflow/models/renderedtifields.py
index 65f96ca7b7b68..c7bad78b5f3f4 100644
--- a/airflow/models/renderedtifields.py
+++ b/airflow/models/renderedtifields.py
@@ -72,6 +72,7 @@ class RenderedTaskInstanceFields(Base):
RenderedTaskInstanceFields.dag_id == foreign(DagRun.dag_id),
RenderedTaskInstanceFields.run_id == foreign(DagRun.run_id),
)""",
+ viewonly=True,
)
execution_date = association_proxy("dag_run", "execution_date")
@@ -203,12 +204,12 @@ def delete_old_records(
if session.bind.dialect.name in ["postgresql", "sqlite"]:
# Fetch Top X records given dag_id & task_id ordered by Execution Date
- subq1 = tis_to_keep_query.subquery('subq1')
-
+ subq1 = tis_to_keep_query.subquery()
+ excluded = session.query(subq1.c.dag_id, subq1.c.task_id, subq1.c.run_id)
session.query(cls).filter(
cls.dag_id == dag_id,
cls.task_id == task_id,
- tuple_(cls.dag_id, cls.task_id, cls.run_id).notin_(subq1),
+ tuple_(cls.dag_id, cls.task_id, cls.run_id).notin_(excluded),
).delete(synchronize_session=False)
elif session.bind.dialect.name in ["mysql"]:
cls._remove_old_rendered_ti_fields_mysql(dag_id, session, task_id, tis_to_keep_query)
diff --git a/airflow/models/serialized_dag.py b/airflow/models/serialized_dag.py
index 23d04992d3c75..45424709a8421 100644
--- a/airflow/models/serialized_dag.py
+++ b/airflow/models/serialized_dag.py
@@ -250,6 +250,14 @@ def has_dag(cls, dag_id: str, session: Session = None) -> bool:
"""
return session.query(literal(True)).filter(cls.dag_id == dag_id).first() is not None
+ @classmethod
+ @provide_session
+ def get_dag(cls, dag_id: str, session: Session = None) -> Optional['SerializedDAG']:
+ row = cls.get(dag_id, session=session)
+ if row:
+ return row.dag
+ return None
+
@classmethod
@provide_session
def get(cls, dag_id: str, session: Session = None) -> Optional['SerializedDagModel']:
diff --git a/airflow/models/skipmixin.py b/airflow/models/skipmixin.py
index 79d02592a7c62..d5b1481cb19b2 100644
--- a/airflow/models/skipmixin.py
+++ b/airflow/models/skipmixin.py
@@ -128,7 +128,7 @@ def skip(
session=session,
)
- def skip_all_except(self, ti: TaskInstance, branch_task_ids: Union[str, Iterable[str]]):
+ def skip_all_except(self, ti: TaskInstance, branch_task_ids: Union[None, str, Iterable[str]]):
"""
This method implements the logic for a branching operator; given a single
task ID or list of task IDs to follow, this skips all other tasks
@@ -140,6 +140,8 @@ def skip_all_except(self, ti: TaskInstance, branch_task_ids: Union[str, Iterable
self.log.info("Following branch %s", branch_task_ids)
if isinstance(branch_task_ids, str):
branch_task_ids = {branch_task_ids}
+ elif branch_task_ids is None:
+ branch_task_ids = ()
branch_task_ids = set(branch_task_ids)
diff --git a/airflow/models/taskfail.py b/airflow/models/taskfail.py
index 23d3086e2d013..f7de99c308cac 100644
--- a/airflow/models/taskfail.py
+++ b/airflow/models/taskfail.py
@@ -16,9 +16,11 @@
# specific language governing permissions and limitations
# under the License.
"""Taskfail tracks the failed run durations of each task instance"""
-from sqlalchemy import Column, Index, Integer, String
-from airflow.models.base import COLLATION_ARGS, ID_LEN, Base
+from sqlalchemy import Column, ForeignKeyConstraint, Integer
+from sqlalchemy.orm import relationship
+
+from airflow.models.base import Base, StringID
from airflow.utils.sqlalchemy import UtcDateTime
@@ -28,22 +30,53 @@ class TaskFail(Base):
__tablename__ = "task_fail"
id = Column(Integer, primary_key=True)
- task_id = Column(String(ID_LEN, **COLLATION_ARGS), nullable=False)
- dag_id = Column(String(ID_LEN, **COLLATION_ARGS), nullable=False)
- execution_date = Column(UtcDateTime, nullable=False)
+ task_id = Column(StringID(), nullable=False)
+ dag_id = Column(StringID(), nullable=False)
+ run_id = Column(StringID(), nullable=False)
+ map_index = Column(Integer, nullable=False)
start_date = Column(UtcDateTime)
end_date = Column(UtcDateTime)
duration = Column(Integer)
- __table_args__ = (Index('idx_task_fail_dag_task_date', dag_id, task_id, execution_date, unique=False),)
+ __table_args__ = (
+ ForeignKeyConstraint(
+ [dag_id, task_id, run_id, map_index],
+ [
+ "task_instance.dag_id",
+ "task_instance.task_id",
+ "task_instance.run_id",
+ "task_instance.map_index",
+ ],
+ name='task_fail_ti_fkey',
+ ondelete="CASCADE",
+ ),
+ )
+
+ # We don't need a DB level FK here, as we already have that to TI (which has one to DR) but by defining
+ # the relationship we can more easily find the execution date for these rows
+ dag_run = relationship(
+ "DagRun",
+ primaryjoin="""and_(
+ TaskFail.dag_id == foreign(DagRun.dag_id),
+ TaskFail.run_id == foreign(DagRun.run_id),
+ )""",
+ viewonly=True,
+ )
- def __init__(self, task, execution_date, start_date, end_date):
- self.dag_id = task.dag_id
- self.task_id = task.task_id
- self.execution_date = execution_date
- self.start_date = start_date
- self.end_date = end_date
+ def __init__(self, ti):
+ self.dag_id = ti.dag_id
+ self.task_id = ti.task_id
+ self.run_id = ti.run_id
+ self.map_index = ti.map_index
+ self.start_date = ti.start_date
+ self.end_date = ti.end_date
if self.end_date and self.start_date:
self.duration = int((self.end_date - self.start_date).total_seconds())
else:
self.duration = None
+
+ def __repr__(self):
+ prefix = f"<{self.__class__.__name__}: {self.dag_id}.{self.task_id} {self.run_id}"
+ if self.map_index != -1:
+ prefix += f" map_index={self.map_index}"
+ return prefix + '>'
diff --git a/airflow/models/taskinstance.py b/airflow/models/taskinstance.py
index 9361e24343289..0f5d49b819762 100644
--- a/airflow/models/taskinstance.py
+++ b/airflow/models/taskinstance.py
@@ -20,24 +20,23 @@
import hashlib
import logging
import math
+import operator
import os
-import pickle
import signal
-import threading
import warnings
from collections import defaultdict
from datetime import datetime, timedelta
from functools import partial
-from inspect import currentframe
-from tempfile import NamedTemporaryFile
from types import TracebackType
from typing import (
- IO,
TYPE_CHECKING,
Any,
+ Callable,
+ Collection,
+ ContextManager,
Dict,
+ Generator,
Iterable,
- Iterator,
List,
NamedTuple,
Optional,
@@ -47,6 +46,7 @@
)
from urllib.parse import quote
+import attr
import dill
import jinja2
import pendulum
@@ -60,17 +60,21 @@
PickleType,
String,
and_,
+ false,
func,
inspect,
or_,
text,
- tuple_,
)
from sqlalchemy.ext.associationproxy import association_proxy
+from sqlalchemy.ext.mutable import MutableDict
from sqlalchemy.orm import reconstructor, relationship
from sqlalchemy.orm.attributes import NO_VALUE, set_committed_value
+from sqlalchemy.orm.exc import NoResultFound
+from sqlalchemy.orm.query import Query
from sqlalchemy.orm.session import Session
from sqlalchemy.sql.elements import BooleanClauseList
+from sqlalchemy.sql.expression import ColumnOperators
from sqlalchemy.sql.sqltypes import BigInteger
from airflow import settings
@@ -89,6 +93,7 @@
TaskDeferred,
UnmappableXComLengthPushed,
UnmappableXComTypePushed,
+ XComForMappingNotPushed,
)
from airflow.models.base import COLLATION_ARGS, ID_LEN, Base
from airflow.models.log import Log
@@ -100,6 +105,7 @@
from airflow.plugins_manager import integrate_macros_plugins
from airflow.sentry import Sentry
from airflow.stats import Stats
+from airflow.templates import SandboxedEnvironment
from airflow.ti_deps.dep_context import DepContext
from airflow.ti_deps.dependencies_deps import REQUEUEABLE_DEPS, RUNNING_DEPS
from airflow.timetables.base import DataInterval
@@ -114,7 +120,7 @@
from airflow.utils.platform import getuser
from airflow.utils.retries import run_with_db_retries
from airflow.utils.session import NEW_SESSION, create_session, provide_session
-from airflow.utils.sqlalchemy import ExtendedJSON, UtcDateTime, with_row_locks
+from airflow.utils.sqlalchemy import ExtendedJSON, UtcDateTime, tuple_in_condition, with_row_locks
from airflow.utils.state import DagRunState, State, TaskInstanceState
from airflow.utils.timeout import timeout
@@ -125,15 +131,14 @@
if TYPE_CHECKING:
+ from airflow.models.baseoperator import BaseOperator
from airflow.models.dag import DAG, DagModel
from airflow.models.dagrun import DagRun
from airflow.models.operator import Operator
-_TASK_EXECUTION_FRAME_LOCAL_STORAGE = threading.local()
-
@contextlib.contextmanager
-def set_current_context(context: Context) -> Iterator[Context]:
+def set_current_context(context: Context) -> Generator[Context, None, None]:
"""
Sets the current execution context to the provided context object.
This method should be called once per Task execution, before calling operator.execute.
@@ -151,31 +156,6 @@ def set_current_context(context: Context) -> Iterator[Context]:
)
-def load_error_file(fd: IO[bytes]) -> Optional[Union[str, Exception]]:
- """Load and return error from error file"""
- if fd.closed:
- return None
- fd.seek(0, os.SEEK_SET)
- data = fd.read()
- if not data:
- return None
- try:
- return pickle.loads(data)
- except Exception:
- return "Failed to load task run error"
-
-
-def set_error_file(error_file: str, error: Union[str, BaseException]) -> None:
- """Write error into error file by path"""
- with open(error_file, "wb") as fd:
- try:
- pickle.dump(error, fd)
- except Exception:
- # local class objects cannot be pickled, so we fallback
- # to store the string representation instead
- pickle.dump(str(error), fd)
-
-
def clear_task_instances(
tis,
session,
@@ -195,8 +175,9 @@ def clear_task_instances(
:param activate_dag_runs: Deprecated parameter, do not pass
"""
job_ids = []
- task_id_by_key: Dict[str, Dict[str, Dict[int, Set[str]]]] = defaultdict(
- lambda: defaultdict(lambda: defaultdict(set))
+ # Keys: dag_id -> run_id -> map_indexes -> try_numbers -> task_id
+ task_id_by_key: Dict[str, Dict[str, Dict[int, Dict[int, Set[str]]]]] = defaultdict(
+ lambda: defaultdict(lambda: defaultdict(lambda: defaultdict(set)))
)
for ti in tis:
if ti.state == TaskInstanceState.RUNNING:
@@ -222,14 +203,14 @@ def clear_task_instances(
ti.external_executor_id = None
session.merge(ti)
- task_id_by_key[ti.dag_id][ti.run_id][ti.try_number].add(ti.task_id)
+ task_id_by_key[ti.dag_id][ti.run_id][ti.map_index][ti.try_number].add(ti.task_id)
if task_id_by_key:
# Clear all reschedules related to the ti to clear
# This is an optimization for the common case where all tis are for a small number
- # of dag_id, run_id and try_number. Use a nested dict of dag_id,
- # run_id, try_number and task_id to construct the where clause in a
+ # of dag_id, run_id, try_number, and map_index. Use a nested dict of dag_id,
+ # run_id, try_number, map_index, and task_id to construct the where clause in a
# hierarchical manner. This speeds up the delete statement by more than 40x for
# large number of tis (50k+).
conditions = or_(
@@ -239,11 +220,17 @@ def clear_task_instances(
and_(
TR.run_id == run_id,
or_(
- and_(TR.try_number == try_number, TR.task_id.in_(task_ids))
- for try_number, task_ids in task_tries.items()
+ and_(
+ TR.map_index == map_index,
+ or_(
+ and_(TR.try_number == try_number, TR.task_id.in_(task_ids))
+ for try_number, task_ids in task_tries.items()
+ ),
+ )
+ for map_index, task_tries in map_indexes.items()
),
)
- for run_id, task_tries in run_ids.items()
+ for run_id, map_indexes in run_ids.items()
),
)
for dag_id, run_ids in task_id_by_key.items()
@@ -294,6 +281,91 @@ def clear_task_instances(
dr.start_date = None
+class _LazyXComAccessIterator(collections.abc.Iterator):
+ __slots__ = ['_cm', '_it']
+
+ def __init__(self, cm: ContextManager[Query]):
+ self._cm = cm
+ self._it = None
+
+ def __del__(self):
+ if self._it:
+ self._cm.__exit__(None, None, None)
+
+ def __iter__(self):
+ return self
+
+ def __next__(self):
+ if not self._it:
+ self._it = iter(self._cm.__enter__())
+ return XCom.deserialize_value(next(self._it))
+
+
+@attr.define
+class _LazyXComAccess(collections.abc.Sequence):
+ """Wrapper to lazily pull XCom with a sequence-like interface.
+
+ Note that since the session bound to the parent query may have died when we
+ actually access the sequence's content, we must create a new session
+ for every function call with ``with_session()``.
+ """
+
+ dag_id: str
+ run_id: str
+ task_id: str
+ _query: Query = attr.ib(repr=False)
+ _len: Optional[int] = attr.ib(init=False, repr=False, default=None)
+
+ @classmethod
+ def build_from_single_xcom(cls, first: "XCom", query: Query) -> "_LazyXComAccess":
+ return cls(
+ dag_id=first.dag_id,
+ run_id=first.run_id,
+ task_id=first.task_id,
+ query=query.with_entities(XCom.value)
+ .filter(
+ XCom.run_id == first.run_id,
+ XCom.task_id == first.task_id,
+ XCom.dag_id == first.dag_id,
+ XCom.map_index >= 0,
+ )
+ .order_by(None)
+ .order_by(XCom.map_index.asc()),
+ )
+
+ def __len__(self):
+ if self._len is None:
+ with self._get_bound_query() as query:
+ self._len = query.count()
+ return self._len
+
+ def __iter__(self):
+ return _LazyXComAccessIterator(self._get_bound_query())
+
+ def __getitem__(self, key):
+ if not isinstance(key, int):
+ raise ValueError("only support index access for now")
+ try:
+ with self._get_bound_query() as query:
+ r = query.offset(key).limit(1).one()
+ except NoResultFound:
+ raise IndexError(key) from None
+ return XCom.deserialize_value(r)
+
+ @contextlib.contextmanager
+ def _get_bound_query(self) -> Generator[Query, None, None]:
+ # Do we have a valid session already?
+ if self._query.session and self._query.session.is_active:
+ yield self._query
+ return
+
+ session = settings.Session()
+ try:
+ yield self._query.with_session(session)
+ finally:
+ session.close()
+
+
class TaskInstanceKey(NamedTuple):
"""Key used to identify task instance."""
@@ -340,6 +412,10 @@ class TaskInstance(Base, LoggingMixin):
Database transactions on this table should insure double triggers and
any confusion around what task instances are or aren't ready to run
even while multiple schedulers may be firing task instances.
+
+ A value of -1 in map_index represents any of: a TI without mapped tasks;
+ a TI with mapped tasks that has yet to be expanded (state=pending);
+ a TI with mapped tasks that expanded to an empty list (state=skipped).
"""
__tablename__ = "task_instance"
@@ -379,7 +455,7 @@ class TaskInstance(Base, LoggingMixin):
# The method to call next, and any extra arguments to pass to it.
# Usually used when resuming from DEFERRED.
next_method = Column(String(1000))
- next_kwargs = Column(ExtendedJSON)
+ next_kwargs = Column(MutableDict.as_mutable(ExtendedJSON))
# If adding new fields here then remember to add them to
# refresh_from_db() or they won't display in the UI correctly
@@ -443,7 +519,8 @@ def __init__(
self.task_id = task.task_id
self.map_index = map_index
self.refresh_from_task(task)
- self._log = logging.getLogger("airflow.task")
+ # init_on_load will config the log
+ self.init_on_load()
if run_id is None and execution_date is not None:
from airflow.models.dagrun import DagRun # Avoid circular import
@@ -486,7 +563,6 @@ def __init__(
if state:
self.state = state
self.hostname = ''
- self.init_on_load()
# Is this TaskInstance being currently running within `airflow tasks run --raw`.
# Not persisted to the database so only valid for the current process
self.raw = False
@@ -517,6 +593,8 @@ def insert_mapping(run_id: str, task: "Operator", map_index: int) -> dict:
@reconstructor
def init_on_load(self):
"""Initialize the attributes that aren't stored in the DB"""
+ # correctly config the ti log
+ self._log = logging.getLogger("airflow.task")
self.test_mode = False # can be changed when calling 'run'
@property
@@ -698,13 +776,12 @@ def log_url(self):
@property
def mark_success_url(self):
"""URL to mark TI success"""
- iso = quote(self.execution_date.isoformat())
base_url = conf.get('webserver', 'BASE_URL')
return base_url + (
"/confirm"
f"?task_id={self.task_id}"
f"&dag_id={self.dag_id}"
- f"&execution_date={iso}"
+ f"&dag_run_id={quote(self.run_id)}"
"&upstream=false"
"&downstream=false"
"&state=success"
@@ -719,20 +796,15 @@ def current_state(self, session=NEW_SESSION) -> str:
:param session: SQLAlchemy ORM Session
"""
- ti = (
- session.query(TaskInstance)
+ return (
+ session.query(TaskInstance.state)
.filter(
TaskInstance.dag_id == self.dag_id,
TaskInstance.task_id == self.task_id,
TaskInstance.run_id == self.run_id,
)
- .all()
+ .scalar()
)
- if ti:
- state = ti[0].state
- else:
- state = None
- return state
@provide_session
def error(self, session=NEW_SESSION):
@@ -819,17 +891,25 @@ def refresh_from_task(self, task: "Operator", pool_override=None):
self.operator = task.task_type
@provide_session
- def clear_xcom_data(self, session=NEW_SESSION):
- """
- Clears all XCom data from the database for the task instance
+ def clear_xcom_data(self, session: Session = NEW_SESSION):
+ """Clear all XCom data from the database for the task instance.
+
+ If the task is unmapped, all XComs matching this task ID in the same DAG
+ run are removed. If the task is mapped, only the one with matching map
+ index is removed.
:param session: SQLAlchemy ORM Session
"""
self.log.debug("Clearing XCom data")
+ if self.map_index < 0:
+ map_index: Optional[int] = None
+ else:
+ map_index = self.map_index
XCom.clear(
dag_id=self.dag_id,
task_id=self.task_id,
run_id=self.run_id,
+ map_index=map_index,
session=session,
)
@@ -1274,14 +1354,19 @@ def _date_or_empty(self, attr: str):
return result.strftime('%Y%m%dT%H%M%S') if result else ''
def _log_state(self, lead_msg: str = ''):
- self.log.info(
- '%sMarking task as %s.'
- ' dag_id=%s, task_id=%s,'
- ' execution_date=%s, start_date=%s, end_date=%s',
+ params = [
lead_msg,
str(self.state).upper(),
self.dag_id,
self.task_id,
+ ]
+ message = '%sMarking task as %s. dag_id=%s, task_id=%s, '
+ if self.map_index >= 0:
+ params.append(self.map_index)
+ message += 'map_index=%d, '
+ self.log.info(
+ message + 'execution_date=%s, start_date=%s, end_date=%s',
+ *params,
self._date_or_empty('execution_date'),
self._date_or_empty('start_date'),
self._date_or_empty('end_date'),
@@ -1303,7 +1388,6 @@ def _run_raw_task(
test_mode: bool = False,
job_id: Optional[str] = None,
pool: Optional[str] = None,
- error_file: Optional[str] = None,
session=NEW_SESSION,
) -> None:
"""
@@ -1328,10 +1412,14 @@ def _run_raw_task(
session.commit()
actual_start_date = timezone.utcnow()
Stats.incr(f'ti.start.{self.task.dag_id}.{self.task.task_id}')
+ # Initialize final state counters at zero
+ for state in State.task_states:
+ Stats.incr(f'ti.finish.{self.task.dag_id}.{self.task.task_id}.{state}', count=0)
+
+ self.task = self.task.prepare_for_execution()
+ context = self.get_template_context(ignore_param_exceptions=False)
try:
if not mark_success:
- self.task = self.task.prepare_for_execution()
- context = self.get_template_context(ignore_param_exceptions=False)
self._execute_task_with_callbacks(context, test_mode)
if not test_mode:
self.refresh_from_db(lock_for_update=True, session=session)
@@ -1370,7 +1458,7 @@ def _run_raw_task(
except (AirflowFailException, AirflowSensorTimeout) as e:
# If AirflowFailException is raised, task should not retry.
# If a sensor in reschedule mode reaches timeout, task should not retry.
- self.handle_failure(e, test_mode, force_fail=True, error_file=error_file, session=session)
+ self.handle_failure(e, test_mode, context, force_fail=True, session=session)
session.commit()
raise
except AirflowException as e:
@@ -1385,11 +1473,11 @@ def _run_raw_task(
session.commit()
return
else:
- self.handle_failure(e, test_mode, error_file=error_file, session=session)
+ self.handle_failure(e, test_mode, context, session=session)
session.commit()
raise
except (Exception, KeyboardInterrupt) as e:
- self.handle_failure(e, test_mode, error_file=error_file, session=session)
+ self.handle_failure(e, test_mode, context, session=session)
session.commit()
raise
finally:
@@ -1400,6 +1488,12 @@ def _run_raw_task(
self.end_date = timezone.utcnow()
self._log_state()
self.set_duration()
+
+ # run on_success_callback before db committing
+ # otherwise, the LocalTaskJob sees the state is changed to `success`,
+ # but the task_runner is still running, LocalTaskJob then treats the state is set externally!
+ self._run_finished_callback(self.task.on_success_callback, context, 'on_success')
+
if not test_mode:
session.add(Log(self.state, self))
session.merge(self)
@@ -1428,13 +1522,15 @@ def signal_handler(signum, frame):
signal.signal(signal.SIGTERM, signal_handler)
- # Don't clear Xcom until the task is certain to execute
- self.clear_xcom_data()
+ # Don't clear Xcom until the task is certain to execute, and check if we are resuming from deferral.
+ if not self.next_method:
+ self.clear_xcom_data()
+
with Stats.timer(f'dag.{self.task.dag_id}.{self.task.task_id}.duration'):
# Set the validated/merged params on the task object.
self.task.params = context['params']
- self.render_templates(context=context)
+ task_orig = self.render_templates(context=context)
if not test_mode:
rtif = RenderedTaskInstanceFields(ti=self, render_templates=False)
RenderedTaskInstanceFields.write(rtif)
@@ -1477,7 +1573,7 @@ def signal_handler(signum, frame):
# Execute the task
with set_current_context(context):
- result = self._execute_task(context, self.task)
+ result = self._execute_task(context, task_orig)
# Run post_execute callback
self.task.post_execute(context=context, result=result)
@@ -1495,8 +1591,17 @@ def _update_ti_state_for_sensing(self, session=NEW_SESSION):
# Raise exception for sensing state
raise AirflowSmartSensorException("Task successfully registered in smart sensor.")
- def _execute_task(self, context, task_copy):
+ def _run_finished_callback(self, callback, context, callback_type):
+ """Run callback after task finishes"""
+ try:
+ if callback:
+ callback(context)
+ except Exception: # pylint: disable=broad-except
+ self.log.exception(f"Error when executing {callback_type} callback")
+
+ def _execute_task(self, context, task_orig):
"""Executes Task (optionally with a Timeout) and pushes Xcom results"""
+ task_to_execute = self.task
# If the task has been deferred and is being executed due to a trigger,
# then we need to pick the right method to come back to, otherwise
# we go for the default execute
@@ -1510,43 +1615,42 @@ def _execute_task(self, context, task_copy):
self.log.error("Trigger failed:\n%s", "\n".join(traceback))
raise TaskDeferralError(next_kwargs.get("error", "Unknown"))
# Grab the callable off the Operator/Task and add in any kwargs
- execute_callable = getattr(task_copy, self.next_method)
+ execute_callable = getattr(task_to_execute, self.next_method)
if self.next_kwargs:
execute_callable = partial(execute_callable, **self.next_kwargs)
else:
- execute_callable = task_copy.execute
+ execute_callable = task_to_execute.execute
# If a timeout is specified for the task, make it fail
# if it goes beyond
- try:
- if task_copy.execution_timeout:
- # If we are coming in with a next_method (i.e. from a deferral),
- # calculate the timeout from our start_date.
- if self.next_method:
- timeout_seconds = (
- task_copy.execution_timeout - (timezone.utcnow() - self.start_date)
- ).total_seconds()
- else:
- timeout_seconds = task_copy.execution_timeout.total_seconds()
- try:
- # It's possible we're already timed out, so fast-fail if true
- if timeout_seconds <= 0:
- raise AirflowTaskTimeout()
- # Run task in timeout wrapper
- with timeout(timeout_seconds):
- result = execute_callable(context=context)
- except AirflowTaskTimeout:
- task_copy.on_kill()
- raise
+ if task_to_execute.execution_timeout:
+ # If we are coming in with a next_method (i.e. from a deferral),
+ # calculate the timeout from our start_date.
+ if self.next_method:
+ timeout_seconds = (
+ task_to_execute.execution_timeout - (timezone.utcnow() - self.start_date)
+ ).total_seconds()
else:
- result = execute_callable(context=context)
- except: # noqa: E722
- _TASK_EXECUTION_FRAME_LOCAL_STORAGE.frame = currentframe()
- raise
- # If the task returns a result, push an XCom containing it
- if task_copy.do_xcom_push and result is not None:
- with create_session() as session:
- self.xcom_push(key=XCOM_RETURN_KEY, value=result, session=session)
- self._record_task_map_for_downstreams(result, session=session)
+ timeout_seconds = task_to_execute.execution_timeout.total_seconds()
+ try:
+ # It's possible we're already timed out, so fast-fail if true
+ if timeout_seconds <= 0:
+ raise AirflowTaskTimeout()
+ # Run task in timeout wrapper
+ with timeout(timeout_seconds):
+ result = execute_callable(context=context)
+ except AirflowTaskTimeout:
+ task_to_execute.on_kill()
+ raise
+ else:
+ result = execute_callable(context=context)
+ with create_session() as session:
+ if task_to_execute.do_xcom_push:
+ xcom_value = result
+ else:
+ xcom_value = None
+ if xcom_value is not None: # If the task returns a result, push an XCom containing it.
+ self.xcom_push(key=XCOM_RETURN_KEY, value=xcom_value, session=session)
+ self._record_task_map_for_downstreams(task_orig, xcom_value, session=session)
return result
@provide_session
@@ -1596,40 +1700,6 @@ def _run_execute_callback(self, context: Context, task):
except Exception:
self.log.exception("Failed when executing execute callback")
- def _run_finished_callback(self, error: Optional[Union[str, Exception]] = None) -> None:
- """
- Call callback defined for finished state change.
-
- NOTE: Only invoke this function from caller of self._run_raw_task or
- self.run
- """
- if self.state == State.FAILED:
- task = self.task
- if task.on_failure_callback is not None:
- context = self.get_template_context()
- context["exception"] = error
- try:
- task.on_failure_callback(context)
- except Exception:
- self.log.exception("Error when executing on_failure_callback")
- elif self.state == State.SUCCESS:
- task = self.task
- if task.on_success_callback is not None:
- context = self.get_template_context()
- try:
- task.on_success_callback(context)
- except Exception:
- self.log.exception("Error when executing on_success_callback")
- elif self.state == State.UP_FOR_RETRY:
- task = self.task
- if task.on_retry_callback is not None:
- context = self.get_template_context()
- context["exception"] = error
- try:
- task.on_retry_callback(context)
- except Exception:
- self.log.exception("Error when executing on_retry_callback")
-
@provide_session
def run(
self,
@@ -1660,20 +1730,9 @@ def run(
if not res:
return
- try:
- error_fd = NamedTemporaryFile(delete=True)
- self._run_raw_task(
- mark_success=mark_success,
- test_mode=test_mode,
- job_id=job_id,
- pool=pool,
- error_file=error_fd.name,
- session=session,
- )
- finally:
- error = None if self.state == State.SUCCESS else load_error_file(error_fd)
- error_fd.close()
- self._run_finished_callback(error=error)
+ self._run_raw_task(
+ mark_success=mark_success, test_mode=test_mode, job_id=job_id, pool=pool, session=session
+ )
def dry_run(self):
"""Only Renders Templates for the TI"""
@@ -1734,76 +1793,57 @@ def _handle_reschedule(
session.commit()
self.log.info('Rescheduling task, marking task as UP_FOR_RESCHEDULE')
- def get_truncated_error_traceback(self, error: BaseException) -> Optional[TracebackType]:
+ @staticmethod
+ def get_truncated_error_traceback(error: BaseException, truncate_to: Callable) -> Optional[TracebackType]:
"""
- Returns truncated error traceback.
-
- This method returns traceback of the error truncated to the
- frame saved by earlier try/except along the way. If the frame
- is found, the traceback will be truncated to below the frame.
+ Truncates the traceback of an exception to the first frame called from within a given function
:param error: exception to get traceback from
- :return: traceback to print
+ :param truncate_to: Function to truncate TB to. Must have a ``__code__`` attribute
+
+ :meta private:
"""
tb = error.__traceback__
- try:
- execution_frame = _TASK_EXECUTION_FRAME_LOCAL_STORAGE.frame
- except AttributeError:
- self.log.warning(
- "We expected to get frame set in local storage but it was not."
- " Please report this as an issue with full logs"
- " at https://github.com/apache/airflow/issues/new",
- exc_info=True,
- )
- return tb
- _TASK_EXECUTION_FRAME_LOCAL_STORAGE.frame = None
+ code = truncate_to.__func__.__code__ # type: ignore[attr-defined]
while tb is not None:
- if tb.tb_frame is execution_frame:
- tb = tb.tb_next
- break
+ if tb.tb_frame.f_code is code:
+ return tb.tb_next
tb = tb.tb_next
return tb or error.__traceback__
@provide_session
- def handle_failure(
- self,
- error: Union[None, str, BaseException] = None,
- test_mode: Optional[bool] = None,
- force_fail: bool = False,
- error_file: Optional[str] = None,
- session: Session = NEW_SESSION,
- ) -> None:
+ def handle_failure(self, error, test_mode=None, context=None, force_fail=False, session=None) -> None:
"""Handle Failure for the TaskInstance"""
if test_mode is None:
test_mode = self.test_mode
+ if context is None:
+ context = self.get_template_context()
+
if error:
if isinstance(error, BaseException):
- tb = self.get_truncated_error_traceback(error)
+ tb = self.get_truncated_error_traceback(error, truncate_to=self._execute_task)
self.log.error("Task failed with exception", exc_info=(type(error), error, tb))
else:
self.log.error("%s", error)
- # external monitoring process provides pickle file so _run_raw_task
- # can send its runtime errors for access by failure callback
- if error_file:
- set_error_file(error_file, error)
if not test_mode:
self.refresh_from_db(session)
- task = self.task.unmap()
self.end_date = timezone.utcnow()
self.set_duration()
- Stats.incr(f'operator_failures_{task.task_type}', 1, 1)
+ Stats.incr(f'operator_failures_{self.task.task_type}')
Stats.incr('ti_failures')
if not test_mode:
session.add(Log(State.FAILED, self))
# Log failure duration
- dag_run = self.get_dagrun(session=session) # self.dag_run not populated by refresh_from_db
- session.add(TaskFail(task, dag_run.execution_date, self.start_date, self.end_date))
+ session.add(TaskFail(ti=self))
self.clear_next_method_args()
+ if context is not None:
+ context['exception'] = error
+
# Set state correctly and figure out how to log it and decide whether
# to email
@@ -1816,38 +1856,40 @@ def handle_failure(
# only mark task instance as FAILED if the next task instance
# try_number exceeds the max_tries ... or if force_fail is truthy
+ task = None
+ try:
+ task = self.task.unmap()
+ except Exception:
+ self.log.error("Unable to unmap task, can't determine if we need to send an alert email or not")
+
if force_fail or not self.is_eligible_to_retry():
self.state = State.FAILED
- email_for_state = task.email_on_failure
+ email_for_state = operator.attrgetter('email_on_failure')
+ callback = task.on_failure_callback if task else None
+ callback_type = 'on_failure'
else:
if self.state == State.QUEUED:
# We increase the try_number so as to fail the task if it fails to start after sometime
self._try_number += 1
self.state = State.UP_FOR_RETRY
- email_for_state = task.email_on_retry
+ email_for_state = operator.attrgetter('email_on_retry')
+ callback = task.on_retry_callback if task else None
+ callback_type = 'on_retry'
self._log_state('Immediate failure requested. ' if force_fail else '')
- if email_for_state and task.email:
+ if task and email_for_state(task) and task.email:
try:
- self.email_alert(error)
+ self.email_alert(error, task)
except Exception:
self.log.exception('Failed to send email to: %s', task.email)
+ if callback:
+ self._run_finished_callback(callback, context, callback_type)
+
if not test_mode:
session.merge(self)
session.flush()
- @provide_session
- def handle_failure_with_callback(
- self,
- error: Union[None, str, Exception],
- test_mode: Optional[bool] = None,
- force_fail: bool = False,
- session=NEW_SESSION,
- ) -> None:
- self.handle_failure(error=error, test_mode=test_mode, force_fail=force_fail, session=session)
- self._run_finished_callback(error=error)
-
def is_eligible_to_retry(self):
"""Is task instance is eligible for retry"""
if self.state == State.RESTARTING:
@@ -2042,17 +2084,27 @@ def get_prev_ds_nodash() -> Optional[str]:
@provide_session
def get_rendered_template_fields(self, session: Session = NEW_SESSION) -> None:
- """Fetch rendered template fields from DB"""
+ """
+ Update task with rendered template fields for presentation in UI.
+ If task has already run, will fetch from DB; otherwise will render.
+ """
from airflow.models.renderedtifields import RenderedTaskInstanceFields
rendered_task_instance_fields = RenderedTaskInstanceFields.get_templated_fields(self, session=session)
if rendered_task_instance_fields:
- task = self.task.unmap()
+ self.task = self.task.unmap()
for field_name, rendered_value in rendered_task_instance_fields.items():
setattr(self.task, field_name, rendered_value)
- self.task = task
+ return
+
try:
+ # If we get here, either the task hasn't run or the RTIF record was purged.
+ from airflow.utils.log.secrets_masker import redact
+
self.render_templates()
+ for field_name in self.task.template_fields:
+ rendered_value = getattr(self.task, field_name)
+ setattr(self.task, field_name, redact(rendered_value, field_name))
except (TemplateAssertionError, UndefinedError) as e:
raise AirflowException(
"Webserver does not have access to User-defined Macros or Filters "
@@ -2080,17 +2132,20 @@ def overwrite_params_with_dag_run_conf(self, params, dag_run):
self.log.debug("Updating task params (%s) with DagRun.conf (%s)", params, dag_run.conf)
params.update(dag_run.conf)
- def render_templates(self, context: Optional[Context] = None) -> None:
+ def render_templates(self, context: Optional[Context] = None) -> "Operator":
"""Render templates in the operator fields.
If the task was originally mapped, this may replace ``self.task`` with
- the unmapped, fully rendered BaseOperator.
+ the unmapped, fully rendered BaseOperator. The original ``self.task``
+ before replacement is returned.
"""
if not context:
context = self.get_template_context()
- task = self.task.render_template_fields(context)
- if task is not None:
- self.task = task
+ rendered_task = self.task.render_template_fields(context)
+ if rendered_task is None: # Compatibility -- custom renderer, assume unmapped.
+ return self.task
+ original_task, self.task = self.task, rendered_task
+ return original_task
def render_k8s_pod_yaml(self) -> Optional[dict]:
"""Render k8s pod yaml"""
@@ -2120,11 +2175,15 @@ def render_k8s_pod_yaml(self) -> Optional[dict]:
sanitized_pod = ApiClient().sanitize_for_serialization(pod)
return sanitized_pod
- def get_email_subject_content(self, exception: BaseException) -> Tuple[str, str, str]:
+ def get_email_subject_content(
+ self, exception: BaseException, task: Optional["BaseOperator"] = None
+ ) -> Tuple[str, str, str]:
"""Get the email subject content for exceptions."""
# For a ti from DB (without ti.task), return the default value
# Reuse it for smart sensor to send default email alert
- use_default = not hasattr(self, 'task')
+ if task is None:
+ task = getattr(self, 'task')
+ use_default = task is None
exception_html = str(exception).replace('\n', '
') default_subject = 'Airflow alert: {{ti}}' @@ -2167,13 +2226,20 @@ def get_email_subject_content(self, exception: BaseException) -> Tuple[str, str, html_content_err = jinja_env.from_string(default_html_content_err).render(**default_context) else: - jinja_env = self.task.get_template_env() + # Use the DAG's get_template_env() to set force_sandboxed. Don't add + # the flag to the function on task object -- that function can be + # overridden, and adding a flag breaks backward compatibility. + dag = self.task.get_dag() + if dag: + jinja_env = dag.get_template_env(force_sandboxed=True) + else: + jinja_env = SandboxedEnvironment(cache_size=0) jinja_context = self.get_template_context() context_merge(jinja_context, additional_context) def render(key: str, content: str) -> str: if conf.has_option('email', key): - path = conf.get('email', key) + path = conf.get_mandatory_value('email', key) with open(path) as f: content = f.read() return render_template_to_string(jinja_env.from_string(content), jinja_context) @@ -2184,13 +2250,14 @@ def render(key: str, content: str) -> str: return subject, html_content, html_content_err - def email_alert(self, exception): + def email_alert(self, exception, task: "BaseOperator"): """Send alert email with exception information.""" - subject, html_content, html_content_err = self.get_email_subject_content(exception) + subject, html_content, html_content_err = self.get_email_subject_content(exception, task=task) + assert task.email try: - send_email(self.task.email, subject, html_content) + send_email(task.email, subject, html_content) except Exception: - send_email(self.task.email, subject, html_content_err) + send_email(task.email, subject, html_content_err) def set_duration(self) -> None: """Set TI duration""" @@ -2200,15 +2267,24 @@ def set_duration(self) -> None: self.duration = None self.log.debug("Task Duration set to %s", self.duration) - def _record_task_map_for_downstreams(self, value: Any, *, session: Session) -> None: - if not self.task.has_mapped_dependants(): + def _record_task_map_for_downstreams(self, task: "Operator", value: Any, *, session: Session) -> None: + # TODO: We don't push TaskMap for mapped task instances because it's not + # currently possible for a downstream to depend on one individual mapped + # task instance, only a task as a whole. This will change in AIP-42 + # Phase 2, and we'll need to further analyze the mapped task case. + if next(task.iter_mapped_dependants(), None) is None: + return + if value is None: + raise XComForMappingNotPushed() + if task.is_mapped: return if not isinstance(value, collections.abc.Collection) or isinstance(value, (bytes, str)): raise UnmappableXComTypePushed(value) + task_map = TaskMap.from_task_instance_xcom(self, value) max_map_length = conf.getint("core", "max_map_length", fallback=1024) - if len(value) > max_map_length: + if task_map.length > max_map_length: raise UnmappableXComLengthPushed(value, max_map_length) - session.merge(TaskMap.from_task_instance_xcom(self, value)) + session.merge(task_map) @provide_session def xcom_push( @@ -2244,6 +2320,7 @@ def xcom_push( task_id=self.task_id, dag_id=self.dag_id, run_id=self.run_id, + map_index=self.map_index, session=session, ) @@ -2255,32 +2332,39 @@ def xcom_pull( key: str = XCOM_RETURN_KEY, include_prior_dates: bool = False, session: Session = NEW_SESSION, + *, + map_indexes: Optional[Union[int, Iterable[int]]] = None, + default: Any = None, ) -> Any: - """ - Pull XComs that optionally meet certain criteria. - - The default value for `key` limits the search to XComs - that were returned by other tasks (as opposed to those that were pushed - manually). To remove this filter, pass key=None (or any desired value). - - If a single task_id string is provided, the result is the value of the - most recent matching XCom from that task_id. If multiple task_ids are - provided, a tuple of matching values is returned. None is returned - whenever no matches are found. + """Pull XComs that optionally meet certain criteria. :param key: A key for the XCom. If provided, only XComs with matching - keys will be returned. The default key is 'return_value', also - available as a constant XCOM_RETURN_KEY. This key is automatically + keys will be returned. The default key is ``'return_value'``, also + available as constant ``XCOM_RETURN_KEY``. This key is automatically given to XComs returned by tasks (as opposed to being pushed - manually). To remove the filter, pass key=None. + manually). To remove the filter, pass *None*. :param task_ids: Only XComs from tasks with matching ids will be - pulled. Can pass None to remove the filter. - :param dag_id: If provided, only pulls XComs from this DAG. - If None (default), the DAG of the calling task is used. + pulled. Pass *None* to remove the filter. + :param dag_id: If provided, only pulls XComs from this DAG. If *None* + (default), the DAG of the calling task is used. + :param map_indexes: If provided, only pull XComs with matching indexes. + If *None* (default), this is inferred from the task(s) being pulled + (see below for details). :param include_prior_dates: If False, only XComs from the current - execution_date are returned. If True, XComs from previous dates + execution_date are returned. If *True*, XComs from previous dates are returned as well. - :param session: Sqlalchemy ORM Session + + When pulling one single task (``task_id`` is *None* or a str) without + specifying ``map_indexes``, the return value is inferred from whether + the specified task is mapped. If not, value from the one single task + instance is returned. If the task to pull is mapped, an iterator (not a + list) yielding XComs from mapped task instances is returned. In either + case, ``default`` (*None* if not specified) is returned if no matching + XComs are found. + + When pulling multiple tasks (i.e. either ``task_id`` or ``map_index`` is + a non-str iterable), a list of matching XComs is returned. Elements in + the list is ordered by item ordering in ``task_id`` and ``map_index``. """ if dag_id is None: dag_id = self.dag_id @@ -2290,25 +2374,53 @@ def xcom_pull( run_id=self.run_id, dag_ids=dag_id, task_ids=task_ids, + map_indexes=map_indexes, include_prior_dates=include_prior_dates, session=session, ) - # Since we're only fetching the values field, and not the - # whole class, the @recreate annotation does not kick in. - # Therefore we need to deserialize the fields by ourselves. - if task_ids is None or isinstance(task_ids, str): - xcom = query.with_entities(XCom.value).first() - if xcom: - return XCom.deserialize_value(xcom) + # NOTE: Since we're only fetching the value field and not the whole + # class, the @recreate annotation does not kick in. Therefore we need to + # call XCom.deserialize_value() manually. + + # We are only pulling one single task. + if (task_ids is None or isinstance(task_ids, str)) and not isinstance(map_indexes, Iterable): + first = query.with_entities( + XCom.run_id, XCom.task_id, XCom.dag_id, XCom.map_index, XCom.value + ).first() + if first is None: # No matching XCom at all. + return default + if map_indexes is not None or first.map_index < 0: + return XCom.deserialize_value(first) + + return _LazyXComAccess.build_from_single_xcom(first, query) + + # At this point either task_ids or map_indexes is explicitly multi-value. + + results = ( + (r.task_id, r.map_index, XCom.deserialize_value(r)) + for r in query.with_entities(XCom.task_id, XCom.map_index, XCom.value) + ) + + if task_ids is None: + task_id_pos: Dict[str, int] = defaultdict(int) + elif isinstance(task_ids, str): + task_id_pos = {task_ids: 0} + else: + task_id_pos = {task_id: i for i, task_id in enumerate(task_ids)} + if map_indexes is None: + map_index_pos: Dict[int, int] = defaultdict(int) + elif isinstance(map_indexes, int): + map_index_pos = {map_indexes: 0} else: - vals_kv = { - result.task_id: XCom.deserialize_value(result) - for result in query.with_entities(XCom.task_id, XCom.value) - } + map_index_pos = {map_index: i for i, map_index in enumerate(map_indexes)} - values_ordered_by_id = [vals_kv.get(task_id) for task_id in task_ids] - return values_ordered_by_id + def _arg_pos(item: Tuple[str, int, Any]) -> Tuple[int, int]: + task_id, map_index, _ = item + return task_id_pos[task_id], map_index_pos[map_index] + + results_sorted_by_arg_pos = sorted(results, key=_arg_pos) + return [value for _, _, value in results_sorted_by_arg_pos] @provide_session def get_num_running_task_instances(self, session): @@ -2369,20 +2481,35 @@ def filter_for_tis(tis: Iterable[Union["TaskInstance", TaskInstanceKey]]) -> Opt TaskInstance.task_id == first_task_id, ) - if settings.engine.dialect.name == 'mssql': - return or_( - and_( - TaskInstance.dag_id == ti.dag_id, - TaskInstance.task_id == ti.task_id, - TaskInstance.run_id == ti.run_id, - TaskInstance.map_index == ti.map_index, - ) - for ti in tis - ) - else: - return tuple_( - TaskInstance.dag_id, TaskInstance.task_id, TaskInstance.run_id, TaskInstance.map_index - ).in_([ti.key.primary for ti in tis]) + return tuple_in_condition( + (TaskInstance.dag_id, TaskInstance.task_id, TaskInstance.run_id, TaskInstance.map_index), + (ti.key.primary for ti in tis), + ) + + @classmethod + def ti_selector_condition(cls, vals: Collection[Union[str, Tuple[str, int]]]) -> ColumnOperators: + """ + Build an SQLAlchemy filter for a list where each element can contain + whether a task_id, or a tuple of (task_id,map_index) + + :meta private: + """ + # Compute a filter for TI.task_id and TI.map_index based on input values + # For each item, it will either be a task_id, or (task_id, map_index) + task_id_only = [v for v in vals if isinstance(v, str)] + with_map_index = [v for v in vals if not isinstance(v, str)] + + filters: List[ColumnOperators] = [] + if task_id_only: + filters.append(cls.task_id.in_(task_id_only)) + if with_map_index: + filters.append(tuple_in_condition((cls.task_id, cls.map_index), with_map_index)) + + if not filters: + return false() + if len(filters) == 1: + return filters[0] + return or_(*filters) # State of the task instance. @@ -2402,9 +2529,10 @@ def __init__( dag_id: str, task_id: str, run_id: str, - start_date: datetime, - end_date: datetime, + start_date: Optional[datetime], + end_date: Optional[datetime], try_number: int, + map_index: int, state: str, executor_config: Any, pool: str, @@ -2413,21 +2541,20 @@ def __init__( run_as_user: Optional[str] = None, priority_weight: Optional[int] = None, ): - self._dag_id: str = dag_id - self._task_id: str = task_id - self._run_id: str = run_id - self._start_date: datetime = start_date - self._end_date: datetime = end_date - self._try_number: int = try_number - self._state: str = state - self._executor_config: Any = executor_config - self._run_as_user: Optional[str] = None - self._run_as_user = run_as_user - self._pool: str = pool - self._priority_weight: Optional[int] = None - self._priority_weight = priority_weight - self._queue: str = queue - self._key = key + self.dag_id = dag_id + self.task_id = task_id + self.run_id = run_id + self.map_index = map_index + self.start_date = start_date + self.end_date = end_date + self.try_number = try_number + self.state = state + self.executor_config = executor_config + self.run_as_user = run_as_user + self.pool = pool + self.priority_weight = priority_weight + self.queue = queue + self.key = key def __eq__(self, other): if isinstance(other, self.__class__): @@ -2440,6 +2567,7 @@ def from_ti(cls, ti: TaskInstance): dag_id=ti.dag_id, task_id=ti.task_id, run_id=ti.run_id, + map_index=ti.map_index, start_date=ti.start_date, end_date=ti.end_date, try_number=ti.try_number, @@ -2454,80 +2582,16 @@ def from_ti(cls, ti: TaskInstance): @classmethod def from_dict(cls, obj_dict: dict): - ti_key = obj_dict.get('_key', []) - start_date: Union[Any, datetime] = ( - datetime.fromisoformat(str(obj_dict.get('_start_date'))) - if obj_dict.get('_start_date') is not None - else None - ) - end_date: Union[Any, datetime] = ( - datetime.fromisoformat(str(obj_dict.get('_end_date'))) - if obj_dict.get('_end_date') is not None - else None - ) - return cls( - dag_id=str(obj_dict['_dag_id']), - task_id=str(obj_dict.get('_task_id')), - run_id=str(obj_dict.get('_run_id')), - start_date=start_date, - end_date=end_date, - try_number=obj_dict.get('_try_number', 1), - state=str(obj_dict.get('_state')), - executor_config=obj_dict.get('_executor_config'), - run_as_user=obj_dict.get('_run_as_user', None), - pool=str(obj_dict.get('_pool')), - priority_weight=obj_dict.get('_priority_weight', None), - queue=str(obj_dict.get('_queue')), - key=TaskInstanceKey(ti_key[0], ti_key[1], ti_key[2], ti_key[3], ti_key[4]), - ) - - @property - def dag_id(self) -> str: - return self._dag_id - - @property - def task_id(self) -> str: - return self._task_id - - @property - def run_id(self) -> str: - return self._run_id - - @property - def start_date(self) -> datetime: - return self._start_date - - @property - def end_date(self) -> datetime: - return self._end_date - - @property - def try_number(self) -> int: - return self._try_number - - @property - def state(self) -> str: - return self._state - - @property - def pool(self) -> str: - return self._pool - - @property - def priority_weight(self) -> Optional[int]: - return self._priority_weight - - @property - def queue(self) -> str: - return self._queue - - @property - def key(self) -> TaskInstanceKey: - return self._key - - @property - def executor_config(self): - return self._executor_config + ti_key = TaskInstanceKey(*obj_dict.pop('key')) + start_date = None + end_date = None + start_date_str: Optional[str] = obj_dict.pop('start_date') + end_date_str: Optional[str] = obj_dict.pop('end_date') + if start_date_str: + start_date = timezone.parse(start_date_str) + if end_date_str: + end_date = timezone.parse(end_date_str) + return cls(**obj_dict, start_date=start_date, end_date=end_date, key=ti_key) STATICA_HACK = True diff --git a/airflow/models/taskmixin.py b/airflow/models/taskmixin.py index ad6404869eb36..1d66a719a6aec 100644 --- a/airflow/models/taskmixin.py +++ b/airflow/models/taskmixin.py @@ -28,6 +28,7 @@ from logging import Logger from airflow.models.dag import DAG + from airflow.models.mappedoperator import MappedOperator from airflow.utils.edgemodifier import EdgeModifier from airflow.utils.task_group import TaskGroup @@ -54,29 +55,16 @@ def leaves(self) -> Sequence["DependencyMixin"]: raise NotImplementedError() @abstractmethod - def set_upstream( - self, - other: Union["DependencyMixin", Sequence["DependencyMixin"]], - edge_modifier: Optional["EdgeModifier"] = None, - ): + def set_upstream(self, other: Union["DependencyMixin", Sequence["DependencyMixin"]]): """Set a task or a task list to be directly upstream from the current task.""" raise NotImplementedError() @abstractmethod - def set_downstream( - self, - other: Union["DependencyMixin", Sequence["DependencyMixin"]], - edge_modifier: Optional["EdgeModifier"] = None, - ): + def set_downstream(self, other: Union["DependencyMixin", Sequence["DependencyMixin"]]): """Set a task or a task list to be directly downstream from the current task.""" raise NotImplementedError() - def update_relative( - self, - other: "DependencyMixin", - upstream=True, - edge_modifier: Optional["EdgeModifier"] = None, - ) -> None: + def update_relative(self, other: "DependencyMixin", upstream=True) -> None: """ Update relationship information about another TaskMixin. Default is no-op. Override if necessary. @@ -146,6 +134,13 @@ def label(self) -> Optional[str]: def has_dag(self) -> bool: return self.dag is not None + @property + def dag_id(self) -> str: + """Returns dag id if it has one or an adhoc/meaningless ID""" + if self.dag: + return self.dag.dag_id + return "_in_memory_dag_" + @property def log(self) -> "Logger": raise NotImplementedError() @@ -176,7 +171,7 @@ def _set_relatives( task_list: List[Operator] = [] for task_object in task_or_task_list: - task_object.update_relative(self, not upstream, edge_modifier=edge_modifier) + task_object.update_relative(self, not upstream) relatives = task_object.leaves if upstream else task_object.roots for task in relatives: if not isinstance(task, (BaseOperator, MappedOperator)): @@ -283,16 +278,23 @@ def serialize_for_task_group(self) -> Tuple[DagAttributeTypes, Any]: """This is used by SerializedTaskGroup to serialize a task group's content.""" raise NotImplementedError() - def mapped_dependants(self) -> Iterator["DAGNode"]: - """Return any mapped nodes that are direct dependencies of the current task + def _iter_all_mapped_downstreams(self) -> Iterator["MappedOperator"]: + """Return mapped nodes that are direct dependencies of the current task. For now, this walks the entire DAG to find mapped nodes that has this current task as an upstream. We cannot use ``downstream_list`` since it only contains operators, not task groups. In the future, we should provide a way to record an DAG node's all downstream nodes instead. + + Note that this does not guarantee the returned tasks actually use the + current task for task mapping, but only checks those task are mapped + operators, and are downstreams of the current task. + + To get a list of tasks that uses the current task for task mapping, use + :meth:`iter_mapped_dependants` instead. """ from airflow.models.mappedoperator import MappedOperator - from airflow.utils.task_group import MappedTaskGroup, TaskGroup + from airflow.utils.task_group import TaskGroup def _walk_group(group: TaskGroup) -> Iterable[Tuple[str, DAGNode]]: """Recursively walk children in a task group. @@ -307,21 +309,25 @@ def _walk_group(group: TaskGroup) -> Iterable[Tuple[str, DAGNode]]: tg = self.task_group if not tg: - raise RuntimeError("Cannot check for mapped_dependants when not attached to a DAG") + raise RuntimeError("Cannot check for mapped dependants when not attached to a DAG") for key, child in _walk_group(tg): if key == self.node_id: continue - if not isinstance(child, (MappedOperator, MappedTaskGroup)): + if not isinstance(child, MappedOperator): continue if self.node_id in child.upstream_task_ids: yield child - def has_mapped_dependants(self) -> bool: - """Whether any downstream dependencies depend on this task for mapping. + def iter_mapped_dependants(self) -> Iterator["MappedOperator"]: + """Return mapped nodes that depend on the current task the expansion. For now, this walks the entire DAG to find mapped nodes that has this current task as an upstream. We cannot use ``downstream_list`` since it only contains operators, not task groups. In the future, we should provide a way to record an DAG node's all downstream nodes instead. """ - return any(self.mapped_dependants()) + return ( + downstream + for downstream in self._iter_all_mapped_downstreams() + if any(p.node_id == self.node_id for p in downstream.iter_mapped_dependencies()) + ) diff --git a/airflow/models/trigger.py b/airflow/models/trigger.py index c1ccdd4964678..dc91fdccdf2ba 100644 --- a/airflow/models/trigger.py +++ b/airflow/models/trigger.py @@ -24,6 +24,7 @@ from airflow.models.taskinstance import TaskInstance from airflow.triggers.base import BaseTrigger from airflow.utils import timezone +from airflow.utils.retries import run_with_db_retries from airflow.utils.session import provide_session from airflow.utils.sqlalchemy import ExtendedJSON, UtcDateTime from airflow.utils.state import State @@ -88,9 +89,11 @@ def clean_unused(cls, session=None): (triggers have a one-to-many relationship to both) """ # Update all task instances with trigger IDs that are not DEFERRED to remove them - session.query(TaskInstance).filter( - TaskInstance.state != State.DEFERRED, TaskInstance.trigger_id.isnot(None) - ).update({TaskInstance.trigger_id: None}) + for attempt in run_with_db_retries(): + with attempt: + session.query(TaskInstance).filter( + TaskInstance.state != State.DEFERRED, TaskInstance.trigger_id.isnot(None) + ).update({TaskInstance.trigger_id: None}) # Get all triggers that have no task instances depending on them... ids = [ trigger_id @@ -168,7 +171,7 @@ def assign_unassigned(cls, triggerer_id, capacity, session=None): """ from airflow.jobs.base_job import BaseJob # To avoid circular import - count = session.query(cls.id).filter(cls.triggerer_id == triggerer_id).count() + count = session.query(func.count(cls.id)).filter(cls.triggerer_id == triggerer_id).scalar() capacity -= count if capacity <= 0: diff --git a/airflow/models/xcom.py b/airflow/models/xcom.py index 7456fd293d778..d67160d1fa901 100644 --- a/airflow/models/xcom.py +++ b/airflow/models/xcom.py @@ -26,7 +26,7 @@ from typing import TYPE_CHECKING, Any, Iterable, List, Optional, Type, Union, cast, overload import pendulum -from sqlalchemy import Column, Index, Integer, LargeBinary, String +from sqlalchemy import Column, ForeignKeyConstraint, Index, Integer, LargeBinary, String from sqlalchemy.ext.associationproxy import association_proxy from sqlalchemy.orm import Query, Session, reconstructor, relationship from sqlalchemy.orm.exc import NoResultFound @@ -46,10 +46,6 @@ MAX_XCOM_SIZE = 49344 XCOM_RETURN_KEY = 'return_value' -# Stand-in value for 'airflow task test' generating a temporary in-memory DAG -# run without storing it in the database. -IN_MEMORY_RUN_ID = "__airflow_in_memory_dagrun__" - if TYPE_CHECKING: from airflow.models.taskinstance import TaskInstanceKey @@ -71,26 +67,33 @@ class BaseXCom(Base, LoggingMixin): value = Column(LargeBinary) timestamp = Column(UtcDateTime, default=timezone.utcnow, nullable=False) + __table_args__ = ( + # Ideally we should create a unique index over (key, dag_id, task_id, run_id), + # but it goes over MySQL's index length limit. So we instead index 'key' + # separately, and enforce uniqueness with DagRun.id instead. + Index("idx_xcom_key", key), + ForeignKeyConstraint( + [dag_id, task_id, run_id, map_index], + [ + "task_instance.dag_id", + "task_instance.task_id", + "task_instance.run_id", + "task_instance.map_index", + ], + name="xcom_task_instance_fkey", + ondelete="CASCADE", + ), + ) + dag_run = relationship( "DagRun", - primaryjoin="""and_( - BaseXCom.dag_id == foreign(DagRun.dag_id), - BaseXCom.run_id == foreign(DagRun.run_id), - )""", + primaryjoin="BaseXCom.dag_run_id == foreign(DagRun.id)", uselist=False, lazy="joined", passive_deletes="all", ) execution_date = association_proxy("dag_run", "execution_date") - __table_args__ = ( - # Ideally we should create a unique index over (key, dag_id, task_id, run_id), - # but it goes over MySQL's index length limit. So we instead create indexes - # separately, and enforce uniqueness with DagRun.id instead. - Index("idx_xcom_key", key), - Index("idx_xcom_ti_id", dag_id, task_id, run_id, map_index), - ) - @reconstructor def init_on_load(self): """ @@ -100,7 +103,9 @@ def init_on_load(self): self.value = self.orm_deserialize_value() def __repr__(self): - return f''
+ if self.map_index < 0:
+ return f''
+ return f''
@overload
@classmethod
@@ -162,7 +167,10 @@ def set(
from airflow.models.dagrun import DagRun
if not exactly_one(execution_date is not None, run_id is not None):
- raise ValueError("Exactly one of run_id or execution_date must be passed")
+ raise ValueError(
+ f"Exactly one of run_id or execution_date must be passed. "
+ f"Passed execution_date={execution_date}, run_id={run_id}"
+ )
if run_id is None:
message = "Passing 'execution_date' to 'XCom.set()' is deprecated. Use 'run_id' instead."
@@ -175,8 +183,6 @@ def set(
)
except NoResultFound:
raise ValueError(f"DAG run not found on DAG {dag_id!r} at {execution_date}") from None
- elif run_id == IN_MEMORY_RUN_ID:
- dag_run_id = -1
else:
dag_run_id = session.query(DagRun.id).filter_by(dag_id=dag_id, run_id=run_id).scalar()
if dag_run_id is None:
@@ -197,6 +203,7 @@ def set(
cls.run_id == run_id,
cls.task_id == task_id,
cls.dag_id == dag_id,
+ cls.map_index == map_index,
).delete()
new = cast(Any, cls)( # Work around Mypy complaining model not defining '__init__'.
dag_run_id=dag_run_id,
@@ -205,6 +212,7 @@ def set(
run_id=run_id,
task_id=task_id,
dag_id=dag_id,
+ map_index=map_index,
)
session.add(new)
session.flush()
@@ -378,9 +386,9 @@ def get_many(
keys will be returned. Pass *None* (default) to remove the filter.
:param task_ids: Only XComs from task with matching IDs will be pulled.
Pass *None* (default) to remove the filter.
- :param dag_id: Only pulls XComs from this DAG. If *None* (default), the
- DAG of the calling task is used.
- :param map_index: Only XComs from matching map indexes will be pulled.
+ :param dag_ids: Only pulls XComs from specified DAGs. Pass *None*
+ (default) to remove the filter.
+ :param map_indexes: Only XComs from matching map indexes will be pulled.
Pass *None* (default) to remove the filter.
:param include_prior_dates: If *False* (default), only XComs from the
specified DAG run are returned. If *True*, all matching XComs are
@@ -423,7 +431,10 @@ def get_many(
from airflow.models.dagrun import DagRun
if not exactly_one(execution_date is not None, run_id is not None):
- raise ValueError("Exactly one of run_id or execution_date must be passed")
+ raise ValueError(
+ f"Exactly one of run_id or execution_date must be passed. "
+ f"Passed execution_date={execution_date}, run_id={run_id}"
+ )
if execution_date is not None:
message = "Passing 'execution_date' to 'XCom.get_many()' is deprecated. Use 'run_id' instead."
warnings.warn(message, PendingDeprecationWarning, stacklevel=3)
@@ -452,8 +463,6 @@ def get_many(
if execution_date is not None:
query = query.filter(DagRun.execution_date <= execution_date)
else:
- # This returns an empty query result for IN_MEMORY_RUN_ID,
- # but that is impossible to implement. Sorry?
dr = session.query(DagRun.execution_date).filter(DagRun.run_id == run_id).subquery()
query = query.filter(cls.execution_date <= dr.c.execution_date)
elif execution_date is not None:
@@ -537,7 +546,10 @@ def clear(
raise TypeError("clear() missing required argument: task_id")
if not exactly_one(execution_date is not None, run_id is not None):
- raise ValueError("Exactly one of run_id or execution_date must be passed")
+ raise ValueError(
+ f"Exactly one of run_id or execution_date must be passed. "
+ f"Passed execution_date={execution_date}, run_id={run_id}"
+ )
if execution_date is not None:
message = "Passing 'execution_date' to 'XCom.clear()' is deprecated. Use 'run_id' instead."
diff --git a/airflow/models/xcom_arg.py b/airflow/models/xcom_arg.py
index 2234d1dec392f..2c2a8f9b5868d 100644
--- a/airflow/models/xcom_arg.py
+++ b/airflow/models/xcom_arg.py
@@ -14,15 +14,20 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-from typing import TYPE_CHECKING, Any, List, Optional, Sequence, Union
+from typing import TYPE_CHECKING, Any, Iterator, List, Optional, Sequence, Union
from airflow.exceptions import AirflowException
+from airflow.models.abstractoperator import AbstractOperator
from airflow.models.taskmixin import DAGNode, DependencyMixin
from airflow.models.xcom import XCOM_RETURN_KEY
from airflow.utils.context import Context
from airflow.utils.edgemodifier import EdgeModifier
+from airflow.utils.session import NEW_SESSION, provide_session
+from airflow.utils.types import NOTSET
if TYPE_CHECKING:
+ from sqlalchemy.orm import Session
+
from airflow.models.operator import Operator
@@ -134,39 +139,48 @@ def set_downstream(
"""Proxy to underlying operator set_downstream method. Required by TaskMixin."""
self.operator.set_downstream(task_or_task_list, edge_modifier)
- def resolve(self, context: Context) -> Any:
+ @provide_session
+ def resolve(self, context: Context, session: "Session" = NEW_SESSION) -> Any:
"""
Pull XCom value for the existing arg. This method is run during ``op.execute()``
in respectable context.
"""
- resolved_value = context['ti'].xcom_pull(task_ids=[self.operator.task_id], key=str(self.key))
- if not resolved_value:
- if TYPE_CHECKING:
- assert self.operator.dag
+ result = context["ti"].xcom_pull(
+ task_ids=self.operator.task_id, key=str(self.key), default=NOTSET, session=session
+ )
+ if result is NOTSET:
raise AirflowException(
- f'XComArg result from {self.operator.task_id} at {self.operator.dag.dag_id} '
- f'with key="{self.key}"" is not found!'
+ f'XComArg result from {self.operator.task_id} at {context["ti"].dag_id} '
+ f'with key="{self.key}" is not found!'
)
- resolved_value = resolved_value[0]
-
- return resolved_value
+ return result
@staticmethod
- def apply_upstream_relationship(op: "Operator", arg: Any):
- """
- Set dependency for XComArgs.
+ def iter_xcom_args(arg: Any) -> Iterator["XComArg"]:
+ """Return XComArg instances in an arbitrary value.
- This looks for XComArg objects in ``arg`` "deeply" (looking inside lists, dicts and classes decorated
- with "template_fields") and sets the relationship to ``op`` on any found.
+ This recursively traverse ``arg`` and look for XComArg instances in any
+ collection objects, and instances with ``template_fields`` set.
"""
if isinstance(arg, XComArg):
- op.set_upstream(arg.operator)
+ yield arg
elif isinstance(arg, (tuple, set, list)):
for elem in arg:
- XComArg.apply_upstream_relationship(op, elem)
+ yield from XComArg.iter_xcom_args(elem)
elif isinstance(arg, dict):
for elem in arg.values():
- XComArg.apply_upstream_relationship(op, elem)
- elif hasattr(arg, "template_fields"):
+ yield from XComArg.iter_xcom_args(elem)
+ elif isinstance(arg, AbstractOperator):
for elem in arg.template_fields:
- XComArg.apply_upstream_relationship(op, elem)
+ yield from XComArg.iter_xcom_args(elem)
+
+ @staticmethod
+ def apply_upstream_relationship(op: "Operator", arg: Any):
+ """Set dependency for XComArgs.
+
+ This looks for XComArg objects in ``arg`` "deeply" (looking inside
+ collections objects and classes decorated with ``template_fields``), and
+ sets the relationship to ``op`` on any found.
+ """
+ for ref in XComArg.iter_xcom_args(arg):
+ op.set_upstream(ref.operator)
diff --git a/airflow/operators/bash.py b/airflow/operators/bash.py
index 0f4c8b305b95f..a4fac2b7f7eab 100644
--- a/airflow/operators/bash.py
+++ b/airflow/operators/bash.py
@@ -21,7 +21,7 @@
from airflow.compat.functools import cached_property
from airflow.exceptions import AirflowException, AirflowSkipException
from airflow.hooks.subprocess import SubprocessHook
-from airflow.models import BaseOperator
+from airflow.models.baseoperator import BaseOperator
from airflow.utils.context import Context
from airflow.utils.operator_helpers import context_to_airflow_vars
diff --git a/airflow/operators/datetime.py b/airflow/operators/datetime.py
index c37a4f9d50c11..c5a423d563868 100644
--- a/airflow/operators/datetime.py
+++ b/airflow/operators/datetime.py
@@ -16,6 +16,7 @@
# under the License.
import datetime
+import warnings
from typing import Iterable, Union
from airflow.exceptions import AirflowException
@@ -39,7 +40,7 @@ class BranchDateTimeOperator(BaseBranchOperator):
``datetime.datetime.now()`` falls below target_lower or above ``target_upper``.
:param target_lower: target lower bound.
:param target_upper: target upper bound.
- :param use_task_execution_date: If ``True``, uses task's execution day to compare with targets.
+ :param use_task_logical_date: If ``True``, uses task's logical date to compare with targets.
Execution date is useful for backfilling. If ``False``, uses system's date.
"""
@@ -50,6 +51,7 @@ def __init__(
follow_task_ids_if_false: Union[str, Iterable[str]],
target_lower: Union[datetime.datetime, datetime.time, None],
target_upper: Union[datetime.datetime, datetime.time, None],
+ use_task_logical_date: bool = False,
use_task_execution_date: bool = False,
**kwargs,
) -> None:
@@ -64,10 +66,17 @@ def __init__(
self.target_upper = target_upper
self.follow_task_ids_if_true = follow_task_ids_if_true
self.follow_task_ids_if_false = follow_task_ids_if_false
- self.use_task_execution_date = use_task_execution_date
+ self.use_task_logical_date = use_task_logical_date
+ if use_task_execution_date:
+ self.use_task_logical_date = use_task_execution_date
+ warnings.warn(
+ "Parameter ``use_task_execution_date`` is deprecated. Use ``use_task_logical_date``.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
def choose_branch(self, context: Context) -> Union[str, Iterable[str]]:
- if self.use_task_execution_date is True:
+ if self.use_task_logical_date:
now = timezone.make_naive(context["logical_date"], self.dag.timezone)
else:
now = timezone.make_naive(timezone.utcnow(), self.dag.timezone)
diff --git a/airflow/operators/dummy.py b/airflow/operators/dummy.py
index 61907c08569c2..b2912e92586f0 100644
--- a/airflow/operators/dummy.py
+++ b/airflow/operators/dummy.py
@@ -15,23 +15,31 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-from airflow.models import BaseOperator
-from airflow.utils.context import Context
+"""This module is deprecated. Please use :mod:`airflow.operators.empty`."""
+import warnings
-class DummyOperator(BaseOperator):
- """
- Operator that does literally nothing. It can be used to group tasks in a
- DAG.
+from airflow.operators.empty import EmptyOperator
- The task is evaluated by the scheduler but never processed by the executor.
- """
+warnings.warn(
+ "This module is deprecated. Please use `airflow.operators.empty`.",
+ DeprecationWarning,
+ stacklevel=2,
+)
- ui_color = '#e8f7e4'
- inherits_from_dummy_operator = True
- def __init__(self, **kwargs) -> None:
- super().__init__(**kwargs)
+class DummyOperator(EmptyOperator):
+ """This class is deprecated. Please use `airflow.operators.empty.EmptyOperator`."""
+
+ @property
+ def inherits_from_dummy_operator(self):
+ return True
- def execute(self, context: Context):
- pass
+ def __init__(self, **kwargs):
+ warnings.warn(
+ """This class is deprecated. Please use `airflow.operators.empty.EmptyOperator`.""",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ self.inherits_from_empty_operator = self.inherits_from_dummy_operator
+ super().__init__(**kwargs)
diff --git a/airflow/operators/dummy_operator.py b/airflow/operators/dummy_operator.py
index 6b2fbeeb5bff4..2b46095027875 100644
--- a/airflow/operators/dummy_operator.py
+++ b/airflow/operators/dummy_operator.py
@@ -15,12 +15,24 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-"""This module is deprecated. Please use :mod:`airflow.operators.dummy`."""
+"""This module is deprecated. Please use :mod:`airflow.operators.empty`."""
import warnings
-from airflow.operators.dummy import DummyOperator # noqa
+from airflow.operators.empty import EmptyOperator
warnings.warn(
- "This module is deprecated. Please use `airflow.operators.dummy`.", DeprecationWarning, stacklevel=2
+ "This module is deprecated. Please use `airflow.operators.empty`.", DeprecationWarning, stacklevel=2
)
+
+
+class DummyOperator(EmptyOperator):
+ """This class is deprecated. Please use `airflow.operators.empty.EmptyOperator`."""
+
+ def __init__(self, *args, **kwargs):
+ warnings.warn(
+ """This class is deprecated. Please use `airflow.operators.empty.EmptyOperator`.""",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ super().__init__(*args, **kwargs)
diff --git a/airflow/compat/asyncio.py b/airflow/operators/empty.py
similarity index 64%
rename from airflow/compat/asyncio.py
rename to airflow/operators/empty.py
index a999ce604d268..1f396d0f6a664 100644
--- a/airflow/compat/asyncio.py
+++ b/airflow/operators/empty.py
@@ -15,14 +15,20 @@
# specific language governing permissions and limitations
# under the License.
-try:
- from asyncio import create_task
-except ImportError:
- # create_task is not present in Python 3.6. Once Airflow is at 3.7+, we can
- # remove this helper.
- def create_task(*args, **kwargs): # type: ignore
- """A version of create_task that always errors."""
- raise RuntimeError("Airflow's async functionality is only available on Python 3.7+")
+from airflow.models.baseoperator import BaseOperator
+from airflow.utils.context import Context
-__all__ = ["create_task"]
+class EmptyOperator(BaseOperator):
+ """
+ Operator that does literally nothing. It can be used to group tasks in a
+ DAG.
+
+ The task is evaluated by the scheduler but never processed by the executor.
+ """
+
+ ui_color = '#e8f7e4'
+ inherits_from_empty_operator = True
+
+ def execute(self, context: Context):
+ pass
diff --git a/airflow/operators/python.py b/airflow/operators/python.py
index 3619033f3ca55..920a75d08f766 100644
--- a/airflow/operators/python.py
+++ b/airflow/operators/python.py
@@ -29,7 +29,7 @@
import dill
from airflow.exceptions import AirflowException
-from airflow.models import BaseOperator
+from airflow.models.baseoperator import BaseOperator
from airflow.models.skipmixin import SkipMixin
from airflow.models.taskinstance import _CURRENT_CONTEXT
from airflow.utils.context import Context, context_copy_partial, context_merge
@@ -210,8 +210,10 @@ def execute(self, context: Context) -> Any:
branches = {branch}
elif isinstance(branch, list):
branches = set(branch)
+ elif branch is None:
+ branches = set()
else:
- raise AirflowException("Branch callable must return either a task ID or a list of IDs")
+ raise AirflowException("Branch callable must return either None, a task ID, or a list of IDs")
valid_task_ids = set(context["dag"].task_ids)
invalid_task_ids = branches - valid_task_ids
if invalid_task_ids:
@@ -325,7 +327,8 @@ class PythonVirtualenvOperator(PythonOperator):
processing templated fields, for examples ``['.sql', '.hql']``
"""
- template_fields: Sequence[str] = ('requirements',)
+ template_fields: Sequence[str] = tuple({'requirements'} | set(PythonOperator.template_fields))
+
template_ext: Sequence[str] = ('.txt',)
BASE_SERIALIZABLE_CONTEXT_KEYS = {
'ds',
diff --git a/scripts/ci/static_checks/ui_lint.sh b/airflow/operators/smooth.py
old mode 100755
new mode 100644
similarity index 59%
rename from scripts/ci/static_checks/ui_lint.sh
rename to airflow/operators/smooth.py
index 808467b758016..9dcbccb9e0a2f
--- a/scripts/ci/static_checks/ui_lint.sh
+++ b/airflow/operators/smooth.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,17 +15,21 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+from airflow.models.baseoperator import BaseOperator
+from airflow.utils.context import Context
-export MOUNT_ALL_LOCAL_SOURCES="false"
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
+class SmoothOperator(BaseOperator):
+ """
+ Operator that does literally nothing but it logs YouTube link to
+ Sade song "Smooth Operator".
+ """
-build_images::prepare_ci_build
+ ui_color = '#e8f7e4'
+ yt_link: str = "https://www.youtube.com/watch?v=4TYv2PhG89A"
-build_images::rebuild_ci_image_if_needed
+ def __init__(self, **kwargs) -> None:
+ super().__init__(**kwargs)
-docker run "${EXTRA_DOCKER_FLAGS[@]}" \
- --entrypoint "/bin/bash" \
- "${AIRFLOW_CI_IMAGE_WITH_TAG}" \
- -c 'cd airflow/ui && yarn --frozen-lockfile --non-interactive && yarn run lint "${@}"' "${@#airflow/ui/}"
+ def execute(self, context: Context):
+ self.log.info("Enjoy Sade - Smooth Operator: %s", self.yt_link)
diff --git a/airflow/operators/subdag.py b/airflow/operators/subdag.py
index 217ec5c7cd77a..bd81314dda8fd 100644
--- a/airflow/operators/subdag.py
+++ b/airflow/operators/subdag.py
@@ -115,14 +115,9 @@ def _validate_pool(self, session):
pool = session.query(Pool).filter(Pool.slots == 1).filter(Pool.pool == self.pool).first()
if pool and any(t.pool == self.pool for t in self.subdag.tasks):
raise AirflowException(
- 'SubDagOperator {sd} and subdag task{plural} {t} both '
- 'use pool {p}, but the pool only has 1 slot. The '
- 'subdag tasks will never run.'.format(
- sd=self.task_id,
- plural=len(conflicts) > 1,
- t=', '.join(t.task_id for t in conflicts),
- p=self.pool,
- )
+ f"SubDagOperator {self.task_id} and subdag task{'s' if len(conflicts) > 1 else ''} "
+ f"{', '.join(t.task_id for t in conflicts)} both use pool {self.pool}, "
+ f"but the pool only has 1 slot. The subdag tasks will never run."
)
def _get_dagrun(self, execution_date):
diff --git a/airflow/operators/weekday.py b/airflow/operators/weekday.py
index fb35079fe0d3e..b23d57e9fb1d4 100644
--- a/airflow/operators/weekday.py
+++ b/airflow/operators/weekday.py
@@ -15,7 +15,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
+import warnings
from typing import Iterable, Union
from airflow.operators.branch import BaseBranchOperator
@@ -41,7 +41,7 @@ class BranchDayOfWeekOperator(BaseBranchOperator):
* ``{WeekDay.TUESDAY}``
* ``{WeekDay.SATURDAY, WeekDay.SUNDAY}``
- :param use_task_execution_day: If ``True``, uses task's execution day to compare
+ :param use_task_logical_date: If ``True``, uses task's logical date to compare
with is_today. Execution Date is Useful for backfilling.
If ``False``, uses system's day of the week.
"""
@@ -52,6 +52,7 @@ def __init__(
follow_task_ids_if_true: Union[str, Iterable[str]],
follow_task_ids_if_false: Union[str, Iterable[str]],
week_day: Union[str, Iterable[str]],
+ use_task_logical_date: bool = False,
use_task_execution_day: bool = False,
**kwargs,
) -> None:
@@ -59,11 +60,18 @@ def __init__(
self.follow_task_ids_if_true = follow_task_ids_if_true
self.follow_task_ids_if_false = follow_task_ids_if_false
self.week_day = week_day
- self.use_task_execution_day = use_task_execution_day
+ self.use_task_logical_date = use_task_logical_date
+ if use_task_execution_day:
+ self.use_task_logical_date = use_task_execution_day
+ warnings.warn(
+ "Parameter ``use_task_execution_day`` is deprecated. Use ``use_task_logical_date``.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
self._week_day_num = WeekDay.validate_week_day(week_day)
def choose_branch(self, context: Context) -> Union[str, Iterable[str]]:
- if self.use_task_execution_day:
+ if self.use_task_logical_date:
now = context["logical_date"]
else:
now = timezone.make_naive(timezone.utcnow(), self.dag.timezone)
diff --git a/airflow/plugins_manager.py b/airflow/plugins_manager.py
index 87709d5286d13..82e295fa1906c 100644
--- a/airflow/plugins_manager.py
+++ b/airflow/plugins_manager.py
@@ -63,6 +63,7 @@
global_operator_extra_links: Optional[List[Any]] = None
operator_extra_links: Optional[List[Any]] = None
registered_operator_link_classes: Optional[Dict[str, Type]] = None
+registered_ti_dep_classes: Optional[Dict[str, Type]] = None
timetable_classes: Optional[Dict[str, Type["Timetable"]]] = None
"""Mapping of class names to class of OperatorLinks registered by plugins.
@@ -78,6 +79,7 @@
"appbuilder_menu_items",
"global_operator_extra_links",
"operator_extra_links",
+ "ti_deps",
"timetables",
"source",
"listeners",
@@ -154,6 +156,8 @@ class AirflowPlugin:
# buttons.
operator_extra_links: List[Any] = []
+ ti_deps: List[Any] = []
+
# A list of timetable classes that can be used for DAG scheduling.
timetables: List[Type["Timetable"]] = []
@@ -350,6 +354,27 @@ def initialize_web_ui_plugins():
)
+def initialize_ti_deps_plugins():
+ """Creates modules for loaded extension from custom task instance dependency rule plugins"""
+ global registered_ti_dep_classes
+ if registered_ti_dep_classes is not None:
+ return
+
+ ensure_plugins_loaded()
+
+ if plugins is None:
+ raise AirflowPluginException("Can't load plugins.")
+
+ log.debug("Initialize custom taskinstance deps plugins")
+
+ registered_ti_dep_classes = {}
+
+ for plugin in plugins:
+ registered_ti_dep_classes.update(
+ {as_importable_string(ti_dep.__class__): ti_dep.__class__ for ti_dep in plugin.ti_deps}
+ )
+
+
def initialize_extra_operators_links_plugins():
"""Creates modules for loaded extension from extra operators links plugins"""
global global_operator_extra_links
diff --git a/airflow/providers/airbyte/CHANGELOG.rst b/airflow/providers/airbyte/CHANGELOG.rst
index 5d07f8b9d35ba..deb0e79e968e0 100644
--- a/airflow/providers/airbyte/CHANGELOG.rst
+++ b/airflow/providers/airbyte/CHANGELOG.rst
@@ -19,6 +19,14 @@
Changelog
---------
+2.1.4
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix mistakenly added install_requires for all providers (#22382)``
+
2.1.3
.....
diff --git a/airflow/providers/airbyte/provider.yaml b/airflow/providers/airbyte/provider.yaml
index 5480b68bbf473..a292b5378b14e 100644
--- a/airflow/providers/airbyte/provider.yaml
+++ b/airflow/providers/airbyte/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Airbyte /")
+
+with DAG(
+ dag_id="example_gcs_to_s3",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ tags=["example"],
+ catchup=False,
+) as dag:
+ # [START howto_transfer_gcs_to_s3]
+ gcs_to_s3 = GCSToS3Operator(
+ task_id="gcs_to_s3",
+ bucket=BUCKET,
+ dest_s3_key=S3_KEY,
+ replace=True,
+ )
+ # [END howto_transfer_gcs_to_s3]
diff --git a/airflow/providers/amazon/aws/example_dags/example_glacier_to_gcs.py b/airflow/providers/amazon/aws/example_dags/example_glacier_to_gcs.py
index 274c5737c9cac..2df40a7d0c37b 100644
--- a/airflow/providers/amazon/aws/example_dags/example_glacier_to_gcs.py
+++ b/airflow/providers/amazon/aws/example_dags/example_glacier_to_gcs.py
@@ -17,7 +17,7 @@
import os
from datetime import datetime
-from airflow import models
+from airflow import DAG
from airflow.providers.amazon.aws.operators.glacier import GlacierCreateJobOperator
from airflow.providers.amazon.aws.sensors.glacier import GlacierJobOperationSensor
from airflow.providers.amazon.aws.transfers.glacier_to_gcs import GlacierToGCSOperator
@@ -26,26 +26,26 @@
BUCKET_NAME = os.environ.get("GLACIER_GCS_BUCKET_NAME", "gs://INVALID BUCKET NAME")
OBJECT_NAME = os.environ.get("GLACIER_OBJECT", "example-text.txt")
-with models.DAG(
+with DAG(
"example_glacier_to_gcs",
schedule_interval=None,
start_date=datetime(2021, 1, 1), # Override to match your needs
catchup=False,
) as dag:
- # [START howto_glacier_create_job_operator]
+ # [START howto_operator_glacier_create_job]
create_glacier_job = GlacierCreateJobOperator(task_id="create_glacier_job", vault_name=VAULT_NAME)
JOB_ID = '{{ task_instance.xcom_pull("create_glacier_job")["jobId"] }}'
- # [END howto_glacier_create_job_operator]
+ # [END howto_operator_glacier_create_job]
- # [START howto_glacier_job_operation_sensor]
+ # [START howto_sensor_glacier_job_operation]
wait_for_operation_complete = GlacierJobOperationSensor(
vault_name=VAULT_NAME,
job_id=JOB_ID,
task_id="wait_for_operation_complete",
)
- # [END howto_glacier_job_operation_sensor]
+ # [END howto_sensor_glacier_job_operation]
- # [START howto_glacier_transfer_data_to_gcs]
+ # [START howto_transfer_glacier_to_gcs]
transfer_archive_to_gcs = GlacierToGCSOperator(
task_id="transfer_archive_to_gcs",
vault_name=VAULT_NAME,
@@ -56,9 +56,7 @@
# If chunk size is bigger than actual file size
# then whole file will be downloaded
chunk_size=1024,
- delegate_to=None,
- google_impersonation_chain=None,
)
- # [END howto_glacier_transfer_data_to_gcs]
+ # [END howto_transfer_glacier_to_gcs]
create_glacier_job >> wait_for_operation_complete >> transfer_archive_to_gcs
diff --git a/airflow/providers/amazon/aws/example_dags/example_glue.py b/airflow/providers/amazon/aws/example_dags/example_glue.py
new file mode 100644
index 0000000000000..65417dc24f97c
--- /dev/null
+++ b/airflow/providers/amazon/aws/example_dags/example_glue.py
@@ -0,0 +1,123 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from datetime import datetime
+from os import getenv
+
+from airflow import DAG
+from airflow.decorators import task
+from airflow.models.baseoperator import chain
+from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
+from airflow.providers.amazon.aws.operators.glue_crawler import GlueCrawlerOperator
+from airflow.providers.amazon.aws.sensors.glue import GlueJobSensor
+from airflow.providers.amazon.aws.sensors.glue_crawler import GlueCrawlerSensor
+
+GLUE_DATABASE_NAME = getenv('GLUE_DATABASE_NAME', 'glue_database_name')
+GLUE_EXAMPLE_S3_BUCKET = getenv('GLUE_EXAMPLE_S3_BUCKET', 'glue_example_s3_bucket')
+
+# Role needs putobject/getobject access to the above bucket as well as the glue
+# service role, see docs here: https://docs.aws.amazon.com/glue/latest/dg/create-an-iam-role.html
+GLUE_CRAWLER_ROLE = getenv('GLUE_CRAWLER_ROLE', 'glue_crawler_role')
+GLUE_CRAWLER_NAME = 'example_crawler'
+GLUE_CRAWLER_CONFIG = {
+ 'Name': GLUE_CRAWLER_NAME,
+ 'Role': GLUE_CRAWLER_ROLE,
+ 'DatabaseName': GLUE_DATABASE_NAME,
+ 'Targets': {
+ 'S3Targets': [
+ {
+ 'Path': f'{GLUE_EXAMPLE_S3_BUCKET}/input',
+ }
+ ]
+ },
+}
+
+# Example csv data used as input to the example AWS Glue Job.
+EXAMPLE_CSV = '''
+apple,0.5
+milk,2.5
+bread,4.0
+'''
+
+# Example Spark script to operate on the above sample csv data.
+EXAMPLE_SCRIPT = f'''
+from pyspark.context import SparkContext
+from awsglue.context import GlueContext
+
+glueContext = GlueContext(SparkContext.getOrCreate())
+datasource = glueContext.create_dynamic_frame.from_catalog(
+ database='{GLUE_DATABASE_NAME}', table_name='input')
+print('There are %s items in the table' % datasource.count())
+
+datasource.toDF().write.format('csv').mode("append").save('s3://{GLUE_EXAMPLE_S3_BUCKET}/output')
+'''
+
+
+@task(task_id='setup__upload_artifacts_to_s3')
+def upload_artifacts_to_s3():
+ '''Upload example CSV input data and an example Spark script to be used by the Glue Job'''
+ s3_hook = S3Hook()
+ s3_load_kwargs = {"replace": True, "bucket_name": GLUE_EXAMPLE_S3_BUCKET}
+ s3_hook.load_string(string_data=EXAMPLE_CSV, key='input/input.csv', **s3_load_kwargs)
+ s3_hook.load_string(string_data=EXAMPLE_SCRIPT, key='etl_script.py', **s3_load_kwargs)
+
+
+with DAG(
+ dag_id='example_glue',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ tags=['example'],
+ catchup=False,
+) as glue_dag:
+
+ setup_upload_artifacts_to_s3 = upload_artifacts_to_s3()
+
+ # [START howto_operator_glue_crawler]
+ crawl_s3 = GlueCrawlerOperator(
+ task_id='crawl_s3',
+ config=GLUE_CRAWLER_CONFIG,
+ wait_for_completion=False,
+ )
+ # [END howto_operator_glue_crawler]
+
+ # [START howto_sensor_glue_crawler]
+ wait_for_crawl = GlueCrawlerSensor(task_id='wait_for_crawl', crawler_name=GLUE_CRAWLER_NAME)
+ # [END howto_sensor_glue_crawler]
+
+ # [START howto_operator_glue]
+ job_name = 'example_glue_job'
+ submit_glue_job = GlueJobOperator(
+ task_id='submit_glue_job',
+ job_name=job_name,
+ wait_for_completion=False,
+ script_location=f's3://{GLUE_EXAMPLE_S3_BUCKET}/etl_script.py',
+ s3_bucket=GLUE_EXAMPLE_S3_BUCKET,
+ iam_role_name=GLUE_CRAWLER_ROLE.split('/')[-1],
+ create_job_kwargs={'GlueVersion': '3.0', 'NumberOfWorkers': 2, 'WorkerType': 'G.1X'},
+ )
+ # [END howto_operator_glue]
+
+ # [START howto_sensor_glue]
+ wait_for_job = GlueJobSensor(
+ task_id='wait_for_job',
+ job_name=job_name,
+ # Job ID extracted from previous Glue Job Operator task
+ run_id=submit_glue_job.output,
+ )
+ # [END howto_sensor_glue]
+
+ chain(setup_upload_artifacts_to_s3, crawl_s3, wait_for_crawl, submit_glue_job, wait_for_job)
diff --git a/airflow/providers/amazon/aws/example_dags/example_google_api_to_s3_transfer_basic.py b/airflow/providers/amazon/aws/example_dags/example_google_api_sheets_to_s3.py
similarity index 70%
rename from airflow/providers/amazon/aws/example_dags/example_google_api_to_s3_transfer_basic.py
rename to airflow/providers/amazon/aws/example_dags/example_google_api_sheets_to_s3.py
index 4e093e3e73b77..7b6e4b291a66c 100644
--- a/airflow/providers/amazon/aws/example_dags/example_google_api_to_s3_transfer_basic.py
+++ b/airflow/providers/amazon/aws/example_dags/example_google_api_sheets_to_s3.py
@@ -15,8 +15,7 @@
# specific language governing permissions and limitations
# under the License.
"""
-This is a basic example dag for using `GoogleApiToS3Transfer` to retrieve Google Sheets data:
-
+This is a basic example dag for using `GoogleApiToS3Operator` to retrieve Google Sheets data
You need to set all env variables to request the data.
"""
@@ -26,27 +25,24 @@
from airflow import DAG
from airflow.providers.amazon.aws.transfers.google_api_to_s3 import GoogleApiToS3Operator
-# [START howto_operator_google_api_to_s3_transfer_basic_env_variables]
-GOOGLE_SHEET_ID = getenv("GOOGLE_SHEET_ID")
-GOOGLE_SHEET_RANGE = getenv("GOOGLE_SHEET_RANGE")
-S3_DESTINATION_KEY = getenv("S3_DESTINATION_KEY", "s3://bucket/key.json")
-# [END howto_operator_google_api_to_s3_transfer_basic_env_variables]
-
+GOOGLE_SHEET_ID = getenv("GOOGLE_SHEET_ID", "test-google-sheet-id")
+GOOGLE_SHEET_RANGE = getenv("GOOGLE_SHEET_RANGE", "test-google-sheet-range")
+S3_DESTINATION_KEY = getenv("S3_DESTINATION_KEY", "s3://test-bucket/key.json")
with DAG(
- dag_id="example_google_api_to_s3_transfer_basic",
+ dag_id="example_google_api_sheets_to_s3",
schedule_interval=None,
start_date=datetime(2021, 1, 1),
catchup=False,
tags=['example'],
) as dag:
- # [START howto_operator_google_api_to_s3_transfer_basic_task_1]
+ # [START howto_transfer_google_api_sheets_to_s3]
task_google_sheets_values_to_s3 = GoogleApiToS3Operator(
+ task_id='google_sheet_data_to_s3',
google_api_service_name='sheets',
google_api_service_version='v4',
google_api_endpoint_path='sheets.spreadsheets.values.get',
google_api_endpoint_params={'spreadsheetId': GOOGLE_SHEET_ID, 'range': GOOGLE_SHEET_RANGE},
s3_destination_key=S3_DESTINATION_KEY,
- task_id='google_sheets_values_to_s3',
)
- # [END howto_operator_google_api_to_s3_transfer_basic_task_1]
+ # [END howto_transfer_google_api_sheets_to_s3]
diff --git a/airflow/providers/amazon/aws/example_dags/example_google_api_to_s3_transfer_advanced.py b/airflow/providers/amazon/aws/example_dags/example_google_api_youtube_to_s3.py
similarity index 58%
rename from airflow/providers/amazon/aws/example_dags/example_google_api_to_s3_transfer_advanced.py
rename to airflow/providers/amazon/aws/example_dags/example_google_api_youtube_to_s3.py
index 2101426a64d06..682cc83919129 100644
--- a/airflow/providers/amazon/aws/example_dags/example_google_api_to_s3_transfer_advanced.py
+++ b/airflow/providers/amazon/aws/example_dags/example_google_api_youtube_to_s3.py
@@ -15,7 +15,7 @@
# specific language governing permissions and limitations
# under the License.
"""
-This is a more advanced example dag for using `GoogleApiToS3Transfer` which uses xcom to pass data between
+This is a more advanced example dag for using `GoogleApiToS3Operator` which uses xcom to pass data between
tasks to retrieve specific information about YouTube videos:
First it searches for up to 50 videos (due to pagination) in a given time range
@@ -37,48 +37,42 @@
from os import getenv
from airflow import DAG
-from airflow.operators.dummy import DummyOperator
-from airflow.operators.python import BranchPythonOperator
+from airflow.decorators import task
+from airflow.models.baseoperator import chain
from airflow.providers.amazon.aws.transfers.google_api_to_s3 import GoogleApiToS3Operator
-# [START howto_operator_google_api_to_s3_transfer_advanced_env_variables]
-YOUTUBE_CONN_ID = getenv("YOUTUBE_CONN_ID", "google_cloud_default")
-YOUTUBE_CHANNEL_ID = getenv("YOUTUBE_CHANNEL_ID", "UCSXwxpWZQ7XZ1WL3wqevChA") # "Apache Airflow"
+YOUTUBE_CHANNEL_ID = getenv(
+ "YOUTUBE_CHANNEL_ID", "UCSXwxpWZQ7XZ1WL3wqevChA"
+) # Youtube channel "Apache Airflow"
YOUTUBE_VIDEO_PUBLISHED_AFTER = getenv("YOUTUBE_VIDEO_PUBLISHED_AFTER", "2019-09-25T00:00:00Z")
YOUTUBE_VIDEO_PUBLISHED_BEFORE = getenv("YOUTUBE_VIDEO_PUBLISHED_BEFORE", "2019-10-18T00:00:00Z")
-S3_DESTINATION_KEY = getenv("S3_DESTINATION_KEY", "s3://bucket/key.json")
+S3_BUCKET_NAME = getenv("S3_DESTINATION_KEY", "s3://bucket-test")
YOUTUBE_VIDEO_PARTS = getenv("YOUTUBE_VIDEO_PARTS", "snippet")
YOUTUBE_VIDEO_FIELDS = getenv("YOUTUBE_VIDEO_FIELDS", "items(id,snippet(description,publishedAt,tags,title))")
-# [END howto_operator_google_api_to_s3_transfer_advanced_env_variables]
-# [START howto_operator_google_api_to_s3_transfer_advanced_task_1_2]
-def _check_and_transform_video_ids(task_output, task_instance):
- video_ids_response = task_output
- video_ids = [item['id']['videoId'] for item in video_ids_response['items']]
+@task(task_id='transform_video_ids')
+def transform_video_ids(**kwargs):
+ task_instance = kwargs['task_instance']
+ output = task_instance.xcom_pull(task_ids="video_ids_to_s3", key="video_ids_response")
+ video_ids = [item['id']['videoId'] for item in output['items']]
- if video_ids:
- task_instance.xcom_push(key='video_ids', value={'id': ','.join(video_ids)})
- return 'video_data_to_s3'
- return 'no_video_ids'
+ if not video_ids:
+ video_ids = []
+ kwargs['task_instance'].xcom_push(key='video_ids', value={'id': ','.join(video_ids)})
-# [END howto_operator_google_api_to_s3_transfer_advanced_task_1_2]
-
-
-s3_directory, s3_file = S3_DESTINATION_KEY.rsplit('/', 1)
-s3_file_name, _ = s3_file.rsplit('.', 1)
with DAG(
- dag_id="example_google_api_to_s3_transfer_advanced",
+ dag_id="example_google_api_youtube_to_s3",
schedule_interval=None,
start_date=datetime(2021, 1, 1),
catchup=False,
tags=['example'],
) as dag:
- # [START howto_operator_google_api_to_s3_transfer_advanced_task_1]
+ # [START howto_transfer_google_api_youtube_search_to_s3]
task_video_ids_to_s3 = GoogleApiToS3Operator(
- gcp_conn_id=YOUTUBE_CONN_ID,
+ task_id='video_ids_to_s3',
google_api_service_name='youtube',
google_api_service_version='v3',
google_api_endpoint_path='youtube.search.list',
@@ -92,20 +86,16 @@ def _check_and_transform_video_ids(task_output, task_instance):
'fields': 'items/id/videoId',
},
google_api_response_via_xcom='video_ids_response',
- s3_destination_key=f'{s3_directory}/youtube_search_{s3_file_name}.json',
- task_id='video_ids_to_s3',
- )
- # [END howto_operator_google_api_to_s3_transfer_advanced_task_1]
- # [START howto_operator_google_api_to_s3_transfer_advanced_task_1_1]
- task_check_and_transform_video_ids = BranchPythonOperator(
- python_callable=_check_and_transform_video_ids,
- op_args=[task_video_ids_to_s3.output[task_video_ids_to_s3.google_api_response_via_xcom]],
- task_id='check_and_transform_video_ids',
+ s3_destination_key=f'{S3_BUCKET_NAME}/youtube_search.json',
+ s3_overwrite=True,
)
- # [END howto_operator_google_api_to_s3_transfer_advanced_task_1_1]
- # [START howto_operator_google_api_to_s3_transfer_advanced_task_2]
+ # [END howto_transfer_google_api_youtube_search_to_s3]
+
+ task_transform_video_ids = transform_video_ids()
+
+ # [START howto_transfer_google_api_youtube_list_to_s3]
task_video_data_to_s3 = GoogleApiToS3Operator(
- gcp_conn_id=YOUTUBE_CONN_ID,
+ task_id='video_data_to_s3',
google_api_service_name='youtube',
google_api_service_version='v3',
google_api_endpoint_path='youtube.videos.list',
@@ -115,14 +105,9 @@ def _check_and_transform_video_ids(task_output, task_instance):
'fields': YOUTUBE_VIDEO_FIELDS,
},
google_api_endpoint_params_via_xcom='video_ids',
- s3_destination_key=f'{s3_directory}/youtube_videos_{s3_file_name}.json',
- task_id='video_data_to_s3',
+ s3_destination_key=f'{S3_BUCKET_NAME}/youtube_videos.json',
+ s3_overwrite=True,
)
- # [END howto_operator_google_api_to_s3_transfer_advanced_task_2]
- # [START howto_operator_google_api_to_s3_transfer_advanced_task_2_1]
- task_no_video_ids = DummyOperator(task_id='no_video_ids')
- # [END howto_operator_google_api_to_s3_transfer_advanced_task_2_1]
- task_check_and_transform_video_ids >> [task_video_data_to_s3, task_no_video_ids]
+ # [END howto_transfer_google_api_youtube_list_to_s3]
- # Task dependency created via `XComArgs`:
- # task_video_ids_to_s3 >> task_check_and_transform_video_ids
+ chain(task_video_ids_to_s3, task_transform_video_ids, task_video_data_to_s3)
diff --git a/airflow/providers/amazon/aws/example_dags/example_hive_to_dynamodb.py b/airflow/providers/amazon/aws/example_dags/example_hive_to_dynamodb.py
new file mode 100644
index 0000000000000..6fccd64c6d596
--- /dev/null
+++ b/airflow/providers/amazon/aws/example_dags/example_hive_to_dynamodb.py
@@ -0,0 +1,132 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+ This DAG will not work unless you create an Amazon EMR cluster running
+ Apache Hive and copy data into it following steps 1-4 (inclusive) here:
+ https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/EMRforDynamoDB.Tutorial.html
+"""
+
+import os
+from datetime import datetime
+
+from airflow import DAG
+from airflow.decorators import task
+from airflow.models import Connection
+from airflow.providers.amazon.aws.hooks.dynamodb import DynamoDBHook
+from airflow.providers.amazon.aws.transfers.hive_to_dynamodb import HiveToDynamoDBOperator
+from airflow.utils import db
+
+DYNAMODB_TABLE_NAME = 'example_hive_to_dynamodb_table'
+HIVE_CONNECTION_ID = os.getenv('HIVE_CONNECTION_ID', 'hive_on_emr')
+HIVE_HOSTNAME = os.getenv('HIVE_HOSTNAME', 'ec2-123-45-67-890.compute-1.amazonaws.com')
+
+# These values assume you set up the Hive data source following the link above.
+DYNAMODB_TABLE_HASH_KEY = 'feature_id'
+HIVE_SQL = 'SELECT feature_id, feature_name, feature_class, state_alpha FROM hive_features'
+
+
+@task
+def create_dynamodb_table():
+ client = DynamoDBHook(client_type='dynamodb').conn
+ client.create_table(
+ TableName=DYNAMODB_TABLE_NAME,
+ KeySchema=[
+ {'AttributeName': DYNAMODB_TABLE_HASH_KEY, 'KeyType': 'HASH'},
+ ],
+ AttributeDefinitions=[
+ {'AttributeName': DYNAMODB_TABLE_HASH_KEY, 'AttributeType': 'N'},
+ ],
+ ProvisionedThroughput={'ReadCapacityUnits': 20, 'WriteCapacityUnits': 20},
+ )
+
+ # DynamoDB table creation is nearly, but not quite, instantaneous.
+ # Wait for the table to be active to avoid race conditions writing to it.
+ waiter = client.get_waiter('table_exists')
+ waiter.wait(TableName=DYNAMODB_TABLE_NAME, WaiterConfig={'Delay': 1})
+
+
+@task
+def get_dynamodb_item_count():
+ """
+ A DynamoDB table has an ItemCount value, but it is only updated every six hours.
+ To verify this DAG worked, we will scan the table and count the items manually.
+ """
+ table = DynamoDBHook(resource_type='dynamodb').conn.Table(DYNAMODB_TABLE_NAME)
+
+ response = table.scan(Select='COUNT')
+ item_count = response['Count']
+
+ while 'LastEvaluatedKey' in response:
+ response = table.scan(Select='COUNT', ExclusiveStartKey=response['LastEvaluatedKey'])
+ item_count += response['Count']
+
+ print(f'DynamoDB table contains {item_count} items.')
+
+
+# Included for sample purposes only; in production you wouldn't delete
+# the table you just backed your data up to. Using 'all_done' so even
+# if an intermediate step fails, the DAG will clean up after itself.
+@task(trigger_rule='all_done')
+def delete_dynamodb_table():
+ DynamoDBHook(client_type='dynamodb').conn.delete_table(TableName=DYNAMODB_TABLE_NAME)
+
+
+# Included for sample purposes only; in production this should
+# be configured in the environment and not be part of the DAG.
+# Note: The 'hiveserver2_default' connection will not work if Hive
+# is hosted on EMR. You must set the host name of the connection
+# to match your EMR cluster's hostname.
+@task
+def configure_hive_connection():
+ db.merge_conn(
+ Connection(
+ conn_id=HIVE_CONNECTION_ID,
+ conn_type='hiveserver2',
+ host=HIVE_HOSTNAME,
+ port=10000,
+ )
+ )
+
+
+with DAG(
+ dag_id='example_hive_to_dynamodb',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ tags=['example'],
+ catchup=False,
+) as dag:
+ # Add the prerequisites docstring to the DAG in the UI.
+ dag.doc_md = __doc__
+
+ # [START howto_transfer_hive_to_dynamodb]
+ backup_to_dynamodb = HiveToDynamoDBOperator(
+ task_id='backup_to_dynamodb',
+ hiveserver2_conn_id=HIVE_CONNECTION_ID,
+ sql=HIVE_SQL,
+ table_name=DYNAMODB_TABLE_NAME,
+ table_keys=[DYNAMODB_TABLE_HASH_KEY],
+ )
+ # [END howto_transfer_hive_to_dynamodb]
+
+ (
+ configure_hive_connection()
+ >> create_dynamodb_table()
+ >> backup_to_dynamodb
+ >> get_dynamodb_item_count()
+ >> delete_dynamodb_table()
+ )
diff --git a/airflow/providers/amazon/aws/example_dags/example_imap_attachment_to_s3.py b/airflow/providers/amazon/aws/example_dags/example_imap_attachment_to_s3.py
index 2f02dd0d29c44..357d92a6f5694 100644
--- a/airflow/providers/amazon/aws/example_dags/example_imap_attachment_to_s3.py
+++ b/airflow/providers/amazon/aws/example_dags/example_imap_attachment_to_s3.py
@@ -26,12 +26,11 @@
from airflow import DAG
from airflow.providers.amazon.aws.transfers.imap_attachment_to_s3 import ImapAttachmentToS3Operator
-# [START howto_operator_imap_attachment_to_s3_env_variables]
IMAP_ATTACHMENT_NAME = getenv("IMAP_ATTACHMENT_NAME", "test.txt")
IMAP_MAIL_FOLDER = getenv("IMAP_MAIL_FOLDER", "INBOX")
IMAP_MAIL_FILTER = getenv("IMAP_MAIL_FILTER", "All")
-S3_DESTINATION_KEY = getenv("S3_DESTINATION_KEY", "s3://bucket/key.json")
-# [END howto_operator_imap_attachment_to_s3_env_variables]
+S3_BUCKET = getenv("S3_BUCKET", "test-bucket")
+S3_KEY = getenv("S3_KEY", "key")
with DAG(
dag_id="example_imap_attachment_to_s3",
@@ -40,12 +39,13 @@
catchup=False,
tags=['example'],
) as dag:
- # [START howto_operator_imap_attachment_to_s3_task_1]
+ # [START howto_transfer_imap_attachment_to_s3]
task_transfer_imap_attachment_to_s3 = ImapAttachmentToS3Operator(
+ task_id='transfer_imap_attachment_to_s3',
imap_attachment_name=IMAP_ATTACHMENT_NAME,
- s3_key=S3_DESTINATION_KEY,
+ s3_bucket=S3_BUCKET,
+ s3_key=S3_KEY,
imap_mail_folder=IMAP_MAIL_FOLDER,
imap_mail_filter=IMAP_MAIL_FILTER,
- task_id='transfer_imap_attachment_to_s3',
)
- # [END howto_operator_imap_attachment_to_s3_task_1]
+ # [END howto_transfer_imap_attachment_to_s3]
diff --git a/airflow/providers/amazon/aws/example_dags/example_lambda.py b/airflow/providers/amazon/aws/example_dags/example_lambda.py
index 17ba0dab48eb9..3b87c3aa3c3fb 100644
--- a/airflow/providers/amazon/aws/example_dags/example_lambda.py
+++ b/airflow/providers/amazon/aws/example_dags/example_lambda.py
@@ -36,10 +36,10 @@
tags=['example'],
catchup=False,
) as dag:
- # [START howto_lambda_operator]
+ # [START howto_operator_lambda]
invoke_lambda_function = AwsLambdaInvokeFunctionOperator(
task_id='setup__invoke_lambda_function',
function_name=LAMBDA_FUNCTION_NAME,
payload=SAMPLE_EVENT,
)
- # [END howto_lambda_operator]
+ # [END howto_operator_lambda]
diff --git a/airflow/providers/amazon/aws/example_dags/example_local_to_s3.py b/airflow/providers/amazon/aws/example_dags/example_local_to_s3.py
index 647f802ad6c62..05f9c74d7685b 100644
--- a/airflow/providers/amazon/aws/example_dags/example_local_to_s3.py
+++ b/airflow/providers/amazon/aws/example_dags/example_local_to_s3.py
@@ -31,11 +31,12 @@
start_date=datetime(2021, 1, 1), # Override to match your needs
catchup=False,
) as dag:
- # [START howto_local_transfer_data_to_s3]
+ # [START howto_transfer_local_to_s3]
create_local_to_s3_job = LocalFilesystemToS3Operator(
task_id="create_local_to_s3_job",
filename="relative/path/to/file.csv",
dest_key=S3_KEY,
dest_bucket=S3_BUCKET,
+ replace=True,
)
- # [END howto_local_transfer_data_to_s3]
+ # [END howto_transfer_local_to_s3]
diff --git a/airflow/providers/google/cloud/example_dags/example_local_to_gcs.py b/airflow/providers/amazon/aws/example_dags/example_mongo_to_s3.py
similarity index 51%
rename from airflow/providers/google/cloud/example_dags/example_local_to_gcs.py
rename to airflow/providers/amazon/aws/example_dags/example_mongo_to_s3.py
index 68740befbcd03..e95964b59cee1 100644
--- a/airflow/providers/google/cloud/example_dags/example_local_to_gcs.py
+++ b/airflow/providers/amazon/aws/example_dags/example_mongo_to_s3.py
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -16,30 +15,34 @@
# specific language governing permissions and limitations
# under the License.
+
import os
-from datetime import datetime
from airflow import models
-from airflow.providers.google.cloud.transfers.local_to_gcs import LocalFilesystemToGCSOperator
+from airflow.providers.amazon.aws.transfers.mongo_to_s3 import MongoToS3Operator
+from airflow.utils.dates import datetime
-# [START howto_gcs_environment_variables]
-BUCKET_NAME = os.environ.get('GCP_GCS_BUCKET', 'example-bucket-name')
-PATH_TO_UPLOAD_FILE = os.environ.get('GCP_GCS_PATH_TO_UPLOAD_FILE', 'example-text.txt')
-DESTINATION_FILE_LOCATION = os.environ.get('GCP_GCS_DESTINATION_FILE_LOCATION', 'example-text.txt')
-# [END howto_gcs_environment_variables]
+S3_BUCKET = os.environ.get("S3_BUCKET", "test-bucket")
+S3_KEY = os.environ.get("S3_KEY", "key")
+MONGO_DATABASE = os.environ.get("MONGO_DATABASE", "Test")
+MONGO_COLLECTION = os.environ.get("MONGO_COLLECTION", "Test")
with models.DAG(
- 'example_local_to_gcs',
- schedule_interval='@once',
+ "example_mongo_to_s3",
+ schedule_interval=None,
start_date=datetime(2021, 1, 1),
catchup=False,
- tags=['example'],
) as dag:
- # [START howto_operator_local_filesystem_to_gcs]
- upload_file = LocalFilesystemToGCSOperator(
- task_id="upload_file",
- src=PATH_TO_UPLOAD_FILE,
- dst=DESTINATION_FILE_LOCATION,
- bucket=BUCKET_NAME,
+ # [START howto_transfer_mongo_to_s3]
+ create_local_to_s3_job = MongoToS3Operator(
+ task_id="create_mongo_to_s3_job",
+ mongo_collection=MONGO_COLLECTION,
+ # Mongo query by matching values
+ # Here returns all documents which have "OK" as value for the key "status"
+ mongo_query={"status": "OK"},
+ s3_bucket=S3_BUCKET,
+ s3_key=S3_KEY,
+ mongo_db=MONGO_DATABASE,
+ replace=True,
)
- # [END howto_operator_local_filesystem_to_gcs]
+ # [END howto_transfer_mongo_to_s3]
diff --git a/airflow/providers/amazon/aws/example_dags/example_quicksight.py b/airflow/providers/amazon/aws/example_dags/example_quicksight.py
new file mode 100644
index 0000000000000..5c50a54492617
--- /dev/null
+++ b/airflow/providers/amazon/aws/example_dags/example_quicksight.py
@@ -0,0 +1,57 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from datetime import datetime
+
+from airflow import DAG
+from airflow.models.baseoperator import chain
+from airflow.providers.amazon.aws.operators.quicksight import QuickSightCreateIngestionOperator
+from airflow.providers.amazon.aws.sensors.quicksight import QuickSightSensor
+
+DATA_SET_ID = os.getenv("DATA_SET_ID", "data-set-id")
+INGESTION_ID = os.getenv("INGESTION_ID", "ingestion-id")
+
+with DAG(
+ dag_id="example_quicksight",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ tags=["example"],
+ catchup=False,
+) as dag:
+ # Create and Start the QuickSight SPICE data ingestion
+ # and does not wait for its completion
+ # [START howto_operator_quicksight_create_ingestion]
+ quicksight_create_ingestion_no_waiting = QuickSightCreateIngestionOperator(
+ task_id="quicksight_create_ingestion_no_waiting",
+ data_set_id=DATA_SET_ID,
+ ingestion_id=INGESTION_ID,
+ wait_for_completion=False,
+ )
+ # [END howto_operator_quicksight_create_ingestion]
+
+ # The following task checks the status of the QuickSight SPICE ingestion
+ # job until it succeeds.
+ # [START howto_sensor_quicksight]
+ quicksight_job_status = QuickSightSensor(
+ task_id="quicksight_job_status",
+ data_set_id=DATA_SET_ID,
+ ingestion_id=INGESTION_ID,
+ )
+ # [END howto_sensor_quicksight]
+
+ chain(quicksight_create_ingestion_no_waiting, quicksight_job_status)
diff --git a/airflow/providers/amazon/aws/example_dags/example_rds.py b/airflow/providers/amazon/aws/example_dags/example_rds.py
deleted file mode 100644
index f30404b9d6867..0000000000000
--- a/airflow/providers/amazon/aws/example_dags/example_rds.py
+++ /dev/null
@@ -1,149 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-from datetime import datetime
-
-from airflow import DAG
-from airflow.providers.amazon.aws.operators.rds import (
- RdsCancelExportTaskOperator,
- RdsCopyDbSnapshotOperator,
- RdsCreateDbSnapshotOperator,
- RdsCreateEventSubscriptionOperator,
- RdsDeleteDbSnapshotOperator,
- RdsDeleteEventSubscriptionOperator,
- RdsStartExportTaskOperator,
-)
-from airflow.providers.amazon.aws.sensors.rds import RdsExportTaskExistenceSensor, RdsSnapshotExistenceSensor
-
-# [START rds_snapshots_howto_guide]
-with DAG(
- dag_id='rds_snapshots', start_date=datetime(2021, 1, 1), schedule_interval=None, catchup=False
-) as dag:
- # [START howto_guide_rds_create_snapshot]
- create_snapshot = RdsCreateDbSnapshotOperator(
- task_id='create_snapshot',
- db_type='instance',
- db_identifier='auth-db',
- db_snapshot_identifier='auth-db-snap',
- aws_conn_id='aws_default',
- hook_params={'region_name': 'us-east-1'},
- )
- # [END howto_guide_rds_create_snapshot]
-
- # [START howto_guide_rds_copy_snapshot]
- copy_snapshot = RdsCopyDbSnapshotOperator(
- task_id='copy_snapshot',
- db_type='instance',
- target_db_snapshot_identifier='auth-db-snap-backup',
- source_db_snapshot_identifier='auth-db-snap',
- aws_conn_id='aws_default',
- hook_params={'region_name': 'us-east-1'},
- )
- # [END howto_guide_rds_copy_snapshot]
-
- # [START howto_guide_rds_delete_snapshot]
- delete_snapshot = RdsDeleteDbSnapshotOperator(
- task_id='delete_snapshot',
- db_type='instance',
- db_snapshot_identifier='auth-db-snap-backup',
- aws_conn_id='aws_default',
- hook_params={'region_name': 'us-east-1'},
- )
- # [END howto_guide_rds_delete_snapshot]
-
- create_snapshot >> copy_snapshot >> delete_snapshot
-# [END rds_snapshots_howto_guide]
-
-# [START rds_exports_howto_guide]
-with DAG(dag_id='rds_exports', start_date=datetime(2021, 1, 1), schedule_interval=None, catchup=False) as dag:
- # [START howto_guide_rds_start_export]
- start_export = RdsStartExportTaskOperator(
- task_id='start_export',
- export_task_identifier='export-auth-db-snap-{{ ds }}',
- source_arn='arn:aws:rds:::snapshot:auth-db-snap',
- s3_bucket_name='my_s3_bucket',
- s3_prefix='some/prefix',
- iam_role_arn='arn:aws:iam:::role/MyRole',
- kms_key_id='arn:aws:kms:::key/*****-****-****-****-********',
- aws_conn_id='aws_default',
- hook_params={'region_name': 'us-east-1'},
- )
- # [END howto_guide_rds_start_export]
-
- # [START howto_guide_rds_cancel_export]
- cancel_export = RdsCancelExportTaskOperator(
- task_id='cancel_export',
- export_task_identifier='export-auth-db-snap-{{ ds }}',
- aws_conn_id='aws_default',
- hook_params={'region_name': 'us-east-1'},
- )
- # [END howto_guide_rds_cancel_export]
-
- start_export >> cancel_export
-# [END rds_exports_howto_guide]
-
-# [START rds_events_howto_guide]
-with DAG(dag_id='rds_events', start_date=datetime(2021, 1, 1), schedule_interval=None, catchup=False) as dag:
- # [START howto_guide_rds_create_subscription]
- create_subscription = RdsCreateEventSubscriptionOperator(
- task_id='create_subscription',
- subscription_name='my-topic-subscription',
- sns_topic_arn='arn:aws:sns:::MyTopic',
- source_type='db-instance',
- source_ids=['auth-db'],
- event_categories=['Availability', 'Backup'],
- aws_conn_id='aws_default',
- hook_params={'region_name': 'us-east-1'},
- )
- # [END howto_guide_rds_create_subscription]
-
- # [START howto_guide_rds_delete_subscription]
- delete_subscription = RdsDeleteEventSubscriptionOperator(
- task_id='delete_subscription',
- subscription_name='my-topic-subscription',
- aws_conn_id='aws_default',
- hook_params={'region_name': 'us-east-1'},
- )
- # [END howto_guide_rds_delete_subscription]
-
- create_subscription >> delete_subscription
-# [END rds_events_howto_guide]
-
-# [START rds_sensors_howto_guide]
-with DAG(dag_id='rds_events', start_date=datetime(2021, 1, 1), schedule_interval=None, catchup=False) as dag:
- # [START howto_guide_rds_snapshot_sensor]
- snapshot_sensor = RdsSnapshotExistenceSensor(
- task_id='snapshot_sensor',
- db_type='instance',
- db_snapshot_identifier='auth-db-snap-{{ ds }}',
- target_statuses=['available'],
- aws_conn_id='aws_default',
- hook_params={'region_name': 'us-east-1'},
- )
- # [END howto_guide_rds_snapshot_sensor]
-
- # [START howto_guide_rds_export_sensor]
- export_sensor = RdsExportTaskExistenceSensor(
- task_id='export_sensor',
- export_task_identifier='export-auth-db-snap-{{ ds }}',
- target_statuses=['starting', 'in_progress', 'complete', 'canceling', 'canceled'],
- aws_conn_id='aws_default',
- hook_params={'region_name': 'us-east-1'},
- )
- # [END howto_guide_rds_export_sensor]
-# [END rds_sensors_howto_guide]
diff --git a/airflow/providers/amazon/aws/example_dags/example_rds_event.py b/airflow/providers/amazon/aws/example_dags/example_rds_event.py
new file mode 100644
index 0000000000000..4ec8b6f5be3c4
--- /dev/null
+++ b/airflow/providers/amazon/aws/example_dags/example_rds_event.py
@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from datetime import datetime
+from os import getenv
+
+from airflow import DAG
+from airflow.models.baseoperator import chain
+from airflow.providers.amazon.aws.operators.rds import (
+ RdsCreateEventSubscriptionOperator,
+ RdsDeleteEventSubscriptionOperator,
+)
+
+SUBSCRIPTION_NAME = getenv("SUBSCRIPTION_NAME", "subscription-name")
+SNS_TOPIC_ARN = getenv("SNS_TOPIC_ARN", "arn:aws:sns:::MyTopic")
+RDS_DB_IDENTIFIER = getenv("RDS_DB_IDENTIFIER", "database-identifier")
+
+with DAG(
+ dag_id='example_rds_event',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ tags=['example'],
+ catchup=False,
+) as dag:
+ # [START howto_operator_rds_create_event_subscription]
+ create_subscription = RdsCreateEventSubscriptionOperator(
+ task_id='create_subscription',
+ subscription_name=SUBSCRIPTION_NAME,
+ sns_topic_arn=SNS_TOPIC_ARN,
+ source_type='db-instance',
+ source_ids=[RDS_DB_IDENTIFIER],
+ event_categories=['availability'],
+ )
+ # [END howto_operator_rds_create_event_subscription]
+
+ # [START howto_operator_rds_delete_event_subscription]
+ delete_subscription = RdsDeleteEventSubscriptionOperator(
+ task_id='delete_subscription',
+ subscription_name=SUBSCRIPTION_NAME,
+ )
+ # [END howto_operator_rds_delete_event_subscription]
+
+ chain(create_subscription, delete_subscription)
diff --git a/airflow/providers/amazon/aws/example_dags/example_rds_export.py b/airflow/providers/amazon/aws/example_dags/example_rds_export.py
new file mode 100644
index 0000000000000..1dce5804910af
--- /dev/null
+++ b/airflow/providers/amazon/aws/example_dags/example_rds_export.py
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from datetime import datetime
+from os import getenv
+
+from airflow import DAG
+from airflow.models.baseoperator import chain
+from airflow.providers.amazon.aws.operators.rds import RdsCancelExportTaskOperator, RdsStartExportTaskOperator
+from airflow.providers.amazon.aws.sensors.rds import RdsExportTaskExistenceSensor
+
+RDS_EXPORT_TASK_IDENTIFIER = getenv("RDS_EXPORT_TASK_IDENTIFIER", "export-task-identifier")
+RDS_EXPORT_SOURCE_ARN = getenv(
+ "RDS_EXPORT_SOURCE_ARN", "arn:aws:rds:::snapshot:snap-id"
+)
+BUCKET_NAME = getenv("BUCKET_NAME", "bucket-name")
+BUCKET_PREFIX = getenv("BUCKET_PREFIX", "bucket-prefix")
+ROLE_ARN = getenv("ROLE_ARN", "arn:aws:iam:::role/Role")
+KMS_KEY_ID = getenv("KMS_KEY_ID", "arn:aws:kms:::key/key-id")
+
+
+with DAG(
+ dag_id='example_rds_export',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ tags=['example'],
+ catchup=False,
+) as dag:
+ # [START howto_operator_rds_start_export_task]
+ start_export = RdsStartExportTaskOperator(
+ task_id='start_export',
+ export_task_identifier=RDS_EXPORT_TASK_IDENTIFIER,
+ source_arn=RDS_EXPORT_SOURCE_ARN,
+ s3_bucket_name=BUCKET_NAME,
+ s3_prefix=BUCKET_PREFIX,
+ iam_role_arn=ROLE_ARN,
+ kms_key_id=KMS_KEY_ID,
+ )
+ # [END howto_operator_rds_start_export_task]
+
+ # [START howto_operator_rds_cancel_export]
+ cancel_export = RdsCancelExportTaskOperator(
+ task_id='cancel_export',
+ export_task_identifier=RDS_EXPORT_TASK_IDENTIFIER,
+ )
+ # [END howto_operator_rds_cancel_export]
+
+ # [START howto_sensor_rds_export_task_existence]
+ export_sensor = RdsExportTaskExistenceSensor(
+ task_id='export_sensor',
+ export_task_identifier=RDS_EXPORT_TASK_IDENTIFIER,
+ target_statuses=['canceled'],
+ )
+ # [END howto_sensor_rds_export_task_existence]
+
+ chain(start_export, cancel_export, export_sensor)
diff --git a/airflow/providers/amazon/aws/example_dags/example_rds_snapshot.py b/airflow/providers/amazon/aws/example_dags/example_rds_snapshot.py
new file mode 100644
index 0000000000000..f7e1d02e07d49
--- /dev/null
+++ b/airflow/providers/amazon/aws/example_dags/example_rds_snapshot.py
@@ -0,0 +1,76 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from datetime import datetime
+from os import getenv
+
+from airflow import DAG
+from airflow.models.baseoperator import chain
+from airflow.providers.amazon.aws.operators.rds import (
+ RdsCopyDbSnapshotOperator,
+ RdsCreateDbSnapshotOperator,
+ RdsDeleteDbSnapshotOperator,
+)
+from airflow.providers.amazon.aws.sensors.rds import RdsSnapshotExistenceSensor
+
+RDS_DB_IDENTIFIER = getenv("RDS_DB_IDENTIFIER", "database-identifier")
+RDS_DB_SNAPSHOT_IDENTIFIER = getenv("RDS_DB_SNAPSHOT_IDENTIFIER", "database-1-snap")
+
+with DAG(
+ dag_id='example_rds_snapshot',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ tags=['example'],
+ catchup=False,
+) as dag:
+ # [START howto_operator_rds_create_db_snapshot]
+ create_snapshot = RdsCreateDbSnapshotOperator(
+ task_id='create_snapshot',
+ db_type='instance',
+ db_identifier=RDS_DB_IDENTIFIER,
+ db_snapshot_identifier=RDS_DB_SNAPSHOT_IDENTIFIER,
+ )
+ # [END howto_operator_rds_create_db_snapshot]
+
+ # [START howto_sensor_rds_snapshot_existence]
+ snapshot_sensor = RdsSnapshotExistenceSensor(
+ task_id='snapshot_sensor',
+ db_type='instance',
+ db_snapshot_identifier=RDS_DB_IDENTIFIER,
+ target_statuses=['available'],
+ )
+ # [END howto_sensor_rds_snapshot_existence]
+
+ # [START howto_operator_rds_copy_snapshot]
+ copy_snapshot = RdsCopyDbSnapshotOperator(
+ task_id='copy_snapshot',
+ db_type='instance',
+ source_db_snapshot_identifier=RDS_DB_IDENTIFIER,
+ target_db_snapshot_identifier=f'{RDS_DB_IDENTIFIER}-copy',
+ )
+ # [END howto_operator_rds_copy_snapshot]
+
+ # [START howto_operator_rds_delete_snapshot]
+ delete_snapshot = RdsDeleteDbSnapshotOperator(
+ task_id='delete_snapshot',
+ db_type='instance',
+ db_snapshot_identifier=RDS_DB_IDENTIFIER,
+ )
+ # [END howto_operator_rds_delete_snapshot]
+
+ chain(create_snapshot, snapshot_sensor, copy_snapshot, delete_snapshot)
diff --git a/airflow/providers/amazon/aws/example_dags/example_redshift_cluster.py b/airflow/providers/amazon/aws/example_dags/example_redshift_cluster.py
index 1469688266bd9..dc6deead3c618 100644
--- a/airflow/providers/amazon/aws/example_dags/example_redshift_cluster.py
+++ b/airflow/providers/amazon/aws/example_dags/example_redshift_cluster.py
@@ -22,6 +22,8 @@
from airflow import DAG
from airflow.models.baseoperator import chain
from airflow.providers.amazon.aws.operators.redshift_cluster import (
+ RedshiftCreateClusterOperator,
+ RedshiftDeleteClusterOperator,
RedshiftPauseClusterOperator,
RedshiftResumeClusterOperator,
)
@@ -36,6 +38,17 @@
catchup=False,
tags=['example'],
) as dag:
+ # [START howto_operator_redshift_cluster]
+ task_create_cluster = RedshiftCreateClusterOperator(
+ task_id="redshift_create_cluster",
+ cluster_identifier=REDSHIFT_CLUSTER_IDENTIFIER,
+ cluster_type="single-node",
+ node_type="dc2.large",
+ master_username="adminuser",
+ master_user_password="dummypass",
+ )
+ # [END howto_operator_redshift_cluster]
+
# [START howto_sensor_redshift_cluster]
task_wait_cluster_available = RedshiftClusterSensor(
task_id='sensor_redshift_cluster_available',
@@ -68,4 +81,18 @@
)
# [END howto_operator_redshift_resume_cluster]
- chain(task_wait_cluster_available, task_pause_cluster, task_wait_cluster_paused, task_resume_cluster)
+ # [START howto_operator_redshift_delete_cluster]
+ task_delete_cluster = RedshiftDeleteClusterOperator(
+ task_id="delete_cluster",
+ cluster_identifier=REDSHIFT_CLUSTER_IDENTIFIER,
+ )
+ # [END howto_operator_redshift_delete_cluster]
+
+ chain(
+ task_create_cluster,
+ task_wait_cluster_available,
+ task_pause_cluster,
+ task_wait_cluster_paused,
+ task_resume_cluster,
+ task_delete_cluster,
+ )
diff --git a/airflow/providers/amazon/aws/example_dags/example_redshift_data_execute_sql.py b/airflow/providers/amazon/aws/example_dags/example_redshift_data_execute_sql.py
index 38ac3ef1424c7..cfa3e4cefcb2d 100644
--- a/airflow/providers/amazon/aws/example_dags/example_redshift_data_execute_sql.py
+++ b/airflow/providers/amazon/aws/example_dags/example_redshift_data_execute_sql.py
@@ -57,7 +57,7 @@ def output_query_results(statement_id):
catchup=False,
tags=['example'],
) as dag:
- # [START howto_redshift_data]
+ # [START howto_operator_redshift_data]
task_query = RedshiftDataOperator(
task_id='redshift_query',
cluster_identifier=REDSHIFT_CLUSTER_IDENTIFIER,
@@ -67,6 +67,6 @@ def output_query_results(statement_id):
poll_interval=POLL_INTERVAL,
await_result=True,
)
- # [END howto_redshift_data]
+ # [END howto_operator_redshift_data]
task_output = output_query_results(task_query.output)
diff --git a/airflow/providers/amazon/aws/example_dags/example_s3.py b/airflow/providers/amazon/aws/example_dags/example_s3.py
new file mode 100644
index 0000000000000..7e06575d4a58b
--- /dev/null
+++ b/airflow/providers/amazon/aws/example_dags/example_s3.py
@@ -0,0 +1,224 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from datetime import datetime
+from typing import List
+
+from airflow.models.baseoperator import chain
+from airflow.models.dag import DAG
+from airflow.providers.amazon.aws.operators.s3 import (
+ S3CopyObjectOperator,
+ S3CreateBucketOperator,
+ S3CreateObjectOperator,
+ S3DeleteBucketOperator,
+ S3DeleteBucketTaggingOperator,
+ S3DeleteObjectsOperator,
+ S3FileTransformOperator,
+ S3GetBucketTaggingOperator,
+ S3ListOperator,
+ S3ListPrefixesOperator,
+ S3PutBucketTaggingOperator,
+)
+from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor, S3KeysUnchangedSensor
+
+BUCKET_NAME = os.environ.get('BUCKET_NAME', 'test-airflow-12345')
+BUCKET_NAME_2 = os.environ.get('BUCKET_NAME_2', 'test-airflow-123456')
+KEY = os.environ.get('KEY', 'key')
+KEY_2 = os.environ.get('KEY_2', 'key2')
+# Empty string prefix refers to the bucket root
+# See what prefix is here https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html
+PREFIX = os.environ.get('PREFIX', '')
+DELIMITER = os.environ.get('DELIMITER', '/')
+TAG_KEY = os.environ.get('TAG_KEY', 'test-s3-bucket-tagging-key')
+TAG_VALUE = os.environ.get('TAG_VALUE', 'test-s3-bucket-tagging-value')
+DATA = os.environ.get(
+ 'DATA',
+ '''
+apple,0.5
+milk,2.5
+bread,4.0
+''',
+)
+
+with DAG(
+ dag_id='example_s3',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=['example'],
+) as dag:
+ # [START howto_sensor_s3_key_function_definition]
+ def check_fn(files: List) -> bool:
+ """
+ Example of custom check: check if all files are bigger than 1kB
+
+ :param files: List of S3 object attributes.
+ Format: [{
+ 'Size': int
+ }]
+ :return: true if the criteria is met
+ :rtype: bool
+ """
+ return all(f.get('Size', 0) > 1024 for f in files)
+
+ # [END howto_sensor_s3_key_function_definition]
+
+ # [START howto_operator_s3_create_bucket]
+ create_bucket = S3CreateBucketOperator(
+ task_id='s3_create_bucket',
+ bucket_name=BUCKET_NAME,
+ )
+ # [END howto_operator_s3_create_bucket]
+
+ # [START howto_operator_s3_put_bucket_tagging]
+ put_tagging = S3PutBucketTaggingOperator(
+ task_id='s3_put_bucket_tagging',
+ bucket_name=BUCKET_NAME,
+ key=TAG_KEY,
+ value=TAG_VALUE,
+ )
+ # [END howto_operator_s3_put_bucket_tagging]
+
+ # [START howto_operator_s3_get_bucket_tagging]
+ get_tagging = S3GetBucketTaggingOperator(
+ task_id='s3_get_bucket_tagging',
+ bucket_name=BUCKET_NAME,
+ )
+ # [END howto_operator_s3_get_bucket_tagging]
+
+ # [START howto_operator_s3_delete_bucket_tagging]
+ delete_tagging = S3DeleteBucketTaggingOperator(
+ task_id='s3_delete_bucket_tagging',
+ bucket_name=BUCKET_NAME,
+ )
+ # [END howto_operator_s3_delete_bucket_tagging]
+
+ # [START howto_operator_s3_create_object]
+ create_object = S3CreateObjectOperator(
+ task_id="s3_create_object",
+ s3_bucket=BUCKET_NAME,
+ s3_key=KEY,
+ data=DATA,
+ replace=True,
+ )
+ # [END howto_operator_s3_create_object]
+
+ # [START howto_operator_s3_list_prefixes]
+ list_prefixes = S3ListPrefixesOperator(
+ task_id="s3_list_prefix_operator",
+ bucket=BUCKET_NAME,
+ prefix=PREFIX,
+ delimiter=DELIMITER,
+ )
+ # [END howto_operator_s3_list_prefixes]
+
+ # [START howto_operator_s3_list]
+ list_keys = S3ListOperator(
+ task_id="s3_list_operator",
+ bucket=BUCKET_NAME,
+ prefix=PREFIX,
+ )
+ # [END howto_operator_s3_list]
+
+ # [START howto_sensor_s3_key_single_key]
+ # Check if a file exists
+ sensor_one_key = S3KeySensor(
+ task_id="s3_sensor_one_key",
+ bucket_name=BUCKET_NAME,
+ bucket_key=KEY,
+ )
+ # [END howto_sensor_s3_key_single_key]
+
+ # [START howto_sensor_s3_key_multiple_keys]
+ # Check if both files exist
+ sensor_two_keys = S3KeySensor(
+ task_id="s3_sensor_two_keys",
+ bucket_name=BUCKET_NAME,
+ bucket_key=[KEY, KEY_2],
+ )
+ # [END howto_sensor_s3_key_multiple_keys]
+
+ # [START howto_sensor_s3_key_function]
+ # Check if a file exists and match a certain pattern defined in check_fn
+ sensor_key_with_function = S3KeySensor(
+ task_id="s3_sensor_key_function",
+ bucket_name=BUCKET_NAME,
+ bucket_key=KEY,
+ check_fn=check_fn,
+ )
+ # [END howto_sensor_s3_key_function]
+
+ # [START howto_sensor_s3_keys_unchanged]
+ sensor_keys_unchanged = S3KeysUnchangedSensor(
+ task_id="s3_sensor_one_key_size",
+ bucket_name=BUCKET_NAME_2,
+ prefix=PREFIX,
+ inactivity_period=10,
+ )
+ # [END howto_sensor_s3_keys_unchanged]
+
+ # [START howto_operator_s3_copy_object]
+ copy_object = S3CopyObjectOperator(
+ task_id="s3_copy_object",
+ source_bucket_name=BUCKET_NAME,
+ dest_bucket_name=BUCKET_NAME_2,
+ source_bucket_key=KEY,
+ dest_bucket_key=KEY_2,
+ )
+ # [END howto_operator_s3_copy_object]
+
+ # [START howto_operator_s3_file_transform]
+ transforms_file = S3FileTransformOperator(
+ task_id="s3_file_transform",
+ source_s3_key=f's3://{BUCKET_NAME}/{KEY}',
+ dest_s3_key=f's3://{BUCKET_NAME_2}/{KEY_2}',
+ # Use `cp` command as transform script as an example
+ transform_script='cp',
+ replace=True,
+ )
+ # [END howto_operator_s3_file_transform]
+
+ # [START howto_operator_s3_delete_objects]
+ delete_objects = S3DeleteObjectsOperator(
+ task_id="s3_delete_objects",
+ bucket=BUCKET_NAME_2,
+ keys=KEY_2,
+ )
+ # [END howto_operator_s3_delete_objects]
+
+ # [START howto_operator_s3_delete_bucket]
+ delete_bucket = S3DeleteBucketOperator(
+ task_id='s3_delete_bucket', bucket_name=BUCKET_NAME, force_delete=True
+ )
+ # [END howto_operator_s3_delete_bucket]
+
+ chain(
+ create_bucket,
+ put_tagging,
+ get_tagging,
+ delete_tagging,
+ create_object,
+ list_prefixes,
+ list_keys,
+ [sensor_one_key, sensor_two_keys, sensor_key_with_function],
+ copy_object,
+ transforms_file,
+ sensor_keys_unchanged,
+ delete_objects,
+ delete_bucket,
+ )
diff --git a/airflow/providers/amazon/aws/example_dags/example_s3_bucket.py b/airflow/providers/amazon/aws/example_dags/example_s3_bucket.py
deleted file mode 100644
index 538538d6c0ea8..0000000000000
--- a/airflow/providers/amazon/aws/example_dags/example_s3_bucket.py
+++ /dev/null
@@ -1,63 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-# Ignore missing args provided by default_args
-# type: ignore[call-arg]
-
-import os
-from datetime import datetime
-
-from airflow.decorators import task
-from airflow.models.dag import DAG
-from airflow.providers.amazon.aws.hooks.s3 import S3Hook
-from airflow.providers.amazon.aws.operators.s3 import S3CreateBucketOperator, S3DeleteBucketOperator
-
-BUCKET_NAME = os.environ.get('BUCKET_NAME', 'test-airflow-12345')
-
-
-# [START howto_operator_s3_bucket]
-@task(task_id="s3_bucket_dag_add_keys_to_bucket")
-def upload_keys():
- """This is a python callback to add keys into the s3 bucket"""
- # add keys to bucket
- s3_hook = S3Hook()
- for i in range(0, 3):
- s3_hook.load_string(
- string_data="input",
- key=f"path/data{i}",
- bucket_name=BUCKET_NAME,
- )
-
-
-with DAG(
- dag_id='s3_bucket_dag',
- schedule_interval=None,
- start_date=datetime(2021, 1, 1),
- catchup=False,
- default_args={"bucket_name": BUCKET_NAME},
- max_active_runs=1,
- tags=['example'],
-) as dag:
-
- create_bucket = S3CreateBucketOperator(task_id='s3_bucket_dag_create', region_name='us-east-1')
-
- # Using a task-decorated function to add keys
- add_keys_to_bucket = upload_keys()
-
- delete_bucket = S3DeleteBucketOperator(task_id='s3_bucket_dag_delete', force_delete=True)
-
- create_bucket >> add_keys_to_bucket >> delete_bucket
- # [END howto_operator_s3_bucket]
diff --git a/airflow/providers/amazon/aws/example_dags/example_s3_bucket_tagging.py b/airflow/providers/amazon/aws/example_dags/example_s3_bucket_tagging.py
deleted file mode 100644
index 7312dcc1a5b53..0000000000000
--- a/airflow/providers/amazon/aws/example_dags/example_s3_bucket_tagging.py
+++ /dev/null
@@ -1,62 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# Ignore missing args provided by default_args
-# type: ignore[call-arg]
-
-import os
-from datetime import datetime
-
-from airflow.models.dag import DAG
-from airflow.providers.amazon.aws.operators.s3 import (
- S3CreateBucketOperator,
- S3DeleteBucketOperator,
- S3DeleteBucketTaggingOperator,
- S3GetBucketTaggingOperator,
- S3PutBucketTaggingOperator,
-)
-
-BUCKET_NAME = os.environ.get('BUCKET_NAME', 'test-s3-bucket-tagging')
-TAG_KEY = os.environ.get('TAG_KEY', 'test-s3-bucket-tagging-key')
-TAG_VALUE = os.environ.get('TAG_VALUE', 'test-s3-bucket-tagging-value')
-
-
-# [START howto_operator_s3_bucket_tagging]
-with DAG(
- dag_id='s3_bucket_tagging_dag',
- schedule_interval=None,
- start_date=datetime(2021, 1, 1),
- catchup=False,
- default_args={"bucket_name": BUCKET_NAME},
- max_active_runs=1,
- tags=['example'],
-) as dag:
-
- create_bucket = S3CreateBucketOperator(task_id='s3_bucket_tagging_dag_create', region_name='us-east-1')
-
- delete_bucket = S3DeleteBucketOperator(task_id='s3_bucket_tagging_dag_delete', force_delete=True)
-
- get_tagging = S3GetBucketTaggingOperator(task_id='s3_bucket_tagging_dag_get_tagging')
-
- put_tagging = S3PutBucketTaggingOperator(
- task_id='s3_bucket_tagging_dag_put_tagging', key=TAG_KEY, value=TAG_VALUE
- )
-
- delete_tagging = S3DeleteBucketTaggingOperator(task_id='s3_bucket_tagging_dag_delete_tagging')
-
- create_bucket >> put_tagging >> get_tagging >> delete_tagging >> delete_bucket
- # [END howto_operator_s3_bucket_tagging]
diff --git a/airflow/providers/amazon/aws/example_dags/example_s3_to_ftp.py b/airflow/providers/amazon/aws/example_dags/example_s3_to_ftp.py
new file mode 100644
index 0000000000000..6ebe2501c613b
--- /dev/null
+++ b/airflow/providers/amazon/aws/example_dags/example_s3_to_ftp.py
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import os
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.amazon.aws.transfers.s3_to_ftp import S3ToFTPOperator
+
+S3_BUCKET = os.environ.get("S3_BUCKET", "test-bucket")
+S3_KEY = os.environ.get("S3_KEY", "key")
+
+with models.DAG(
+ "example_s3_to_ftp",
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+) as dag:
+ # [START howto_transfer_s3_to_ftp]
+ s3_to_ftp_task = S3ToFTPOperator(
+ task_id="ftp_to_s3_task",
+ ftp_path="/tmp/ftp_path",
+ s3_bucket=S3_BUCKET,
+ s3_key=S3_KEY,
+ )
+ # [END howto_transfer_s3_to_ftp]
diff --git a/airflow/providers/amazon/aws/example_dags/example_s3_to_sftp.py b/airflow/providers/amazon/aws/example_dags/example_s3_to_sftp.py
index 965644eaf4a64..d7983265e7697 100644
--- a/airflow/providers/amazon/aws/example_dags/example_s3_to_sftp.py
+++ b/airflow/providers/amazon/aws/example_dags/example_s3_to_sftp.py
@@ -27,17 +27,14 @@
with models.DAG(
"example_s3_to_sftp",
schedule_interval=None,
- start_date=datetime(2021, 1, 1), # Override to match your needs
+ start_date=datetime(2021, 1, 1),
catchup=False,
) as dag:
-
- # [START howto_s3_transfer_data_to_sftp]
+ # [START howto_transfer_s3_to_sftp]
create_s3_to_sftp_job = S3ToSFTPOperator(
- task_id="create_to_s3_sftp_job",
- sftp_conn_id="sftp_conn_id",
+ task_id="create_s3_to_sftp_job",
sftp_path="sftp_path",
- s3_conn_id="s3_conn_id",
s3_bucket=S3_BUCKET,
s3_key=S3_KEY,
)
- # [END howto_s3_transfer_data_to_sftp]
+ # [END howto_transfer_s3_to_sftp]
diff --git a/airflow/providers/amazon/aws/example_dags/example_sagemaker.py b/airflow/providers/amazon/aws/example_dags/example_sagemaker.py
index 6162b061db465..41e7666eabb8a 100644
--- a/airflow/providers/amazon/aws/example_dags/example_sagemaker.py
+++ b/airflow/providers/amazon/aws/example_dags/example_sagemaker.py
@@ -14,164 +14,454 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
+import base64
+import os
+import subprocess
from datetime import datetime
-from os import environ
+from tempfile import NamedTemporaryFile
+
+import boto3
from airflow import DAG
+from airflow.decorators import task
+from airflow.providers.amazon.aws.hooks.s3 import S3Hook
from airflow.providers.amazon.aws.operators.sagemaker import (
SageMakerDeleteModelOperator,
SageMakerModelOperator,
SageMakerProcessingOperator,
SageMakerTrainingOperator,
SageMakerTransformOperator,
+ SageMakerTuningOperator,
+)
+from airflow.providers.amazon.aws.sensors.sagemaker import (
+ SageMakerTrainingSensor,
+ SageMakerTransformSensor,
+ SageMakerTuningSensor,
+)
+
+# Project name will be used in naming the S3 buckets and various tasks.
+# The dataset used in this example is identifying varieties of the Iris flower.
+PROJECT_NAME = 'iris'
+TIMESTAMP = '{{ ts_nodash }}'
+
+S3_BUCKET = os.getenv('S3_BUCKET', 'S3_bucket')
+RAW_DATA_S3_KEY = f'{PROJECT_NAME}/preprocessing/input.csv'
+INPUT_DATA_S3_KEY = f'{PROJECT_NAME}/processed-input-data'
+TRAINING_OUTPUT_S3_KEY = f'{PROJECT_NAME}/results'
+PREDICTION_OUTPUT_S3_KEY = f'{PROJECT_NAME}/transform'
+
+PROCESSING_LOCAL_INPUT_PATH = '/opt/ml/processing/input'
+PROCESSING_LOCAL_OUTPUT_PATH = '/opt/ml/processing/output'
+
+MODEL_NAME = f'{PROJECT_NAME}-KNN-model'
+# Job names can't be reused, so appending a timestamp ensures it is unique.
+PROCESSING_JOB_NAME = f'{PROJECT_NAME}-processing-{TIMESTAMP}'
+TRAINING_JOB_NAME = f'{PROJECT_NAME}-train-{TIMESTAMP}'
+TRANSFORM_JOB_NAME = f'{PROJECT_NAME}-transform-{TIMESTAMP}'
+TUNING_JOB_NAME = f'{PROJECT_NAME}-tune-{TIMESTAMP}'
+
+ROLE_ARN = os.getenv(
+ 'SAGEMAKER_ROLE_ARN',
+ 'arn:aws:iam::1234567890:role/service-role/AmazonSageMaker-ExecutionRole',
)
+ECR_REPOSITORY = os.getenv('ECR_REPOSITORY', '1234567890.dkr.ecr.us-west-2.amazonaws.com/process_data')
+REGION = ECR_REPOSITORY.split('.')[3]
+
+# For this example we are using a subset of Fischer's Iris Data Set.
+# The full dataset can be found at UC Irvine's machine learning repository:
+# https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data
+DATASET = """
+ 5.1,3.5,1.4,0.2,Iris-setosa
+ 4.9,3.0,1.4,0.2,Iris-setosa
+ 7.0,3.2,4.7,1.4,Iris-versicolor
+ 6.4,3.2,4.5,1.5,Iris-versicolor
+ 4.9,2.5,4.5,1.7,Iris-virginica
+ 7.3,2.9,6.3,1.8,Iris-virginica
+ """
+SAMPLE_SIZE = DATASET.count('\n') - 1
-MODEL_NAME = "sample_model"
-TRAINING_JOB_NAME = "sample_training"
-IMAGE_URI = environ.get("ECR_IMAGE_URI", "123456789012.dkr.ecr.us-east-1.amazonaws.com/repo_name")
-S3_BUCKET = environ.get("BUCKET_NAME", "test-airflow-12345")
-ROLE = environ.get("SAGEMAKER_ROLE_ARN", "arn:aws:iam::123456789012:role/role_name")
+# The URI of an Amazon-provided docker image for handling KNN model training. This is a public ECR
+# repo cited in public SageMaker documentation, so the account number does not need to be redacted.
+# For more info see: https://docs.aws.amazon.com/sagemaker/latest/dg/ecr-us-west-2.html#knn-us-west-2.title
+KNN_IMAGE_URI = '174872318107.dkr.ecr.us-west-2.amazonaws.com/knn'
+TASK_TIMEOUT = {'MaxRuntimeInSeconds': 6 * 60}
+
+RESOURCE_CONFIG = {
+ 'InstanceCount': 1,
+ 'InstanceType': 'ml.m5.large',
+ 'VolumeSizeInGB': 1,
+}
+
+TRAINING_DATA_SOURCE = {
+ 'CompressionType': 'None',
+ 'ContentType': 'text/csv',
+ 'DataSource': { # type: ignore
+ 'S3DataSource': {
+ 'S3DataDistributionType': 'FullyReplicated',
+ 'S3DataType': 'S3Prefix',
+ 'S3Uri': f's3://{S3_BUCKET}/{INPUT_DATA_S3_KEY}/train.csv',
+ }
+ },
+}
+
+# Define configs for processing, training, model creation, and batch transform jobs
SAGEMAKER_PROCESSING_JOB_CONFIG = {
- "ProcessingJobName": "sample_processing_job",
- "ProcessingInputs": [
+ 'ProcessingJobName': PROCESSING_JOB_NAME,
+ 'RoleArn': f'{ROLE_ARN}',
+ 'ProcessingInputs': [
{
- "InputName": "input",
- "AppManaged": False,
- "S3Input": {
- "S3Uri": f"s3://{S3_BUCKET}/preprocessing/input/",
- "LocalPath": "/opt/ml/processing/input/",
- "S3DataType": "S3Prefix",
- "S3InputMode": "File",
- "S3DataDistributionType": "FullyReplicated",
- "S3CompressionType": "None",
+ 'InputName': 'input',
+ 'AppManaged': False,
+ 'S3Input': {
+ 'S3Uri': f's3://{S3_BUCKET}/{RAW_DATA_S3_KEY}',
+ 'LocalPath': PROCESSING_LOCAL_INPUT_PATH,
+ 'S3DataType': 'S3Prefix',
+ 'S3InputMode': 'File',
+ 'S3DataDistributionType': 'FullyReplicated',
+ 'S3CompressionType': 'None',
},
},
],
- "ProcessingOutputConfig": {
- "Outputs": [
+ 'ProcessingOutputConfig': {
+ 'Outputs': [
{
- "OutputName": "output",
- "S3Output": {
- "S3Uri": f"s3://{S3_BUCKET}/preprocessing/output/",
- "LocalPath": "/opt/ml/processing/output/",
- "S3UploadMode": "EndOfJob",
+ 'OutputName': 'output',
+ 'S3Output': {
+ 'S3Uri': f's3://{S3_BUCKET}/{INPUT_DATA_S3_KEY}',
+ 'LocalPath': PROCESSING_LOCAL_OUTPUT_PATH,
+ 'S3UploadMode': 'EndOfJob',
},
- "AppManaged": False,
+ 'AppManaged': False,
}
]
},
- "ProcessingResources": {
- "ClusterConfig": {
- "InstanceCount": 1,
- "InstanceType": "ml.m5.large",
- "VolumeSizeInGB": 5,
- }
+ 'ProcessingResources': {
+ 'ClusterConfig': RESOURCE_CONFIG,
},
- "StoppingCondition": {"MaxRuntimeInSeconds": 3600},
- "AppSpecification": {
- "ImageUri": f"{IMAGE_URI}",
- "ContainerEntrypoint": ["python3", "./preprocessing.py"],
+ 'StoppingCondition': TASK_TIMEOUT,
+ 'AppSpecification': {
+ 'ImageUri': ECR_REPOSITORY,
},
- "RoleArn": f"{ROLE}",
}
-SAGEMAKER_TRAINING_JOB_CONFIG = {
- "AlgorithmSpecification": {
- "TrainingImage": f"{IMAGE_URI}",
+TRAINING_CONFIG = {
+ 'TrainingJobName': TRAINING_JOB_NAME,
+ 'RoleArn': ROLE_ARN,
+ 'AlgorithmSpecification': {
+ "TrainingImage": KNN_IMAGE_URI,
"TrainingInputMode": "File",
},
- "InputDataConfig": [
- {
- "ChannelName": "config",
- "DataSource": {
- "S3DataSource": {
- "S3DataType": "S3Prefix",
- "S3Uri": f"s3://{S3_BUCKET}/config/",
- "S3DataDistributionType": "FullyReplicated",
- }
- },
- "CompressionType": "None",
- "RecordWrapperType": "None",
- },
- ],
- "OutputDataConfig": {
- "KmsKeyId": "",
- "S3OutputPath": f"s3://{S3_BUCKET}/training/",
- },
- "ResourceConfig": {
- "InstanceType": "ml.m5.large",
- "InstanceCount": 1,
- "VolumeSizeInGB": 5,
+ 'HyperParameters': {
+ 'predictor_type': 'classifier',
+ 'feature_dim': '4',
+ 'k': '3',
+ 'sample_size': str(SAMPLE_SIZE),
},
- "StoppingCondition": {"MaxRuntimeInSeconds": 6000},
- "RoleArn": f"{ROLE}",
- "EnableNetworkIsolation": False,
- "EnableInterContainerTrafficEncryption": False,
- "EnableManagedSpotTraining": False,
- "TrainingJobName": TRAINING_JOB_NAME,
-}
-
-SAGEMAKER_CREATE_MODEL_CONFIG = {
- "ModelName": MODEL_NAME,
- "Containers": [
+ 'InputDataConfig': [
{
- "Image": f"{IMAGE_URI}",
- "Mode": "SingleModel",
- "ModelDataUrl": f"s3://{S3_BUCKET}/training/{TRAINING_JOB_NAME}/output/model.tar.gz",
+ 'ChannelName': 'train',
+ **TRAINING_DATA_SOURCE, # type: ignore [arg-type]
}
],
- "ExecutionRoleArn": f"{ROLE}",
- "EnableNetworkIsolation": False,
+ 'OutputDataConfig': {'S3OutputPath': f's3://{S3_BUCKET}/{TRAINING_OUTPUT_S3_KEY}/'},
+ 'ResourceConfig': RESOURCE_CONFIG,
+ 'StoppingCondition': TASK_TIMEOUT,
}
-SAGEMAKER_INFERENCE_CONFIG = {
- "TransformJobName": "sample_transform_job",
- "ModelName": MODEL_NAME,
- "TransformInput": {
- "DataSource": {
- "S3DataSource": {
- "S3DataType": "S3Prefix",
- "S3Uri": f"s3://{S3_BUCKET}/config/config_date.yml",
+MODEL_CONFIG = {
+ 'ModelName': MODEL_NAME,
+ 'ExecutionRoleArn': ROLE_ARN,
+ 'PrimaryContainer': {
+ 'Mode': 'SingleModel',
+ 'Image': KNN_IMAGE_URI,
+ 'ModelDataUrl': f's3://{S3_BUCKET}/{TRAINING_OUTPUT_S3_KEY}/{TRAINING_JOB_NAME}/output/model.tar.gz',
+ },
+}
+
+TRANSFORM_CONFIG = {
+ 'TransformJobName': TRANSFORM_JOB_NAME,
+ 'ModelName': MODEL_NAME,
+ 'TransformInput': {
+ 'DataSource': {
+ 'S3DataSource': {
+ 'S3DataType': 'S3Prefix',
+ 'S3Uri': f's3://{S3_BUCKET}/{INPUT_DATA_S3_KEY}/test.csv',
}
},
- "ContentType": "application/x-yaml",
- "CompressionType": "None",
- "SplitType": "None",
+ 'SplitType': 'Line',
+ 'ContentType': 'text/csv',
+ },
+ 'TransformOutput': {'S3OutputPath': f's3://{S3_BUCKET}/{PREDICTION_OUTPUT_S3_KEY}'},
+ 'TransformResources': {
+ 'InstanceCount': 1,
+ 'InstanceType': 'ml.m5.large',
},
- "TransformOutput": {"S3OutputPath": f"s3://{S3_BUCKET}/inferencing/output/"},
- "TransformResources": {"InstanceType": "ml.m5.large", "InstanceCount": 1},
}
-# [START howto_operator_sagemaker]
+TUNING_CONFIG = {
+ 'HyperParameterTuningJobName': TUNING_JOB_NAME,
+ 'HyperParameterTuningJobConfig': {
+ 'Strategy': 'Bayesian',
+ 'HyperParameterTuningJobObjective': {
+ 'MetricName': 'test:accuracy',
+ 'Type': 'Maximize',
+ },
+ 'ResourceLimits': {
+ # You would bump these up in production as appropriate.
+ 'MaxNumberOfTrainingJobs': 1,
+ 'MaxParallelTrainingJobs': 1,
+ },
+ 'ParameterRanges': {
+ 'CategoricalParameterRanges': [],
+ 'IntegerParameterRanges': [
+ # Set the min and max values of the hyperparameters you want to tune.
+ {
+ 'Name': 'k',
+ 'MinValue': '1',
+ "MaxValue": str(SAMPLE_SIZE),
+ },
+ {
+ 'Name': 'sample_size',
+ 'MinValue': '1',
+ 'MaxValue': str(SAMPLE_SIZE),
+ },
+ ],
+ },
+ },
+ 'TrainingJobDefinition': {
+ 'StaticHyperParameters': {
+ 'predictor_type': 'classifier',
+ 'feature_dim': '4',
+ },
+ 'AlgorithmSpecification': {'TrainingImage': KNN_IMAGE_URI, 'TrainingInputMode': 'File'},
+ 'InputDataConfig': [
+ {
+ 'ChannelName': 'train',
+ **TRAINING_DATA_SOURCE, # type: ignore [arg-type]
+ },
+ {
+ 'ChannelName': 'test',
+ **TRAINING_DATA_SOURCE, # type: ignore [arg-type]
+ },
+ ],
+ 'OutputDataConfig': {'S3OutputPath': f's3://{S3_BUCKET}/{TRAINING_OUTPUT_S3_KEY}'},
+ 'ResourceConfig': RESOURCE_CONFIG,
+ 'StoppingCondition': TASK_TIMEOUT,
+ 'RoleArn': ROLE_ARN,
+ },
+}
+
+
+# This script will be the entrypoint for the docker image which will handle preprocessing the raw data
+# NOTE: The following string must remain dedented as it is being written to a file.
+PREPROCESS_SCRIPT = (
+ """
+import boto3
+import numpy as np
+import pandas as pd
+
+def main():
+ # Load the Iris dataset from {input_path}/input.csv, split it into train/test
+ # subsets, and write them to {output_path}/ for the Processing Operator.
+
+ columns = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'species']
+ iris = pd.read_csv('{input_path}/input.csv', names=columns)
+
+ # Process data
+ iris['species'] = iris['species'].replace({{'Iris-virginica': 0, 'Iris-versicolor': 1, 'Iris-setosa': 2}})
+ iris = iris[['species', 'sepal_length', 'sepal_width', 'petal_length', 'petal_width']]
+
+ # Split into test and train data
+ iris_train, iris_test = np.split(
+ iris.sample(frac=1, random_state=np.random.RandomState()), [int(0.7 * len(iris))]
+ )
+
+ # Remove the "answers" from the test set
+ iris_test.drop(['species'], axis=1, inplace=True)
+
+ # Write the splits to disk
+ iris_train.to_csv('{output_path}/train.csv', index=False, header=False)
+ iris_test.to_csv('{output_path}/test.csv', index=False, header=False)
+
+ print('Preprocessing Done.')
+
+if __name__ == "__main__":
+ main()
+
+ """
+).format(input_path=PROCESSING_LOCAL_INPUT_PATH, output_path=PROCESSING_LOCAL_OUTPUT_PATH)
+
+
+@task
+def upload_dataset_to_s3():
+ """Uploads the provided dataset to a designated Amazon S3 bucket."""
+ S3Hook().load_string(
+ string_data=DATASET,
+ bucket_name=S3_BUCKET,
+ key=RAW_DATA_S3_KEY,
+ replace=True,
+ )
+
+
+@task
+def build_and_upload_docker_image():
+ """
+ We need a Docker image with the following requirements:
+ - Has numpy, pandas, requests, and boto3 installed
+ - Has our data preprocessing script mounted and set as the entry point
+ """
+
+ # Fetch and parse ECR Token to be used for the docker push
+ ecr_client = boto3.client('ecr', region_name=REGION)
+ token = ecr_client.get_authorization_token()
+ credentials = (base64.b64decode(token['authorizationData'][0]['authorizationToken'])).decode('utf-8')
+ username, password = credentials.split(':')
+
+ with NamedTemporaryFile(mode='w+t') as preprocessing_script, NamedTemporaryFile(mode='w+t') as dockerfile:
+
+ preprocessing_script.write(PREPROCESS_SCRIPT)
+ preprocessing_script.flush()
+
+ dockerfile.write(
+ f"""
+ FROM amazonlinux
+ COPY {preprocessing_script.name.split('/')[2]} /preprocessing.py
+ ADD credentials /credentials
+ ENV AWS_SHARED_CREDENTIALS_FILE=/credentials
+ RUN yum install python3 pip -y
+ RUN pip3 install boto3 pandas requests
+ CMD [ "python3", "/preprocessing.py"]
+ """
+ )
+ dockerfile.flush()
+
+ docker_build_and_push_commands = f"""
+ cp /root/.aws/credentials /tmp/credentials &&
+ docker build -f {dockerfile.name} -t {ECR_REPOSITORY} /tmp &&
+ rm /tmp/credentials &&
+ aws ecr get-login-password --region {REGION} |
+ docker login --username {username} --password {password} {ECR_REPOSITORY} &&
+ docker push {ECR_REPOSITORY}
+ """
+ docker_build = subprocess.Popen(
+ docker_build_and_push_commands,
+ shell=True,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ )
+ _, err = docker_build.communicate()
+
+ if docker_build.returncode != 0:
+ raise RuntimeError(err)
+
+
+@task(trigger_rule='all_done')
+def cleanup():
+ # Delete S3 Artifacts
+ client = boto3.client('s3')
+ object_keys = [
+ key['Key'] for key in client.list_objects_v2(Bucket=S3_BUCKET, Prefix=PROJECT_NAME)['Contents']
+ ]
+ for key in object_keys:
+ client.delete_objects(Bucket=S3_BUCKET, Delete={'Objects': [{'Key': key}]})
+
+
with DAG(
- "sample_sagemaker_dag",
+ dag_id='example_sagemaker',
schedule_interval=None,
- start_date=datetime(2022, 2, 21),
+ start_date=datetime(2021, 1, 1),
+ tags=['example'],
catchup=False,
) as dag:
- sagemaker_processing_task = SageMakerProcessingOperator(
+
+ # [START howto_operator_sagemaker_processing]
+ preprocess_raw_data = SageMakerProcessingOperator(
+ task_id='preprocess_raw_data',
config=SAGEMAKER_PROCESSING_JOB_CONFIG,
- aws_conn_id="aws_default",
- task_id="sagemaker_preprocessing_task",
+ do_xcom_push=False,
)
+ # [END howto_operator_sagemaker_processing]
- training_task = SageMakerTrainingOperator(
- config=SAGEMAKER_TRAINING_JOB_CONFIG, aws_conn_id="aws_default", task_id="sagemaker_training_task"
+ # [START howto_operator_sagemaker_training]
+ train_model = SageMakerTrainingOperator(
+ task_id='train_model',
+ config=TRAINING_CONFIG,
+ # Waits by default, setting as False to demonstrate the Sensor below.
+ wait_for_completion=False,
+ do_xcom_push=False,
)
+ # [END howto_operator_sagemaker_training]
- model_create_task = SageMakerModelOperator(
- config=SAGEMAKER_CREATE_MODEL_CONFIG, aws_conn_id="aws_default", task_id="sagemaker_create_model_task"
+ # [START howto_sensor_sagemaker_training]
+ await_training = SageMakerTrainingSensor(
+ task_id='await_training',
+ job_name=TRAINING_JOB_NAME,
)
+ # [END howto_sensor_sagemaker_training]
- inference_task = SageMakerTransformOperator(
- config=SAGEMAKER_INFERENCE_CONFIG, aws_conn_id="aws_default", task_id="sagemaker_inference_task"
+ # [START howto_operator_sagemaker_model]
+ create_model = SageMakerModelOperator(
+ task_id='create_model',
+ config=MODEL_CONFIG,
+ do_xcom_push=False,
)
+ # [END howto_operator_sagemaker_model]
- model_delete_task = SageMakerDeleteModelOperator(
- task_id="sagemaker_delete_model_task", config={'ModelName': MODEL_NAME}, aws_conn_id="aws_default"
+ # [START howto_operator_sagemaker_tuning]
+ tune_model = SageMakerTuningOperator(
+ task_id='tune_model',
+ config=TUNING_CONFIG,
+ # Waits by default, setting as False to demonstrate the Sensor below.
+ wait_for_completion=False,
+ do_xcom_push=False,
)
+ # [END howto_operator_sagemaker_tuning]
- sagemaker_processing_task >> training_task >> model_create_task >> inference_task >> model_delete_task
- # [END howto_operator_sagemaker]
+ # [START howto_sensor_sagemaker_tuning]
+ await_tune = SageMakerTuningSensor(
+ task_id='await_tuning',
+ job_name=TUNING_JOB_NAME,
+ )
+ # [END howto_sensor_sagemaker_tuning]
+
+ # [START howto_operator_sagemaker_transform]
+ test_model = SageMakerTransformOperator(
+ task_id='test_model',
+ config=TRANSFORM_CONFIG,
+ # Waits by default, setting as False to demonstrate the Sensor below.
+ wait_for_completion=False,
+ do_xcom_push=False,
+ )
+ # [END howto_operator_sagemaker_transform]
+
+ # [START howto_sensor_sagemaker_transform]
+ await_transform = SageMakerTransformSensor(
+ task_id='await_transform',
+ job_name=TRANSFORM_JOB_NAME,
+ )
+ # [END howto_sensor_sagemaker_transform]
+
+ # Trigger rule set to "all_done" so clean up will run regardless of success on other tasks.
+ # [START howto_operator_sagemaker_delete_model]
+ delete_model = SageMakerDeleteModelOperator(
+ task_id='delete_model',
+ config={'ModelName': MODEL_NAME},
+ trigger_rule='all_done',
+ )
+ # [END howto_operator_sagemaker_delete_model]
+
+ (
+ upload_dataset_to_s3()
+ >> build_and_upload_docker_image()
+ >> preprocess_raw_data
+ >> train_model
+ >> await_training
+ >> create_model
+ >> tune_model
+ >> await_tune
+ >> test_model
+ >> await_transform
+ >> cleanup()
+ >> delete_model
+ )
diff --git a/airflow/providers/amazon/aws/example_dags/example_sagemaker_endpoint.py b/airflow/providers/amazon/aws/example_dags/example_sagemaker_endpoint.py
new file mode 100644
index 0000000000000..52ee5303f2553
--- /dev/null
+++ b/airflow/providers/amazon/aws/example_dags/example_sagemaker_endpoint.py
@@ -0,0 +1,230 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import json
+import os
+from datetime import datetime
+
+import boto3
+
+from airflow import DAG
+from airflow.decorators import task
+from airflow.providers.amazon.aws.operators.s3 import S3CreateObjectOperator
+from airflow.providers.amazon.aws.operators.sagemaker import (
+ SageMakerDeleteModelOperator,
+ SageMakerEndpointConfigOperator,
+ SageMakerEndpointOperator,
+ SageMakerModelOperator,
+ SageMakerTrainingOperator,
+)
+from airflow.providers.amazon.aws.sensors.sagemaker import SageMakerEndpointSensor
+
+# Project name will be used in naming the S3 buckets and various tasks.
+# The dataset used in this example is identifying varieties of the Iris flower.
+PROJECT_NAME = 'iris'
+TIMESTAMP = '{{ ts_nodash }}'
+
+S3_BUCKET = os.getenv('S3_BUCKET', 'S3_bucket')
+ROLE_ARN = os.getenv(
+ 'SAGEMAKER_ROLE_ARN',
+ 'arn:aws:iam::1234567890:role/service-role/AmazonSageMaker-ExecutionRole',
+)
+INPUT_DATA_S3_KEY = f'{PROJECT_NAME}/processed-input-data'
+TRAINING_OUTPUT_S3_KEY = f'{PROJECT_NAME}/training-results'
+
+MODEL_NAME = f'{PROJECT_NAME}-KNN-model'
+ENDPOINT_NAME = f'{PROJECT_NAME}-endpoint'
+# Job names can't be reused, so appending a timestamp ensures it is unique.
+ENDPOINT_CONFIG_JOB_NAME = f'{PROJECT_NAME}-endpoint-config-{TIMESTAMP}'
+TRAINING_JOB_NAME = f'{PROJECT_NAME}-train-{TIMESTAMP}'
+
+# For an example of how to obtain the following train and test data, please see
+# https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/example_dags/example_sagemaker.py
+TRAIN_DATA = '0,4.9,2.5,4.5,1.7\n1,7.0,3.2,4.7,1.4\n0,7.3,2.9,6.3,1.8\n2,5.1,3.5,1.4,0.2\n'
+SAMPLE_TEST_DATA = '6.4,3.2,4.5,1.5'
+
+# The URI of an Amazon-provided docker image for handling KNN model training. This is a public ECR
+# repo cited in public SageMaker documentation, so the account number does not need to be redacted.
+# For more info see: https://docs.aws.amazon.com/sagemaker/latest/dg/ecr-us-west-2.html#knn-us-west-2.title
+KNN_IMAGE_URI = '174872318107.dkr.ecr.us-west-2.amazonaws.com/knn'
+
+# Define configs for processing, training, model creation, and batch transform jobs
+TRAINING_CONFIG = {
+ 'TrainingJobName': TRAINING_JOB_NAME,
+ 'RoleArn': ROLE_ARN,
+ 'AlgorithmSpecification': {
+ "TrainingImage": KNN_IMAGE_URI,
+ "TrainingInputMode": "File",
+ },
+ 'HyperParameters': {
+ 'predictor_type': 'classifier',
+ 'feature_dim': '4',
+ 'k': '3',
+ 'sample_size': '6',
+ },
+ 'InputDataConfig': [
+ {
+ 'ChannelName': 'train',
+ 'CompressionType': 'None',
+ 'ContentType': 'text/csv',
+ 'DataSource': {
+ 'S3DataSource': {
+ 'S3DataDistributionType': 'FullyReplicated',
+ 'S3DataType': 'S3Prefix',
+ 'S3Uri': f's3://{S3_BUCKET}/{INPUT_DATA_S3_KEY}/train.csv',
+ }
+ },
+ }
+ ],
+ 'OutputDataConfig': {'S3OutputPath': f's3://{S3_BUCKET}/{TRAINING_OUTPUT_S3_KEY}/'},
+ 'ResourceConfig': {
+ 'InstanceCount': 1,
+ 'InstanceType': 'ml.m5.large',
+ 'VolumeSizeInGB': 1,
+ },
+ 'StoppingCondition': {'MaxRuntimeInSeconds': 6 * 60},
+}
+
+MODEL_CONFIG = {
+ 'ModelName': MODEL_NAME,
+ 'ExecutionRoleArn': ROLE_ARN,
+ 'PrimaryContainer': {
+ 'Mode': 'SingleModel',
+ 'Image': KNN_IMAGE_URI,
+ 'ModelDataUrl': f's3://{S3_BUCKET}/{TRAINING_OUTPUT_S3_KEY}/{TRAINING_JOB_NAME}/output/model.tar.gz',
+ },
+}
+
+ENDPOINT_CONFIG_CONFIG = {
+ 'EndpointConfigName': ENDPOINT_CONFIG_JOB_NAME,
+ 'ProductionVariants': [
+ {
+ 'VariantName': f'{PROJECT_NAME}-demo',
+ 'ModelName': MODEL_NAME,
+ 'InstanceType': 'ml.t2.medium',
+ 'InitialInstanceCount': 1,
+ },
+ ],
+}
+
+DEPLOY_ENDPOINT_CONFIG = {
+ 'EndpointName': ENDPOINT_NAME,
+ 'EndpointConfigName': ENDPOINT_CONFIG_JOB_NAME,
+}
+
+
+@task
+def call_endpoint():
+ runtime = boto3.Session().client('sagemaker-runtime')
+
+ response = runtime.invoke_endpoint(
+ EndpointName=ENDPOINT_NAME,
+ ContentType='text/csv',
+ Body=SAMPLE_TEST_DATA,
+ )
+
+ return json.loads(response["Body"].read().decode())['predictions']
+
+
+@task(trigger_rule='all_done')
+def cleanup():
+ # Delete Endpoint and Endpoint Config
+ client = boto3.client('sagemaker')
+ endpoint_config_name = client.list_endpoint_configs()['EndpointConfigs'][0]['EndpointConfigName']
+ client.delete_endpoint_config(EndpointConfigName=endpoint_config_name)
+ client.delete_endpoint(EndpointName=ENDPOINT_NAME)
+
+ # Delete S3 Artifacts
+ client = boto3.client('s3')
+ object_keys = [
+ key['Key'] for key in client.list_objects_v2(Bucket=S3_BUCKET, Prefix=PROJECT_NAME)['Contents']
+ ]
+ for key in object_keys:
+ client.delete_objects(Bucket=S3_BUCKET, Delete={'Objects': [{'Key': key}]})
+
+
+with DAG(
+ dag_id='example_sagemaker_endpoint',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ tags=['example'],
+ catchup=False,
+) as dag:
+
+ upload_data = S3CreateObjectOperator(
+ task_id='upload_data',
+ s3_bucket=S3_BUCKET,
+ s3_key=f'{INPUT_DATA_S3_KEY}/train.csv',
+ data=TRAIN_DATA,
+ replace=True,
+ )
+
+ train_model = SageMakerTrainingOperator(
+ task_id='train_model',
+ config=TRAINING_CONFIG,
+ do_xcom_push=False,
+ )
+
+ create_model = SageMakerModelOperator(
+ task_id='create_model',
+ config=MODEL_CONFIG,
+ do_xcom_push=False,
+ )
+
+ # [START howto_operator_sagemaker_endpoint_config]
+ configure_endpoint = SageMakerEndpointConfigOperator(
+ task_id='configure_endpoint',
+ config=ENDPOINT_CONFIG_CONFIG,
+ do_xcom_push=False,
+ )
+ # [END howto_operator_sagemaker_endpoint_config]
+
+ # [START howto_operator_sagemaker_endpoint]
+ deploy_endpoint = SageMakerEndpointOperator(
+ task_id='deploy_endpoint',
+ config=DEPLOY_ENDPOINT_CONFIG,
+ # Waits by default, > train_model
+ >> create_model
+ >> configure_endpoint
+ >> deploy_endpoint
+ >> await_endpoint
+ >> call_endpoint()
+ >> cleanup()
+ >> delete_model
+ )
diff --git a/airflow/providers/amazon/aws/example_dags/example_salesforce_to_s3.py b/airflow/providers/amazon/aws/example_dags/example_salesforce_to_s3.py
index 49755316ffd5b..735cda4d3af14 100644
--- a/airflow/providers/amazon/aws/example_dags/example_salesforce_to_s3.py
+++ b/airflow/providers/amazon/aws/example_dags/example_salesforce_to_s3.py
@@ -15,60 +15,35 @@
# specific language governing permissions and limitations
# under the License.
-# Ignore missing args provided by default_args
-# type: ignore[call-arg]
-
"""
-This is a basic example DAG for using `SalesforceToS3Operator` to retrieve Salesforce customer
-data and upload to an S3 bucket.
+This is a basic example DAG for using `SalesforceToS3Operator` to retrieve Salesforce account
+data and upload it to an Amazon S3 bucket.
"""
from datetime import datetime
+from os import getenv
from airflow import DAG
-from airflow.providers.amazon.aws.operators.s3 import S3CopyObjectOperator, S3DeleteObjectsOperator
from airflow.providers.amazon.aws.transfers.salesforce_to_s3 import SalesforceToS3Operator
-BASE_PATH = "salesforce/customers"
-FILE_NAME = "customer_daily_extract_{{ ds_nodash }}.csv"
+S3_BUCKET_NAME = getenv("S3_BUCKET_NAME", "s3_bucket_name")
+S3_KEY = getenv("S3_KEY", "s3_filename")
with DAG(
- dag_id="example_salesforce_to_s3_transfer",
- schedule_interval="@daily",
+ dag_id="example_salesforce_to_s3",
+ schedule_interval=None,
start_date=datetime(2021, 7, 8),
catchup=False,
- default_args={"retries": 1, "aws_conn_id": "s3"},
tags=["example"],
- default_view="graph",
) as dag:
- # [START howto_operator_salesforce_to_s3_transfer]
- upload_salesforce_data_to_s3_landing = SalesforceToS3Operator(
- task_id="upload_salesforce_data_to_s3",
- salesforce_query="SELECT Id, Name, Company, Phone, Email, LastModifiedDate, IsActive FROM Customers",
- s3_bucket_name="landing-bucket",
- s3_key=f"{BASE_PATH}/{FILE_NAME}",
+ # [START howto_transfer_salesforce_to_s3]
+ upload_salesforce_data_to_s3 = SalesforceToS3Operator(
+ task_id="upload_salesforce_to_s3",
+ salesforce_query="SELECT AccountNumber, Name FROM Account",
+ s3_bucket_name=S3_BUCKET_NAME,
+ s3_key=S3_KEY,
salesforce_conn_id="salesforce",
replace=True,
)
- # [END howto_operator_salesforce_to_s3_transfer]
-
- date_prefixes = "{{ execution_date.strftime('%Y/%m/%d') }}"
-
- store_to_s3_data_lake = S3CopyObjectOperator(
- task_id="store_to_s3_data_lake",
- source_bucket_key=upload_salesforce_data_to_s3_landing.output,
- dest_bucket_name="data_lake",
- dest_bucket_key=f"{BASE_PATH}/{date_prefixes}/{FILE_NAME}",
- )
-
- delete_data_from_s3_landing = S3DeleteObjectsOperator(
- task_id="delete_data_from_s3_landing",
- bucket=upload_salesforce_data_to_s3_landing.s3_bucket_name,
- keys=upload_salesforce_data_to_s3_landing.s3_key,
- )
-
- store_to_s3_data_lake >> delete_data_from_s3_landing
-
- # Task dependencies created via `XComArgs`:
- # upload_salesforce_data_to_s3_landing >> store_to_s3_data_lake
+ # [END howto_transfer_salesforce_to_s3]
diff --git a/airflow/providers/amazon/aws/example_dags/example_sftp_to_s3.py b/airflow/providers/amazon/aws/example_dags/example_sftp_to_s3.py
index f36c5a2934a6c..0e2407a7d3546 100644
--- a/airflow/providers/amazon/aws/example_dags/example_sftp_to_s3.py
+++ b/airflow/providers/amazon/aws/example_dags/example_sftp_to_s3.py
@@ -28,16 +28,14 @@
with models.DAG(
"example_sftp_to_s3",
schedule_interval=None,
- start_date=datetime(2021, 1, 1), # Override to match your needs
+ start_date=datetime(2021, 1, 1),
catchup=False,
) as dag:
- # [START howto_sftp_transfer_data_to_s3]
+ # [START howto_transfer_sftp_to_s3]
create_sftp_to_s3_job = SFTPToS3Operator(
task_id="create_sftp_to_s3_job",
- sftp_conn_id="sftp_conn_id",
- sftp_path="sftp_path",
- s3_conn_id="s3_conn_id",
+ sftp_path="/tmp/sftp_path",
s3_bucket=S3_BUCKET,
s3_key=S3_KEY,
)
- # [END howto_sftp_transfer_data_to_s3]
+ # [END howto_transfer_sftp_to_s3]
diff --git a/airflow/providers/google/cloud/example_dags/example_gcs_to_local.py b/airflow/providers/amazon/aws/example_dags/example_sql_to_s3.py
similarity index 54%
rename from airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
rename to airflow/providers/amazon/aws/example_dags/example_sql_to_s3.py
index 50f62d23e00e0..df2abee0f3052 100644
--- a/airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
+++ b/airflow/providers/amazon/aws/example_dags/example_sql_to_s3.py
@@ -14,30 +14,31 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+
+
import os
from datetime import datetime
from airflow import models
-from airflow.providers.google.cloud.transfers.gcs_to_local import GCSToLocalFilesystemOperator
-
-PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "example-id")
-BUCKET = os.environ.get("GCP_GCS_BUCKET", "test-gcs-example-bucket")
+from airflow.providers.amazon.aws.transfers.sql_to_s3 import SqlToS3Operator
-PATH_TO_REMOTE_FILE = os.environ.get("GCP_GCS_PATH_TO_UPLOAD_FILE", "test-gcs-example-remote.txt")
-PATH_TO_LOCAL_FILE = os.environ.get("GCP_GCS_PATH_TO_SAVED_FILE", "test-gcs-example-local.txt")
+S3_BUCKET = os.environ.get("S3_BUCKET", "test-bucket")
+S3_KEY = os.environ.get("S3_KEY", "key")
+SQL_QUERY = os.environ.get("SQL_QUERY", "SHOW tables")
with models.DAG(
- "example_gcs_to_local",
- schedule_interval='@once',
+ "example_sql_to_s3",
+ schedule_interval=None,
start_date=datetime(2021, 1, 1),
catchup=False,
- tags=['example'],
) as dag:
- # [START howto_operator_gcs_download_file_task]
- download_file = GCSToLocalFilesystemOperator(
- task_id="download_file",
- object_name=PATH_TO_REMOTE_FILE,
- bucket=BUCKET,
- filename=PATH_TO_LOCAL_FILE,
+ # [START howto_transfer_sql_to_s3]
+ sql_to_s3_task = SqlToS3Operator(
+ task_id="sql_to_s3_task",
+ sql_conn_id="mysql_default",
+ query=SQL_QUERY,
+ s3_bucket=S3_BUCKET,
+ s3_key=S3_KEY,
+ replace=True,
)
- # [END howto_operator_gcs_download_file_task]
+ # [END howto_transfer_sql_to_s3]
diff --git a/airflow/providers/amazon/aws/example_dags/example_sqs.py b/airflow/providers/amazon/aws/example_dags/example_sqs.py
index acf8c12e3292d..69ff1e5b75831 100644
--- a/airflow/providers/amazon/aws/example_dags/example_sqs.py
+++ b/airflow/providers/amazon/aws/example_dags/example_sqs.py
@@ -14,22 +14,22 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-from datetime import datetime, timedelta
+from datetime import datetime
+from os import getenv
from airflow import DAG
from airflow.decorators import task
+from airflow.models.baseoperator import chain
from airflow.providers.amazon.aws.hooks.sqs import SqsHook
from airflow.providers.amazon.aws.operators.sqs import SqsPublishOperator
from airflow.providers.amazon.aws.sensors.sqs import SqsSensor
-QUEUE_NAME = 'Airflow-Example-Queue'
-AWS_CONN_ID = 'aws_default'
+QUEUE_NAME = getenv('QUEUE_NAME', 'Airflow-Example-Queue')
@task(task_id="create_queue")
def create_queue_fn():
- """This is a Python function that creates an SQS queue"""
+ """Create the example queue"""
hook = SqsHook()
result = hook.create_queue(queue_name=QUEUE_NAME)
return result['QueueUrl']
@@ -37,7 +37,7 @@ def create_queue_fn():
@task(task_id="delete_queue")
def delete_queue_fn(queue_url):
- """This is a Python function that deletes an SQS queue"""
+ """Delete the example queue"""
hook = SqsHook()
hook.get_conn().delete_queue(QueueUrl=queue_url)
@@ -46,36 +46,27 @@ def delete_queue_fn(queue_url):
dag_id='example_sqs',
schedule_interval=None,
start_date=datetime(2021, 1, 1),
- dagrun_timeout=timedelta(minutes=60),
tags=['example'],
catchup=False,
) as dag:
- # [START howto_sqs_operator_and_sensor]
- # Using a task-decorated function to create an SQS queue
create_queue = create_queue_fn()
+ # [START howto_operator_sqs]
publish_to_queue = SqsPublishOperator(
task_id='publish_to_queue',
sqs_queue=create_queue,
message_content="{{ task_instance }}-{{ execution_date }}",
- message_attributes=None,
- delay_seconds=0,
)
+ # [END howto_operator_sqs]
+ # [START howto_sensor_sqs]
read_from_queue = SqsSensor(
task_id='read_from_queue',
sqs_queue=create_queue,
- max_messages=5,
- wait_time_seconds=1,
- visibility_timeout=None,
- message_filtering=None,
- message_filtering_match_values=None,
- message_filtering_config=None,
)
+ # [END howto_sensor_sqs]
- # Using a task-decorated function to delete the SQS queue we created earlier
delete_queue = delete_queue_fn(create_queue)
- create_queue >> publish_to_queue >> read_from_queue >> delete_queue
- # [END howto_sqs_operator_and_sensor]
+ chain(create_queue, publish_to_queue, read_from_queue, delete_queue)
diff --git a/airflow/providers/amazon/aws/example_dags/example_step_functions.py b/airflow/providers/amazon/aws/example_dags/example_step_functions.py
new file mode 100644
index 0000000000000..02763e3ea13f1
--- /dev/null
+++ b/airflow/providers/amazon/aws/example_dags/example_step_functions.py
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from datetime import datetime
+from os import environ
+
+from airflow import DAG
+from airflow.models.baseoperator import chain
+from airflow.providers.amazon.aws.operators.step_function import (
+ StepFunctionGetExecutionOutputOperator,
+ StepFunctionStartExecutionOperator,
+)
+from airflow.providers.amazon.aws.sensors.step_function import StepFunctionExecutionSensor
+
+STEP_FUNCTIONS_STATE_MACHINE_ARN = environ.get('STEP_FUNCTIONS_STATE_MACHINE_ARN', 'state_machine_arn')
+
+with DAG(
+ dag_id='example_step_functions',
+ schedule_interval=None,
+ start_date=datetime(2021, 1, 1),
+ tags=['example'],
+ catchup=False,
+) as dag:
+
+ # [START howto_operator_step_function_start_execution]
+ start_execution = StepFunctionStartExecutionOperator(
+ task_id='start_execution', state_machine_arn=STEP_FUNCTIONS_STATE_MACHINE_ARN
+ )
+ # [END howto_operator_step_function_start_execution]
+
+ # [START howto_sensor_step_function_execution]
+ wait_for_execution = StepFunctionExecutionSensor(
+ task_id='wait_for_execution', execution_arn=start_execution.output
+ )
+ # [END howto_sensor_step_function_execution]
+
+ # [START howto_operator_step_function_get_execution_output]
+ get_execution_output = StepFunctionGetExecutionOutputOperator(
+ task_id='get_execution_output', execution_arn=start_execution.output
+ )
+ # [END howto_operator_step_function_get_execution_output]
+
+ chain(start_execution, wait_for_execution, get_execution_output)
diff --git a/airflow/providers/amazon/aws/hooks/athena.py b/airflow/providers/amazon/aws/hooks/athena.py
index bbd08f9034529..82e69ccf6f001 100644
--- a/airflow/providers/amazon/aws/hooks/athena.py
+++ b/airflow/providers/amazon/aws/hooks/athena.py
@@ -91,8 +91,7 @@ def run_query(
if client_request_token:
params['ClientRequestToken'] = client_request_token
response = self.get_conn().start_query_execution(**params)
- query_execution_id = response['QueryExecutionId']
- return query_execution_id
+ return response['QueryExecutionId']
def check_query_status(self, query_execution_id: str) -> Optional[str]:
"""
diff --git a/airflow/providers/amazon/aws/hooks/batch_waiters.py b/airflow/providers/amazon/aws/hooks/batch_waiters.py
index 29c10e4f464d5..59ba0e431f25e 100644
--- a/airflow/providers/amazon/aws/hooks/batch_waiters.py
+++ b/airflow/providers/amazon/aws/hooks/batch_waiters.py
@@ -120,7 +120,7 @@ def default_config(self) -> Dict:
:rtype: Dict
"""
if self._default_config is None:
- config_path = Path(__file__).with_name("batch_waiters.json").absolute()
+ config_path = Path(__file__).with_name("batch_waiters.json").resolve()
with open(config_path) as config_file:
self._default_config = json.load(config_file)
return deepcopy(self._default_config) # avoid accidental mutation
diff --git a/airflow/providers/amazon/aws/hooks/eks.py b/airflow/providers/amazon/aws/hooks/eks.py
index 1475d3d1f5354..d2a795e498dfd 100644
--- a/airflow/providers/amazon/aws/hooks/eks.py
+++ b/airflow/providers/amazon/aws/hooks/eks.py
@@ -638,7 +638,7 @@ class EKSHook(EksHook):
def __init__(self, *args, **kwargs):
warnings.warn(
- "This hook is deprecated. " "Please use `airflow.providers.amazon.aws.hooks.eks.EksHook`.",
+ "This hook is deprecated. Please use `airflow.providers.amazon.aws.hooks.eks.EksHook`.",
DeprecationWarning,
stacklevel=2,
)
diff --git a/airflow/providers/amazon/aws/hooks/emr.py b/airflow/providers/amazon/aws/hooks/emr.py
index 9a2c13ca9d684..143bdcdcc8913 100644
--- a/airflow/providers/amazon/aws/hooks/emr.py
+++ b/airflow/providers/amazon/aws/hooks/emr.py
@@ -133,6 +133,7 @@ def submit_job(
job_driver: dict,
configuration_overrides: Optional[dict] = None,
client_request_token: Optional[str] = None,
+ tags: Optional[dict] = None,
) -> str:
"""
Submit a job to the EMR Containers API and return the job ID.
@@ -148,6 +149,7 @@ def submit_job(
specifically either application configuration or monitoring configuration.
:param client_request_token: The client idempotency token of the job run request.
Use this if you want to specify a unique ID to prevent two jobs from getting started.
+ :param tags: The tags assigned to job runs.
:return: Job ID
"""
params = {
@@ -157,6 +159,7 @@ def submit_job(
"releaseLabel": release_label,
"jobDriver": job_driver,
"configurationOverrides": configuration_overrides or {},
+ "tags": tags or {},
}
if client_request_token:
params["clientToken"] = client_request_token
diff --git a/airflow/providers/amazon/aws/hooks/glue.py b/airflow/providers/amazon/aws/hooks/glue.py
index f0170f358f73a..dcd6d7c4661cd 100644
--- a/airflow/providers/amazon/aws/hooks/glue.py
+++ b/airflow/providers/amazon/aws/hooks/glue.py
@@ -118,8 +118,8 @@ def initialize_job(
try:
job_name = self.get_or_create_glue_job()
- job_run = glue_client.start_job_run(JobName=job_name, Arguments=script_arguments, **run_kwargs)
- return job_run
+ return glue_client.start_job_run(JobName=job_name, Arguments=script_arguments, **run_kwargs)
+
except Exception as general_error:
self.log.error("Failed to run aws glue job, error: %s", general_error)
raise
@@ -134,8 +134,7 @@ def get_job_state(self, job_name: str, run_id: str) -> str:
"""
glue_client = self.get_conn()
job_run = glue_client.get_job_run(JobName=job_name, RunId=run_id, PredecessorsIncluded=True)
- job_run_state = job_run['JobRun']['JobRunState']
- return job_run_state
+ return job_run['JobRun']['JobRunState']
def job_completion(self, job_name: str, run_id: str) -> Dict[str, str]:
"""
@@ -155,7 +154,7 @@ def job_completion(self, job_name: str, run_id: str) -> Dict[str, str]:
self.log.info("Exiting Job %s Run State: %s", run_id, job_run_state)
return {'JobRunState': job_run_state, 'JobRunId': run_id}
if job_run_state in failed_states:
- job_error_message = "Exiting Job " + run_id + " Run State: " + job_run_state
+ job_error_message = f"Exiting Job {run_id} Run State: {job_run_state}"
self.log.info(job_error_message)
raise AirflowException(job_error_message)
else:
diff --git a/airflow/providers/amazon/aws/hooks/glue_catalog.py b/airflow/providers/amazon/aws/hooks/glue_catalog.py
index fc9c353e084b0..e77916d09e3b9 100644
--- a/airflow/providers/amazon/aws/hooks/glue_catalog.py
+++ b/airflow/providers/amazon/aws/hooks/glue_catalog.py
@@ -18,8 +18,11 @@
"""This module contains AWS Glue Catalog Hook"""
import warnings
-from typing import Optional, Set
+from typing import Dict, List, Optional, Set
+from botocore.exceptions import ClientError
+
+from airflow.exceptions import AirflowException
from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
@@ -123,6 +126,59 @@ def get_table_location(self, database_name: str, table_name: str) -> str:
return table['StorageDescriptor']['Location']
+ def get_partition(self, database_name: str, table_name: str, partition_values: List[str]) -> Dict:
+ """
+ Gets a Partition
+
+ :param database_name: Database name
+ :param table_name: Database's Table name
+ :param partition_values: List of utf-8 strings that define the partition
+ Please see official AWS documentation for further information.
+ https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html#aws-glue-api-catalog-partitions-GetPartition
+
+ :rtype: dict
+
+ :raises: AirflowException
+
+ >>> hook = GlueCatalogHook()
+ >>> partition = hook.get_partition('db', 'table', ['string'])
+ >>> partition['Values']
+ """
+ try:
+ response = self.get_conn().get_partition(
+ DatabaseName=database_name, TableName=table_name, PartitionValues=partition_values
+ )
+ return response["Partition"]
+ except ClientError as e:
+ self.log.error("Client error: %s", e)
+ raise AirflowException("AWS request failed, check logs for more info")
+
+ def create_partition(self, database_name: str, table_name: str, partition_input: Dict) -> Dict:
+ """
+ Creates a new Partition
+
+ :param database_name: Database name
+ :param table_name: Database's Table name
+ :param partition_input: Definition of how the partition is created
+ Please see official AWS documentation for further information.
+ https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html#aws-glue-api-catalog-partitions-CreatePartition
+
+ :rtype: dict
+
+ :raises: AirflowException
+
+ >>> hook = GlueCatalogHook()
+ >>> partition_input = {"Values": []}
+ >>> hook.create_partition(database_name="db", table_name="table", partition_input=partition_input)
+ """
+ try:
+ return self.get_conn().create_partition(
+ DatabaseName=database_name, TableName=table_name, PartitionInput=partition_input
+ )
+ except ClientError as e:
+ self.log.error("Client error: %s", e)
+ raise AirflowException("AWS request failed, check logs for more info")
+
class AwsGlueCatalogHook(GlueCatalogHook):
"""
diff --git a/airflow/providers/amazon/aws/hooks/glue_crawler.py b/airflow/providers/amazon/aws/hooks/glue_crawler.py
index 00d438aaf56aa..65f7df8d28566 100644
--- a/airflow/providers/amazon/aws/hooks/glue_crawler.py
+++ b/airflow/providers/amazon/aws/hooks/glue_crawler.py
@@ -102,8 +102,7 @@ def create_crawler(self, **crawler_kwargs) -> str:
"""
crawler_name = crawler_kwargs['Name']
self.log.info("Creating crawler: %s", crawler_name)
- crawler = self.glue_client.create_crawler(**crawler_kwargs)
- return crawler
+ return self.glue_client.create_crawler(**crawler_kwargs)
def start_crawler(self, crawler_name: str) -> dict:
"""
@@ -113,8 +112,7 @@ def start_crawler(self, crawler_name: str) -> dict:
:return: Empty dictionary
"""
self.log.info("Starting crawler %s", crawler_name)
- crawler = self.glue_client.start_crawler(Name=crawler_name)
- return crawler
+ return self.glue_client.start_crawler(Name=crawler_name)
def wait_for_crawler_completion(self, crawler_name: str, poll_interval: int = 5) -> str:
"""
@@ -137,18 +135,17 @@ def wait_for_crawler_completion(self, crawler_name: str, poll_interval: int = 5)
crawler_status = crawler['LastCrawl']['Status']
if crawler_status in failed_status:
raise AirflowException(f"Status: {crawler_status}")
- else:
- metrics = self.glue_client.get_crawler_metrics(CrawlerNameList=[crawler_name])[
- 'CrawlerMetricsList'
- ][0]
- self.log.info("Status: %s", crawler_status)
- self.log.info("Last Runtime Duration (seconds): %s", metrics['LastRuntimeSeconds'])
- self.log.info("Median Runtime Duration (seconds): %s", metrics['MedianRuntimeSeconds'])
- self.log.info("Tables Created: %s", metrics['TablesCreated'])
- self.log.info("Tables Updated: %s", metrics['TablesUpdated'])
- self.log.info("Tables Deleted: %s", metrics['TablesDeleted'])
-
- return crawler_status
+ metrics = self.glue_client.get_crawler_metrics(CrawlerNameList=[crawler_name])[
+ 'CrawlerMetricsList'
+ ][0]
+ self.log.info("Status: %s", crawler_status)
+ self.log.info("Last Runtime Duration (seconds): %s", metrics['LastRuntimeSeconds'])
+ self.log.info("Median Runtime Duration (seconds): %s", metrics['MedianRuntimeSeconds'])
+ self.log.info("Tables Created: %s", metrics['TablesCreated'])
+ self.log.info("Tables Updated: %s", metrics['TablesUpdated'])
+ self.log.info("Tables Deleted: %s", metrics['TablesDeleted'])
+
+ return crawler_status
else:
self.log.info("Polling for AWS Glue crawler: %s ", crawler_name)
diff --git a/airflow/providers/amazon/aws/hooks/kinesis.py b/airflow/providers/amazon/aws/hooks/kinesis.py
index 8f26b54d64b12..f15457c6c28d7 100644
--- a/airflow/providers/amazon/aws/hooks/kinesis.py
+++ b/airflow/providers/amazon/aws/hooks/kinesis.py
@@ -43,9 +43,7 @@ def __init__(self, delivery_stream: str, *args, **kwargs) -> None:
def put_records(self, records: Iterable):
"""Write batch records to Kinesis Firehose"""
- response = self.get_conn().put_record_batch(DeliveryStreamName=self.delivery_stream, Records=records)
-
- return response
+ return self.get_conn().put_record_batch(DeliveryStreamName=self.delivery_stream, Records=records)
class AwsFirehoseHook(FirehoseHook):
diff --git a/airflow/providers/amazon/aws/hooks/lambda_function.py b/airflow/providers/amazon/aws/hooks/lambda_function.py
index 0f04af7e3b57b..b6819d9dba263 100644
--- a/airflow/providers/amazon/aws/hooks/lambda_function.py
+++ b/airflow/providers/amazon/aws/hooks/lambda_function.py
@@ -66,8 +66,7 @@ def invoke_lambda(
"Payload": payload,
"Qualifier": qualifier,
}
- response = self.conn.invoke(**{k: v for k, v in invoke_args.items() if v is not None})
- return response
+ return self.conn.invoke(**{k: v for k, v in invoke_args.items() if v is not None})
def create_lambda(
self,
@@ -118,10 +117,9 @@ def create_lambda(
"CodeSigningConfigArn": code_signing_config_arn,
"Architectures": architectures,
}
- response = self.conn.create_function(
+ return self.conn.create_function(
**{k: v for k, v in create_function_args.items() if v is not None},
)
- return response
class AwsLambdaHook(LambdaHook):
diff --git a/airflow/providers/amazon/aws/hooks/quicksight.py b/airflow/providers/amazon/aws/hooks/quicksight.py
new file mode 100644
index 0000000000000..a7e90c36cf92a
--- /dev/null
+++ b/airflow/providers/amazon/aws/hooks/quicksight.py
@@ -0,0 +1,153 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sys
+import time
+
+from botocore.exceptions import ClientError
+
+from airflow import AirflowException
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+
+if sys.version_info >= (3, 8):
+ from functools import cached_property
+else:
+ from cached_property import cached_property
+
+
+class QuickSightHook(AwsBaseHook):
+ """
+ Interact with Amazon QuickSight.
+
+ Additional arguments (such as ``aws_conn_id``) may be specified and
+ are passed down to the underlying AwsBaseHook.
+ .. seealso::
+ :class:`~airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook`
+ """
+
+ NON_TERMINAL_STATES = {"INITIALIZED", "QUEUED", "RUNNING"}
+ FAILED_STATES = {"FAILED"}
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(client_type="quicksight", *args, **kwargs)
+
+ @cached_property
+ def sts_hook(self):
+ return StsHook(aws_conn_id=self.aws_conn_id)
+
+ def create_ingestion(
+ self,
+ data_set_id: str,
+ ingestion_id: str,
+ ingestion_type: str,
+ wait_for_completion: bool = True,
+ check_interval: int = 30,
+ ):
+ """
+ Creates and starts a new SPICE ingestion for a dataset. Refreshes the SPICE datasets
+
+ :param data_set_id: ID of the dataset used in the ingestion.
+ :param ingestion_id: ID for the ingestion.
+ :param ingestion_type: Type of ingestion . "INCREMENTAL_REFRESH"|"FULL_REFRESH"
+ :param wait_for_completion: if the program should keep running until job finishes
+ :param check_interval: the time interval in seconds which the operator
+ will check the status of QuickSight Ingestion
+ :return: Returns descriptive information about the created data ingestion
+ having Ingestion ARN, HTTP status, ingestion ID and ingestion status.
+ :rtype: Dict
+ """
+
+ self.log.info("Creating QuickSight Ingestion for data set id %s.", data_set_id)
+ quicksight_client = self.get_conn()
+ try:
+ aws_account_id = self.sts_hook.get_account_number()
+ create_ingestion_response = quicksight_client.create_ingestion(
+ DataSetId=data_set_id,
+ IngestionId=ingestion_id,
+ IngestionType=ingestion_type,
+ AwsAccountId=aws_account_id,
+ )
+
+ if wait_for_completion:
+ self.wait_for_state(
+ aws_account_id=aws_account_id,
+ data_set_id=data_set_id,
+ ingestion_id=ingestion_id,
+ target_state={"COMPLETED"},
+ check_interval=check_interval,
+ )
+ return create_ingestion_response
+ except Exception as general_error:
+ self.log.error("Failed to run Amazon QuickSight create_ingestion API, error: %s", general_error)
+ raise
+
+ def get_status(self, aws_account_id: str, data_set_id: str, ingestion_id: str):
+ """
+ Get the current status of QuickSight Create Ingestion API.
+
+ :param aws_account_id: An AWS Account ID
+ :param data_set_id: QuickSight Data Set ID
+ :param ingestion_id: QuickSight Ingestion ID
+ :return: An QuickSight Ingestion Status
+ :rtype: str
+ """
+ try:
+ describe_ingestion_response = self.get_conn().describe_ingestion(
+ AwsAccountId=aws_account_id, DataSetId=data_set_id, IngestionId=ingestion_id
+ )
+ return describe_ingestion_response["Ingestion"]["IngestionStatus"]
+ except KeyError:
+ raise AirflowException("Could not get status of the Amazon QuickSight Ingestion")
+ except ClientError:
+ raise AirflowException("AWS request failed, check logs for more info")
+
+ def wait_for_state(
+ self,
+ aws_account_id: str,
+ data_set_id: str,
+ ingestion_id: str,
+ target_state: set,
+ check_interval: int,
+ ):
+ """
+ Check status of a QuickSight Create Ingestion API
+
+ :param aws_account_id: An AWS Account ID
+ :param data_set_id: QuickSight Data Set ID
+ :param ingestion_id: QuickSight Ingestion ID
+ :param target_state: Describes the QuickSight Job's Target State
+ :param check_interval: the time interval in seconds which the operator
+ will check the status of QuickSight Ingestion
+ :return: response of describe_ingestion call after Ingestion is is done
+ """
+
+ sec = 0
+ status = self.get_status(aws_account_id, data_set_id, ingestion_id)
+ while status in self.NON_TERMINAL_STATES and status != target_state:
+ self.log.info("Current status is %s", status)
+ time.sleep(check_interval)
+ sec += check_interval
+ if status in self.FAILED_STATES:
+ raise AirflowException("The Amazon QuickSight Ingestion failed!")
+ if status == "CANCELLED":
+ raise AirflowException("The Amazon QuickSight SPICE ingestion cancelled!")
+ status = self.get_status(aws_account_id, data_set_id, ingestion_id)
+
+ self.log.info("QuickSight Ingestion completed")
+ return status
diff --git a/airflow/providers/amazon/aws/hooks/redshift_cluster.py b/airflow/providers/amazon/aws/hooks/redshift_cluster.py
index 88b0e85308ab7..c7b3c51722141 100644
--- a/airflow/providers/amazon/aws/hooks/redshift_cluster.py
+++ b/airflow/providers/amazon/aws/hooks/redshift_cluster.py
@@ -15,7 +15,9 @@
# specific language governing permissions and limitations
# under the License.
-from typing import List, Optional
+from typing import Any, Dict, List, Optional
+
+from botocore.exceptions import ClientError
from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
@@ -37,6 +39,40 @@ def __init__(self, *args, **kwargs) -> None:
kwargs["client_type"] = "redshift"
super().__init__(*args, **kwargs)
+ def create_cluster(
+ self,
+ cluster_identifier: str,
+ node_type: str,
+ master_username: str,
+ master_user_password: str,
+ params: Dict[str, Any],
+ ) -> Dict[str, Any]:
+ """
+ Creates a new cluster with the specified parameters
+
+ :param cluster_identifier: A unique identifier for the cluster.
+ :param node_type: The node type to be provisioned for the cluster.
+ Valid Values: ``ds2.xlarge``, ``ds2.8xlarge``, ``dc1.large``,
+ ``dc1.8xlarge``, ``dc2.large``, ``dc2.8xlarge``, ``ra3.xlplus``,
+ ``ra3.4xlarge``, and ``ra3.16xlarge``.
+ :param master_username: The username associated with the admin user account
+ for the cluster that is being created.
+ :param master_user_password: password associated with the admin user account
+ for the cluster that is being created.
+ :param params: Remaining AWS Create cluster API params.
+ """
+ try:
+ response = self.get_conn().create_cluster(
+ ClusterIdentifier=cluster_identifier,
+ NodeType=node_type,
+ MasterUsername=master_username,
+ MasterUserPassword=master_user_password,
+ **params,
+ )
+ return response
+ except ClientError as e:
+ raise e
+
# TODO: Wrap create_cluster_snapshot
def cluster_status(self, cluster_identifier: str) -> str:
"""
diff --git a/airflow/providers/amazon/aws/hooks/redshift_sql.py b/airflow/providers/amazon/aws/hooks/redshift_sql.py
index 0b889063cc10a..03bb45f7ee128 100644
--- a/airflow/providers/amazon/aws/hooks/redshift_sql.py
+++ b/airflow/providers/amazon/aws/hooks/redshift_sql.py
@@ -130,6 +130,4 @@ def get_conn(self) -> RedshiftConnection:
conn_params = self._get_conn_params()
conn_kwargs_dejson = self.conn.extra_dejson
conn_kwargs: Dict = {**conn_params, **conn_kwargs_dejson}
- conn: RedshiftConnection = redshift_connector.connect(**conn_kwargs)
-
- return conn
+ return redshift_connector.connect(**conn_kwargs)
diff --git a/airflow/providers/amazon/aws/hooks/s3.py b/airflow/providers/amazon/aws/hooks/s3.py
index fc23255a78da4..e7e9f2de508da 100644
--- a/airflow/providers/amazon/aws/hooks/s3.py
+++ b/airflow/providers/amazon/aws/hooks/s3.py
@@ -146,6 +146,35 @@ def parse_s3_url(s3url: str) -> Tuple[str, str]:
return bucket_name, key
+ @staticmethod
+ def get_s3_bucket_key(
+ bucket: Optional[str], key: str, bucket_param_name: str, key_param_name: str
+ ) -> Tuple[str, str]:
+ """
+ Get the S3 bucket name and key from either:
+ - bucket name and key. Return the info as it is after checking `key` is a relative path
+ - key. Must be a full s3:// url
+
+ :param bucket: The S3 bucket name
+ :param key: The S3 key
+ :param bucket_param_name: The parameter name containing the bucket name
+ :param key_param_name: The parameter name containing the key name
+ :return: the parsed bucket name and key
+ :rtype: tuple of str
+ """
+
+ if bucket is None:
+ return S3Hook.parse_s3_url(key)
+
+ parsed_url = urlparse(key)
+ if parsed_url.scheme != '' or parsed_url.netloc != '':
+ raise TypeError(
+ f'If `{bucket_param_name}` is provided, {key_param_name} should be a relative path '
+ 'from root level, rather than a full s3:// url'
+ )
+
+ return bucket, key
+
@provide_bucket_name
def check_for_bucket(self, bucket_name: Optional[str] = None) -> bool:
"""
@@ -248,11 +277,10 @@ def list_prefixes(
Bucket=bucket_name, Prefix=prefix, Delimiter=delimiter, PaginationConfig=config
)
- prefixes = []
+ prefixes = [] # type: List[str]
for page in response:
if 'CommonPrefixes' in page:
- for common_prefix in page['CommonPrefixes']:
- prefixes.append(common_prefix['Prefix'])
+ prefixes.extend(common_prefix['Prefix'] for common_prefix in page['CommonPrefixes'])
return prefixes
@@ -337,37 +365,80 @@ def _is_in_period(input_date: datetime) -> bool:
StartAfter=start_after_key,
)
- keys = []
+ keys = [] # type: List[str]
for page in response:
if 'Contents' in page:
- for k in page['Contents']:
- keys.append(k)
-
+ keys.extend(iter(page['Contents']))
if self.object_filter_usr is not None:
return self.object_filter_usr(keys, from_datetime, to_datetime)
return self._list_key_object_filter(keys, from_datetime, to_datetime)
+ @provide_bucket_name
+ def get_file_metadata(
+ self,
+ prefix: str,
+ bucket_name: Optional[str] = None,
+ page_size: Optional[int] = None,
+ max_items: Optional[int] = None,
+ ) -> List:
+ """
+ Lists metadata objects in a bucket under prefix
+
+ :param prefix: a key prefix
+ :param bucket_name: the name of the bucket
+ :param page_size: pagination size
+ :param max_items: maximum items to return
+ :return: a list of metadata of objects
+ :rtype: list
+ """
+ config = {
+ 'PageSize': page_size,
+ 'MaxItems': max_items,
+ }
+
+ paginator = self.get_conn().get_paginator('list_objects_v2')
+ response = paginator.paginate(Bucket=bucket_name, Prefix=prefix, PaginationConfig=config)
+
+ files = []
+ for page in response:
+ if 'Contents' in page:
+ files += page['Contents']
+ return files
+
@provide_bucket_name
@unify_bucket_name_and_key
- def check_for_key(self, key: str, bucket_name: Optional[str] = None) -> bool:
+ def head_object(self, key: str, bucket_name: Optional[str] = None) -> Optional[dict]:
"""
- Checks if a key exists in a bucket
+ Retrieves metadata of an object
:param key: S3 key that will point to the file
:param bucket_name: Name of the bucket in which the file is stored
- :return: True if the key exists and False if not.
- :rtype: bool
+ :return: metadata of an object
+ :rtype: dict
"""
try:
- self.get_conn().head_object(Bucket=bucket_name, Key=key)
- return True
+ return self.get_conn().head_object(Bucket=bucket_name, Key=key)
except ClientError as e:
if e.response["ResponseMetadata"]["HTTPStatusCode"] == 404:
- return False
+ return None
else:
raise e
+ @provide_bucket_name
+ @unify_bucket_name_and_key
+ def check_for_key(self, key: str, bucket_name: Optional[str] = None) -> bool:
+ """
+ Checks if a key exists in a bucket
+
+ :param key: S3 key that will point to the file
+ :param bucket_name: Name of the bucket in which the file is stored
+ :return: True if the key exists and False if not.
+ :rtype: bool
+ """
+ obj = self.head_object(key, bucket_name)
+ return obj is not None
+
@provide_bucket_name
@unify_bucket_name_and_key
def get_key(self, key: str, bucket_name: Optional[str] = None) -> S3Transfer:
@@ -530,7 +601,7 @@ def load_file(
extra_args['ServerSideEncryption'] = "AES256"
if gzip:
with open(filename, 'rb') as f_in:
- filename_gz = f_in.name + '.gz'
+ filename_gz = f'{f_in.name}.gz'
with gz.open(filename_gz, 'wb') as f_out:
shutil.copyfileobj(f_in, f_out)
filename = filename_gz
@@ -604,7 +675,7 @@ def load_bytes(
"""
Loads bytes to S3
- This is provided as a convenience to drop a string in S3. It uses the
+ This is provided as a convenience to drop bytes data into S3. It uses the
boto infrastructure to ship a file to s3.
:param bytes_data: bytes to set as content for the key.
@@ -710,27 +781,13 @@ def copy_object(
"""
acl_policy = acl_policy or 'private'
- if dest_bucket_name is None:
- dest_bucket_name, dest_bucket_key = self.parse_s3_url(dest_bucket_key)
- else:
- parsed_url = urlparse(dest_bucket_key)
- if parsed_url.scheme != '' or parsed_url.netloc != '':
- raise AirflowException(
- 'If dest_bucket_name is provided, '
- 'dest_bucket_key should be relative path '
- 'from root level, rather than a full s3:// url'
- )
+ dest_bucket_name, dest_bucket_key = self.get_s3_bucket_key(
+ dest_bucket_name, dest_bucket_key, 'dest_bucket_name', 'dest_bucket_key'
+ )
- if source_bucket_name is None:
- source_bucket_name, source_bucket_key = self.parse_s3_url(source_bucket_key)
- else:
- parsed_url = urlparse(source_bucket_key)
- if parsed_url.scheme != '' or parsed_url.netloc != '':
- raise AirflowException(
- 'If source_bucket_name is provided, '
- 'source_bucket_key should be relative path '
- 'from root level, rather than a full s3:// url'
- )
+ source_bucket_name, source_bucket_key = self.get_s3_bucket_key(
+ source_bucket_name, source_bucket_key, 'source_bucket_name', 'source_bucket_key'
+ )
copy_source = {'Bucket': source_bucket_name, 'Key': source_bucket_key, 'VersionId': source_version_id}
response = self.get_conn().copy_object(
diff --git a/airflow/providers/amazon/aws/hooks/sagemaker.py b/airflow/providers/amazon/aws/hooks/sagemaker.py
index 73348c8555aa6..2c8c28a738ec3 100644
--- a/airflow/providers/amazon/aws/hooks/sagemaker.py
+++ b/airflow/providers/amazon/aws/hooks/sagemaker.py
@@ -310,7 +310,8 @@ def create_training_job(
max_ingestion_time: Optional[int] = None,
):
"""
- Create a training job
+ Starts a model training job. After training completes, Amazon SageMaker saves
+ the resulting model artifacts to an Amazon S3 location that you specify.
:param config: the config for training
:param wait_for_completion: if the program should keep running until job finishes
@@ -357,7 +358,11 @@ def create_tuning_job(
max_ingestion_time: Optional[int] = None,
):
"""
- Create a tuning job
+ Starts a hyperparameter tuning job. A hyperparameter tuning job finds the
+ best version of a model by running many training jobs on your dataset using
+ the algorithm you choose and values for hyperparameters within ranges that
+ you specify. It then chooses the hyperparameter values that result in a model
+ that performs the best, as measured by an objective metric that you choose.
:param config: the config for tuning
:param wait_for_completion: if the program should keep running until job finishes
@@ -389,7 +394,8 @@ def create_transform_job(
max_ingestion_time: Optional[int] = None,
):
"""
- Create a transform job
+ Starts a transform job. A transform job uses a trained model to get inferences
+ on a dataset and saves these results to an Amazon S3 location that you specify.
:param config: the config for transform job
:param wait_for_completion: if the program should keep running until job finishes
@@ -422,7 +428,10 @@ def create_processing_job(
max_ingestion_time: Optional[int] = None,
):
"""
- Create a processing job
+ Use Amazon SageMaker Processing to analyze data and evaluate machine learning
+ models on Amazon SageMaker. With Processing, you can use a simplified, managed
+ experience on SageMaker to run your data processing workloads, such as feature
+ engineering, data validation, model evaluation, and model interpretation.
:param config: the config for processing job
:param wait_for_completion: if the program should keep running until job finishes
@@ -446,7 +455,10 @@ def create_processing_job(
def create_model(self, config: dict):
"""
- Create a model job
+ Creates a model in Amazon SageMaker. In the request, you name the model and
+ describe a primary container. For the primary container, you specify the Docker
+ image that contains inference code, artifacts (from prior training), and a custom
+ environment map that the inference code uses when you deploy the model for predictions.
:param config: the config for model
:return: A response to model creation
@@ -455,7 +467,14 @@ def create_model(self, config: dict):
def create_endpoint_config(self, config: dict):
"""
- Create an endpoint config
+ Creates an endpoint configuration that Amazon SageMaker hosting
+ services uses to deploy models. In the configuration, you identify
+ one or more models, created using the CreateModel API, to deploy and
+ the resources that you want Amazon SageMaker to provision.
+
+ .. seealso::
+ :class:`~airflow.providers.amazon.aws.hooks.sagemaker.SageMakerHook.create_model`
+ :class:`~airflow.providers.amazon.aws.hooks.sagemaker.SageMakerHook.create_endpoint`
:param config: the config for endpoint-config
:return: A response to endpoint config creation
@@ -470,7 +489,15 @@ def create_endpoint(
max_ingestion_time: Optional[int] = None,
):
"""
- Create an endpoint
+ When you create a serverless endpoint, SageMaker provisions and manages
+ the compute resources for you. Then, you can make inference requests to
+ the endpoint and receive model predictions in response. SageMaker scales
+ the compute resources up and down as needed to handle your request traffic.
+
+ Requires an Endpoint Config.
+ .. seealso::
+ :class:`~airflow.providers.amazon.aws.hooks.sagemaker.SageMakerHook.create_endpoint_config`
+
:param config: the config for endpoint
:param wait_for_completion: if the program should keep running until job finishes
@@ -501,7 +528,9 @@ def update_endpoint(
max_ingestion_time: Optional[int] = None,
):
"""
- Update an endpoint
+ Deploys the new EndpointConfig specified in the request, switches to using
+ newly created endpoint, and then deletes resources provisioned for the
+ endpoint using the previous EndpointConfig (there is no availability loss).
:param config: the config for endpoint
:param wait_for_completion: if the program should keep running until job finishes
diff --git a/airflow/providers/amazon/aws/hooks/ses.py b/airflow/providers/amazon/aws/hooks/ses.py
index 21efa14107071..92dcce7ecb617 100644
--- a/airflow/providers/amazon/aws/hooks/ses.py
+++ b/airflow/providers/amazon/aws/hooks/ses.py
@@ -106,7 +106,7 @@ class SESHook(SesHook):
def __init__(self, *args, **kwargs):
warnings.warn(
- "This hook is deprecated. " "Please use :class:`airflow.providers.amazon.aws.hooks.ses.SesHook`.",
+ "This hook is deprecated. Please use :class:`airflow.providers.amazon.aws.hooks.ses.SesHook`.",
DeprecationWarning,
stacklevel=2,
)
diff --git a/airflow/providers/amazon/aws/hooks/sns.py b/airflow/providers/amazon/aws/hooks/sns.py
index 94c83d8d31300..fc009d9f9bdae 100644
--- a/airflow/providers/amazon/aws/hooks/sns.py
+++ b/airflow/providers/amazon/aws/hooks/sns.py
@@ -100,7 +100,7 @@ class AwsSnsHook(SnsHook):
def __init__(self, *args, **kwargs):
warnings.warn(
- "This hook is deprecated. " "Please use :class:`airflow.providers.amazon.aws.hooks.sns.SnsHook`.",
+ "This hook is deprecated. Please use :class:`airflow.providers.amazon.aws.hooks.sns.SnsHook`.",
DeprecationWarning,
stacklevel=2,
)
diff --git a/airflow/providers/amazon/aws/hooks/sqs.py b/airflow/providers/amazon/aws/hooks/sqs.py
index c6de9f904879a..b94756f63aa26 100644
--- a/airflow/providers/amazon/aws/hooks/sqs.py
+++ b/airflow/providers/amazon/aws/hooks/sqs.py
@@ -88,7 +88,7 @@ class SQSHook(SqsHook):
def __init__(self, *args, **kwargs):
warnings.warn(
- "This hook is deprecated. " "Please use :class:`airflow.providers.amazon.aws.hooks.sqs.SqsHook`.",
+ "This hook is deprecated. Please use :class:`airflow.providers.amazon.aws.hooks.sqs.SqsHook`.",
DeprecationWarning,
stacklevel=2,
)
diff --git a/airflow/providers/amazon/aws/hooks/sts.py b/airflow/providers/amazon/aws/hooks/sts.py
new file mode 100644
index 0000000000000..aff787ee5d70e
--- /dev/null
+++ b/airflow/providers/amazon/aws/hooks/sts.py
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+
+
+class StsHook(AwsBaseHook):
+ """
+ Interact with AWS Security Token Service (STS)
+
+ Additional arguments (such as ``aws_conn_id``) may be specified and
+ are passed down to the underlying AwsBaseHook.
+ .. seealso::
+ :class:`~airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook`
+ """
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(client_type="sts", *args, **kwargs)
+
+ def get_account_number(self) -> str:
+ """Get the account Number"""
+
+ try:
+ return self.get_conn().get_caller_identity()['Account']
+ except Exception as general_error:
+ self.log.error("Failed to get the AWS Account Number, error: %s", general_error)
+ raise
diff --git a/airflow/providers/amazon/aws/log/s3_task_handler.py b/airflow/providers/amazon/aws/log/s3_task_handler.py
index 695c4623d97b2..ce3da88f16916 100644
--- a/airflow/providers/amazon/aws/log/s3_task_handler.py
+++ b/airflow/providers/amazon/aws/log/s3_task_handler.py
@@ -16,6 +16,7 @@
# specific language governing permissions and limitations
# under the License.
import os
+import pathlib
import sys
if sys.version_info >= (3, 8):
@@ -92,8 +93,7 @@ def close(self):
remote_loc = os.path.join(self.remote_base, self.log_relative_path)
if os.path.exists(local_loc):
# read log and remove old logs to get just the latest additions
- with open(local_loc) as logfile:
- log = logfile.read()
+ log = pathlib.Path(local_loc).read_text()
self.s3_write(log, remote_loc)
# Mark closed so we don't double write if close is called twice
diff --git a/airflow/providers/amazon/aws/operators/cloud_formation.py b/airflow/providers/amazon/aws/operators/cloud_formation.py
index 677ac81ddf602..2e7bb8ae121c6 100644
--- a/airflow/providers/amazon/aws/operators/cloud_formation.py
+++ b/airflow/providers/amazon/aws/operators/cloud_formation.py
@@ -29,11 +29,13 @@ class CloudFormationCreateStackOperator(BaseOperator):
"""
An operator that creates a CloudFormation stack.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:CloudFormationCreateStackOperator`
+
+
:param stack_name: stack name (templated)
:param cloudformation_parameters: parameters to be passed to CloudFormation.
-
- .. seealso::
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.create_stack
:param aws_conn_id: aws connection to uses
"""
@@ -63,8 +65,10 @@ class CloudFormationDeleteStackOperator(BaseOperator):
:param stack_name: stack name (templated)
:param cloudformation_parameters: parameters to be passed to CloudFormation.
- .. seealso::
- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/cloudformation.html#CloudFormation.Client.delete_stack
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:CloudFormationDeleteStackOperator`
+
:param aws_conn_id: aws connection to uses
"""
diff --git a/airflow/providers/amazon/aws/operators/datasync.py b/airflow/providers/amazon/aws/operators/datasync.py
index 932f7b31abc25..5a0c86071139f 100644
--- a/airflow/providers/amazon/aws/operators/datasync.py
+++ b/airflow/providers/amazon/aws/operators/datasync.py
@@ -31,7 +31,7 @@
class DataSyncOperator(BaseOperator):
- r"""Find, Create, Update, Execute and Delete AWS DataSync Tasks.
+ """Find, Create, Update, Execute and Delete AWS DataSync Tasks.
If ``do_xcom_push`` is True, then the DataSync TaskArn and TaskExecutionArn
which were executed will be pushed to an XCom.
diff --git a/airflow/providers/amazon/aws/operators/dms.py b/airflow/providers/amazon/aws/operators/dms.py
index aad16155525e8..aca515cfed3a9 100644
--- a/airflow/providers/amazon/aws/operators/dms.py
+++ b/airflow/providers/amazon/aws/operators/dms.py
@@ -15,8 +15,6 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-
from typing import TYPE_CHECKING, Dict, Optional, Sequence
from airflow.models import BaseOperator
@@ -154,6 +152,10 @@ class DmsDescribeTasksOperator(BaseOperator):
"""
Describes AWS DMS replication tasks.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:DmsDescribeTasksOperator`
+
:param describe_tasks_kwargs: Describe tasks command arguments
:param aws_conn_id: The Airflow connection used for AWS credentials.
If this is None or empty then the default boto3 behaviour is used. If
@@ -250,6 +252,10 @@ class DmsStopTaskOperator(BaseOperator):
"""
Stops AWS DMS replication task.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:DmsStopTaskOperator`
+
:param replication_task_arn: Replication task ARN
:param aws_conn_id: The Airflow connection used for AWS credentials.
If this is None or empty then the default boto3 behaviour is used. If
diff --git a/airflow/providers/amazon/aws/operators/ec2.py b/airflow/providers/amazon/aws/operators/ec2.py
index 033760be5b560..133596929e423 100644
--- a/airflow/providers/amazon/aws/operators/ec2.py
+++ b/airflow/providers/amazon/aws/operators/ec2.py
@@ -30,6 +30,10 @@ class EC2StartInstanceOperator(BaseOperator):
"""
Start AWS EC2 instance using boto3.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:EC2StartInstanceOperator`
+
:param instance_id: id of the AWS EC2 instance
:param aws_conn_id: aws connection to use
:param region_name: (optional) aws region name associated with the client
@@ -72,6 +76,10 @@ class EC2StopInstanceOperator(BaseOperator):
"""
Stop AWS EC2 instance using boto3.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:EC2StopInstanceOperator`
+
:param instance_id: id of the AWS EC2 instance
:param aws_conn_id: aws connection to use
:param region_name: (optional) aws region name associated with the client
diff --git a/airflow/providers/amazon/aws/operators/ecs.py b/airflow/providers/amazon/aws/operators/ecs.py
index b6bae92388190..d1112edf445ee 100644
--- a/airflow/providers/amazon/aws/operators/ecs.py
+++ b/airflow/providers/amazon/aws/operators/ecs.py
@@ -25,7 +25,7 @@
from threading import Event, Thread
from typing import Dict, Generator, Optional, Sequence
-from botocore.exceptions import ClientError
+from botocore.exceptions import ClientError, ConnectionClosedError
from botocore.waiter import Waiter
from airflow.exceptions import AirflowException
@@ -139,6 +139,9 @@ def _get_log_events(self, skip: int = 0) -> Generator:
self.logger.warning('Error on retrieving Cloudwatch log events', error)
yield from ()
+ except ConnectionClosedError as error:
+ self.logger.warning('ConnectionClosedError on retrieving Cloudwatch log events', error)
+ yield from ()
def _event_to_str(self, event: dict) -> str:
event_dt = datetime.utcfromtimestamp(event['timestamp'] / 1000.0)
@@ -372,7 +375,7 @@ def _start_task(self, context):
self.arn = response['tasks'][0]['taskArn']
self.ecs_task_id = self.arn.split("/")[-1]
- self.log.info(f"ECS task ID is: {self.ecs_task_id}")
+ self.log.info("ECS task ID is: %s", self.ecs_task_id)
if self.reattach:
# Save the task ARN in XCom to be able to reattach it if needed
@@ -409,6 +412,7 @@ def _try_reattach_task(self, context):
)
if previous_task_arn in running_tasks:
self.arn = previous_task_arn
+ self.ecs_task_id = self.arn.split("/")[-1]
self.log.info("Reattaching previously launched task: %s", self.arn)
else:
self.log.info("No active previously launched task found to reattach")
@@ -466,9 +470,8 @@ def _check_success_task(self) -> None:
# https://docs.aws.amazon.com/AmazonECS/latest/developerguide/stopped-task-errors.html
if re.match(r'Host EC2 \(instance .+?\) (stopped|terminated)\.', task.get('stoppedReason', '')):
raise AirflowException(
- 'The task was stopped because the host instance terminated: {}'.format(
- task.get('stoppedReason', '')
- )
+ f"The task was stopped because the host instance terminated:"
+ f" {task.get('stoppedReason', '')}"
)
containers = task['containers']
for container in containers:
@@ -487,9 +490,8 @@ def _check_success_task(self) -> None:
raise AirflowException(f'This task is still pending {task}')
elif 'error' in container.get('reason', '').lower():
raise AirflowException(
- 'This containers encounter an error during launching : {}'.format(
- container.get('reason', '').lower()
- )
+ f"This containers encounter an error during launching: "
+ f"{container.get('reason', '').lower()}"
)
def get_hook(self) -> AwsBaseHook:
diff --git a/airflow/providers/amazon/aws/operators/eks.py b/airflow/providers/amazon/aws/operators/eks.py
index ef40cab2e1e8a..8e97015d86182 100644
--- a/airflow/providers/amazon/aws/operators/eks.py
+++ b/airflow/providers/amazon/aws/operators/eks.py
@@ -139,20 +139,6 @@ def __init__(
region: Optional[str] = None,
**kwargs,
) -> None:
- if compute:
- if compute not in SUPPORTED_COMPUTE_VALUES:
- raise ValueError("Provided compute type is not supported.")
- elif (compute == 'nodegroup') and not nodegroup_role_arn:
- raise ValueError(
- MISSING_ARN_MSG.format(compute=NODEGROUP_FULL_NAME, requirement='nodegroup_role_arn')
- )
- elif (compute == 'fargate') and not fargate_pod_execution_role_arn:
- raise ValueError(
- MISSING_ARN_MSG.format(
- compute=FARGATE_FULL_NAME, requirement='fargate_pod_execution_role_arn'
- )
- )
-
self.compute = compute
self.cluster_name = cluster_name
self.cluster_role_arn = cluster_role_arn
@@ -170,6 +156,20 @@ def __init__(
super().__init__(**kwargs)
def execute(self, context: 'Context'):
+ if self.compute:
+ if self.compute not in SUPPORTED_COMPUTE_VALUES:
+ raise ValueError("Provided compute type is not supported.")
+ elif (self.compute == 'nodegroup') and not self.nodegroup_role_arn:
+ raise ValueError(
+ MISSING_ARN_MSG.format(compute=NODEGROUP_FULL_NAME, requirement='nodegroup_role_arn')
+ )
+ elif (self.compute == 'fargate') and not self.fargate_pod_execution_role_arn:
+ raise ValueError(
+ MISSING_ARN_MSG.format(
+ compute=FARGATE_FULL_NAME, requirement='fargate_pod_execution_role_arn'
+ )
+ )
+
eks_hook = EksHook(
aws_conn_id=self.aws_conn_id,
region_name=self.region,
@@ -275,23 +275,23 @@ def __init__(
self.create_nodegroup_kwargs = create_nodegroup_kwargs or {}
self.aws_conn_id = aws_conn_id
self.region = region
- nodegroup_subnets_list: List[str] = []
- if isinstance(nodegroup_subnets, str):
- if nodegroup_subnets != "":
+ self.nodegroup_subnets = nodegroup_subnets
+ super().__init__(**kwargs)
+
+ def execute(self, context: 'Context'):
+ if isinstance(self.nodegroup_subnets, str):
+ nodegroup_subnets_list: List[str] = []
+ if self.nodegroup_subnets != "":
try:
- nodegroup_subnets_list = cast(List, literal_eval(nodegroup_subnets))
+ nodegroup_subnets_list = cast(List, literal_eval(self.nodegroup_subnets))
except ValueError:
self.log.warning(
"The nodegroup_subnets should be List or string representing "
"Python list and is %s. Defaulting to []",
- nodegroup_subnets,
+ self.nodegroup_subnets,
)
- else:
- nodegroup_subnets_list = nodegroup_subnets
- self.nodegroup_subnets = nodegroup_subnets_list
- super().__init__(**kwargs)
+ self.nodegroup_subnets = nodegroup_subnets_list
- def execute(self, context: 'Context'):
eks_hook = EksHook(
aws_conn_id=self.aws_conn_id,
region_name=self.region,
diff --git a/airflow/providers/amazon/aws/operators/emr.py b/airflow/providers/amazon/aws/operators/emr.py
index 6f942e297fb9c..a1f3fa753d817 100644
--- a/airflow/providers/amazon/aws/operators/emr.py
+++ b/airflow/providers/amazon/aws/operators/emr.py
@@ -136,6 +136,8 @@ class EmrContainerOperator(BaseOperator):
:param poll_interval: Time (in seconds) to wait between two consecutive calls to check query status on EMR
:param max_tries: Maximum number of times to wait for the job run to finish.
Defaults to None, which will poll until the job is *not* in a pending, submitted, or running state.
+ :param tags: The tags assigned to job runs.
+ Defaults to None
"""
template_fields: Sequence[str] = (
@@ -160,6 +162,7 @@ def __init__(
aws_conn_id: str = "aws_default",
poll_interval: int = 30,
max_tries: Optional[int] = None,
+ tags: Optional[dict] = None,
**kwargs: Any,
) -> None:
super().__init__(**kwargs)
@@ -173,6 +176,7 @@ def __init__(
self.client_request_token = client_request_token or str(uuid4())
self.poll_interval = poll_interval
self.max_tries = max_tries
+ self.tags = tags
self.job_id: Optional[str] = None
@cached_property
@@ -192,6 +196,7 @@ def execute(self, context: 'Context') -> Optional[str]:
self.job_driver,
self.configuration_overrides,
self.client_request_token,
+ self.tags,
)
query_status = self.hook.poll_query_status(self.job_id, self.max_tries, self.poll_interval)
diff --git a/airflow/providers/amazon/aws/operators/glacier.py b/airflow/providers/amazon/aws/operators/glacier.py
index 27904e4d1d0a3..337492a4523e1 100644
--- a/airflow/providers/amazon/aws/operators/glacier.py
+++ b/airflow/providers/amazon/aws/operators/glacier.py
@@ -51,5 +51,4 @@ def __init__(
def execute(self, context: 'Context'):
hook = GlacierHook(aws_conn_id=self.aws_conn_id)
- response = hook.retrieve_inventory(vault_name=self.vault_name)
- return response
+ return hook.retrieve_inventory(vault_name=self.vault_name)
diff --git a/airflow/providers/amazon/aws/operators/glue.py b/airflow/providers/amazon/aws/operators/glue.py
index 10e0530df7754..cb76e784300e0 100644
--- a/airflow/providers/amazon/aws/operators/glue.py
+++ b/airflow/providers/amazon/aws/operators/glue.py
@@ -34,6 +34,10 @@ class GlueJobOperator(BaseOperator):
ETL service for running Spark Jobs on the AWS cloud.
Language support: Python and Scala
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:GlueJobOperator`
+
:param job_name: unique job name per AWS Account
:param script_location: location of ETL script. Must be a local or S3 path
:param job_desc: job description details
@@ -100,7 +104,9 @@ def execute(self, context: 'Context'):
:return: the id of the current glue job.
"""
- if not self.script_location.startswith(self.s3_protocol):
+ if self.script_location is None:
+ s3_script_location = None
+ elif not self.script_location.startswith(self.s3_protocol):
s3_hook = S3Hook(aws_conn_id=self.aws_conn_id)
script_name = os.path.basename(self.script_location)
s3_hook.load_file(
diff --git a/airflow/providers/amazon/aws/operators/glue_crawler.py b/airflow/providers/amazon/aws/operators/glue_crawler.py
index 0bd27aaeaf0b6..a584301f926d1 100644
--- a/airflow/providers/amazon/aws/operators/glue_crawler.py
+++ b/airflow/providers/amazon/aws/operators/glue_crawler.py
@@ -38,9 +38,14 @@ class GlueCrawlerOperator(BaseOperator):
service that manages a catalog of metadata tables that contain the inferred
schema, format and data types of data stores within the AWS cloud.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:GlueCrawlerOperator`
+
:param config: Configurations for the AWS Glue crawler
:param aws_conn_id: aws connection to use
:param poll_interval: Time (in seconds) to wait between two consecutive calls to check crawler status
+ :param wait_for_completion: Whether or not wait for crawl execution completion. (default: True)
"""
ui_color = '#ededed'
@@ -50,11 +55,13 @@ def __init__(
config,
aws_conn_id='aws_default',
poll_interval: int = 5,
+ wait_for_completion: bool = True,
**kwargs,
):
super().__init__(**kwargs)
self.aws_conn_id = aws_conn_id
self.poll_interval = poll_interval
+ self.wait_for_completion = wait_for_completion
self.config = config
@cached_property
@@ -76,8 +83,9 @@ def execute(self, context: 'Context'):
self.log.info("Triggering AWS Glue Crawler")
self.hook.start_crawler(crawler_name)
- self.log.info("Waiting for AWS Glue Crawler")
- self.hook.wait_for_crawler_completion(crawler_name=crawler_name, poll_interval=self.poll_interval)
+ if self.wait_for_completion:
+ self.log.info("Waiting for AWS Glue Crawler")
+ self.hook.wait_for_crawler_completion(crawler_name=crawler_name, poll_interval=self.poll_interval)
return crawler_name
diff --git a/airflow/providers/amazon/aws/operators/quicksight.py b/airflow/providers/amazon/aws/operators/quicksight.py
new file mode 100644
index 0000000000000..a9da61bdfc3bf
--- /dev/null
+++ b/airflow/providers/amazon/aws/operators/quicksight.py
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Optional, Sequence
+
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.quicksight import QuickSightHook
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+DEFAULT_CONN_ID = "aws_default"
+
+
+class QuickSightCreateIngestionOperator(BaseOperator):
+ """
+ Creates and starts a new SPICE ingestion for a dataset.
+ Also, helps to Refresh existing SPICE datasets.
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:QuickSightCreateIngestionOperator`
+
+ :param data_set_id: ID of the dataset used in the ingestion.
+ :param ingestion_id: ID for the ingestion.
+ :param ingestion_type: Type of ingestion. Values Can be INCREMENTAL_REFRESH or FULL_REFRESH.
+ Default FULL_REFRESH.
+ :param wait_for_completion: If wait is set to True, the time interval, in seconds,
+ that the operation waits to check the status of the Amazon QuickSight Ingestion.
+ :param check_interval: if wait is set to be true, this is the time interval
+ in seconds which the operator will check the status of the Amazon QuickSight Ingestion
+ :param aws_conn_id: The Airflow connection used for AWS credentials. (templated)
+ If this is None or empty then the default boto3 behaviour is used. If
+ running Airflow in a distributed manner and aws_conn_id is None or
+ empty, then the default boto3 configuration would be used (and must be
+ maintained on each worker node).
+ :param region: Which AWS region the connection should use. (templated)
+ If this is None or empty then the default boto3 behaviour is used.
+ """
+
+ template_fields: Sequence[str] = (
+ "data_set_id",
+ "ingestion_id",
+ "ingestion_type",
+ "wait_for_completion",
+ "check_interval",
+ "aws_conn_id",
+ "region",
+ )
+ ui_color = "#ffd700"
+
+ def __init__(
+ self,
+ data_set_id: str,
+ ingestion_id: str,
+ ingestion_type: str = "FULL_REFRESH",
+ wait_for_completion: bool = True,
+ check_interval: int = 30,
+ aws_conn_id: str = DEFAULT_CONN_ID,
+ region: Optional[str] = None,
+ **kwargs,
+ ):
+ self.data_set_id = data_set_id
+ self.ingestion_id = ingestion_id
+ self.ingestion_type = ingestion_type
+ self.wait_for_completion = wait_for_completion
+ self.check_interval = check_interval
+ self.aws_conn_id = aws_conn_id
+ self.region = region
+ super().__init__(**kwargs)
+
+ def execute(self, context: "Context"):
+ hook = QuickSightHook(
+ aws_conn_id=self.aws_conn_id,
+ region_name=self.region,
+ )
+ self.log.info("Running the Amazon QuickSight SPICE Ingestion on Dataset ID: %s)", self.data_set_id)
+ return hook.create_ingestion(
+ data_set_id=self.data_set_id,
+ ingestion_id=self.ingestion_id,
+ ingestion_type=self.ingestion_type,
+ wait_for_completion=self.wait_for_completion,
+ check_interval=self.check_interval,
+ )
diff --git a/airflow/providers/amazon/aws/operators/redshift_cluster.py b/airflow/providers/amazon/aws/operators/redshift_cluster.py
index ec4a88f498c56..340e9577efef4 100644
--- a/airflow/providers/amazon/aws/operators/redshift_cluster.py
+++ b/airflow/providers/amazon/aws/operators/redshift_cluster.py
@@ -14,7 +14,8 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-from typing import TYPE_CHECKING, Sequence
+import time
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Sequence
from airflow.models import BaseOperator
from airflow.providers.amazon.aws.hooks.redshift_cluster import RedshiftHook
@@ -23,6 +24,224 @@
from airflow.utils.context import Context
+class RedshiftCreateClusterOperator(BaseOperator):
+ """Creates a new cluster with the specified parameters.
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:RedshiftCreateClusterOperator`
+
+ :param cluster_identifier: A unique identifier for the cluster.
+ :param node_type: The node type to be provisioned for the cluster.
+ Valid Values: ``ds2.xlarge``, ``ds2.8xlarge``, ``dc1.large``,
+ ``dc1.8xlarge``, ``dc2.large``, ``dc2.8xlarge``, ``ra3.xlplus``,
+ ``ra3.4xlarge``, and ``ra3.16xlarge``.
+ :param master_username: The username associated with the admin user account for
+ the cluster that is being created.
+ :param master_user_password: The password associated with the admin user account for
+ the cluster that is being created.
+ :param cluster_type: The type of the cluster ``single-node`` or ``multi-node``.
+ The default value is ``multi-node``.
+ :param db_name: The name of the first database to be created when the cluster is created.
+ :param number_of_nodes: The number of compute nodes in the cluster.
+ This param require when ``cluster_type`` is ``multi-node``.
+ :param cluster_security_groups: A list of security groups to be associated with this cluster.
+ :param vpc_security_group_ids: A list of VPC security groups to be associated with the cluster.
+ :param cluster_subnet_group_name: The name of a cluster subnet group to be associated with this cluster.
+ :param availability_zone: The EC2 Availability Zone (AZ).
+ :param preferred_maintenance_window: The time range (in UTC) during which automated cluster
+ maintenance can occur.
+ :param cluster_parameter_group_name: The name of the parameter group to be associated with this cluster.
+ :param automated_snapshot_retention_period: The number of days that automated snapshots are retained.
+ The default value is ``1``.
+ :param manual_snapshot_retention_period: The default number of days to retain a manual snapshot.
+ :param port: The port number on which the cluster accepts incoming connections.
+ The Default value is ``5439``.
+ :param cluster_version: The version of a Redshift engine software that you want to deploy on the cluster.
+ :param allow_version_upgrade: Whether major version upgrades can be applied during the maintenance window.
+ The Default value is ``True``.
+ :parma publicly_accessible: Whether cluster can be accessed from a public network.
+ :parma encrypted: Whether data in the cluster is encrypted at rest.
+ The default value is ``False``.
+ :parma hsm_client_certificate_identifier: Name of the HSM client certificate
+ the Amazon Redshift cluster uses to retrieve the data.
+ :parma hsm_configuration_identifier: Name of the HSM configuration
+ :parma elastic_ip: The Elastic IP (EIP) address for the cluster.
+ :parma tags: A list of tag instances
+ :parma kms_key_id: KMS key id of encryption key.
+ :param enhanced_vpc_routing: Whether to create the cluster with enhanced VPC routing enabled
+ Default value is ``False``.
+ :param additional_info: Reserved
+ :param iam_roles: A list of IAM roles that can be used by the cluster to access other AWS services.
+ :param maintenance_track_name: Name of the maintenance track for the cluster.
+ :param snapshot_schedule_identifier: A unique identifier for the snapshot schedule.
+ :param availability_zone_relocation: Enable relocation for a Redshift cluster
+ between Availability Zones after the cluster is created.
+ :param aqua_configuration_status: The cluster is configured to use AQUA .
+ :param default_iam_role_arn: ARN for the IAM role.
+ :param aws_conn_id: str = The Airflow connection used for AWS credentials.
+ The default connection id is ``aws_default``.
+ """
+
+ template_fields: Sequence[str] = (
+ "cluster_identifier",
+ "cluster_type",
+ "node_type",
+ "number_of_nodes",
+ )
+ ui_color = "#eeaa11"
+ ui_fgcolor = "#ffffff"
+
+ def __init__(
+ self,
+ *,
+ cluster_identifier: str,
+ node_type: str,
+ master_username: str,
+ master_user_password: str,
+ cluster_type: str = "multi-node",
+ db_name: str = "dev",
+ number_of_nodes: int = 1,
+ cluster_security_groups: Optional[List[str]] = None,
+ vpc_security_group_ids: Optional[List[str]] = None,
+ cluster_subnet_group_name: Optional[str] = None,
+ availability_zone: Optional[str] = None,
+ preferred_maintenance_window: Optional[str] = None,
+ cluster_parameter_group_name: Optional[str] = None,
+ automated_snapshot_retention_period: int = 1,
+ manual_snapshot_retention_period: Optional[int] = None,
+ port: int = 5439,
+ cluster_version: str = "1.0",
+ allow_version_upgrade: bool = True,
+ publicly_accessible: bool = True,
+ encrypted: bool = False,
+ hsm_client_certificate_identifier: Optional[str] = None,
+ hsm_configuration_identifier: Optional[str] = None,
+ elastic_ip: Optional[str] = None,
+ tags: Optional[List[Any]] = None,
+ kms_key_id: Optional[str] = None,
+ enhanced_vpc_routing: bool = False,
+ additional_info: Optional[str] = None,
+ iam_roles: Optional[List[str]] = None,
+ maintenance_track_name: Optional[str] = None,
+ snapshot_schedule_identifier: Optional[str] = None,
+ availability_zone_relocation: Optional[bool] = None,
+ aqua_configuration_status: Optional[str] = None,
+ default_iam_role_arn: Optional[str] = None,
+ aws_conn_id: str = "aws_default",
+ **kwargs,
+ ):
+ super().__init__(**kwargs)
+ self.cluster_identifier = cluster_identifier
+ self.node_type = node_type
+ self.master_username = master_username
+ self.master_user_password = master_user_password
+ self.cluster_type = cluster_type
+ self.db_name = db_name
+ self.number_of_nodes = number_of_nodes
+ self.cluster_security_groups = cluster_security_groups
+ self.vpc_security_group_ids = vpc_security_group_ids
+ self.cluster_subnet_group_name = cluster_subnet_group_name
+ self.availability_zone = availability_zone
+ self.preferred_maintenance_window = preferred_maintenance_window
+ self.cluster_parameter_group_name = cluster_parameter_group_name
+ self.automated_snapshot_retention_period = automated_snapshot_retention_period
+ self.manual_snapshot_retention_period = manual_snapshot_retention_period
+ self.port = port
+ self.cluster_version = cluster_version
+ self.allow_version_upgrade = allow_version_upgrade
+ self.publicly_accessible = publicly_accessible
+ self.encrypted = encrypted
+ self.hsm_client_certificate_identifier = hsm_client_certificate_identifier
+ self.hsm_configuration_identifier = hsm_configuration_identifier
+ self.elastic_ip = elastic_ip
+ self.tags = tags
+ self.kms_key_id = kms_key_id
+ self.enhanced_vpc_routing = enhanced_vpc_routing
+ self.additional_info = additional_info
+ self.iam_roles = iam_roles
+ self.maintenance_track_name = maintenance_track_name
+ self.snapshot_schedule_identifier = snapshot_schedule_identifier
+ self.availability_zone_relocation = availability_zone_relocation
+ self.aqua_configuration_status = aqua_configuration_status
+ self.default_iam_role_arn = default_iam_role_arn
+ self.aws_conn_id = aws_conn_id
+ self.kwargs = kwargs
+
+ def execute(self, context: 'Context'):
+ redshift_hook = RedshiftHook(aws_conn_id=self.aws_conn_id)
+ self.log.info("Creating Redshift cluster %s", self.cluster_identifier)
+ params: Dict[str, Any] = {}
+ if self.db_name:
+ params["DBName"] = self.db_name
+ if self.cluster_type:
+ params["ClusterType"] = self.cluster_type
+ if self.cluster_type == "multi-node":
+ params["NumberOfNodes"] = self.number_of_nodes
+ if self.cluster_security_groups:
+ params["ClusterSecurityGroups"] = self.cluster_security_groups
+ if self.vpc_security_group_ids:
+ params["VpcSecurityGroupIds"] = self.vpc_security_group_ids
+ if self.cluster_subnet_group_name:
+ params["ClusterSubnetGroupName"] = self.cluster_subnet_group_name
+ if self.availability_zone:
+ params["AvailabilityZone"] = self.availability_zone
+ if self.preferred_maintenance_window:
+ params["PreferredMaintenanceWindow"] = self.preferred_maintenance_window
+ if self.cluster_parameter_group_name:
+ params["ClusterParameterGroupName"] = self.cluster_parameter_group_name
+ if self.automated_snapshot_retention_period:
+ params["AutomatedSnapshotRetentionPeriod"] = self.automated_snapshot_retention_period
+ if self.manual_snapshot_retention_period:
+ params["ManualSnapshotRetentionPeriod"] = self.manual_snapshot_retention_period
+ if self.port:
+ params["Port"] = self.port
+ if self.cluster_version:
+ params["ClusterVersion"] = self.cluster_version
+ if self.allow_version_upgrade:
+ params["AllowVersionUpgrade"] = self.allow_version_upgrade
+ if self.publicly_accessible:
+ params["PubliclyAccessible"] = self.publicly_accessible
+ if self.encrypted:
+ params["Encrypted"] = self.encrypted
+ if self.hsm_client_certificate_identifier:
+ params["HsmClientCertificateIdentifier"] = self.hsm_client_certificate_identifier
+ if self.hsm_configuration_identifier:
+ params["HsmConfigurationIdentifier"] = self.hsm_configuration_identifier
+ if self.elastic_ip:
+ params["ElasticIp"] = self.elastic_ip
+ if self.tags:
+ params["Tags"] = self.tags
+ if self.kms_key_id:
+ params["KmsKeyId"] = self.kms_key_id
+ if self.enhanced_vpc_routing:
+ params["EnhancedVpcRouting"] = self.enhanced_vpc_routing
+ if self.additional_info:
+ params["AdditionalInfo"] = self.additional_info
+ if self.iam_roles:
+ params["IamRoles"] = self.iam_roles
+ if self.maintenance_track_name:
+ params["MaintenanceTrackName"] = self.maintenance_track_name
+ if self.snapshot_schedule_identifier:
+ params["SnapshotScheduleIdentifier"] = self.snapshot_schedule_identifier
+ if self.availability_zone_relocation:
+ params["AvailabilityZoneRelocation"] = self.availability_zone_relocation
+ if self.aqua_configuration_status:
+ params["AquaConfigurationStatus"] = self.aqua_configuration_status
+ if self.default_iam_role_arn:
+ params["DefaultIamRoleArn"] = self.default_iam_role_arn
+
+ cluster = redshift_hook.create_cluster(
+ self.cluster_identifier,
+ self.node_type,
+ self.master_username,
+ self.master_user_password,
+ params,
+ )
+ self.log.info("Created Redshift cluster %s", self.cluster_identifier)
+ self.log.info(cluster)
+
+
class RedshiftResumeClusterOperator(BaseOperator):
"""
Resume a paused AWS Redshift Cluster
@@ -99,3 +318,66 @@ def execute(self, context: 'Context'):
self.log.warning(
"Unable to pause cluster since cluster is currently in status: %s", cluster_state
)
+
+
+class RedshiftDeleteClusterOperator(BaseOperator):
+ """
+ Delete an AWS Redshift cluster.
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:RedshiftDeleteClusterOperator`
+
+ :param cluster_identifier: unique identifier of a cluster
+ :param skip_final_cluster_snapshot: determines cluster snapshot creation
+ :param final_cluster_snapshot_identifier: name of final cluster snapshot
+ :param wait_for_completion: Whether wait for cluster deletion or not
+ The default value is ``True``
+ :param aws_conn_id: aws connection to use
+ :param poll_interval: Time (in seconds) to wait between two consecutive calls to check cluster state
+ """
+
+ template_fields: Sequence[str] = ("cluster_identifier",)
+ ui_color = "#eeaa11"
+ ui_fgcolor = "#ffffff"
+
+ def __init__(
+ self,
+ *,
+ cluster_identifier: str,
+ skip_final_cluster_snapshot: bool = True,
+ final_cluster_snapshot_identifier: Optional[str] = None,
+ wait_for_completion: bool = True,
+ aws_conn_id: str = "aws_default",
+ poll_interval: float = 30.0,
+ **kwargs,
+ ):
+ super().__init__(**kwargs)
+ self.cluster_identifier = cluster_identifier
+ self.skip_final_cluster_snapshot = skip_final_cluster_snapshot
+ self.final_cluster_snapshot_identifier = final_cluster_snapshot_identifier
+ self.wait_for_completion = wait_for_completion
+ self.redshift_hook = RedshiftHook(aws_conn_id=aws_conn_id)
+ self.poll_interval = poll_interval
+
+ def execute(self, context: 'Context'):
+ self.delete_cluster()
+
+ if self.wait_for_completion:
+ cluster_status: str = self.check_status()
+ while cluster_status != "cluster_not_found":
+ self.log.info(
+ "cluster status is %s. Sleeping for %s seconds.", cluster_status, self.poll_interval
+ )
+ time.sleep(self.poll_interval)
+ cluster_status = self.check_status()
+
+ def delete_cluster(self) -> None:
+ self.redshift_hook.delete_cluster(
+ cluster_identifier=self.cluster_identifier,
+ skip_final_cluster_snapshot=self.skip_final_cluster_snapshot,
+ final_cluster_snapshot_identifier=self.final_cluster_snapshot_identifier,
+ )
+
+ def check_status(self) -> str:
+ return self.redshift_hook.cluster_status(self.cluster_identifier)
diff --git a/airflow/providers/amazon/aws/operators/redshift_data.py b/airflow/providers/amazon/aws/operators/redshift_data.py
index 3961833475ae3..f2d47da655835 100644
--- a/airflow/providers/amazon/aws/operators/redshift_data.py
+++ b/airflow/providers/amazon/aws/operators/redshift_data.py
@@ -136,12 +136,12 @@ def wait_for_results(self, statement_id):
elif status == 'FAILED' or status == 'ABORTED':
raise ValueError(f"Statement {statement_id!r} terminated with status {status}.")
else:
- self.log.info(f"Query {status}")
+ self.log.info("Query %s", status)
sleep(self.poll_interval)
def execute(self, context: 'Context') -> None:
"""Execute a statement against Amazon Redshift"""
- self.log.info(f"Executing statement: {self.sql}")
+ self.log.info("Executing statement: %s", self.sql)
self.statement_id = self.execute_query()
diff --git a/airflow/providers/amazon/aws/operators/redshift_sql.py b/airflow/providers/amazon/aws/operators/redshift_sql.py
index e525e7963aa90..c7ad77acb5341 100644
--- a/airflow/providers/amazon/aws/operators/redshift_sql.py
+++ b/airflow/providers/amazon/aws/operators/redshift_sql.py
@@ -73,6 +73,6 @@ def get_hook(self) -> RedshiftSQLHook:
def execute(self, context: 'Context') -> None:
"""Execute a statement against Amazon Redshift"""
- self.log.info(f"Executing statement: {self.sql}")
+ self.log.info("Executing statement: %s", self.sql)
hook = self.get_hook()
hook.run(self.sql, autocommit=self.autocommit, parameters=self.parameters)
diff --git a/airflow/providers/amazon/aws/operators/s3.py b/airflow/providers/amazon/aws/operators/s3.py
index 90812e26011ba..7fbef6629c7f7 100644
--- a/airflow/providers/amazon/aws/operators/s3.py
+++ b/airflow/providers/amazon/aws/operators/s3.py
@@ -244,6 +244,10 @@ class S3CopyObjectOperator(BaseOperator):
Note: the S3 connection used here needs to have access to both
source and destination bucket/key.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:S3CopyObjectOperator`
+
:param source_bucket_key: The key of the source object. (templated)
It can be either full s3:// style url or relative path from root level.
@@ -318,11 +322,103 @@ def execute(self, context: 'Context'):
)
+class S3CreateObjectOperator(BaseOperator):
+ """
+ Creates a new object from `data` as string or bytes.
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:S3CreateObjectOperator`
+
+ :param s3_bucket: Name of the S3 bucket where to save the object. (templated)
+ It should be omitted when `bucket_key` is provided as a full s3:// url.
+ :param s3_key: The key of the object to be created. (templated)
+ It can be either full s3:// style url or relative path from root level.
+ When it's specified as a full s3:// url, please omit bucket_name.
+ :param data: string or bytes to save as content.
+ :param replace: If True, it will overwrite the key if it already exists
+ :param encrypt: If True, the file will be encrypted on the server-side
+ by S3 and will be stored in an encrypted form while at rest in S3.
+ :param acl_policy: String specifying the canned ACL policy for the file being
+ uploaded to the S3 bucket.
+ :param encoding: The string to byte encoding.
+ It should be specified only when `data` is provided as string.
+ :param compression: Type of compression to use, currently only gzip is supported.
+ It can be specified only when `data` is provided as string.
+ :param aws_conn_id: Connection id of the S3 connection to use
+ :param verify: Whether or not to verify SSL certificates for S3 connection.
+ By default SSL certificates are verified.
+
+ You can provide the following values:
+
+ - False: do not validate SSL certificates. SSL will still be used,
+ but SSL certificates will not be
+ verified.
+ - path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.
+ You can specify this argument if you want to use a different
+ CA cert bundle than the one used by botocore.
+
+ """
+
+ template_fields: Sequence[str] = ('s3_bucket', 's3_key', 'data')
+
+ def __init__(
+ self,
+ *,
+ s3_bucket: Optional[str] = None,
+ s3_key: str,
+ data: Union[str, bytes],
+ replace: bool = False,
+ encrypt: bool = False,
+ acl_policy: Optional[str] = None,
+ encoding: Optional[str] = None,
+ compression: Optional[str] = None,
+ aws_conn_id: str = 'aws_default',
+ verify: Optional[Union[str, bool]] = None,
+ **kwargs,
+ ):
+ super().__init__(**kwargs)
+
+ self.s3_bucket = s3_bucket
+ self.s3_key = s3_key
+ self.data = data
+ self.replace = replace
+ self.encrypt = encrypt
+ self.acl_policy = acl_policy
+ self.encoding = encoding
+ self.compression = compression
+ self.aws_conn_id = aws_conn_id
+ self.verify = verify
+
+ def execute(self, context: 'Context'):
+ s3_hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
+
+ s3_bucket, s3_key = s3_hook.get_s3_bucket_key(self.s3_bucket, self.s3_key, 'dest_bucket', 'dest_key')
+
+ if isinstance(self.data, str):
+ s3_hook.load_string(
+ self.data,
+ s3_key,
+ s3_bucket,
+ self.replace,
+ self.encrypt,
+ self.encoding,
+ self.acl_policy,
+ self.compression,
+ )
+ else:
+ s3_hook.load_bytes(self.data, s3_key, s3_bucket, self.replace, self.encrypt, self.acl_policy)
+
+
class S3DeleteObjectsOperator(BaseOperator):
"""
To enable users to delete single object or multiple objects from
a bucket using a single HTTP request.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:S3DeleteObjectsOperator`
+
:param bucket: Name of the bucket in which you are going to delete object(s). (templated)
:param keys: The key(s) to delete from S3 bucket. (templated)
@@ -392,7 +488,7 @@ class S3FileTransformOperator(BaseOperator):
location.
The locations of the source and the destination files in the local
- filesystem is provided as an first and second arguments to the
+ filesystem is provided as a first and second arguments to the
transformation script. The transformation script is expected to read the
data from source, transform it and write the output to the local
destination file. The operator then takes over control and uploads the
@@ -401,6 +497,10 @@ class S3FileTransformOperator(BaseOperator):
S3 Select is also available to filter the source contents. Users can
omit the transformation script if S3 Select expression is specified.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:S3FileTransformOperator`
+
:param source_s3_key: The key to be retrieved from S3. (templated)
:param dest_s3_key: The key to be written from S3. (templated)
:param transform_script: location of the executable transformation script
@@ -518,6 +618,10 @@ class S3ListOperator(BaseOperator):
This operator returns a python list with the name of objects which can be
used by `xcom` in the downstream task.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:S3ListOperator`
+
:param bucket: The S3 bucket where to find the objects. (templated)
:param prefix: Prefix string to filters the objects whose name begin with
such prefix. (templated)
@@ -589,6 +693,10 @@ class S3ListPrefixesOperator(BaseOperator):
This operator returns a python list with the name of all subfolders which
can be used by `xcom` in the downstream task.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:S3ListPrefixesOperator`
+
:param bucket: The S3 bucket where to find the subfolders. (templated)
:param prefix: Prefix string to filter the subfolders whose name begin with
such prefix. (templated)
diff --git a/airflow/providers/amazon/aws/operators/sagemaker.py b/airflow/providers/amazon/aws/operators/sagemaker.py
index 071c167e9a0d1..084e3e857e9be 100644
--- a/airflow/providers/amazon/aws/operators/sagemaker.py
+++ b/airflow/providers/amazon/aws/operators/sagemaker.py
@@ -34,12 +34,14 @@
if TYPE_CHECKING:
from airflow.utils.context import Context
+DEFAULT_CONN_ID = 'aws_default'
+CHECK_INTERVAL_SECOND = 30
+
class SageMakerBaseOperator(BaseOperator):
"""This is the base operator for all SageMaker operators.
:param config: The configuration necessary to start a training job (templated)
- :param aws_conn_id: The AWS connection ID to use.
"""
template_fields: Sequence[str] = ('config',)
@@ -48,9 +50,8 @@ class SageMakerBaseOperator(BaseOperator):
ui_color = '#ededed'
integer_fields: List[List[Any]] = []
- def __init__(self, *, config: dict, aws_conn_id: str = 'aws_default', **kwargs):
+ def __init__(self, *, config: dict, **kwargs):
super().__init__(**kwargs)
- self.aws_conn_id = aws_conn_id
self.config = config
def parse_integer(self, config, field):
@@ -105,12 +106,17 @@ def hook(self):
class SageMakerProcessingOperator(SageMakerBaseOperator):
- """Initiate a SageMaker processing job.
+ """
+ Use Amazon SageMaker Processing to analyze data and evaluate machine learning
+ models on Amazon SageMake. With Processing, you can use a simplified, managed
+ experience on SageMaker to run your data processing workloads, such as feature
+ engineering, data validation, model evaluation, and model interpretation.
- This operator returns The ARN of the processing job created in Amazon SageMaker.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:SageMakerProcessingOperator`
:param config: The configuration necessary to start a processing job (templated).
-
For details of the configuration parameter see :py:meth:`SageMaker.Client.create_processing_job`
:param aws_conn_id: The AWS connection ID to use.
:param wait_for_completion: If wait is set to True, the time interval, in seconds,
@@ -123,27 +129,29 @@ class SageMakerProcessingOperator(SageMakerBaseOperator):
the operation does not timeout.
:param action_if_job_exists: Behaviour if the job name already exists. Possible options are "increment"
(default) and "fail".
+ :return Dict: Returns The ARN of the processing job created in Amazon SageMaker.
"""
def __init__(
self,
*,
config: dict,
- aws_conn_id: str,
+ aws_conn_id: str = DEFAULT_CONN_ID,
wait_for_completion: bool = True,
print_log: bool = True,
- check_interval: int = 30,
+ check_interval: int = CHECK_INTERVAL_SECOND,
max_ingestion_time: Optional[int] = None,
action_if_job_exists: str = 'increment',
**kwargs,
):
- super().__init__(config=config, aws_conn_id=aws_conn_id, **kwargs)
+ super().__init__(config=config, **kwargs)
if action_if_job_exists not in ('increment', 'fail'):
raise AirflowException(
f"Argument action_if_job_exists accepts only 'increment' and 'fail'. \
Provided value: '{action_if_job_exists}'."
)
self.action_if_job_exists = action_if_job_exists
+ self.aws_conn_id = aws_conn_id
self.wait_for_completion = wait_for_completion
self.print_log = print_log
self.check_interval = check_interval
@@ -185,21 +193,34 @@ def execute(self, context: 'Context') -> dict:
class SageMakerEndpointConfigOperator(SageMakerBaseOperator):
"""
- Create a SageMaker endpoint config.
+ Creates an endpoint configuration that Amazon SageMaker hosting
+ services uses to deploy models. In the configuration, you identify
+ one or more models, created using the CreateModel API, to deploy and
+ the resources that you want Amazon SageMaker to provision.
- This operator returns The ARN of the endpoint config created in Amazon SageMaker
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:SageMakerEndpointConfigOperator`
:param config: The configuration necessary to create an endpoint config.
For details of the configuration parameter see :py:meth:`SageMaker.Client.create_endpoint_config`
:param aws_conn_id: The AWS connection ID to use.
+ :return Dict: Returns The ARN of the endpoint config created in Amazon SageMaker.
"""
integer_fields = [['ProductionVariants', 'InitialInstanceCount']]
- def __init__(self, *, config: dict, **kwargs):
+ def __init__(
+ self,
+ *,
+ config: dict,
+ aws_conn_id: str = DEFAULT_CONN_ID,
+ **kwargs,
+ ):
super().__init__(config=config, **kwargs)
self.config = config
+ self.aws_conn_id = aws_conn_id
def execute(self, context: 'Context') -> dict:
self.preprocess_config()
@@ -213,9 +234,16 @@ def execute(self, context: 'Context') -> dict:
class SageMakerEndpointOperator(SageMakerBaseOperator):
"""
- Create a SageMaker endpoint.
+ When you create a serverless endpoint, SageMaker provisions and manages
+ the compute resources for you. Then, you can make inference requests to
+ the endpoint and receive model predictions in response. SageMaker scales
+ the compute resources up and down as needed to handle your request traffic.
- This operator returns The ARN of the endpoint created in Amazon SageMaker
+ Requires an Endpoint Config.
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:SageMakerEndpointOperator`
:param config:
The configuration necessary to create an endpoint.
@@ -242,27 +270,30 @@ class SageMakerEndpointOperator(SageMakerBaseOperator):
For details of the configuration parameter of endpoint_configuration see
:py:meth:`SageMaker.Client.create_endpoint`
- :param aws_conn_id: The AWS connection ID to use.
:param wait_for_completion: Whether the operator should wait until the endpoint creation finishes.
:param check_interval: If wait is set to True, this is the time interval, in seconds, that this operation
waits before polling the status of the endpoint creation.
:param max_ingestion_time: If wait is set to True, this operation fails if the endpoint creation doesn't
finish within max_ingestion_time seconds. If you set this parameter to None it never times out.
:param operation: Whether to create an endpoint or update an endpoint. Must be either 'create or 'update'.
+ :param aws_conn_id: The AWS connection ID to use.
+ :return Dict: Returns The ARN of the endpoint created in Amazon SageMaker.
"""
def __init__(
self,
*,
config: dict,
+ aws_conn_id: str = DEFAULT_CONN_ID,
wait_for_completion: bool = True,
- check_interval: int = 30,
+ check_interval: int = CHECK_INTERVAL_SECOND,
max_ingestion_time: Optional[int] = None,
operation: str = 'create',
**kwargs,
):
super().__init__(config=config, **kwargs)
self.config = config
+ self.aws_conn_id = aws_conn_id
self.wait_for_completion = wait_for_completion
self.check_interval = check_interval
self.max_ingestion_time = max_ingestion_time
@@ -331,9 +362,13 @@ def execute(self, context: 'Context') -> dict:
class SageMakerTransformOperator(SageMakerBaseOperator):
- """Initiate a SageMaker transform job.
+ """
+ Starts a transform job. A transform job uses a trained model to get inferences
+ on a dataset and saves these results to an Amazon S3 location that you specify.
- This operator returns The ARN of the model created in Amazon SageMaker.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:SageMakerTransformOperator`
:param config: The configuration necessary to start a transform job (templated).
@@ -361,19 +396,22 @@ class SageMakerTransformOperator(SageMakerBaseOperator):
:param max_ingestion_time: If wait is set to True, the operation fails
if the transform job doesn't finish within max_ingestion_time seconds. If you
set this parameter to None, the operation does not timeout.
+ :return Dict: Returns The ARN of the model created in Amazon SageMaker.
"""
def __init__(
self,
*,
config: dict,
+ aws_conn_id: str = DEFAULT_CONN_ID,
wait_for_completion: bool = True,
- check_interval: int = 30,
+ check_interval: int = CHECK_INTERVAL_SECOND,
max_ingestion_time: Optional[int] = None,
**kwargs,
):
super().__init__(config=config, **kwargs)
self.config = config
+ self.aws_conn_id = aws_conn_id
self.wait_for_completion = wait_for_completion
self.check_interval = check_interval
self.max_ingestion_time = max_ingestion_time
@@ -422,9 +460,16 @@ def execute(self, context: 'Context') -> dict:
class SageMakerTuningOperator(SageMakerBaseOperator):
- """Initiate a SageMaker hyperparameter tuning job.
+ """
+ Starts a hyperparameter tuning job. A hyperparameter tuning job finds the
+ best version of a model by running many training jobs on your dataset using
+ the algorithm you choose and values for hyperparameters within ranges that
+ you specify. It then chooses the hyperparameter values that result in a model
+ that performs the best, as measured by an objective metric that you choose.
- This operator returns The ARN of the tuning job created in Amazon SageMaker.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:SageMakerTuningOperator`
:param config: The configuration necessary to start a tuning job (templated).
@@ -437,6 +482,7 @@ class SageMakerTuningOperator(SageMakerBaseOperator):
:param max_ingestion_time: If wait is set to True, the operation fails
if the tuning job doesn't finish within max_ingestion_time seconds. If you
set this parameter to None, the operation does not timeout.
+ :return Dict: Returns The ARN of the tuning job created in Amazon SageMaker.
"""
integer_fields = [
@@ -451,13 +497,15 @@ def __init__(
self,
*,
config: dict,
+ aws_conn_id: str = DEFAULT_CONN_ID,
wait_for_completion: bool = True,
- check_interval: int = 30,
+ check_interval: int = CHECK_INTERVAL_SECOND,
max_ingestion_time: Optional[int] = None,
**kwargs,
):
super().__init__(config=config, **kwargs)
self.config = config
+ self.aws_conn_id = aws_conn_id
self.wait_for_completion = wait_for_completion
self.check_interval = check_interval
self.max_ingestion_time = max_ingestion_time
@@ -487,19 +535,27 @@ def execute(self, context: 'Context') -> dict:
class SageMakerModelOperator(SageMakerBaseOperator):
- """Create a SageMaker model.
+ """
+ Creates a model in Amazon SageMaker. In the request, you name the model and
+ describe a primary container. For the primary container, you specify the Docker
+ image that contains inference code, artifacts (from prior training), and a custom
+ environment map that the inference code uses when you deploy the model for predictions.
- This operator returns The ARN of the model created in Amazon SageMaker
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:SageMakerModelOperator`
:param config: The configuration necessary to create a model.
For details of the configuration parameter see :py:meth:`SageMaker.Client.create_model`
:param aws_conn_id: The AWS connection ID to use.
+ :return Dict: Returns The ARN of the model created in Amazon SageMaker.
"""
- def __init__(self, *, config, **kwargs):
+ def __init__(self, *, config, aws_conn_id: str = DEFAULT_CONN_ID, **kwargs):
super().__init__(config=config, **kwargs)
self.config = config
+ self.aws_conn_id = aws_conn_id
def expand_role(self) -> None:
if 'ExecutionRoleArn' in self.config:
@@ -518,9 +574,12 @@ def execute(self, context: 'Context') -> dict:
class SageMakerTrainingOperator(SageMakerBaseOperator):
"""
- Initiate a SageMaker training job.
+ Starts a model training job. After training completes, Amazon SageMaker saves
+ the resulting model artifacts to an Amazon S3 location that you specify.
- This operator returns The ARN of the training job created in Amazon SageMaker.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:SageMakerTrainingOperator`
:param config: The configuration necessary to start a training job (templated).
@@ -539,6 +598,7 @@ class SageMakerTrainingOperator(SageMakerBaseOperator):
:param action_if_job_exists: Behaviour if the job name already exists. Possible options are "increment"
(default) and "fail".
This is only relevant if check_if
+ :return Dict: Returns The ARN of the training job created in Amazon SageMaker.
"""
integer_fields = [
@@ -551,15 +611,17 @@ def __init__(
self,
*,
config: dict,
+ aws_conn_id: str = DEFAULT_CONN_ID,
wait_for_completion: bool = True,
print_log: bool = True,
- check_interval: int = 30,
+ check_interval: int = CHECK_INTERVAL_SECOND,
max_ingestion_time: Optional[int] = None,
check_if_job_exists: bool = True,
action_if_job_exists: str = 'increment',
**kwargs,
):
super().__init__(config=config, **kwargs)
+ self.aws_conn_id = aws_conn_id
self.wait_for_completion = wait_for_completion
self.print_log = print_log
self.check_interval = check_interval
@@ -611,22 +673,24 @@ def _check_if_job_exists(self) -> None:
class SageMakerDeleteModelOperator(SageMakerBaseOperator):
- """Deletes a SageMaker model.
+ """
+ Deletes a SageMaker model.
- This operator deletes the Model entry created in SageMaker.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:SageMakerDeleteModelOperator`
:param config: The configuration necessary to delete the model.
-
For details of the configuration parameter see :py:meth:`SageMaker.Client.delete_model`
:param aws_conn_id: The AWS connection ID to use.
"""
- def __init__(self, *, config, aws_conn_id: str, **kwargs):
+ def __init__(self, *, config, aws_conn_id: str = DEFAULT_CONN_ID, **kwargs):
super().__init__(config=config, **kwargs)
- self.aws_conn_id = aws_conn_id
self.config = config
+ self.aws_conn_id = aws_conn_id
def execute(self, context: 'Context') -> Any:
sagemaker_hook = SageMakerHook(aws_conn_id=self.aws_conn_id)
sagemaker_hook.delete_model(model_name=self.config['ModelName'])
- self.log.info(f"Model {self.config['ModelName']} deleted Successfully.")
+ self.log.info("Model %s deleted successfully.", self.config['ModelName'])
diff --git a/airflow/providers/amazon/aws/operators/sqs.py b/airflow/providers/amazon/aws/operators/sqs.py
index 6bc2f8da09704..6eff54134a75d 100644
--- a/airflow/providers/amazon/aws/operators/sqs.py
+++ b/airflow/providers/amazon/aws/operators/sqs.py
@@ -28,7 +28,7 @@
class SqsPublishOperator(BaseOperator):
"""
- Publish message to a SQS queue.
+ Publish a message to an Amazon SQS queue.
.. seealso::
For more information on how to use this operator, take a look at the guide:
@@ -65,7 +65,7 @@ def __init__(
def execute(self, context: 'Context'):
"""
- Publish the message to SQS queue
+ Publish the message to the Amazon SQS queue
:param context: the context object
:return: dict with information about the message sent
@@ -81,7 +81,7 @@ def execute(self, context: 'Context'):
message_attributes=self.message_attributes,
)
- self.log.info('result is send_message is %s', result)
+ self.log.info('send_message result: %s', result)
return result
diff --git a/airflow/providers/amazon/aws/operators/step_function.py b/airflow/providers/amazon/aws/operators/step_function.py
index b800ea90d34b8..7c32b33890117 100644
--- a/airflow/providers/amazon/aws/operators/step_function.py
+++ b/airflow/providers/amazon/aws/operators/step_function.py
@@ -29,12 +29,13 @@
class StepFunctionStartExecutionOperator(BaseOperator):
"""
- An Operator that begins execution of an Step Function State Machine
+ An Operator that begins execution of an AWS Step Function State Machine.
Additional arguments may be specified and are passed down to the underlying BaseOperator.
.. seealso::
- :class:`~airflow.models.BaseOperator`
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:StepFunctionStartExecutionOperator`
:param state_machine_arn: ARN of the Step Function State Machine
:param name: The name of the execution.
@@ -79,12 +80,13 @@ def execute(self, context: 'Context'):
class StepFunctionGetExecutionOutputOperator(BaseOperator):
"""
- An Operator that begins execution of an Step Function State Machine
+ An Operator that returns the output of an AWS Step Function State Machine execution.
Additional arguments may be specified and are passed down to the underlying BaseOperator.
.. seealso::
- :class:`~airflow.models.BaseOperator`
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:StepFunctionGetExecutionOutputOperator`
:param execution_arn: ARN of the Step Function State Machine Execution
:param aws_conn_id: aws connection to use, defaults to 'aws_default'
diff --git a/airflow/providers/amazon/aws/secrets/secrets_manager.py b/airflow/providers/amazon/aws/secrets/secrets_manager.py
index 262abac781d0f..8b72f955ac6d5 100644
--- a/airflow/providers/amazon/aws/secrets/secrets_manager.py
+++ b/airflow/providers/amazon/aws/secrets/secrets_manager.py
@@ -19,12 +19,16 @@
import ast
import json
+import re
import sys
+import warnings
from typing import Optional
from urllib.parse import urlencode
import boto3
+from airflow.version import version as airflow_version
+
if sys.version_info >= (3, 8):
from functools import cached_property
else:
@@ -34,6 +38,11 @@
from airflow.utils.log.logging_mixin import LoggingMixin
+def _parse_version(val):
+ val = re.sub(r'(\d+\.\d+\.\d+).*', lambda x: x.group(1), val)
+ return tuple(int(x) for x in val.split('.'))
+
+
class SecretsManagerBackend(BaseSecretsBackend, LoggingMixin):
"""
Retrieves Connection or Variables from AWS Secrets Manager
@@ -125,7 +134,7 @@ def __init__(
self.profile_name = profile_name
self.sep = sep
self.full_url_mode = full_url_mode
- self.extra_conn_words = extra_conn_words if extra_conn_words else {}
+ self.extra_conn_words = extra_conn_words or {}
self.kwargs = kwargs
@cached_property
@@ -169,13 +178,11 @@ def get_uri_from_secret(self, secret):
conn_string = "{conn_type}://{user}:{password}@{host}:{port}/{schema}".format(**conn_d)
- connection = self._format_uri_with_extra(secret, conn_string)
-
- return connection
+ return self._format_uri_with_extra(secret, conn_string)
- def get_conn_uri(self, conn_id: str):
+ def get_conn_value(self, conn_id: str):
"""
- Get Connection Value
+ Get serialized representation of Connection
:param conn_id: connection id
"""
@@ -184,20 +191,37 @@ def get_conn_uri(self, conn_id: str):
if self.full_url_mode:
return self._get_secret(self.connections_prefix, conn_id)
- else:
- try:
- secret_string = self._get_secret(self.connections_prefix, conn_id)
- # json.loads gives error
- secret = ast.literal_eval(secret_string) if secret_string else None
- except ValueError: # 'malformed node or string: ' error, for empty conns
- connection = None
- secret = None
-
- # These lines will check if we have with some denomination stored an username, password and host
- if secret:
- connection = self.get_uri_from_secret(secret)
-
- return connection
+ try:
+ secret_string = self._get_secret(self.connections_prefix, conn_id)
+ # json.loads gives error
+ secret = ast.literal_eval(secret_string) if secret_string else None
+ except ValueError: # 'malformed node or string: ' error, for empty conns
+ connection = None
+ secret = None
+
+ # These lines will check if we have with some denomination stored an username, password and host
+ if secret:
+ connection = self.get_uri_from_secret(secret)
+
+ return connection
+
+ def get_conn_uri(self, conn_id: str) -> Optional[str]:
+ """
+ Return URI representation of Connection conn_id.
+
+ As of Airflow version 2.3.0 this method is deprecated.
+
+ :param conn_id: the connection id
+ :return: deserialized Connection
+ """
+ if _parse_version(airflow_version) >= (2, 3):
+ warnings.warn(
+ f"Method `{self.__class__.__name__}.get_conn_uri` is deprecated and will be removed "
+ "in a future release. Please use method `get_conn_value` instead.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ return self.get_conn_value(conn_id)
def get_variable(self, key: str) -> Optional[str]:
"""
diff --git a/airflow/providers/amazon/aws/secrets/systems_manager.py b/airflow/providers/amazon/aws/secrets/systems_manager.py
index 85b06330f0d0a..e45a5500ab003 100644
--- a/airflow/providers/amazon/aws/secrets/systems_manager.py
+++ b/airflow/providers/amazon/aws/secrets/systems_manager.py
@@ -16,11 +16,15 @@
# specific language governing permissions and limitations
# under the License.
"""Objects relating to sourcing connections from AWS SSM Parameter Store"""
+import re
import sys
+import warnings
from typing import Optional
import boto3
+from airflow.version import version as airflow_version
+
if sys.version_info >= (3, 8):
from functools import cached_property
else:
@@ -30,6 +34,11 @@
from airflow.utils.log.logging_mixin import LoggingMixin
+def _parse_version(val):
+ val = re.sub(r'(\d+\.\d+\.\d+).*', lambda x: x.group(1), val)
+ return tuple(int(x) for x in val.split('.'))
+
+
class SystemsManagerParameterStoreBackend(BaseSecretsBackend, LoggingMixin):
"""
Retrieves Connection or Variables from AWS SSM Parameter Store
@@ -86,7 +95,7 @@ def client(self):
session = boto3.Session(profile_name=self.profile_name)
return session.client("ssm", **self.kwargs)
- def get_conn_uri(self, conn_id: str) -> Optional[str]:
+ def get_conn_value(self, conn_id: str) -> Optional[str]:
"""
Get param value
@@ -97,6 +106,24 @@ def get_conn_uri(self, conn_id: str) -> Optional[str]:
return self._get_secret(self.connections_prefix, conn_id)
+ def get_conn_uri(self, conn_id: str) -> Optional[str]:
+ """
+ Return URI representation of Connection conn_id.
+
+ As of Airflow version 2.3.0 this method is deprecated.
+
+ :param conn_id: the connection id
+ :return: deserialized Connection
+ """
+ if _parse_version(airflow_version) >= (2, 3):
+ warnings.warn(
+ f"Method `{self.__class__.__name__}.get_conn_uri` is deprecated and will be removed "
+ "in a future release. Please use method `get_conn_value` instead.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ return self.get_conn_value(conn_id)
+
def get_variable(self, key: str) -> Optional[str]:
"""
Get Airflow Variable from Environment Variable
@@ -131,8 +158,7 @@ def _get_secret(self, path_prefix: str, secret_id: str) -> Optional[str]:
ssm_path = self.build_path(path_prefix, secret_id)
try:
response = self.client.get_parameter(Name=ssm_path, WithDecryption=True)
- value = response["Parameter"]["Value"]
- return value
+ return response["Parameter"]["Value"]
except self.client.exceptions.ParameterNotFound:
self.log.debug("Parameter %s not found.", ssm_path)
return None
diff --git a/airflow/providers/amazon/aws/sensors/athena.py b/airflow/providers/amazon/aws/sensors/athena.py
index 64e9279b4b225..927f512143362 100644
--- a/airflow/providers/amazon/aws/sensors/athena.py
+++ b/airflow/providers/amazon/aws/sensors/athena.py
@@ -36,6 +36,11 @@ class AthenaSensor(BaseSensorOperator):
Asks for the state of the Query until it reaches a failure state or success state.
If the query fails, the task will fail.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/sensor:AthenaSensor`
+
+
:param query_execution_id: query_execution_id to check the state of
:param max_retries: Number of times to poll for query state before
returning the current state, defaults to None
diff --git a/airflow/providers/amazon/aws/sensors/cloud_formation.py b/airflow/providers/amazon/aws/sensors/cloud_formation.py
index bf78b3bcfdd07..fb01bdc7f6827 100644
--- a/airflow/providers/amazon/aws/sensors/cloud_formation.py
+++ b/airflow/providers/amazon/aws/sensors/cloud_formation.py
@@ -35,6 +35,11 @@ class CloudFormationCreateStackSensor(BaseSensorOperator):
"""
Waits for a stack to be created successfully on AWS CloudFormation.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/sensor:CloudFormationCreateStackSensor`
+
+
:param stack_name: The name of the stack to wait for (templated)
:param aws_conn_id: ID of the Airflow connection where credentials and extra configuration are
stored
@@ -68,6 +73,10 @@ class CloudFormationDeleteStackSensor(BaseSensorOperator):
"""
Waits for a stack to be deleted successfully on AWS CloudFormation.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/sensor:CloudFormationDeleteStackSensor`
+
:param stack_name: The name of the stack to wait for (templated)
:param aws_conn_id: ID of the Airflow connection where credentials and extra configuration are
stored
diff --git a/airflow/providers/amazon/aws/sensors/ec2.py b/airflow/providers/amazon/aws/sensors/ec2.py
index f955f713f1ab4..7d4a640593fec 100644
--- a/airflow/providers/amazon/aws/sensors/ec2.py
+++ b/airflow/providers/amazon/aws/sensors/ec2.py
@@ -31,6 +31,10 @@ class EC2InstanceStateSensor(BaseSensorOperator):
Check the state of the AWS EC2 instance until
state of the instance become equal to the target state.
+ .. seealso::
+ For more information on how to use this sensor, take a look at the guide:
+ :ref:`howto/sensor:EC2InstanceStateSensor`
+
:param target_state: target state of instance
:param instance_id: id of the AWS EC2 instance
:param region_name: (optional) aws region name associated with the client
diff --git a/airflow/providers/amazon/aws/sensors/emr.py b/airflow/providers/amazon/aws/sensors/emr.py
index 2204efcfaed50..a4c2b3a71142b 100644
--- a/airflow/providers/amazon/aws/sensors/emr.py
+++ b/airflow/providers/amazon/aws/sensors/emr.py
@@ -66,7 +66,7 @@ def get_hook(self) -> EmrHook:
def poke(self, context: 'Context'):
response = self.get_emr_response()
- if not response['ResponseMetadata']['HTTPStatusCode'] == 200:
+ if response['ResponseMetadata']['HTTPStatusCode'] != 200:
self.log.info('Bad HTTP response: %s', response)
return False
@@ -258,8 +258,9 @@ def failure_message_from_response(response: Dict[str, Any]) -> Optional[str]:
cluster_status = response['Cluster']['Status']
state_change_reason = cluster_status.get('StateChangeReason')
if state_change_reason:
- return 'for code: {} with message {}'.format(
- state_change_reason.get('Code', 'No code'), state_change_reason.get('Message', 'Unknown')
+ return (
+ f"for code: {state_change_reason.get('Code', 'No code')} "
+ f"with message {state_change_reason.get('Message', 'Unknown')}"
)
return None
@@ -338,7 +339,8 @@ def failure_message_from_response(response: Dict[str, Any]) -> Optional[str]:
"""
fail_details = response['Step']['Status'].get('FailureDetails')
if fail_details:
- return 'for reason {} with message {} and log file {}'.format(
- fail_details.get('Reason'), fail_details.get('Message'), fail_details.get('LogFile')
+ return (
+ f"for reason {fail_details.get('Reason')} "
+ f"with message {fail_details.get('Message')} and log file {fail_details.get('LogFile')}"
)
return None
diff --git a/airflow/providers/amazon/aws/sensors/glacier.py b/airflow/providers/amazon/aws/sensors/glacier.py
index dec97f7293a79..e92f5a4326b3a 100644
--- a/airflow/providers/amazon/aws/sensors/glacier.py
+++ b/airflow/providers/amazon/aws/sensors/glacier.py
@@ -39,7 +39,7 @@ class GlacierJobOperationSensor(BaseSensorOperator):
.. seealso::
For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:GlacierJobOperationSensor`
+ :ref:`howto/sensor:GlacierJobOperationSensor`
:param aws_conn_id: The reference to the AWS connection details
:param vault_name: name of Glacier vault on which job is executed
diff --git a/airflow/providers/amazon/aws/sensors/glue.py b/airflow/providers/amazon/aws/sensors/glue.py
index 149c77cf52a16..525e7b8ee6234 100644
--- a/airflow/providers/amazon/aws/sensors/glue.py
+++ b/airflow/providers/amazon/aws/sensors/glue.py
@@ -31,6 +31,10 @@ class GlueJobSensor(BaseSensorOperator):
Waits for an AWS Glue Job to reach any of the status below
'FAILED', 'STOPPED', 'SUCCEEDED'
+ .. seealso::
+ For more information on how to use this sensor, take a look at the guide:
+ :ref:`howto/sensor:GlueJobSensor`
+
:param job_name: The AWS Glue Job unique name
:param run_id: The AWS Glue current running job identifier
"""
@@ -53,7 +57,7 @@ def poke(self, context: 'Context'):
self.log.info("Exiting Job %s Run State: %s", self.run_id, job_state)
return True
elif job_state in self.errored_states:
- job_error_message = "Exiting Job " + self.run_id + " Run State: " + job_state
+ job_error_message = f"Exiting Job {self.run_id} Run State: {job_state}"
raise AirflowException(job_error_message)
else:
return False
diff --git a/airflow/providers/amazon/aws/sensors/glue_crawler.py b/airflow/providers/amazon/aws/sensors/glue_crawler.py
index 10ff0a074f40d..52944ce2eb30e 100644
--- a/airflow/providers/amazon/aws/sensors/glue_crawler.py
+++ b/airflow/providers/amazon/aws/sensors/glue_crawler.py
@@ -31,6 +31,10 @@ class GlueCrawlerSensor(BaseSensorOperator):
Waits for an AWS Glue crawler to reach any of the statuses below
'FAILED', 'CANCELLED', 'SUCCEEDED'
+ .. seealso::
+ For more information on how to use this sensor, take a look at the guide:
+ :ref:`howto/sensor:GlueCrawlerSensor`
+
:param crawler_name: The AWS Glue crawler unique name
:param aws_conn_id: aws connection to use, defaults to 'aws_default'
"""
diff --git a/airflow/providers/amazon/aws/sensors/quicksight.py b/airflow/providers/amazon/aws/sensors/quicksight.py
new file mode 100644
index 0000000000000..da94980e80650
--- /dev/null
+++ b/airflow/providers/amazon/aws/sensors/quicksight.py
@@ -0,0 +1,96 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import sys
+from typing import TYPE_CHECKING, Optional
+
+from airflow.exceptions import AirflowException
+from airflow.providers.amazon.aws.hooks.quicksight import QuickSightHook
+from airflow.providers.amazon.aws.hooks.sts import StsHook
+from airflow.sensors.base import BaseSensorOperator
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+if sys.version_info >= (3, 8):
+ from functools import cached_property
+else:
+ from cached_property import cached_property
+
+
+class QuickSightSensor(BaseSensorOperator):
+ """
+ Watches for the status of an Amazon QuickSight Ingestion.
+
+ .. seealso::
+ For more information on how to use this sensor, take a look at the guide:
+ :ref:`howto/sensor:QuickSightSensor`
+
+ :param data_set_id: ID of the dataset used in the ingestion.
+ :param ingestion_id: ID for the ingestion.
+ :param aws_conn_id: The Airflow connection used for AWS credentials. (templated)
+ If this is None or empty then the default boto3 behaviour is used. If
+ running Airflow in a distributed manner and aws_conn_id is None or
+ empty, then the default boto3 configuration would be used (and must be
+ maintained on each worker node).
+ """
+
+ def __init__(
+ self,
+ *,
+ data_set_id: str,
+ ingestion_id: str,
+ aws_conn_id: str = "aws_default",
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.data_set_id = data_set_id
+ self.ingestion_id = ingestion_id
+ self.aws_conn_id = aws_conn_id
+ self.success_status = "COMPLETED"
+ self.errored_statuses = ("FAILED", "CANCELLED")
+ self.quicksight_hook: Optional[QuickSightHook] = None
+ self.sts_hook: Optional[StsHook] = None
+
+ def poke(self, context: "Context"):
+ """
+ Pokes until the QuickSight Ingestion has successfully finished.
+
+ :param context: The task context during execution.
+ :return: True if it COMPLETED and False if not.
+ :rtype: bool
+ """
+ quicksight_hook = self.get_quicksight_hook
+ sts_hook = self.get_sts_hook
+ self.log.info("Poking for Amazon QuickSight Ingestion ID: %s", self.ingestion_id)
+ aws_account_id = sts_hook.get_account_number()
+ quicksight_ingestion_state = quicksight_hook.get_status(
+ aws_account_id, self.data_set_id, self.ingestion_id
+ )
+ self.log.info("QuickSight Status: %s", quicksight_ingestion_state)
+ if quicksight_ingestion_state in self.errored_statuses:
+ raise AirflowException("The QuickSight Ingestion failed!")
+ return quicksight_ingestion_state == self.success_status
+
+ @cached_property
+ def get_quicksight_hook(self):
+ return QuickSightHook(aws_conn_id=self.aws_conn_id)
+
+ @cached_property
+ def get_sts_hook(self):
+ return StsHook(aws_conn_id=self.aws_conn_id)
diff --git a/airflow/providers/amazon/aws/sensors/rds.py b/airflow/providers/amazon/aws/sensors/rds.py
index 1c74d5ae8fc14..3c24c82fbf940 100644
--- a/airflow/providers/amazon/aws/sensors/rds.py
+++ b/airflow/providers/amazon/aws/sensors/rds.py
@@ -71,7 +71,7 @@ class RdsSnapshotExistenceSensor(RdsBaseSensor):
.. seealso::
For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:RdsSnapshotExistenceSensor`
+ :ref:`howto/sensor:RdsSnapshotExistenceSensor`
:param db_type: Type of the DB - either "instance" or "cluster"
:param db_snapshot_identifier: The identifier for the DB snapshot
@@ -113,7 +113,7 @@ class RdsExportTaskExistenceSensor(RdsBaseSensor):
.. seealso::
For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:RdsExportTaskExistenceSensor`
+ :ref:`howto/sensor:RdsExportTaskExistenceSensor`
:param export_task_identifier: A unique identifier for the snapshot export task.
:param target_statuses: Target status of export task
diff --git a/airflow/providers/amazon/aws/sensors/s3.py b/airflow/providers/amazon/aws/sensors/s3.py
index 2bda06639f5dc..182b05864cf1d 100644
--- a/airflow/providers/amazon/aws/sensors/s3.py
+++ b/airflow/providers/amazon/aws/sensors/s3.py
@@ -20,9 +20,9 @@
import os
import re
import sys
+import warnings
from datetime import datetime
from typing import TYPE_CHECKING, Callable, List, Optional, Sequence, Set, Union
-from urllib.parse import urlparse
if TYPE_CHECKING:
from airflow.utils.context import Context
@@ -40,17 +40,30 @@
class S3KeySensor(BaseSensorOperator):
"""
- Waits for a key (a file-like instance on S3) to be present in a S3 bucket.
+ Waits for one or multiple keys (a file-like instance on S3) to be present in a S3 bucket.
S3 being a key/value it does not support folders. The path is just a key
a resource.
- :param bucket_key: The key being waited on. Supports full s3:// style url
+ .. seealso::
+ For more information on how to use this sensor, take a look at the guide:
+ :ref:`howto/sensor:S3KeySensor`
+
+ :param bucket_key: The key(s) being waited on. Supports full s3:// style url
or relative path from root level. When it's specified as a full s3://
- url, please leave bucket_name as `None`.
+ url, please leave bucket_name as `None`
:param bucket_name: Name of the S3 bucket. Only needed when ``bucket_key``
- is not provided as a full s3:// url.
+ is not provided as a full s3:// url. When specified, all the keys passed to ``bucket_key``
+ refers to this bucket
:param wildcard_match: whether the bucket_key should be interpreted as a
Unix wildcard pattern
+ :param check_fn: Function that receives the list of the S3 objects,
+ and returns a boolean:
+ - ``True``: the criteria is met
+ - ``False``: the criteria isn't met
+ **Example**: Wait for any S3 object size more than 1 megabyte ::
+
+ def check_fn(files: List) -> bool:
+ return any(f.get('Size', 0) > 1048576 for f in files)
:param aws_conn_id: a reference to the s3 connection
:param verify: Whether or not to verify SSL certificates for S3 connection.
By default SSL certificates are verified.
@@ -69,36 +82,55 @@ class S3KeySensor(BaseSensorOperator):
def __init__(
self,
*,
- bucket_key: str,
+ bucket_key: Union[str, List[str]],
bucket_name: Optional[str] = None,
wildcard_match: bool = False,
+ check_fn: Optional[Callable[..., bool]] = None,
aws_conn_id: str = 'aws_default',
verify: Optional[Union[str, bool]] = None,
**kwargs,
):
super().__init__(**kwargs)
self.bucket_name = bucket_name
- self.bucket_key = bucket_key
+ self.bucket_key = [bucket_key] if isinstance(bucket_key, str) else bucket_key
self.wildcard_match = wildcard_match
+ self.check_fn = check_fn
self.aws_conn_id = aws_conn_id
self.verify = verify
self.hook: Optional[S3Hook] = None
- def _resolve_bucket_and_key(self):
- """If key is URI, parse bucket"""
- if self.bucket_name is None:
- self.bucket_name, self.bucket_key = S3Hook.parse_s3_url(self.bucket_key)
+ def _check_key(self, key):
+ bucket_name, key = S3Hook.get_s3_bucket_key(self.bucket_name, key, 'bucket_name', 'bucket_key')
+ self.log.info('Poking for key : s3://%s/%s', bucket_name, key)
+
+ """
+ Set variable `files` which contains a list of dict which contains only the size
+ If needed we might want to add other attributes later
+ Format: [{
+ 'Size': int
+ }]
+ """
+ if self.wildcard_match:
+ prefix = re.split(r'[\[\*\?]', key, 1)[0]
+ files = self.get_hook().get_file_metadata(prefix, bucket_name)
+ if len(files) == 0:
+ return False
+
+ # Reduce the set of metadata to size only
+ files = list(map(lambda f: {'Size': f['Size']}, files))
else:
- parsed_url = urlparse(self.bucket_key)
- if parsed_url.scheme != '' or parsed_url.netloc != '':
- raise AirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')
+ obj = self.get_hook().head_object(key, bucket_name)
+ if obj is None:
+ return False
+ files = [{'Size': obj['ContentLength']}]
+
+ if self.check_fn is not None:
+ return self.check_fn(files)
+
+ return True
def poke(self, context: 'Context'):
- self._resolve_bucket_and_key()
- self.log.info('Poking for key : s3://%s/%s', self.bucket_name, self.bucket_key)
- if self.wildcard_match:
- return self.get_hook().check_for_wildcard_key(self.bucket_key, self.bucket_name)
- return self.get_hook().check_for_key(self.bucket_key, self.bucket_name)
+ return all(self._check_key(key) for key in self.bucket_key)
def get_hook(self) -> S3Hook:
"""Create and return an S3Hook"""
@@ -111,37 +143,8 @@ def get_hook(self) -> S3Hook:
class S3KeySizeSensor(S3KeySensor):
"""
- Waits for a key (a file-like instance on S3) to be present and be more than
- some size in a S3 bucket.
- S3 being a key/value it does not support folders. The path is just a key
- a resource.
-
- :param bucket_key: The key being waited on. Supports full s3:// style url
- or relative path from root level. When it's specified as a full s3://
- url, please leave bucket_name as `None`.
- :param bucket_name: Name of the S3 bucket. Only needed when ``bucket_key``
- is not provided as a full s3:// url.
- :param wildcard_match: whether the bucket_key should be interpreted as a
- Unix wildcard pattern
- :param aws_conn_id: a reference to the s3 connection
- :param verify: Whether or not to verify SSL certificates for S3 connection.
- By default SSL certificates are verified.
- You can provide the following values:
-
- - ``False``: do not validate SSL certificates. SSL will still be used
- (unless use_ssl is False), but SSL certificates will not be
- verified.
- - ``path/to/cert/bundle.pem``: A filename of the CA cert bundle to uses.
- You can specify this argument if you want to use a different
- CA cert bundle than the one used by botocore.
- :param check_fn: Function that receives the list of the S3 objects,
- and returns the boolean:
- - ``True``: a certain criteria is met
- - ``False``: the criteria isn't met
- **Example**: Wait for any S3 object size more than 1 megabyte ::
-
- def check_fn(self, data: List) -> bool:
- return any(f.get('Size', 0) > 1048576 for f in data if isinstance(f, dict))
+ This class is deprecated.
+ Please use :class:`~airflow.providers.amazon.aws.sensors.s3.S3KeySensor`.
"""
def __init__(
@@ -150,48 +153,26 @@ def __init__(
check_fn: Optional[Callable[..., bool]] = None,
**kwargs,
):
- super().__init__(**kwargs)
- self.check_fn_user = check_fn
-
- def poke(self, context: 'Context'):
- if super().poke(context=context) is False:
- return False
-
- s3_objects = self.get_files(s3_hook=self.get_hook())
- if not s3_objects:
- return False
- if self.check_fn_user is not None:
- return self.check_fn_user(s3_objects)
- return self.check_fn(s3_objects)
-
- def get_files(self, s3_hook: S3Hook, delimiter: Optional[str] = '/') -> List:
- """Gets a list of files in the bucket"""
- prefix = self.bucket_key
- config = {
- 'PageSize': None,
- 'MaxItems': None,
- }
- if self.wildcard_match:
- prefix = re.split(r'[\[\*\?]', self.bucket_key, 1)[0]
+ warnings.warn(
+ """
+ S3KeySizeSensor is deprecated.
+ Please use `airflow.providers.amazon.aws.sensors.s3.S3KeySensor`.
+ """,
+ DeprecationWarning,
+ stacklevel=2,
+ )
- paginator = s3_hook.get_conn().get_paginator('list_objects_v2')
- response = paginator.paginate(
- Bucket=self.bucket_name, Prefix=prefix, Delimiter=delimiter, PaginationConfig=config
+ super().__init__(
+ check_fn=check_fn if check_fn is not None else S3KeySizeSensor.default_check_fn, **kwargs
)
- keys: List = []
- for page in response:
- if 'Contents' in page:
- _temp = [k for k in page['Contents'] if isinstance(k.get('Size', None), (int, float))]
- keys = keys + _temp
- return keys
-
- def check_fn(self, data: List, object_min_size: Optional[Union[int, float]] = 0) -> bool:
+
+ @staticmethod
+ def default_check_fn(data: List) -> bool:
"""Default function for checking that S3 Objects have size more than 0
:param data: List of the objects in S3 bucket.
- :param object_min_size: Checks if the objects sizes are greater then this value.
"""
- return all(f.get('Size', 0) > object_min_size for f in data if isinstance(f, dict))
+ return all(f.get('Size', 0) > 0 for f in data)
@poke_mode_only
@@ -203,6 +184,10 @@ class S3KeysUnchangedSensor(BaseSensorOperator):
in reschedule mode, as the state of the listed objects in the S3 bucket will
be lost between rescheduled invocations.
+ .. seealso::
+ For more information on how to use this sensor, take a look at the guide:
+ :ref:`howto/sensor:S3KeysUnchangedSensor`
+
:param bucket_name: Name of the S3 bucket
:param prefix: The prefix being waited on. Relative path from bucket root level.
:param aws_conn_id: a reference to the s3 connection
@@ -332,30 +317,10 @@ def poke(self, context: 'Context'):
return self.is_keys_unchanged(set(self.hook.list_keys(self.bucket_name, prefix=self.prefix)))
-class S3PrefixSensor(BaseSensorOperator):
+class S3PrefixSensor(S3KeySensor):
"""
- Waits for a prefix or all prefixes to exist. A prefix is the first part of a key,
- thus enabling checking of constructs similar to glob ``airfl*`` or
- SQL LIKE ``'airfl%'``. There is the possibility to precise a delimiter to
- indicate the hierarchy or keys, meaning that the match will stop at that
- delimiter. Current code accepts sane delimiters, i.e. characters that
- are NOT special characters in the Python regex engine.
-
- :param bucket_name: Name of the S3 bucket
- :param prefix: The prefix being waited on. Relative path from bucket root level.
- :param delimiter: The delimiter intended to show hierarchy.
- Defaults to '/'.
- :param aws_conn_id: a reference to the s3 connection
- :param verify: Whether or not to verify SSL certificates for S3 connection.
- By default SSL certificates are verified.
- You can provide the following values:
-
- - ``False``: do not validate SSL certificates. SSL will still be used
- (unless use_ssl is False), but SSL certificates will not be
- verified.
- - ``path/to/cert/bundle.pem``: A filename of the CA cert bundle to uses.
- You can specify this argument if you want to use a different
- CA cert bundle than the one used by botocore.
+ This class is deprecated.
+ Please use :class:`~airflow.providers.amazon.aws.sensors.s3.S3KeySensor`.
"""
template_fields: Sequence[str] = ('prefix', 'bucket_name')
@@ -363,35 +328,21 @@ class S3PrefixSensor(BaseSensorOperator):
def __init__(
self,
*,
- bucket_name: str,
prefix: Union[str, List[str]],
delimiter: str = '/',
- aws_conn_id: str = 'aws_default',
- verify: Optional[Union[str, bool]] = None,
**kwargs,
):
- super().__init__(**kwargs)
- # Parse
- self.bucket_name = bucket_name
- self.prefix = [prefix] if isinstance(prefix, str) else prefix
- self.delimiter = delimiter
- self.aws_conn_id = aws_conn_id
- self.verify = verify
- self.hook: Optional[S3Hook] = None
-
- def poke(self, context: 'Context'):
- self.log.info('Poking for prefix : %s in bucket s3://%s', self.prefix, self.bucket_name)
- return all(self._check_for_prefix(prefix) for prefix in self.prefix)
-
- def get_hook(self) -> S3Hook:
- """Create and return an S3Hook"""
- if self.hook:
- return self.hook
+ warnings.warn(
+ """
+ S3PrefixSensor is deprecated.
+ Please use `airflow.providers.amazon.aws.sensors.s3.S3KeySensor`.
+ """,
+ DeprecationWarning,
+ stacklevel=2,
+ )
- self.hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
- return self.hook
+ self.prefix = prefix
+ prefixes = [self.prefix] if isinstance(self.prefix, str) else self.prefix
+ keys = [pref if pref.endswith(delimiter) else pref + delimiter for pref in prefixes]
- def _check_for_prefix(self, prefix: str) -> bool:
- return self.get_hook().check_for_prefix(
- prefix=prefix, delimiter=self.delimiter, bucket_name=self.bucket_name
- )
+ super().__init__(bucket_key=keys, **kwargs)
diff --git a/airflow/providers/amazon/aws/sensors/sagemaker.py b/airflow/providers/amazon/aws/sensors/sagemaker.py
index 054b139cc2eb9..925ddaed17fd9 100644
--- a/airflow/providers/amazon/aws/sensors/sagemaker.py
+++ b/airflow/providers/amazon/aws/sensors/sagemaker.py
@@ -27,10 +27,10 @@
class SageMakerBaseSensor(BaseSensorOperator):
- """Contains general sensor behavior for SageMaker.
+ """
+ Contains general sensor behavior for SageMaker.
- Subclasses should implement get_sagemaker_response()
- and state_from_response() methods.
+ Subclasses should implement get_sagemaker_response() and state_from_response() methods.
Subclasses should also implement NON_TERMINAL_STATES and FAILED_STATE methods.
"""
@@ -50,7 +50,7 @@ def get_hook(self) -> SageMakerHook:
def poke(self, context: 'Context'):
response = self.get_sagemaker_response()
- if not (response['ResponseMetadata']['HTTPStatusCode'] == 200):
+ if response['ResponseMetadata']['HTTPStatusCode'] != 200:
self.log.info('Bad HTTP response: %s', response)
return False
state = self.state_from_response(response)
@@ -84,13 +84,15 @@ def state_from_response(self, response: dict) -> str:
class SageMakerEndpointSensor(SageMakerBaseSensor):
- """Asks for the state of the endpoint state until it reaches a
- terminal state.
- If it fails the sensor errors, the task fails.
-
+ """
+ Polls the endpoint state until it reaches a terminal state. Raises an
+ AirflowException with the failure reason if a failed state is reached.
- :param job_name: job_name of the endpoint instance to check the state of
+ .. seealso::
+ For more information on how to use this sensor, take a look at the guide:
+ :ref:`howto/sensor:SageMakerEndpointSensor`
+ :param endpoint_name: Name of the endpoint instance to watch.
"""
template_fields: Sequence[str] = ('endpoint_name',)
@@ -118,15 +120,15 @@ def state_from_response(self, response):
class SageMakerTransformSensor(SageMakerBaseSensor):
- """Asks for the state of the transform state until it reaches a
- terminal state.
- The sensor will error if the job errors, throwing a
- AirflowException
- containing the failure reason.
+ """
+ Polls the transform job until it reaches a terminal state. Raises an
+ AirflowException with the failure reason if a failed state is reached.
- :param
- job_name: job_name of the transform job instance to check the state of
+ .. seealso::
+ For more information on how to use this sensor, take a look at the guide:
+ :ref:`howto/sensor:SageMakerTransformSensor`
+ :param job_name: Name of the transform job to watch.
"""
template_fields: Sequence[str] = ('job_name',)
@@ -154,16 +156,15 @@ def state_from_response(self, response):
class SageMakerTuningSensor(SageMakerBaseSensor):
- """Asks for the state of the tuning state until it reaches a terminal
- state.
- The sensor will error if the job errors, throwing a
- AirflowException
- containing the failure reason.
-
- :param
- job_name: job_name of the tuning instance to check the state of
- :type
- job_name: str
+ """
+ Asks for the state of the tuning state until it reaches a terminal state.
+ Raises an AirflowException with the failure reason if a failed state is reached.
+
+ .. seealso::
+ For more information on how to use this sensor, take a look at the guide:
+ :ref:`howto/sensor:SageMakerTuningSensor`
+
+ :param job_name: Name of the tuning instance to watch.
"""
template_fields: Sequence[str] = ('job_name',)
@@ -191,14 +192,16 @@ def state_from_response(self, response):
class SageMakerTrainingSensor(SageMakerBaseSensor):
- """Asks for the state of the training state until it reaches a
- terminal state.
- If it fails the sensor errors, failing the task.
-
+ """
+ Polls the training job until it reaches a terminal state. Raises an
+ AirflowException with the failure reason if a failed state is reached.
- :param job_name: name of the SageMaker training job to check the state of
+ .. seealso::
+ For more information on how to use this sensor, take a look at the guide:
+ :ref:`howto/sensor:SageMakerTrainingSensor`
- :param print_log: if the operator should print the cloudwatch log
+ :param job_name: Name of the training job to watch.
+ :param print_log: Prints the cloudwatch log if True; Defaults to True.
"""
template_fields: Sequence[str] = ('job_name',)
@@ -222,7 +225,7 @@ def init_log_resource(self, hook: SageMakerHook) -> None:
self.instance_count = description['ResourceConfig']['InstanceCount']
status = description['TrainingJobStatus']
job_already_completed = status not in self.non_terminal_states()
- self.state = LogState.TAILING if (not job_already_completed) else LogState.COMPLETE
+ self.state = LogState.COMPLETE if job_already_completed else LogState.TAILING
self.last_description = description
self.last_describe_job_call = time.monotonic()
self.log_resource_inited = True
diff --git a/airflow/providers/amazon/aws/sensors/sqs.py b/airflow/providers/amazon/aws/sensors/sqs.py
index 9c3b55ae39a9c..cc026ec5e8306 100644
--- a/airflow/providers/amazon/aws/sensors/sqs.py
+++ b/airflow/providers/amazon/aws/sensors/sqs.py
@@ -33,9 +33,13 @@
class SqsSensor(BaseSensorOperator):
"""
- Get messages from an SQS queue and then deletes the message from the SQS queue.
- If deletion of messages fails an AirflowException is thrown otherwise, the message
- is pushed through XCom with the key ``messages``.
+ Get messages from an Amazon SQS queue and then delete the messages from the queue.
+ If deletion of messages fails an AirflowException is thrown. Otherwise, the messages
+ are pushed through XCom with the key ``messages``.
+
+ .. seealso::
+ For more information on how to use this sensor, take a look at the guide:
+ :ref:`howto/sensor:SqsSensor`
:param aws_conn_id: AWS connection id
:param sqs_queue: The SQS queue url (templated)
@@ -128,21 +132,18 @@ def poke(self, context: 'Context'):
num_messages = len(messages)
self.log.info("Received %d messages", num_messages)
- if not num_messages:
- return False
-
- if self.message_filtering:
+ if num_messages and self.message_filtering:
messages = self.filter_messages(messages)
num_messages = len(messages)
self.log.info("There are %d messages left after filtering", num_messages)
+ if not num_messages:
+ return False
+
if not self.delete_message_on_reception:
context['ti'].xcom_push(key='messages', value=messages)
return True
- if not num_messages:
- return False
-
self.log.info("Deleting %d messages", num_messages)
entries = [
diff --git a/airflow/providers/amazon/aws/sensors/step_function.py b/airflow/providers/amazon/aws/sensors/step_function.py
index 3c170c072762f..6f82c0bc99d8c 100644
--- a/airflow/providers/amazon/aws/sensors/step_function.py
+++ b/airflow/providers/amazon/aws/sensors/step_function.py
@@ -28,13 +28,17 @@
class StepFunctionExecutionSensor(BaseSensorOperator):
"""
- Asks for the state of the Step Function State Machine Execution until it
+ Asks for the state of the AWS Step Function State Machine Execution until it
reaches a failure state or success state.
- If it fails, failing the task.
+ If it fails, then fail the task.
On successful completion of the Execution the Sensor will do an XCom Push
of the State Machine's output to `output`
+ .. seealso::
+ For more information on how to use this sensor, take a look at the guide:
+ :ref:`howto/sensor:StepFunctionExecutionSensor`
+
:param execution_arn: execution_arn to check the state of
:param aws_conn_id: aws connection to use, defaults to 'aws_default'
"""
diff --git a/airflow/providers/amazon/aws/transfers/ftp_to_s3.py b/airflow/providers/amazon/aws/transfers/ftp_to_s3.py
index fae7d5454a5bd..1426599bc4763 100644
--- a/airflow/providers/amazon/aws/transfers/ftp_to_s3.py
+++ b/airflow/providers/amazon/aws/transfers/ftp_to_s3.py
@@ -31,6 +31,10 @@ class FTPToS3Operator(BaseOperator):
This operator enables the transfer of files from a FTP server to S3. It can be used to
transfer one or multiple files.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:FTPToS3Operator`
+
:param ftp_path: The ftp remote path. For one file it is mandatory to include the file as well.
For multiple files, it is the route where the files will be found.
:param s3_bucket: The targeted s3 bucket in which to upload the file(s).
@@ -104,7 +108,7 @@ def __upload_to_s3_from_ftp(self, remote_filename, s3_file_key):
gzip=self.gzip,
acl_policy=self.acl_policy,
)
- self.log.info(f'File upload to {s3_file_key}')
+ self.log.info('File upload to %s', s3_file_key)
def execute(self, context: 'Context'):
self.ftp_hook = FTPHook(ftp_conn_id=self.ftp_conn_id)
@@ -112,7 +116,7 @@ def execute(self, context: 'Context'):
if self.ftp_filenames:
if isinstance(self.ftp_filenames, str):
- self.log.info(f'Getting files in {self.ftp_path}')
+ self.log.info('Getting files in %s', self.ftp_path)
list_dir = self.ftp_hook.list_directory(
path=self.ftp_path,
@@ -125,7 +129,7 @@ def execute(self, context: 'Context'):
files = list(filter(lambda f: ftp_filename in f, list_dir))
for file in files:
- self.log.info(f'Moving file {file}')
+ self.log.info('Moving file %s', file)
if self.s3_filenames and isinstance(self.s3_filenames, str):
filename = file.replace(self.ftp_filenames, self.s3_filenames)
diff --git a/airflow/providers/amazon/aws/transfers/gcs_to_s3.py b/airflow/providers/amazon/aws/transfers/gcs_to_s3.py
index 3dc8aa0b873ae..b521ce5360741 100644
--- a/airflow/providers/amazon/aws/transfers/gcs_to_s3.py
+++ b/airflow/providers/amazon/aws/transfers/gcs_to_s3.py
@@ -32,6 +32,10 @@ class GCSToS3Operator(BaseOperator):
"""
Synchronizes a Google Cloud Storage bucket with an S3 bucket.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:GCSToS3Operator`
+
:param bucket: The Google Cloud Storage bucket to find the objects. (templated)
:param prefix: Prefix string which filters objects whose name begin with
this prefix. (templated)
diff --git a/airflow/providers/amazon/aws/transfers/google_api_to_s3.py b/airflow/providers/amazon/aws/transfers/google_api_to_s3.py
index dd794c348b5bf..f3e10b62b2aa0 100644
--- a/airflow/providers/amazon/aws/transfers/google_api_to_s3.py
+++ b/airflow/providers/amazon/aws/transfers/google_api_to_s3.py
@@ -43,6 +43,10 @@ class GoogleApiToS3Operator(BaseOperator):
Therefore it is recommended that you use the custom Google Cloud Service Operators for working
with the Google Cloud Platform.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:GoogleApiToS3Operator`
+
:param google_api_service_name: The specific API service that is being requested.
:param google_api_service_version: The version of the API that is being requested.
:param google_api_endpoint_path: The client libraries path to the api call's executing method.
@@ -151,13 +155,12 @@ def _retrieve_data_from_google_api(self) -> dict:
api_version=self.google_api_service_version,
impersonation_chain=self.google_impersonation_chain,
)
- google_api_response = google_discovery_api_hook.query(
+ return google_discovery_api_hook.query(
endpoint=self.google_api_endpoint_path,
data=self.google_api_endpoint_params,
paginate=self.google_api_pagination,
num_retries=self.google_api_num_retries,
)
- return google_api_response
def _load_data_to_s3(self, data: dict) -> None:
s3_hook = S3Hook(aws_conn_id=self.aws_conn_id)
diff --git a/airflow/providers/amazon/aws/transfers/hive_to_dynamodb.py b/airflow/providers/amazon/aws/transfers/hive_to_dynamodb.py
index d6aa71e944fa0..652eca22b86df 100644
--- a/airflow/providers/amazon/aws/transfers/hive_to_dynamodb.py
+++ b/airflow/providers/amazon/aws/transfers/hive_to_dynamodb.py
@@ -35,6 +35,10 @@ class HiveToDynamoDBOperator(BaseOperator):
into memory before being pushed to DynamoDB, so this operator should
be used for smallish amount of data.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/transfer:HiveToDynamoDBOperator`
+
:param sql: SQL query to execute against the hive database. (templated)
:param table_name: target DynamoDB table
:param table_keys: partition key and sort key
diff --git a/airflow/providers/amazon/aws/transfers/imap_attachment_to_s3.py b/airflow/providers/amazon/aws/transfers/imap_attachment_to_s3.py
index b3fa33f001d13..e79276dbc7190 100644
--- a/airflow/providers/amazon/aws/transfers/imap_attachment_to_s3.py
+++ b/airflow/providers/amazon/aws/transfers/imap_attachment_to_s3.py
@@ -16,7 +16,8 @@
# specific language governing permissions and limitations
# under the License.
"""This module allows you to transfer mail attachments from a mail server into s3 bucket."""
-from typing import TYPE_CHECKING, Sequence
+import warnings
+from typing import TYPE_CHECKING, Optional, Sequence
from airflow.models import BaseOperator
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
@@ -25,6 +26,10 @@
if TYPE_CHECKING:
from airflow.utils.context import Context
+_DEPRECATION_MSG = (
+ "The s3_conn_id parameter has been deprecated. You should pass instead the aws_conn_id parameter."
+)
+
class ImapAttachmentToS3Operator(BaseOperator):
"""
@@ -35,6 +40,7 @@ class ImapAttachmentToS3Operator(BaseOperator):
:ref:`howto/operator:ImapAttachmentToS3Operator`
:param imap_attachment_name: The file name of the mail attachment that you want to transfer.
+ :param s3_bucket: The targeted s3 bucket. This is the S3 bucket where the file will be downloaded.
:param s3_key: The destination file name in the s3 bucket for the attachment.
:param imap_check_regex: If set checks the `imap_attachment_name` for a regular expression.
:param imap_mail_folder: The folder on the mail server to look for the attachment.
@@ -42,7 +48,7 @@ class ImapAttachmentToS3Operator(BaseOperator):
See :py:meth:`imaplib.IMAP4.search` for details.
:param s3_overwrite: If set overwrites the s3 key if already exists.
:param imap_conn_id: The reference to the connection details of the mail server.
- :param s3_conn_id: The reference to the s3 connection details.
+ :param aws_conn_id: AWS connection to use.
"""
template_fields: Sequence[str] = ('imap_attachment_name', 's3_key', 'imap_mail_filter')
@@ -51,24 +57,31 @@ def __init__(
self,
*,
imap_attachment_name: str,
+ s3_bucket: str,
s3_key: str,
imap_check_regex: bool = False,
imap_mail_folder: str = 'INBOX',
imap_mail_filter: str = 'All',
s3_overwrite: bool = False,
imap_conn_id: str = 'imap_default',
- s3_conn_id: str = 'aws_default',
+ s3_conn_id: Optional[str] = None,
+ aws_conn_id: str = 'aws_default',
**kwargs,
) -> None:
super().__init__(**kwargs)
+ if s3_conn_id:
+ warnings.warn(_DEPRECATION_MSG, DeprecationWarning, stacklevel=3)
+ aws_conn_id = s3_conn_id
+
self.imap_attachment_name = imap_attachment_name
+ self.s3_bucket = s3_bucket
self.s3_key = s3_key
self.imap_check_regex = imap_check_regex
self.imap_mail_folder = imap_mail_folder
self.imap_mail_filter = imap_mail_filter
self.s3_overwrite = s3_overwrite
self.imap_conn_id = imap_conn_id
- self.s3_conn_id = s3_conn_id
+ self.aws_conn_id = aws_conn_id
def execute(self, context: 'Context') -> None:
"""
@@ -91,5 +104,10 @@ def execute(self, context: 'Context') -> None:
mail_filter=self.imap_mail_filter,
)
- s3_hook = S3Hook(aws_conn_id=self.s3_conn_id)
- s3_hook.load_bytes(bytes_data=imap_mail_attachments[0][1], key=self.s3_key, replace=self.s3_overwrite)
+ s3_hook = S3Hook(aws_conn_id=self.aws_conn_id)
+ s3_hook.load_bytes(
+ bytes_data=imap_mail_attachments[0][1],
+ bucket_name=self.s3_bucket,
+ key=self.s3_key,
+ replace=self.s3_overwrite,
+ )
diff --git a/airflow/providers/amazon/aws/transfers/local_to_s3.py b/airflow/providers/amazon/aws/transfers/local_to_s3.py
index 704198b05d1ff..d3e76dcb1d275 100644
--- a/airflow/providers/amazon/aws/transfers/local_to_s3.py
+++ b/airflow/providers/amazon/aws/transfers/local_to_s3.py
@@ -28,16 +28,18 @@ class LocalFilesystemToS3Operator(BaseOperator):
"""
Uploads a file from a local filesystem to Amazon S3.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:LocalFilesystemToS3Operator`
+
:param filename: Path to the local file. Path can be either absolute
(e.g. /path/to/file.ext) or relative (e.g. ../../foo/*/*.csv). (templated)
:param dest_key: The key of the object to copy to. (templated)
It can be either full s3:// style url or relative path from root level.
- When it's specified as a full s3:// url, including dest_bucket results in a TypeError.
+ When it's specified as a full s3:// url, please omit `dest_bucket`.
:param dest_bucket: Name of the S3 bucket to where the object is copied. (templated)
-
- Inclusion when `dest_key` is provided as a full s3:// url results in a TypeError.
:param aws_conn_id: Connection id of the S3 connection to use
:param verify: Whether or not to verify SSL certificates for S3 connection.
By default SSL certificates are verified.
@@ -88,17 +90,15 @@ def __init__(
self.gzip = gzip
self.acl_policy = acl_policy
- def _check_inputs(self):
- if 's3://' in self.dest_key and self.dest_bucket is not None:
- raise TypeError('dest_bucket should be None when dest_key is provided as a full s3:// file path.')
-
def execute(self, context: 'Context'):
- self._check_inputs()
s3_hook = S3Hook(aws_conn_id=self.aws_conn_id, verify=self.verify)
+ s3_bucket, s3_key = s3_hook.get_s3_bucket_key(
+ self.dest_bucket, self.dest_key, 'dest_bucket', 'dest_key'
+ )
s3_hook.load_file(
self.filename,
- self.dest_key,
- self.dest_bucket,
+ s3_key,
+ s3_bucket,
self.replace,
self.encrypt,
self.gzip,
diff --git a/airflow/providers/amazon/aws/transfers/mongo_to_s3.py b/airflow/providers/amazon/aws/transfers/mongo_to_s3.py
index 218d35c892b96..44aae36378a29 100644
--- a/airflow/providers/amazon/aws/transfers/mongo_to_s3.py
+++ b/airflow/providers/amazon/aws/transfers/mongo_to_s3.py
@@ -37,6 +37,10 @@
class MongoToS3Operator(BaseOperator):
"""Operator meant to move data from mongo via pymongo to s3 via boto.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:MongoToS3Operator`
+
:param mongo_conn_id: reference to a specific mongo connection
:param aws_conn_id: reference to a specific S3 connection
:param mongo_collection: reference to a specific collection in your mongo db
diff --git a/airflow/providers/amazon/aws/transfers/mysql_to_s3.py b/airflow/providers/amazon/aws/transfers/mysql_to_s3.py
index 728aaddcba0a9..dc3d84ecb3658 100644
--- a/airflow/providers/amazon/aws/transfers/mysql_to_s3.py
+++ b/airflow/providers/amazon/aws/transfers/mysql_to_s3.py
@@ -66,8 +66,7 @@ def __init__(
if "header" not in pd_kwargs:
pd_kwargs["header"] = header
kwargs["pd_kwargs"] = {**kwargs.get('pd_kwargs', {}), **pd_kwargs}
- else:
- if pd_csv_kwargs is not None:
- raise TypeError("pd_csv_kwargs may not be specified when file_format='parquet'")
+ elif pd_csv_kwargs is not None:
+ raise TypeError("pd_csv_kwargs may not be specified when file_format='parquet'")
super().__init__(sql_conn_id=mysql_conn_id, **kwargs)
diff --git a/airflow/providers/amazon/aws/transfers/redshift_to_s3.py b/airflow/providers/amazon/aws/transfers/redshift_to_s3.py
index 22068be4131af..0bfdda44e78d6 100644
--- a/airflow/providers/amazon/aws/transfers/redshift_to_s3.py
+++ b/airflow/providers/amazon/aws/transfers/redshift_to_s3.py
@@ -139,8 +139,6 @@ def _build_unload_query(
def execute(self, context: 'Context') -> None:
redshift_hook = RedshiftSQLHook(redshift_conn_id=self.redshift_conn_id)
conn = S3Hook.get_connection(conn_id=self.aws_conn_id)
-
- credentials_block = None
if conn.extra_dejson.get('role_arn', False):
credentials_block = f"aws_iam_role={conn.extra_dejson['role_arn']}"
else:
diff --git a/airflow/providers/amazon/aws/transfers/s3_to_ftp.py b/airflow/providers/amazon/aws/transfers/s3_to_ftp.py
index 219d853848dd9..2e07a9575fd26 100644
--- a/airflow/providers/amazon/aws/transfers/s3_to_ftp.py
+++ b/airflow/providers/amazon/aws/transfers/s3_to_ftp.py
@@ -31,6 +31,10 @@ class S3ToFTPOperator(BaseOperator):
"""
This operator enables the transferring of files from S3 to a FTP server.
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:S3ToFTPOperator`
+
:param s3_bucket: The targeted s3 bucket. This is the S3 bucket from
where the file is downloaded.
:param s3_key: The targeted s3 key. This is the specified file path for
diff --git a/airflow/providers/amazon/aws/transfers/s3_to_redshift.py b/airflow/providers/amazon/aws/transfers/s3_to_redshift.py
index 6b54e8902a6cf..014e23ec070f2 100644
--- a/airflow/providers/amazon/aws/transfers/s3_to_redshift.py
+++ b/airflow/providers/amazon/aws/transfers/s3_to_redshift.py
@@ -127,7 +127,6 @@ def execute(self, context: 'Context') -> None:
redshift_hook = RedshiftSQLHook(redshift_conn_id=self.redshift_conn_id)
conn = S3Hook.get_connection(conn_id=self.aws_conn_id)
- credentials_block = None
if conn.extra_dejson.get('role_arn', False):
credentials_block = f"aws_iam_role={conn.extra_dejson['role_arn']}"
else:
diff --git a/airflow/providers/amazon/aws/transfers/s3_to_sftp.py b/airflow/providers/amazon/aws/transfers/s3_to_sftp.py
index 233503020a73e..7c003cfb72608 100644
--- a/airflow/providers/amazon/aws/transfers/s3_to_sftp.py
+++ b/airflow/providers/amazon/aws/transfers/s3_to_sftp.py
@@ -15,9 +15,9 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
+import warnings
from tempfile import NamedTemporaryFile
-from typing import TYPE_CHECKING, Sequence
+from typing import TYPE_CHECKING, Optional, Sequence
from urllib.parse import urlparse
from airflow.models import BaseOperator
@@ -27,6 +27,10 @@
if TYPE_CHECKING:
from airflow.utils.context import Context
+_DEPRECATION_MSG = (
+ "The s3_conn_id parameter has been deprecated. You should pass instead the aws_conn_id parameter."
+)
+
class S3ToSFTPOperator(BaseOperator):
"""
@@ -40,8 +44,7 @@ class S3ToSFTPOperator(BaseOperator):
establishing a connection to the SFTP server.
:param sftp_path: The sftp remote path. This is the specified file path for
uploading file to the SFTP server.
- :param s3_conn_id: The s3 connection id. The name or identifier for
- establishing a connection to S3
+ :param aws_conn_id: aws connection to use
:param s3_bucket: The targeted s3 bucket. This is the S3 bucket from
where the file is downloaded.
:param s3_key: The targeted s3 key. This is the specified file path for
@@ -57,15 +60,20 @@ def __init__(
s3_key: str,
sftp_path: str,
sftp_conn_id: str = 'ssh_default',
- s3_conn_id: str = 'aws_default',
+ s3_conn_id: Optional[str] = None,
+ aws_conn_id: str = 'aws_default',
**kwargs,
) -> None:
super().__init__(**kwargs)
+ if s3_conn_id:
+ warnings.warn(_DEPRECATION_MSG, DeprecationWarning, stacklevel=3)
+ aws_conn_id = s3_conn_id
+
self.sftp_conn_id = sftp_conn_id
self.sftp_path = sftp_path
self.s3_bucket = s3_bucket
self.s3_key = s3_key
- self.s3_conn_id = s3_conn_id
+ self.aws_conn_id = aws_conn_id
@staticmethod
def get_s3_key(s3_key: str) -> str:
@@ -76,7 +84,7 @@ def get_s3_key(s3_key: str) -> str:
def execute(self, context: 'Context') -> None:
self.s3_key = self.get_s3_key(self.s3_key)
ssh_hook = SSHHook(ssh_conn_id=self.sftp_conn_id)
- s3_hook = S3Hook(self.s3_conn_id)
+ s3_hook = S3Hook(self.aws_conn_id)
s3_client = s3_hook.get_conn()
sftp_client = ssh_hook.get_conn().open_sftp()
diff --git a/airflow/providers/amazon/aws/transfers/salesforce_to_s3.py b/airflow/providers/amazon/aws/transfers/salesforce_to_s3.py
index b754bf9564287..a953693f1f193 100644
--- a/airflow/providers/amazon/aws/transfers/salesforce_to_s3.py
+++ b/airflow/providers/amazon/aws/transfers/salesforce_to_s3.py
@@ -127,6 +127,6 @@ def execute(self, context: 'Context') -> str:
)
s3_uri = f"s3://{self.s3_bucket_name}/{self.s3_key}"
- self.log.info(f"Salesforce data uploaded to S3 at {s3_uri}.")
+ self.log.info("Salesforce data uploaded to S3 at %s.", s3_uri)
return s3_uri
diff --git a/airflow/providers/amazon/aws/transfers/sql_to_s3.py b/airflow/providers/amazon/aws/transfers/sql_to_s3.py
index b82263be3420c..f399c271416e4 100644
--- a/airflow/providers/amazon/aws/transfers/sql_to_s3.py
+++ b/airflow/providers/amazon/aws/transfers/sql_to_s3.py
@@ -51,7 +51,11 @@
class SqlToS3Operator(BaseOperator):
"""
- Saves data from an specific SQL query into a file in S3.
+ Saves data from a specific SQL query into a file in S3.
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:SqlToS3Operator`
:param query: the sql query to be executed. If you want to execute a file, place the absolute path of it,
ending with .sql extension. (templated)
diff --git a/airflow/providers/amazon/aws/utils/eks_get_token.py b/airflow/providers/amazon/aws/utils/eks_get_token.py
index 145b8417bbd1e..d9422b35e301c 100644
--- a/airflow/providers/amazon/aws/utils/eks_get_token.py
+++ b/airflow/providers/amazon/aws/utils/eks_get_token.py
@@ -17,7 +17,7 @@
import argparse
import json
-from datetime import datetime, timedelta
+from datetime import datetime, timedelta, timezone
from airflow.providers.amazon.aws.hooks.eks import EksHook
@@ -27,7 +27,8 @@
def get_expiration_time():
- token_expiration = datetime.utcnow() + timedelta(minutes=TOKEN_EXPIRATION_MINUTES)
+ token_expiration = datetime.now(timezone.utc) + timedelta(minutes=TOKEN_EXPIRATION_MINUTES)
+
return token_expiration.strftime('%Y-%m-%dT%H:%M:%SZ')
diff --git a/airflow/providers/amazon/aws/utils/emailer.py b/airflow/providers/amazon/aws/utils/emailer.py
index f4ad65d87a17d..7f0356a3a8d59 100644
--- a/airflow/providers/amazon/aws/utils/emailer.py
+++ b/airflow/providers/amazon/aws/utils/emailer.py
@@ -16,7 +16,7 @@
# specific language governing permissions and limitations
# under the License.
"""Airflow module for email backend using AWS SES"""
-from typing import List, Optional, Union
+from typing import Any, Dict, List, Optional, Union
from airflow.providers.amazon.aws.hooks.ses import SesHook
@@ -32,6 +32,7 @@ def send_email(
mime_charset: str = 'utf-8',
conn_id: str = 'aws_default',
from_email: Optional[str] = None,
+ custom_headers: Optional[Dict[str, Any]] = None,
**kwargs,
) -> None:
"""Email backend for SES."""
@@ -48,4 +49,5 @@ def send_email(
bcc=bcc,
mime_subtype=mime_subtype,
mime_charset=mime_charset,
+ custom_headers=custom_headers,
)
diff --git a/airflow/providers/amazon/aws/utils/redshift.py b/airflow/providers/amazon/aws/utils/redshift.py
index d02dcc5977e61..bb64c9b46f463 100644
--- a/airflow/providers/amazon/aws/utils/redshift.py
+++ b/airflow/providers/amazon/aws/utils/redshift.py
@@ -43,7 +43,7 @@ def build_credentials_block(credentials: ReadOnlyCredentials) -> str:
else:
credentials_line = (
- f"aws_access_key_id={credentials.access_key};" f"aws_secret_access_key={credentials.secret_key}"
+ f"aws_access_key_id={credentials.access_key};aws_secret_access_key={credentials.secret_key}"
)
return credentials_line
diff --git a/airflow/providers/amazon/provider.yaml b/airflow/providers/amazon/provider.yaml
index abf9a249d6ca7..413b6dcfed7ca 100644
--- a/airflow/providers/amazon/provider.yaml
+++ b/airflow/providers/amazon/provider.yaml
@@ -22,6 +22,9 @@ description: |
Amazon integration (including `Amazon Web Services (AWS) `__.
versions:
+ - 2.2.3
- 2.2.2
- 2.2.1
- 2.2.0
diff --git a/airflow/providers/apache/hive/CHANGELOG.rst b/airflow/providers/apache/hive/CHANGELOG.rst
index 9f106218462a9..bb28c620b4126 100644
--- a/airflow/providers/apache/hive/CHANGELOG.rst
+++ b/airflow/providers/apache/hive/CHANGELOG.rst
@@ -19,6 +19,26 @@
Changelog
---------
+2.3.3
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix HiveToMySqlOperator's wrong docstring (#23316)``
+
+.. Below changes are excluded from the changelog. Move them to
+ appropriate section above if needed. Do not delete the lines(!):
+ * ``Bump pre-commit hook versions (#22887)``
+
+2.3.2
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix mistakenly added install_requires for all providers (#22382)``
+
2.3.1
.....
diff --git a/airflow/providers/apache/hive/operators/hive.py b/airflow/providers/apache/hive/operators/hive.py
index b4581aaea3fe3..45cae0fa4e31f 100644
--- a/airflow/providers/apache/hive/operators/hive.py
+++ b/airflow/providers/apache/hive/operators/hive.py
@@ -100,11 +100,15 @@ def __init__(
self.mapred_queue = mapred_queue
self.mapred_queue_priority = mapred_queue_priority
self.mapred_job_name = mapred_job_name
- self.mapred_job_name_template = conf.get(
+
+ job_name_template = conf.get(
'hive',
'mapred_job_name_template',
fallback="Airflow HiveOperator task for {hostname}.{dag_id}.{task_id}.{execution_date}",
)
+ if job_name_template is None:
+ raise ValueError("Job name template should be set !")
+ self.mapred_job_name_template: str = job_name_template
# assigned lazily - just for consistency we can create the attribute with a
# `None` initial value, later it will be populated by the execute method.
diff --git a/airflow/providers/apache/hive/operators/hive_stats.py b/airflow/providers/apache/hive/operators/hive_stats.py
index 56c0f304af050..a1b5539622321 100644
--- a/airflow/providers/apache/hive/operators/hive_stats.py
+++ b/airflow/providers/apache/hive/operators/hive_stats.py
@@ -76,8 +76,9 @@ def __init__(
) -> None:
if 'col_blacklist' in kwargs:
warnings.warn(
- 'col_blacklist kwarg passed to {c} (task_id: {t}) is deprecated, please rename it to '
- 'excluded_columns instead'.format(c=self.__class__.__name__, t=kwargs.get('task_id')),
+ f"col_blacklist kwarg passed to {self.__class__.__name__} "
+ f"(task_id: {kwargs.get('task_id')}) is deprecated, "
+ f"please rename it to excluded_columns instead",
category=FutureWarning,
stacklevel=2,
)
diff --git a/airflow/providers/apache/hive/provider.yaml b/airflow/providers/apache/hive/provider.yaml
index 35a0caa32a7e4..cf00cf2e7cc88 100644
--- a/airflow/providers/apache/hive/provider.yaml
+++ b/airflow/providers/apache/hive/provider.yaml
@@ -22,6 +22,8 @@ description: |
`Apache Hive `.
+ :param hiveserver2_conn_id: Reference to the
+ :ref:`Hive Server2 thrift service connection id `.
:param mysql_preoperator: sql statement to run against mysql prior to
import, typically use to truncate of delete in place
of the data coming in, allowing the task to be idempotent (running
diff --git a/airflow/providers/apache/kylin/CHANGELOG.rst b/airflow/providers/apache/kylin/CHANGELOG.rst
index 655175bed8c9e..5b25eda615175 100644
--- a/airflow/providers/apache/kylin/CHANGELOG.rst
+++ b/airflow/providers/apache/kylin/CHANGELOG.rst
@@ -19,6 +19,14 @@
Changelog
---------
+2.0.4
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix mistakenly added install_requires for all providers (#22382)``
+
2.0.3
.....
diff --git a/airflow/providers/apache/kylin/provider.yaml b/airflow/providers/apache/kylin/provider.yaml
index fb64ed28bcc4c..613042f784a92 100644
--- a/airflow/providers/apache/kylin/provider.yaml
+++ b/airflow/providers/apache/kylin/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Apache Kylin `.
+ """
+
+ conn_name_attr = 'arangodb_conn_id'
+ default_conn_name = 'arangodb_default'
+ conn_type = 'arangodb'
+ hook_name = 'ArangoDB'
+
+ def __init__(self, arangodb_conn_id: str = default_conn_name, *args, **kwargs) -> None:
+ super().__init__(*args, **kwargs)
+ self.hosts = None
+ self.database = None
+ self.username = None
+ self.password = None
+ self.db_conn = None
+ self.arangodb_conn_id = arangodb_conn_id
+ self.client: Optional[ArangoDBClient] = None
+ self.get_conn()
+
+ def get_conn(self) -> ArangoDBClient:
+ """Function that initiates a new ArangoDB connection"""
+ if self.client is not None:
+ return self.client
+
+ conn = self.get_connection(self.arangodb_conn_id)
+ self.hosts = conn.host.split(",")
+ self.database = conn.schema
+ self.username = conn.login
+ self.password = conn.password
+
+ self.client = ArangoDBClient(hosts=self.hosts)
+ self.db_conn = self.client.db(name=self.database, username=self.username, password=self.password)
+ return self.client
+
+ def query(self, query, **kwargs) -> Result:
+ """
+ Function to create a arangodb session
+ and execute the AQL query in the session.
+
+ :param query: AQL query
+ :return: Result
+ """
+ try:
+ if self.db_conn:
+ result = self.db_conn.aql.execute(query, **kwargs)
+ return result
+ else:
+ raise AirflowException(
+ f"Failed to execute AQLQuery, error connecting to database: {self.database}"
+ )
+ except AQLQueryExecuteError as error:
+ raise AirflowException(f"Failed to execute AQLQuery, error: {str(error)}")
+
+ def create_collection(self, name):
+ if not self.db_conn.has_collection(name):
+ self.db_conn.create_collection(name)
+ return True
+ else:
+ self.log.info('Collection already exists: %s', name)
+ return False
+
+ def create_database(self, name):
+ if not self.db_conn.has_database(name):
+ self.db_conn.create_database(name)
+ return True
+ else:
+ self.log.info('Database already exists: %s', name)
+ return False
+
+ def create_graph(self, name):
+ if not self.db_conn.has_graph(name):
+ self.db_conn.create_graph(name)
+ return True
+ else:
+ self.log.info('Graph already exists: %s', name)
+ return False
+
+ @staticmethod
+ def get_ui_field_behaviour() -> Dict[str, Any]:
+ return {
+ "hidden_fields": ['port', 'extra'],
+ "relabeling": {
+ 'host': 'ArangoDB Host URL or comma separated list of URLs (coordinators in a cluster)',
+ 'schema': 'ArangoDB Database',
+ 'login': 'ArangoDB Username',
+ 'password': 'ArangoDB Password',
+ },
+ "placeholders": {
+ 'host': 'eg."http://127.0.0.1:8529" or "http://127.0.0.1:8529,http://127.0.0.1:8530"'
+ ' (coordinators in a cluster)',
+ 'schema': '_system',
+ 'login': 'root',
+ 'password': 'password',
+ },
+ }
diff --git a/dev/breeze/src/airflow_breeze/shell/__init__.py b/airflow/providers/arangodb/operators/__init__.py
similarity index 100%
rename from dev/breeze/src/airflow_breeze/shell/__init__.py
rename to airflow/providers/arangodb/operators/__init__.py
diff --git a/airflow/providers/arangodb/operators/arangodb.py b/airflow/providers/arangodb/operators/arangodb.py
new file mode 100644
index 0000000000000..716ca455d7e0a
--- /dev/null
+++ b/airflow/providers/arangodb/operators/arangodb.py
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import TYPE_CHECKING, Callable, Optional, Sequence
+
+from airflow.models import BaseOperator
+from airflow.providers.arangodb.hooks.arangodb import ArangoDBHook
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+
+class AQLOperator(BaseOperator):
+ """
+ Executes AQL query in a ArangoDB database
+
+ .. seealso::
+ For more information on how to use this operator, take a look at the guide:
+ :ref:`howto/operator:AQLOperator`
+
+ :param query: the AQL query to be executed. Can receive a str representing a
+ AQL statement, or you can provide .sql file having the query
+ :param result_processor: function to further process the Result from ArangoDB
+ :param arangodb_conn_id: Reference to :ref:`ArangoDB connection id `.
+ """
+
+ template_fields: Sequence[str] = ('query',)
+
+ template_ext: Sequence[str] = (".sql",)
+ template_fields_renderers = {"query": "sql"}
+
+ def __init__(
+ self,
+ *,
+ query: str,
+ arangodb_conn_id: str = 'arangodb_default',
+ result_processor: Optional[Callable] = None,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.arangodb_conn_id = arangodb_conn_id
+ self.query = query
+ self.result_processor = result_processor
+
+ def execute(self, context: 'Context'):
+ self.log.info('Executing: %s', self.query)
+ hook = ArangoDBHook(arangodb_conn_id=self.arangodb_conn_id)
+ result = hook.query(self.query)
+ if self.result_processor:
+ self.result_processor(result)
diff --git a/scripts/ci/images/ci_wait_for_and_verify_ci_image.sh b/airflow/providers/arangodb/provider.yaml
old mode 100755
new mode 100644
similarity index 52%
rename from scripts/ci/images/ci_wait_for_and_verify_ci_image.sh
rename to airflow/providers/arangodb/provider.yaml
index 7bbd9c9af048c..129be5b8a0d9c
--- a/scripts/ci/images/ci_wait_for_and_verify_ci_image.sh
+++ b/airflow/providers/arangodb/provider.yaml
@@ -1,4 +1,3 @@
-#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -16,28 +15,34 @@
# specific language governing permissions and limitations
# under the License.
-if [[ $1 == "" ]]; then
- >&2 echo "Requires python MAJOR/MINOR version as first parameter"
- exit 1
-fi
-
-export PYTHON_MAJOR_MINOR_VERSION=$1
-shift
-
-
-# shellcheck source=scripts/ci/libraries/_script_init.sh
-. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"
-
-image_name_with_tag="${AIRFLOW_CI_IMAGE}:${GITHUB_REGISTRY_PULL_IMAGE_TAG}"
-
-build_images::prepare_ci_build
-
-push_pull_remove_images::wait_for_image "${image_name_with_tag}"
-
-if [[ ${VERIFY_IMAGE=} != "false" ]]; then
- verify_image::verify_ci_image "${image_name_with_tag}"
-fi
-
-md5sum::update_all_md5_with_group
-
-docker_v tag "${image_name_with_tag}" "${AIRFLOW_CI_IMAGE}"
+---
+package-name: apache-airflow-providers-arangodb
+name: ArangoDB
+description: |
+ `ArangoDB ` to use
+ when connecting to ArangoDB.
+ :param arangodb_db: Target ArangoDB name.
+ """
+
+ template_fields: Sequence[str] = ('query',)
+
+ template_ext: Sequence[str] = (".sql",)
+ template_fields_renderers = {"query": "sql"}
+
+ def __init__(self, *, query: str, arangodb_conn_id: str = "arangodb_default", **kwargs) -> None:
+ super().__init__(**kwargs)
+ self.arangodb_conn_id = arangodb_conn_id
+ self.query = query
+
+ def poke(self, context: 'Context') -> bool:
+ self.log.info("Sensor running the following query: %s", self.query)
+ hook = ArangoDBHook(self.arangodb_conn_id)
+ records = hook.query(self.query, count=True).count()
+ self.log.info("Total records found: %d", records)
+ return 0 != records
diff --git a/airflow/providers/asana/.latest-doc-only-change.txt b/airflow/providers/asana/.latest-doc-only-change.txt
index 28124098645cf..ab24993f57139 100644
--- a/airflow/providers/asana/.latest-doc-only-change.txt
+++ b/airflow/providers/asana/.latest-doc-only-change.txt
@@ -1 +1 @@
-6c3a67d4fccafe4ab6cd9ec8c7bacf2677f17038
+8b6b0848a3cacf9999477d6af4d2a87463f03026
diff --git a/airflow/providers/asana/CHANGELOG.rst b/airflow/providers/asana/CHANGELOG.rst
index 005a2ba275f23..5262e12f92bc0 100644
--- a/airflow/providers/asana/CHANGELOG.rst
+++ b/airflow/providers/asana/CHANGELOG.rst
@@ -18,6 +18,14 @@
Changelog
---------
+1.1.3
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix mistakenly added install_requires for all providers (#22382)``
+
1.1.2
.....
diff --git a/airflow/providers/asana/hooks/asana.py b/airflow/providers/asana/hooks/asana.py
index 8c9f2e045a64b..45816437f81b8 100644
--- a/airflow/providers/asana/hooks/asana.py
+++ b/airflow/providers/asana/hooks/asana.py
@@ -20,8 +20,8 @@
import sys
from typing import Any, Dict, Optional
-from asana import Client
-from asana.error import NotFoundError
+from asana import Client # type: ignore[attr-defined]
+from asana.error import NotFoundError # type: ignore[attr-defined]
if sys.version_info >= (3, 8):
from functools import cached_property
diff --git a/airflow/providers/asana/provider.yaml b/airflow/providers/asana/provider.yaml
index c7f8efbdcae68..e82dd7c6ecc45 100644
--- a/airflow/providers/asana/provider.yaml
+++ b/airflow/providers/asana/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Asana `__
versions:
+ - 2.0.4
- 2.0.3
- 2.0.2
- 2.0.1
diff --git a/airflow/providers/cncf/kubernetes/.latest-doc-only-change.txt b/airflow/providers/cncf/kubernetes/.latest-doc-only-change.txt
new file mode 100644
index 0000000000000..8be8f80b1cc05
--- /dev/null
+++ b/airflow/providers/cncf/kubernetes/.latest-doc-only-change.txt
@@ -0,0 +1 @@
+c7399c7190750ba705b8255b7a92de2554e6eef3
diff --git a/airflow/providers/cncf/kubernetes/CHANGELOG.rst b/airflow/providers/cncf/kubernetes/CHANGELOG.rst
index a4cb493f11556..bb2d236594044 100644
--- a/airflow/providers/cncf/kubernetes/CHANGELOG.rst
+++ b/airflow/providers/cncf/kubernetes/CHANGELOG.rst
@@ -19,16 +19,109 @@
Changelog
---------
-3.1.1
+main
+....
+
+Features
+~~~~~~~~
+
+KubernetesPodOperator now uses KubernetesHook
+`````````````````````````````````````````````
+
+Previously, KubernetesPodOperator relied on core Airflow configuration (namely setting for kubernetes executor) for certain settings used in client generation. Now KubernetesPodOperator uses KubernetesHook, and the consideration of core k8s settings is officially deprecated.
+
+If you are using the Airflow configuration settings (e.g. as opposed to operator params) to configure the kubernetes client, then prior to the next major release you will need to add an Airflow connection and set your KPO tasks to use that connection.
+
+4.0.2
.....
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix: Exception when parsing log #20966 (#23301)``
+* ``Fixed Kubernetes Operator large xcom content Defect (#23490)``
+* ``Clarify 'reattach_on_restart' behavior (#23377)``
+
+.. Below changes are excluded from the changelog. Move them to
+ appropriate section above if needed. Do not delete the lines(!):
+ * ``Add YANKED to yanked releases of the cncf.kubernetes (#23378)``
+
+.. Review and move the new changes to one of the sections above:
+ * ``Fix k8s pod.execute randomly stuck indefinitely by logs consumption (#23497) (#23618)``
+ * ``Revert "Fix k8s pod.execute randomly stuck indefinitely by logs consumption (#23497) (#23618)" (#23656)``
+
+4.0.1
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Add k8s container's error message in airflow exception (#22871)``
+* ``KubernetesHook should try incluster first when not otherwise configured (#23126)``
+* ``KubernetesPodOperator should patch "already checked" always (#22734)``
+* ``Delete old Spark Application in SparkKubernetesOperator (#21092)``
+* ``Cleanup dup code now that k8s provider requires 2.3.0+ (#22845)``
+* ``Fix ''KubernetesPodOperator'' with 'KubernetesExecutor'' on 2.3.0 (#23371)``
+* ``Fix KPO to have hyphen instead of period (#22982)``
+* ``Fix new MyPy errors in main (#22884)``
+
+.. Below changes are excluded from the changelog. Move them to
+ appropriate section above if needed. Do not delete the lines(!):
+ * ``Use new Breese for building, pulling and verifying the images. (#23104)``
+ * ``Prepare documentation for cncf.kubernetes 4.0.1 release (#23374)``
+
+4.0.0
+.....
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+The provider in version 4.0.0 only works with Airflow 2.3+. Please upgrade
+Airflow to 2.3 version if you want to use the features or fixes in 4.* line
+of the provider.
+
+The main reason for the incompatibility is using latest Kubernetes Libraries.
+The ``cncf.kubernetes`` provider requires newer version of libraries than
+Airflow 2.1 and 2.2 used for Kubernetes Executor and that makes the provider
+incompatible with those Airflow versions.
+
+Features
+~~~~~~~~
+
+* ``Log traceback only on ''DEBUG'' for KPO logs read interruption (#22595)``
+* ``Update our approach for executor-bound dependencies (#22573)``
+* ``Optionally not follow logs in KPO pod_manager (#22412)``
+
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Stop crashing when empty logs are received from kubernetes client (#22566)``
+
+3.1.2 (YANKED)
+..............
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix mistakenly added install_requires for all providers (#22382)``
+* ``Fix "run_id" k8s and elasticsearch compatibility with Airflow 2.1 (#22385)``
+
+Misc
+~~~~
+
+* ``Remove RefreshConfiguration workaround for K8s token refreshing (#20759)``
+
+3.1.1 (YANKED)
+..............
+
Misc
~~~~~
* ``Add Trove classifiers in PyPI (Framework :: Apache Airflow :: Provider)``
-3.1.0
-.....
+3.1.0 (YANKED)
+..............
Features
~~~~~~~~
@@ -47,8 +140,8 @@ Misc
appropriate section above if needed. Do not delete the lines(!):
* ``Add pre-commit check for docstring param types (#21398)``
-3.0.2
-.....
+3.0.2 (YANKED)
+..............
Bug Fixes
~~~~~~~~~
@@ -63,8 +156,8 @@ Bug Fixes
* ``Add optional features in providers. (#21074)``
* ``Add documentation for January 2021 providers release (#21257)``
-3.0.1
-.....
+3.0.1 (YANKED)
+..............
Misc
diff --git a/airflow/providers/cncf/kubernetes/hooks/kubernetes.py b/airflow/providers/cncf/kubernetes/hooks/kubernetes.py
index 38305031dd022..e15dce67ef40a 100644
--- a/airflow/providers/cncf/kubernetes/hooks/kubernetes.py
+++ b/airflow/providers/cncf/kubernetes/hooks/kubernetes.py
@@ -16,7 +16,12 @@
# under the License.
import sys
import tempfile
-from typing import Any, Dict, Generator, Optional, Tuple, Union
+import warnings
+from typing import Any, Dict, Generator, List, Optional, Tuple, Union
+
+from kubernetes.config import ConfigException
+
+from airflow.kubernetes.kube_client import _disable_verify_ssl, _enable_tcp_keepalive
if sys.version_info >= (3, 8):
from functools import cached_property
@@ -61,6 +66,14 @@ class KubernetesHook(BaseHook):
:param conn_id: The :ref:`kubernetes connection `
to Kubernetes cluster.
+ :param client_configuration: Optional dictionary of client configuration params.
+ Passed on to kubernetes client.
+ :param cluster_context: Optionally specify a context to use (e.g. if you have multiple
+ in your kubeconfig.
+ :param config_file: Path to kubeconfig file.
+ :param in_cluster: Set to ``True`` if running from within a kubernetes cluster.
+ :param disable_verify_ssl: Set to ``True`` if SSL verification should be disabled.
+ :param disable_tcp_keepalive: Set to ``True`` if you want to disable keepalive logic.
"""
conn_name_attr = 'kubernetes_conn_id'
@@ -89,6 +102,8 @@ def get_connection_form_widgets() -> Dict[str, Any]:
"extra__kubernetes__cluster_context": StringField(
lazy_gettext('Cluster context'), widget=BS3TextFieldWidget()
),
+ "extra__kubernetes__disable_verify_ssl": BooleanField(lazy_gettext('Disable SSL')),
+ "extra__kubernetes__disable_tcp_keepalive": BooleanField(lazy_gettext('Disable TCP keepalive')),
}
@staticmethod
@@ -106,6 +121,8 @@ def __init__(
cluster_context: Optional[str] = None,
config_file: Optional[str] = None,
in_cluster: Optional[bool] = None,
+ disable_verify_ssl: Optional[bool] = None,
+ disable_tcp_keepalive: Optional[bool] = None,
) -> None:
super().__init__()
self.conn_id = conn_id
@@ -113,6 +130,16 @@ def __init__(
self.cluster_context = cluster_context
self.config_file = config_file
self.in_cluster = in_cluster
+ self.disable_verify_ssl = disable_verify_ssl
+ self.disable_tcp_keepalive = disable_tcp_keepalive
+
+ # these params used for transition in KPO to K8s hook
+ # for a deprecation period we will continue to consider k8s settings from airflow.cfg
+ self._deprecated_core_disable_tcp_keepalive: Optional[bool] = None
+ self._deprecated_core_disable_verify_ssl: Optional[bool] = None
+ self._deprecated_core_in_cluster: Optional[bool] = None
+ self._deprecated_core_cluster_context: Optional[str] = None
+ self._deprecated_core_config_file: Optional[str] = None
@staticmethod
def _coalesce_param(*params):
@@ -120,23 +147,51 @@ def _coalesce_param(*params):
if param is not None:
return param
- def get_conn(self) -> Any:
- """Returns kubernetes api session for use with requests"""
+ @cached_property
+ def conn_extras(self):
if self.conn_id:
connection = self.get_connection(self.conn_id)
extras = connection.extra_dejson
else:
extras = {}
+ return extras
+
+ def _get_field(self, field_name):
+ if field_name.startswith('extra_'):
+ raise ValueError(
+ f"Got prefixed name {field_name}; please remove the 'extra__kubernetes__' prefix "
+ f"when using this method."
+ )
+ if field_name in self.conn_extras:
+ return self.conn_extras[field_name] or None
+ prefixed_name = f"extra__kubernetes__{field_name}"
+ return self.conn_extras.get(prefixed_name) or None
+
+ @staticmethod
+ def _deprecation_warning_core_param(deprecation_warnings):
+ settings_list_str = ''.join([f"\n\t{k}={v!r}" for k, v in deprecation_warnings])
+ warnings.warn(
+ f"\nApplying core Airflow settings from section [kubernetes] with the following keys:"
+ f"{settings_list_str}\n"
+ "In a future release, KubernetesPodOperator will no longer consider core\n"
+ "Airflow settings; define an Airflow connection instead.",
+ DeprecationWarning,
+ )
+
+ def get_conn(self) -> Any:
+ """Returns kubernetes api session for use with requests"""
+
in_cluster = self._coalesce_param(
- self.in_cluster, extras.get("extra__kubernetes__in_cluster") or None
+ self.in_cluster, self.conn_extras.get("extra__kubernetes__in_cluster") or None
)
cluster_context = self._coalesce_param(
- self.cluster_context, extras.get("extra__kubernetes__cluster_context") or None
+ self.cluster_context, self.conn_extras.get("extra__kubernetes__cluster_context") or None
)
kubeconfig_path = self._coalesce_param(
- self.config_file, extras.get("extra__kubernetes__kube_config_path") or None
+ self.config_file, self.conn_extras.get("extra__kubernetes__kube_config_path") or None
)
- kubeconfig = extras.get("extra__kubernetes__kube_config") or None
+
+ kubeconfig = self.conn_extras.get("extra__kubernetes__kube_config") or None
num_selected_configuration = len([o for o in [in_cluster, kubeconfig, kubeconfig_path] if o])
if num_selected_configuration > 1:
@@ -145,6 +200,43 @@ def get_conn(self) -> Any:
"kube_config, in_cluster are mutually exclusive. "
"You can only use one option at a time."
)
+
+ disable_verify_ssl = self._coalesce_param(
+ self.disable_verify_ssl, _get_bool(self._get_field("disable_verify_ssl"))
+ )
+ disable_tcp_keepalive = self._coalesce_param(
+ self.disable_tcp_keepalive, _get_bool(self._get_field("disable_tcp_keepalive"))
+ )
+
+ # BEGIN apply settings from core kubernetes configuration
+ # this section should be removed in next major release
+ deprecation_warnings: List[Tuple[str, Any]] = []
+ if disable_verify_ssl is None and self._deprecated_core_disable_verify_ssl is True:
+ deprecation_warnings.append(('verify_ssl', False))
+ disable_verify_ssl = self._deprecated_core_disable_verify_ssl
+ # by default, hook will try in_cluster first. so we only need to
+ # apply core airflow config and alert when False and in_cluster not otherwise set.
+ if in_cluster is None and self._deprecated_core_in_cluster is False:
+ deprecation_warnings.append(('in_cluster', self._deprecated_core_in_cluster))
+ in_cluster = self._deprecated_core_in_cluster
+ if not cluster_context and self._deprecated_core_cluster_context:
+ deprecation_warnings.append(('cluster_context', self._deprecated_core_cluster_context))
+ cluster_context = self._deprecated_core_cluster_context
+ if not kubeconfig_path and self._deprecated_core_config_file:
+ deprecation_warnings.append(('config_file', self._deprecated_core_config_file))
+ kubeconfig_path = self._deprecated_core_config_file
+ if disable_tcp_keepalive is None and self._deprecated_core_disable_tcp_keepalive is True:
+ deprecation_warnings.append(('enable_tcp_keepalive', False))
+ disable_tcp_keepalive = True
+ if deprecation_warnings:
+ self._deprecation_warning_core_param(deprecation_warnings)
+ # END apply settings from core kubernetes configuration
+
+ if disable_verify_ssl is True:
+ _disable_verify_ssl()
+ if disable_tcp_keepalive is not True:
+ _enable_tcp_keepalive()
+
if in_cluster:
self.log.debug("loading kube_config from: in_cluster configuration")
config.load_incluster_config()
@@ -171,11 +263,21 @@ def get_conn(self) -> Any:
)
return client.ApiClient()
- self.log.debug("loading kube_config from: default file")
- config.load_kube_config(
- client_configuration=self.client_configuration,
- context=cluster_context,
- )
+ return self._get_default_client(cluster_context=cluster_context)
+
+ def _get_default_client(self, *, cluster_context=None):
+ # if we get here, then no configuration has been supplied
+ # we should try in_cluster since that's most likely
+ # but failing that just load assuming a kubeconfig file
+ # in the default location
+ try:
+ config.load_incluster_config(client_configuration=self.client_configuration)
+ except ConfigException:
+ self.log.debug("loading kube_config from: default file")
+ config.load_kube_config(
+ client_configuration=self.client_configuration,
+ context=cluster_context,
+ )
return client.ApiClient()
@cached_property
@@ -203,10 +305,24 @@ def create_custom_object(
if namespace is None:
namespace = self.get_namespace()
if isinstance(body, str):
- body = _load_body_to_dict(body)
+ body_dict = _load_body_to_dict(body)
+ else:
+ body_dict = body
+ try:
+ api.delete_namespaced_custom_object(
+ group=group,
+ version=version,
+ namespace=namespace,
+ plural=plural,
+ name=body_dict["metadata"]["name"],
+ )
+ self.log.warning("Deleted SparkApplication with the same name.")
+ except client.rest.ApiException:
+ self.log.info("SparkApp %s not found.", body_dict['metadata']['name'])
+
try:
response = api.create_namespaced_custom_object(
- group=group, version=version, namespace=namespace, plural=plural, body=body
+ group=group, version=version, namespace=namespace, plural=plural, body=body_dict
)
self.log.debug("Response: %s", response)
return response
@@ -290,3 +406,18 @@ def get_pod_logs(
_preload_content=False,
namespace=namespace if namespace else self.get_namespace(),
)
+
+
+def _get_bool(val) -> Optional[bool]:
+ """
+ Converts val to bool if can be done with certainty.
+ If we cannot infer intention we return None.
+ """
+ if isinstance(val, bool):
+ return val
+ elif isinstance(val, str):
+ if val.strip().lower() == 'true':
+ return True
+ elif val.strip().lower() == 'false':
+ return False
+ return None
diff --git a/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py b/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
index 7ca930b3d8490..69f120e823b3b 100644
--- a/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
+++ b/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
@@ -25,8 +25,9 @@
from kubernetes.client import CoreV1Api, models as k8s
+from airflow.configuration import conf
from airflow.exceptions import AirflowException
-from airflow.kubernetes import kube_client, pod_generator
+from airflow.kubernetes import pod_generator
from airflow.kubernetes.pod_generator import PodGenerator
from airflow.kubernetes.secret import Secret
from airflow.models import BaseOperator
@@ -42,11 +43,17 @@
convert_volume,
convert_volume_mount,
)
-from airflow.providers.cncf.kubernetes.utils import xcom_sidecar
-from airflow.providers.cncf.kubernetes.utils.pod_manager import PodLaunchFailedException, PodManager, PodPhase
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
+from airflow.providers.cncf.kubernetes.utils import xcom_sidecar # type: ignore[attr-defined]
+from airflow.providers.cncf.kubernetes.utils.pod_manager import (
+ PodLaunchFailedException,
+ PodManager,
+ PodPhase,
+ get_container_termination_message,
+)
from airflow.settings import pod_mutation_hook
from airflow.utils import yaml
-from airflow.utils.helpers import validate_key
+from airflow.utils.helpers import prune_dict, validate_key
from airflow.version import version as airflow_version
if sys.version_info >= (3, 8):
@@ -78,6 +85,8 @@ class KubernetesPodOperator(BaseOperator):
:class:`~airflow.providers.google.cloud.operators.kubernetes_engine.GKEStartPodOperator`, which
simplifies the authorization process.
+ :param kubernetes_conn_id: The :ref:`kubernetes connection id `
+ for the Kubernetes cluster.
:param namespace: the namespace to run within kubernetes.
:param image: Docker image you wish to launch. Defaults to hub.docker.com,
but fully qualified URLS will point to custom repositories. (templated)
@@ -98,7 +107,8 @@ class KubernetesPodOperator(BaseOperator):
:param in_cluster: run kubernetes client with in_cluster configuration.
:param cluster_context: context that points to kubernetes cluster.
Ignored when in_cluster is True. If None, current-context is used.
- :param reattach_on_restart: if the scheduler dies while the pod is running, reattach and monitor
+ :param reattach_on_restart: if the worker dies while the pod is running, reattach and monitor
+ during the next try. If False, always create a new pod for each try.
:param labels: labels to apply to the Pod. (templated)
:param startup_timeout_seconds: timeout in seconds to startup the pod.
:param get_logs: get the stdout of the container as logs of the tasks.
@@ -133,6 +143,7 @@ class KubernetesPodOperator(BaseOperator):
:param priority_class_name: priority class name for the launched Pod
:param termination_grace_period: Termination grace period if task killed in UI,
defaults to kubernetes default
+ :param: kubernetes_conn_id: To retrieve credentials for your k8s cluster from an Airflow connection
"""
BASE_CONTAINER_NAME = 'base'
@@ -152,6 +163,7 @@ class KubernetesPodOperator(BaseOperator):
def __init__(
self,
*,
+ kubernetes_conn_id: Optional[str] = None, # 'kubernetes_default',
namespace: Optional[str] = None,
image: Optional[str] = None,
name: Optional[str] = None,
@@ -199,7 +211,7 @@ def __init__(
if kwargs.get('xcom_push') is not None:
raise AirflowException("'xcom_push' was deprecated, use 'do_xcom_push' instead")
super().__init__(resources=None, **kwargs)
-
+ self.kubernetes_conn_id = kubernetes_conn_id
self.do_xcom_push = do_xcom_push
self.image = image
self.namespace = namespace
@@ -283,9 +295,14 @@ def _get_ti_pod_labels(context: Optional[dict] = None, include_try_number: bool
return {}
ti = context['ti']
- run_id = getattr(ti, 'run_id') or context['run_id']
+ run_id = context['run_id']
- labels = {'dag_id': ti.dag_id, 'task_id': ti.task_id, 'run_id': run_id}
+ labels = {
+ 'dag_id': ti.dag_id,
+ 'task_id': ti.task_id,
+ 'run_id': run_id,
+ 'kubernetes_pod_operator': 'True',
+ }
# If running on Airflow 2.3+:
map_index = getattr(ti, 'map_index', -1)
@@ -308,20 +325,24 @@ def _get_ti_pod_labels(context: Optional[dict] = None, include_try_number: bool
def pod_manager(self) -> PodManager:
return PodManager(kube_client=self.client)
- @cached_property
- def client(self) -> CoreV1Api:
- # todo: use airflow Connection / hook to authenticate to the cluster
- kwargs: Dict[str, Any] = dict(
- cluster_context=self.cluster_context,
+ def get_hook(self):
+ hook = KubernetesHook(
+ conn_id=self.kubernetes_conn_id,
+ in_cluster=self.in_cluster,
config_file=self.config_file,
+ cluster_context=self.cluster_context,
)
- if self.in_cluster is not None:
- kwargs.update(in_cluster=self.in_cluster)
- return kube_client.get_kube_client(**kwargs)
+ self._patch_deprecated_k8s_settings(hook)
+ return hook
- def find_pod(self, namespace, context) -> Optional[k8s.V1Pod]:
+ @cached_property
+ def client(self) -> CoreV1Api:
+ hook = self.get_hook()
+ return hook.core_v1_client
+
+ def find_pod(self, namespace, context, *, exclude_checked=True) -> Optional[k8s.V1Pod]:
"""Returns an already-running pod for this task instance if one exists."""
- label_selector = self._build_find_pod_label_selector(context)
+ label_selector = self._build_find_pod_label_selector(context, exclude_checked=exclude_checked)
pod_list = self.client.list_namespaced_pod(
namespace=namespace,
label_selector=label_selector,
@@ -373,9 +394,10 @@ def execute(self, context: 'Context'):
self.await_pod_start(pod=self.pod)
if self.get_logs:
- self.pod_manager.follow_container_logs(
+ self.pod_manager.fetch_container_logs(
pod=self.pod,
container_name=self.BASE_CONTAINER_NAME,
+ follow=True,
)
else:
self.pod_manager.await_container_completion(
@@ -398,17 +420,21 @@ def execute(self, context: 'Context'):
def cleanup(self, pod: k8s.V1Pod, remote_pod: k8s.V1Pod):
pod_phase = remote_pod.status.phase if hasattr(remote_pod, 'status') else None
+ if not self.is_delete_operator_pod:
+ with _suppress(Exception):
+ self.patch_already_checked(remote_pod)
if pod_phase != PodPhase.SUCCEEDED:
if self.log_events_on_failure:
with _suppress(Exception):
for event in self.pod_manager.read_pod_events(pod).items:
self.log.error("Pod Event: %s - %s", event.reason, event.message)
- if not self.is_delete_operator_pod:
- with _suppress(Exception):
- self.patch_already_checked(pod)
with _suppress(Exception):
self.process_pod_deletion(pod)
- raise AirflowException(f'Pod {pod and pod.metadata.name} returned a failure: {remote_pod}')
+ error_message = get_container_termination_message(remote_pod, self.BASE_CONTAINER_NAME)
+ error_message = "\n" + error_message if error_message else ""
+ raise AirflowException(
+ f'Pod {pod and pod.metadata.name} returned a failure:{error_message}\n{remote_pod}'
+ )
else:
with _suppress(Exception):
self.process_pod_deletion(pod)
@@ -420,10 +446,14 @@ def process_pod_deletion(self, pod):
else:
self.log.info("skipping deleting pod: %s", pod.metadata.name)
- def _build_find_pod_label_selector(self, context: Optional[dict] = None) -> str:
+ def _build_find_pod_label_selector(self, context: Optional[dict] = None, *, exclude_checked=True) -> str:
labels = self._get_ti_pod_labels(context, include_try_number=False)
label_strings = [f'{label_id}={label}' for label_id, label in sorted(labels.items())]
- return ','.join(label_strings) + f',{self.POD_CHECKED_KEY}!=True'
+ labels_value = ','.join(label_strings)
+ if exclude_checked:
+ labels_value += f',{self.POD_CHECKED_KEY}!=True'
+ labels_value += ',!airflow-worker'
+ return labels_value
def _set_name(self, name):
if name is None:
@@ -432,7 +462,7 @@ def _set_name(self, name):
raise AirflowException("`name` is required unless `pod_template_file` or `full_pod_spec` is set")
validate_key(name, max_length=220)
- return re.sub(r'[^a-z0-9.-]+', '-', name.lower())
+ return re.sub(r'[^a-z0-9-]+', '-', name.lower())
def patch_already_checked(self, pod: k8s.V1Pod):
"""Add an "already checked" annotation to ensure we don't reattach on retries"""
@@ -531,7 +561,6 @@ def build_pod_request_obj(self, context=None):
pod.metadata.labels.update(
{
'airflow_version': airflow_version.replace('+', '-'),
- 'kubernetes_pod_operator': 'True',
}
)
pod_mutation_hook(pod)
@@ -544,7 +573,40 @@ def dry_run(self) -> None:
one in a dry_run) and excludes all empty elements.
"""
pod = self.build_pod_request_obj()
- print(yaml.dump(_prune_dict(pod.to_dict(), mode='strict')))
+ print(yaml.dump(prune_dict(pod.to_dict(), mode='strict')))
+
+ def _patch_deprecated_k8s_settings(self, hook: KubernetesHook):
+ """
+ Here we read config from core Airflow config [kubernetes] section.
+ In a future release we will stop looking at this section and require users
+ to use Airflow connections to configure KPO.
+
+ When we find values there that we need to apply on the hook, we patch special
+ hook attributes here.
+ """
+
+ # default for enable_tcp_keepalive is True; patch if False
+ if conf.getboolean('kubernetes', 'enable_tcp_keepalive') is False:
+ hook._deprecated_core_disable_tcp_keepalive = True
+
+ # default verify_ssl is True; patch if False.
+ if conf.getboolean('kubernetes', 'verify_ssl') is False:
+ hook._deprecated_core_disable_verify_ssl = True
+
+ # default for in_cluster is True; patch if False and no KPO param.
+ conf_in_cluster = conf.getboolean('kubernetes', 'in_cluster')
+ if self.in_cluster is None and conf_in_cluster is False:
+ hook._deprecated_core_in_cluster = conf_in_cluster
+
+ # there's no default for cluster context; if we get something (and no KPO param) patch it.
+ conf_cluster_context = conf.get('kubernetes', 'cluster_context', fallback=None)
+ if not self.cluster_context and conf_cluster_context:
+ hook._deprecated_core_cluster_context = conf_cluster_context
+
+ # there's no default for config_file; if we get something (and no KPO param) patch it.
+ conf_config_file = conf.get('kubernetes', 'config_file', fallback=None)
+ if not self.config_file and conf_config_file:
+ hook._deprecated_core_config_file = conf_config_file
class _suppress(AbstractContextManager):
@@ -567,56 +629,6 @@ def __exit__(self, exctype, excinst, exctb):
caught_error = exctype is not None and issubclass(exctype, self._exceptions)
if caught_error:
self.exception = excinst
- logger = logging.getLogger()
+ logger = logging.getLogger(__name__)
logger.error(str(excinst), exc_info=True)
return caught_error
-
-
-def _prune_dict(val: Any, mode='strict'):
- """
- Note: this is duplicated from ``airflow.utils.helpers.prune_dict``. That one should
- be the one used if possible, but this one is included to avoid having to
- bump min airflow version. This function will be removed once the min airflow version
- is bumped to 2.3.
-
- Given dict ``val``, returns new dict based on ``val`` with all
- empty elements removed.
-
- What constitutes "empty" is controlled by the ``mode`` parameter. If mode is 'strict'
- then only ``None`` elements will be removed. If mode is ``truthy``, then element ``x``
- will be removed if ``bool(x) is False``.
- """
-
- def is_empty(x):
- if mode == 'strict':
- return x is None
- elif mode == 'truthy':
- return bool(x) is False
- raise ValueError("allowable values for `mode` include 'truthy' and 'strict'")
-
- if isinstance(val, dict):
- new_dict = {}
- for k, v in val.items():
- if is_empty(v):
- continue
- elif isinstance(v, (list, dict)):
- new_val = _prune_dict(v, mode=mode)
- if new_val:
- new_dict[k] = new_val
- else:
- new_dict[k] = v
- return new_dict
- elif isinstance(val, list):
- new_list = []
- for v in val:
- if is_empty(v):
- continue
- elif isinstance(v, (list, dict)):
- new_val = _prune_dict(v, mode=mode)
- if new_val:
- new_list.append(new_val)
- else:
- new_list.append(v)
- return new_list
- else:
- return val
diff --git a/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py b/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py
index 10296871efc60..fbf0aebd4948c 100644
--- a/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py
+++ b/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py
@@ -33,7 +33,7 @@ class SparkKubernetesOperator(BaseOperator):
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/v1beta2-1.1.0-2.4.5/docs/api-docs.md#sparkapplication
:param application_file: Defines Kubernetes 'custom_resource_definition' of 'sparkApplication' as either a
- path to a '.json' file or a JSON string.
+ path to a '.yaml' file, '.json' file, YAML string or JSON string.
:param namespace: kubernetes namespace to put sparkApplication
:param kubernetes_conn_id: The :ref:`kubernetes connection id `
for the to Kubernetes cluster.
@@ -61,14 +61,15 @@ def __init__(
self.kubernetes_conn_id = kubernetes_conn_id
self.api_group = api_group
self.api_version = api_version
+ self.plural = "sparkapplications"
def execute(self, context: 'Context'):
- self.log.info("Creating sparkApplication")
hook = KubernetesHook(conn_id=self.kubernetes_conn_id)
+ self.log.info("Creating sparkApplication")
response = hook.create_custom_object(
group=self.api_group,
version=self.api_version,
- plural="sparkapplications",
+ plural=self.plural,
body=self.application_file,
namespace=self.namespace,
)
diff --git a/airflow/providers/cncf/kubernetes/provider.yaml b/airflow/providers/cncf/kubernetes/provider.yaml
index dad7bf543a8a6..ce2ce89eddeec 100644
--- a/airflow/providers/cncf/kubernetes/provider.yaml
+++ b/airflow/providers/cncf/kubernetes/provider.yaml
@@ -22,6 +22,10 @@ description: |
`Kubernetes `_.
"""
+import json
from typing import Any, Dict, List, Optional
+from requests import exceptions as requests_exceptions
+
from airflow.exceptions import AirflowException
from airflow.providers.databricks.hooks.databricks_base import BaseDatabricksHook
@@ -90,6 +93,13 @@ def __eq__(self, other: object) -> bool:
def __repr__(self) -> str:
return str(self.__dict__)
+ def to_json(self) -> str:
+ return json.dumps(self.__dict__)
+
+ @classmethod
+ def from_json(cls, data: str) -> 'RunState':
+ return RunState(**json.loads(data))
+
class DatabricksHook(BaseDatabricksHook):
"""
@@ -196,6 +206,16 @@ def get_run_page_url(self, run_id: int) -> str:
response = self._do_api_call(GET_RUN_ENDPOINT, json)
return response['run_page_url']
+ async def a_get_run_page_url(self, run_id: int) -> str:
+ """
+ Async version of `get_run_page_url()`.
+ :param run_id: id of the run
+ :return: URL of the run page
+ """
+ json = {'run_id': run_id}
+ response = await self._a_do_api_call(GET_RUN_ENDPOINT, json)
+ return response['run_page_url']
+
def get_job_id(self, run_id: int) -> int:
"""
Retrieves job_id from run_id.
@@ -227,6 +247,17 @@ def get_run_state(self, run_id: int) -> RunState:
state = response['state']
return RunState(**state)
+ async def a_get_run_state(self, run_id: int) -> RunState:
+ """
+ Async version of `get_run_state()`.
+ :param run_id: id of the run
+ :return: state of the run
+ """
+ json = {'run_id': run_id}
+ response = await self._a_do_api_call(GET_RUN_ENDPOINT, json)
+ state = response['state']
+ return RunState(**state)
+
def get_run_state_str(self, run_id: int) -> str:
"""
Return the string representation of RunState.
@@ -333,22 +364,47 @@ def uninstall(self, json: dict) -> None:
def update_repo(self, repo_id: str, json: Dict[str, Any]) -> dict:
"""
+ Updates given Databricks Repos
- :param repo_id:
- :param json:
- :return:
+ :param repo_id: ID of Databricks Repos
+ :param json: payload
+ :return: metadata from update
"""
repos_endpoint = ('PATCH', f'api/2.0/repos/{repo_id}')
return self._do_api_call(repos_endpoint, json)
- def get_repo_by_path(self, path: str) -> Optional[str]:
+ def delete_repo(self, repo_id: str):
+ """
+ Deletes given Databricks Repos
+
+ :param repo_id: ID of Databricks Repos
+ :return:
+ """
+ repos_endpoint = ('DELETE', f'api/2.0/repos/{repo_id}')
+ self._do_api_call(repos_endpoint)
+
+ def create_repo(self, json: Dict[str, Any]) -> dict:
"""
+ Creates a Databricks Repos
- :param path:
+ :param json: payload
:return:
"""
- result = self._do_api_call(WORKSPACE_GET_STATUS_ENDPOINT, {'path': path})
- if result.get('object_type', '') == 'REPO':
- return str(result['object_id'])
+ repos_endpoint = ('POST', 'api/2.0/repos')
+ return self._do_api_call(repos_endpoint, json)
+
+ def get_repo_by_path(self, path: str) -> Optional[str]:
+ """
+ Obtains Repos ID by path
+ :param path: path to a repository
+ :return: Repos ID if it exists, None if doesn't.
+ """
+ try:
+ result = self._do_api_call(WORKSPACE_GET_STATUS_ENDPOINT, {'path': path}, wrap_http_errors=False)
+ if result.get('object_type', '') == 'REPO':
+ return str(result['object_id'])
+ except requests_exceptions.HTTPError as e:
+ if e.response.status_code != 404:
+ raise e
return None
diff --git a/airflow/providers/databricks/hooks/databricks_base.py b/airflow/providers/databricks/hooks/databricks_base.py
index f545f26a25408..5b18dad9303ef 100644
--- a/airflow/providers/databricks/hooks/databricks_base.py
+++ b/airflow/providers/databricks/hooks/databricks_base.py
@@ -28,10 +28,19 @@
from typing import Any, Dict, Optional, Tuple
from urllib.parse import urlparse
+import aiohttp
import requests
from requests import PreparedRequest, exceptions as requests_exceptions
from requests.auth import AuthBase, HTTPBasicAuth
-from tenacity import RetryError, Retrying, retry_if_exception, stop_after_attempt, wait_exponential
+from requests.exceptions import JSONDecodeError
+from tenacity import (
+ AsyncRetrying,
+ RetryError,
+ Retrying,
+ retry_if_exception,
+ stop_after_attempt,
+ wait_exponential,
+)
from airflow import __version__
from airflow.exceptions import AirflowException
@@ -113,7 +122,7 @@ def my_after_func(retry_state):
else:
self.retry_args = dict(
stop=stop_after_attempt(self.retry_limit),
- wait=wait_exponential(min=self.retry_delay, max=(2 ** retry_limit)),
+ wait=wait_exponential(min=self.retry_delay, max=(2**retry_limit)),
retry=retry_if_exception(self._retryable_error),
after=my_after_func,
)
@@ -134,6 +143,14 @@ def host(self) -> str:
return host
+ async def __aenter__(self):
+ self._session = aiohttp.ClientSession()
+ return self
+
+ async def __aexit__(self, *err):
+ await self._session.close()
+ self._session = None
+
@staticmethod
def _parse_host(host: str) -> str:
"""
@@ -168,6 +185,13 @@ def _get_retry_object(self) -> Retrying:
"""
return Retrying(**self.retry_args)
+ def _a_get_retry_object(self) -> AsyncRetrying:
+ """
+ Instantiates an async retry object
+ :return: instance of AsyncRetrying class
+ """
+ return AsyncRetrying(**self.retry_args)
+
def _get_aad_token(self, resource: str) -> str:
"""
Function to get AAD token for given resource. Supports managed identity or service principal auth
@@ -233,6 +257,72 @@ def _get_aad_token(self, resource: str) -> str:
return token
+ async def _a_get_aad_token(self, resource: str) -> str:
+ """
+ Async version of `_get_aad_token()`.
+ :param resource: resource to issue token to
+ :return: AAD token, or raise an exception
+ """
+ aad_token = self.aad_tokens.get(resource)
+ if aad_token and self._is_aad_token_valid(aad_token):
+ return aad_token['token']
+
+ self.log.info('Existing AAD token is expired, or going to expire soon. Refreshing...')
+ try:
+ async for attempt in self._a_get_retry_object():
+ with attempt:
+ if self.databricks_conn.extra_dejson.get('use_azure_managed_identity', False):
+ params = {
+ "api-version": "2018-02-01",
+ "resource": resource,
+ }
+ async with self._session.get(
+ url=AZURE_METADATA_SERVICE_TOKEN_URL,
+ params=params,
+ headers={**USER_AGENT_HEADER, "Metadata": "true"},
+ timeout=self.aad_timeout_seconds,
+ ) as resp:
+ resp.raise_for_status()
+ jsn = await resp.json()
+ else:
+ tenant_id = self.databricks_conn.extra_dejson['azure_tenant_id']
+ data = {
+ "grant_type": "client_credentials",
+ "client_id": self.databricks_conn.login,
+ "resource": resource,
+ "client_secret": self.databricks_conn.password,
+ }
+ azure_ad_endpoint = self.databricks_conn.extra_dejson.get(
+ "azure_ad_endpoint", AZURE_DEFAULT_AD_ENDPOINT
+ )
+ async with self._session.post(
+ url=AZURE_TOKEN_SERVICE_URL.format(azure_ad_endpoint, tenant_id),
+ data=data,
+ headers={
+ **USER_AGENT_HEADER,
+ 'Content-Type': 'application/x-www-form-urlencoded',
+ },
+ timeout=self.aad_timeout_seconds,
+ ) as resp:
+ resp.raise_for_status()
+ jsn = await resp.json()
+ if (
+ 'access_token' not in jsn
+ or jsn.get('token_type') != 'Bearer'
+ or 'expires_on' not in jsn
+ ):
+ raise AirflowException(f"Can't get necessary data from AAD token: {jsn}")
+
+ token = jsn['access_token']
+ self.aad_tokens[resource] = {'token': token, 'expires_on': int(jsn["expires_on"])}
+ break
+ except RetryError:
+ raise AirflowException(f'API requests to Azure failed {self.retry_limit} times. Giving up.')
+ except aiohttp.ClientResponseError as err:
+ raise AirflowException(f'Response: {err.message}, Status Code: {err.status}')
+
+ return token
+
def _get_aad_headers(self) -> dict:
"""
Fills AAD headers if necessary (SPN is outside of the workspace)
@@ -247,6 +337,20 @@ def _get_aad_headers(self) -> dict:
headers['X-Databricks-Azure-SP-Management-Token'] = mgmt_token
return headers
+ async def _a_get_aad_headers(self) -> dict:
+ """
+ Async version of `_get_aad_headers()`.
+ :return: dictionary with filled AAD headers
+ """
+ headers = {}
+ if 'azure_resource_id' in self.databricks_conn.extra_dejson:
+ mgmt_token = await self._a_get_aad_token(AZURE_MANAGEMENT_ENDPOINT)
+ headers['X-Databricks-Azure-Workspace-Resource-Id'] = self.databricks_conn.extra_dejson[
+ 'azure_resource_id'
+ ]
+ headers['X-Databricks-Azure-SP-Management-Token'] = mgmt_token
+ return headers
+
@staticmethod
def _is_aad_token_valid(aad_token: dict) -> bool:
"""
@@ -280,6 +384,23 @@ def _check_azure_metadata_service() -> None:
except (requests_exceptions.RequestException, ValueError) as e:
raise AirflowException(f"Can't reach Azure Metadata Service: {e}")
+ async def _a_check_azure_metadata_service(self):
+ """Async version of `_check_azure_metadata_service()`."""
+ try:
+ async with self._session.get(
+ url=AZURE_METADATA_SERVICE_INSTANCE_URL,
+ params={"api-version": "2021-02-01"},
+ headers={"Metadata": "true"},
+ timeout=2,
+ ) as resp:
+ jsn = await resp.json()
+ if 'compute' not in jsn or 'azEnvironment' not in jsn['compute']:
+ raise AirflowException(
+ f"Was able to fetch some metadata, but it doesn't look like Azure Metadata: {jsn}"
+ )
+ except (requests_exceptions.RequestException, ValueError) as e:
+ raise AirflowException(f"Can't reach Azure Metadata Service: {e}")
+
def _get_token(self, raise_error: bool = False) -> Optional[str]:
if 'token' in self.databricks_conn.extra_dejson:
self.log.info(
@@ -303,10 +424,38 @@ def _get_token(self, raise_error: bool = False) -> Optional[str]:
return None
+ async def _a_get_token(self, raise_error: bool = False) -> Optional[str]:
+ if 'token' in self.databricks_conn.extra_dejson:
+ self.log.info(
+ 'Using token auth. For security reasons, please set token in Password field instead of extra'
+ )
+ return self.databricks_conn.extra_dejson["token"]
+ elif not self.databricks_conn.login and self.databricks_conn.password:
+ self.log.info('Using token auth.')
+ return self.databricks_conn.password
+ elif 'azure_tenant_id' in self.databricks_conn.extra_dejson:
+ if self.databricks_conn.login == "" or self.databricks_conn.password == "":
+ raise AirflowException("Azure SPN credentials aren't provided")
+ self.log.info('Using AAD Token for SPN.')
+ return await self._a_get_aad_token(DEFAULT_DATABRICKS_SCOPE)
+ elif self.databricks_conn.extra_dejson.get('use_azure_managed_identity', False):
+ self.log.info('Using AAD Token for managed identity.')
+ await self._a_check_azure_metadata_service()
+ return await self._a_get_aad_token(DEFAULT_DATABRICKS_SCOPE)
+ elif raise_error:
+ raise AirflowException('Token authentication isn\'t configured')
+
+ return None
+
def _log_request_error(self, attempt_num: int, error: str) -> None:
self.log.error('Attempt %s API Request to Databricks failed with reason: %s', attempt_num, error)
- def _do_api_call(self, endpoint_info: Tuple[str, str], json: Optional[Dict[str, Any]] = None):
+ def _do_api_call(
+ self,
+ endpoint_info: Tuple[str, str],
+ json: Optional[Dict[str, Any]] = None,
+ wrap_http_errors: bool = True,
+ ):
"""
Utility function to perform an API call with retries
@@ -340,6 +489,8 @@ def _do_api_call(self, endpoint_info: Tuple[str, str], json: Optional[Dict[str,
request_func = requests.post
elif method == 'PATCH':
request_func = requests.patch
+ elif method == 'DELETE':
+ request_func = requests.delete
else:
raise AirflowException('Unexpected HTTP Method: ' + method)
@@ -359,16 +510,94 @@ def _do_api_call(self, endpoint_info: Tuple[str, str], json: Optional[Dict[str,
except RetryError:
raise AirflowException(f'API requests to Databricks failed {self.retry_limit} times. Giving up.')
except requests_exceptions.HTTPError as e:
- raise AirflowException(f'Response: {e.response.content}, Status Code: {e.response.status_code}')
+ if wrap_http_errors:
+ raise AirflowException(
+ f'Response: {e.response.content}, Status Code: {e.response.status_code}'
+ )
+ else:
+ raise e
+
+ async def _a_do_api_call(self, endpoint_info: Tuple[str, str], json: Optional[Dict[str, Any]] = None):
+ """
+ Async version of `_do_api_call()`.
+ :param endpoint_info: Tuple of method and endpoint
+ :param json: Parameters for this API call.
+ :return: If the api call returns a OK status code,
+ this function returns the response in JSON. Otherwise, throw an AirflowException.
+ """
+ method, endpoint = endpoint_info
+
+ url = f'https://{self.host}/{endpoint}'
+
+ aad_headers = await self._a_get_aad_headers()
+ headers = {**USER_AGENT_HEADER.copy(), **aad_headers}
+
+ auth: aiohttp.BasicAuth
+ token = await self._a_get_token()
+ if token:
+ auth = BearerAuth(token)
+ else:
+ self.log.info('Using basic auth.')
+ auth = aiohttp.BasicAuth(self.databricks_conn.login, self.databricks_conn.password)
+
+ request_func: Any
+ if method == 'GET':
+ request_func = self._session.get
+ elif method == 'POST':
+ request_func = self._session.post
+ elif method == 'PATCH':
+ request_func = self._session.patch
+ else:
+ raise AirflowException('Unexpected HTTP Method: ' + method)
+ try:
+ async for attempt in self._a_get_retry_object():
+ with attempt:
+ async with request_func(
+ url,
+ json=json,
+ auth=auth,
+ headers={**headers, **USER_AGENT_HEADER},
+ timeout=self.timeout_seconds,
+ ) as response:
+ response.raise_for_status()
+ return await response.json()
+ except RetryError:
+ raise AirflowException(f'API requests to Databricks failed {self.retry_limit} times. Giving up.')
+ except aiohttp.ClientResponseError as err:
+ raise AirflowException(f'Response: {err.message}, Status Code: {err.status}')
+
+ @staticmethod
+ def _get_error_code(exception: BaseException) -> str:
+ if isinstance(exception, requests_exceptions.HTTPError):
+ try:
+ jsn = exception.response.json()
+ return jsn.get('error_code', '')
+ except JSONDecodeError:
+ pass
+
+ return ""
@staticmethod
def _retryable_error(exception: BaseException) -> bool:
- if not isinstance(exception, requests_exceptions.RequestException):
- return False
- return isinstance(exception, (requests_exceptions.ConnectionError, requests_exceptions.Timeout)) or (
- exception.response is not None
- and (exception.response.status_code >= 500 or exception.response.status_code == 429)
- )
+ if isinstance(exception, requests_exceptions.RequestException):
+ if isinstance(exception, (requests_exceptions.ConnectionError, requests_exceptions.Timeout)) or (
+ exception.response is not None
+ and (
+ exception.response.status_code >= 500
+ or exception.response.status_code == 429
+ or (
+ exception.response.status_code == 400
+ and BaseDatabricksHook._get_error_code(exception) == 'COULD_NOT_ACQUIRE_LOCK'
+ )
+ )
+ ):
+ return True
+
+ if isinstance(exception, aiohttp.ClientResponseError):
+ if exception.status >= 500 or exception.status == 429:
+ return True
+
+ return False
class _TokenAuth(AuthBase):
@@ -383,3 +612,16 @@ def __init__(self, token: str) -> None:
def __call__(self, r: PreparedRequest) -> PreparedRequest:
r.headers['Authorization'] = 'Bearer ' + self.token
return r
+
+
+class BearerAuth(aiohttp.BasicAuth):
+ """aiohttp only ships BasicAuth, for Bearer auth we need a subclass of BasicAuth."""
+
+ def __new__(cls, token: str) -> 'BearerAuth':
+ return super().__new__(cls, token) # type: ignore
+
+ def __init__(self, token: str) -> None:
+ self.token = token
+
+ def encode(self) -> str:
+ return f'Bearer {self.token}'
diff --git a/airflow/providers/databricks/hooks/databricks_sql.py b/airflow/providers/databricks/hooks/databricks_sql.py
index ddf279f4b4394..9d86b4dbe3ae6 100644
--- a/airflow/providers/databricks/hooks/databricks_sql.py
+++ b/airflow/providers/databricks/hooks/databricks_sql.py
@@ -18,30 +18,38 @@
import re
from contextlib import closing
from copy import copy
-from typing import Any, Dict, List, Optional, Union
+from typing import Any, Dict, List, Optional, Tuple, Union
-from databricks import sql
-from databricks.sql.client import Connection
+from databricks import sql # type: ignore[attr-defined]
+from databricks.sql.client import Connection # type: ignore[attr-defined]
+from airflow import __version__
from airflow.exceptions import AirflowException
from airflow.hooks.dbapi import DbApiHook
from airflow.providers.databricks.hooks.databricks_base import BaseDatabricksHook
LIST_SQL_ENDPOINTS_ENDPOINT = ('GET', 'api/2.0/sql/endpoints')
+USER_AGENT_STRING = f'airflow-{__version__}'
class DatabricksSqlHook(BaseDatabricksHook, DbApiHook):
"""
- Interact with Databricks SQL.
+ Hook to interact with Databricks SQL.
- :param databricks_conn_id: Reference to the :ref:`Databricks connection `.
+ :param databricks_conn_id: Reference to the
+ :ref:`Databricks connection `.
:param http_path: Optional string specifying HTTP path of Databricks SQL Endpoint or cluster.
If not specified, it should be either specified in the Databricks connection's extra parameters,
or ``sql_endpoint_name`` must be specified.
- :param sql_endpoint_name: Optional name of Databricks SQL Endpoint. If not specified, ``http_path`` must
- be provided as described above.
+ :param sql_endpoint_name: Optional name of Databricks SQL Endpoint. If not specified, ``http_path``
+ must be provided as described above.
:param session_configuration: An optional dictionary of Spark session parameters. Defaults to None.
If not specified, it could be specified in the Databricks connection's extra parameters.
+ :param http_headers: An optional list of (k, v) pairs that will be set as HTTP headers
+ on every request
+ :param catalog: An optional initial catalog to use. Requires DBR version 9.0+
+ :param schema: An optional initial schema to use. Requires DBR version 9.0+
+ :param kwargs: Additional parameters internal to Databricks SQL Connector parameters
"""
hook_name = 'Databricks SQL'
@@ -52,6 +60,10 @@ def __init__(
http_path: Optional[str] = None,
sql_endpoint_name: Optional[str] = None,
session_configuration: Optional[Dict[str, str]] = None,
+ http_headers: Optional[List[Tuple[str, str]]] = None,
+ catalog: Optional[str] = None,
+ schema: Optional[str] = None,
+ **kwargs,
) -> None:
super().__init__(databricks_conn_id)
self._sql_conn = None
@@ -60,6 +72,10 @@ def __init__(
self._sql_endpoint_name = sql_endpoint_name
self.supports_autocommit = True
self.session_config = session_configuration
+ self.http_headers = http_headers
+ self.catalog = catalog
+ self.schema = schema
+ self.additional_params = kwargs
def _get_extra_config(self) -> Dict[str, Optional[Any]]:
extra_params = copy(self.databricks_conn.extra_dejson)
@@ -113,8 +129,13 @@ def get_conn(self) -> Connection:
self.host,
self._http_path,
self._token,
+ schema=self.schema,
+ catalog=self.catalog,
session_configuration=self.session_config,
+ http_headers=self.http_headers,
+ _user_agent_entry=USER_AGENT_STRING,
**self._get_extra_config(),
+ **self.additional_params,
)
return self._sql_conn
@@ -146,7 +167,11 @@ def run(self, sql: Union[str, List[str]], autocommit=True, parameters=None, hand
"""
if isinstance(sql, str):
sql = self.maybe_split_sql_string(sql)
- self.log.debug("Executing %d statements", len(sql))
+
+ if sql:
+ self.log.debug("Executing %d statements", len(sql))
+ else:
+ raise ValueError("List of SQL statements is empty")
conn = None
for sql_statement in sql:
diff --git a/airflow/providers/databricks/operators/databricks.py b/airflow/providers/databricks/operators/databricks.py
index ec0a9a013cab6..8af4474b1315f 100644
--- a/airflow/providers/databricks/operators/databricks.py
+++ b/airflow/providers/databricks/operators/databricks.py
@@ -19,50 +19,24 @@
"""This module contains Databricks operators."""
import time
+from logging import Logger
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Sequence, Union
from airflow.exceptions import AirflowException
-from airflow.models import BaseOperator
-from airflow.providers.databricks.hooks.databricks import DatabricksHook
+from airflow.models import BaseOperator, BaseOperatorLink, XCom
+from airflow.providers.databricks.hooks.databricks import DatabricksHook, RunState
+from airflow.providers.databricks.triggers.databricks import DatabricksExecutionTrigger
+from airflow.providers.databricks.utils.databricks import deep_string_coerce, validate_trigger_event
if TYPE_CHECKING:
+ from airflow.models.taskinstance import TaskInstanceKey
from airflow.utils.context import Context
+DEFER_METHOD_NAME = 'execute_complete'
XCOM_RUN_ID_KEY = 'run_id'
XCOM_RUN_PAGE_URL_KEY = 'run_page_url'
-def _deep_string_coerce(content, json_path: str = 'json') -> Union[str, list, dict]:
- """
- Coerces content or all values of content if it is a dict to a string. The
- function will throw if content contains non-string or non-numeric types.
-
- The reason why we have this function is because the ``self.json`` field must be a
- dict with only string values. This is because ``render_template`` will fail
- for numerical values.
- """
- coerce = _deep_string_coerce
- if isinstance(content, str):
- return content
- elif isinstance(
- content,
- (
- int,
- float,
- ),
- ):
- # Databricks can tolerate either numeric or string types in the API backend.
- return str(content)
- elif isinstance(content, (list, tuple)):
- return [coerce(e, f'{json_path}[{i}]') for i, e in enumerate(content)]
- elif isinstance(content, dict):
- return {k: coerce(v, f'{json_path}[{k}]') for k, v in list(content.items())}
- else:
- param_type = type(content)
- msg = f'Type {param_type} used for parameter {json_path} is not a number or a string'
- raise AirflowException(msg)
-
-
def _handle_databricks_operator_execution(operator, hook, log, context) -> None:
"""
Handles the Airflow + Databricks lifecycle logic for a Databricks operator
@@ -70,11 +44,11 @@ def _handle_databricks_operator_execution(operator, hook, log, context) -> None:
:param operator: Databricks operator being handled
:param context: Airflow context
"""
- if operator.do_xcom_push:
+ if operator.do_xcom_push and context is not None:
context['ti'].xcom_push(key=XCOM_RUN_ID_KEY, value=operator.run_id)
log.info('Run submitted with run_id: %s', operator.run_id)
run_page_url = hook.get_run_page_url(operator.run_id)
- if operator.do_xcom_push:
+ if operator.do_xcom_push and context is not None:
context['ti'].xcom_push(key=XCOM_RUN_PAGE_URL_KEY, value=run_page_url)
if operator.wait_for_termination:
@@ -102,6 +76,73 @@ def _handle_databricks_operator_execution(operator, hook, log, context) -> None:
log.info('View run status, Spark UI, and logs at %s', run_page_url)
+def _handle_deferrable_databricks_operator_execution(operator, hook, log, context) -> None:
+ """
+ Handles the Airflow + Databricks lifecycle logic for deferrable Databricks operators
+
+ :param operator: Databricks async operator being handled
+ :param context: Airflow context
+ """
+ if operator.do_xcom_push and context is not None:
+ context['ti'].xcom_push(key=XCOM_RUN_ID_KEY, value=operator.run_id)
+ log.info(f'Run submitted with run_id: {operator.run_id}')
+
+ run_page_url = hook.get_run_page_url(operator.run_id)
+ if operator.do_xcom_push and context is not None:
+ context['ti'].xcom_push(key=XCOM_RUN_PAGE_URL_KEY, value=run_page_url)
+ log.info(f'View run status, Spark UI, and logs at {run_page_url}')
+
+ if operator.wait_for_termination:
+ operator.defer(
+ trigger=DatabricksExecutionTrigger(
+ run_id=operator.run_id,
+ databricks_conn_id=operator.databricks_conn_id,
+ polling_period_seconds=operator.polling_period_seconds,
+ ),
+ method_name=DEFER_METHOD_NAME,
+ )
+
+
+def _handle_deferrable_databricks_operator_completion(event: dict, log: Logger) -> None:
+ validate_trigger_event(event)
+ run_state = RunState.from_json(event['run_state'])
+ run_page_url = event['run_page_url']
+ log.info(f'View run status, Spark UI, and logs at {run_page_url}')
+
+ if run_state.is_successful:
+ log.info('Job run completed successfully.')
+ return
+ else:
+ error_message = f'Job run failed with terminal state: {run_state}'
+ raise AirflowException(error_message)
+
+
+class DatabricksJobRunLink(BaseOperatorLink):
+ """Constructs a link to monitor a Databricks Job Run."""
+
+ name = "See Databricks Job Run"
+
+ def get_link(
+ self,
+ operator,
+ dttm=None,
+ *,
+ ti_key: Optional["TaskInstanceKey"] = None,
+ ) -> str:
+ if ti_key is not None:
+ run_page_url = XCom.get_value(key=XCOM_RUN_PAGE_URL_KEY, ti_key=ti_key)
+ else:
+ assert dttm
+ run_page_url = XCom.get_one(
+ key=XCOM_RUN_PAGE_URL_KEY,
+ dag_id=operator.dag.dag_id,
+ task_id=operator.task_id,
+ execution_date=dttm,
+ )
+
+ return run_page_url
+
+
class DatabricksSubmitRunOperator(BaseOperator):
"""
Submits a Spark job run to Databricks using the
@@ -232,6 +273,14 @@ class DatabricksSubmitRunOperator(BaseOperator):
By default this will be set to the Airflow ``task_id``. This ``task_id`` is a
required parameter of the superclass ``BaseOperator``.
This field will be templated.
+ :param idempotency_token: an optional token that can be used to guarantee the idempotency of job run
+ requests. If a run with the provided token already exists, the request does not create a new run but
+ returns the ID of the existing run instead. This token must have at most 64 characters.
+ :param access_control_list: optional list of dictionaries representing Access Control List (ACL) for
+ a given job run. Each dictionary consists of following field - specific subject (``user_name`` for
+ users, or ``group_name`` for groups), and ``permission_level`` for that subject. See Jobs API
+ documentation for more details.
+ :param wait_for_termination: if we should wait for termination of the job run. ``True`` by default.
:param timeout_seconds: The timeout for this run. By default a value of 0 is used
which means to have no timeout.
This field will be templated.
@@ -247,6 +296,11 @@ class DatabricksSubmitRunOperator(BaseOperator):
might be a floating point number).
:param databricks_retry_args: An optional dictionary with arguments passed to ``tenacity.Retrying`` class.
:param do_xcom_push: Whether we should push run_id and run_page_url to xcom.
+ :param git_source: Optional specification of a remote git repository from which
+ supported task types are retrieved.
+
+ .. seealso::
+ https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunsSubmit
"""
# Used in airflow.models.BaseOperator
@@ -255,6 +309,7 @@ class DatabricksSubmitRunOperator(BaseOperator):
# Databricks brand color (blue) under white text
ui_color = '#1CB1C2'
ui_fgcolor = '#fff'
+ operator_extra_links = (DatabricksJobRunLink(),)
def __init__(
self,
@@ -276,10 +331,11 @@ def __init__(
databricks_retry_limit: int = 3,
databricks_retry_delay: int = 1,
databricks_retry_args: Optional[Dict[Any, Any]] = None,
- do_xcom_push: bool = False,
+ do_xcom_push: bool = True,
idempotency_token: Optional[str] = None,
access_control_list: Optional[List[Dict[str, str]]] = None,
wait_for_termination: bool = True,
+ git_source: Optional[Dict[str, str]] = None,
**kwargs,
) -> None:
"""Creates a new ``DatabricksSubmitRunOperator``."""
@@ -319,8 +375,10 @@ def __init__(
self.json['idempotency_token'] = idempotency_token
if access_control_list is not None:
self.json['access_control_list'] = access_control_list
+ if git_source is not None:
+ self.json['git_source'] = git_source
- self.json = _deep_string_coerce(self.json)
+ self.json = deep_string_coerce(self.json)
# This variable will be used in case our task gets killed.
self.run_id: Optional[int] = None
self.do_xcom_push = do_xcom_push
@@ -349,6 +407,18 @@ def on_kill(self):
self.log.error('Error: Task: %s with invalid run_id was requested to be cancelled.', self.task_id)
+class DatabricksSubmitRunDeferrableOperator(DatabricksSubmitRunOperator):
+ """Deferrable version of ``DatabricksSubmitRunOperator``"""
+
+ def execute(self, context):
+ hook = self._get_hook()
+ self.run_id = hook.submit_run(self.json)
+ _handle_deferrable_databricks_operator_execution(self, hook, self.log, context)
+
+ def execute_complete(self, context: Optional[dict], event: dict):
+ _handle_deferrable_databricks_operator_completion(event, self.log)
+
+
class DatabricksRunNowOperator(BaseOperator):
"""
Runs an existing Spark job run to Databricks using the
@@ -409,9 +479,10 @@ class DatabricksRunNowOperator(BaseOperator):
- ``json``
- ``notebook_params``
- ``python_params``
+ - ``python_named_parameters``
- ``jar_params``
- ``spark_submit_params``
-
+ - ``idempotency_token``
:param job_id: the job_id of the existing Databricks job.
This field will be templated.
@@ -448,12 +519,18 @@ class DatabricksRunNowOperator(BaseOperator):
:param python_params: A list of parameters for jobs with python tasks,
e.g. "python_params": ["john doe", "35"].
The parameters will be passed to python file as command line parameters.
- If specified upon run-now, it would overwrite the parameters specified in
- job setting.
+ If specified upon run-now, it would overwrite the parameters specified in job setting.
The json representation of this field (i.e. {"python_params":["john doe","35"]})
cannot exceed 10,000 bytes.
This field will be templated.
+ .. seealso::
+ https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunNow
+ :param python_named_parameters: A list of parameters for jobs with python wheel tasks,
+ e.g. "python_named_parameters": {"name": "john doe", "age": "35"}.
+ If specified upon run-now, it would overwrite the parameters specified in job setting.
+ This field will be templated.
+
.. seealso::
https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunNow
:param jar_params: A list of parameters for jobs with JAR tasks,
@@ -477,9 +554,9 @@ class DatabricksRunNowOperator(BaseOperator):
.. seealso::
https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunNow
- :param timeout_seconds: The timeout for this run. By default a value of 0 is used
- which means to have no timeout.
- This field will be templated.
+ :param idempotency_token: an optional token that can be used to guarantee the idempotency of job run
+ requests. If a run with the provided token already exists, the request does not create a new run but
+ returns the ID of the existing run instead. This token must have at most 64 characters.
:param databricks_conn_id: Reference to the :ref:`Databricks connection `.
By default and in the common case this will be ``databricks_default``. To use
token based authentication, provide the key ``token`` in the extra field for the
@@ -488,8 +565,11 @@ class DatabricksRunNowOperator(BaseOperator):
this run. By default the operator will poll every 30 seconds.
:param databricks_retry_limit: Amount of times retry if the Databricks backend is
unreachable. Its value must be greater than or equal to 1.
+ :param databricks_retry_delay: Number of seconds to wait between retries (it
+ might be a floating point number).
:param databricks_retry_args: An optional dictionary with arguments passed to ``tenacity.Retrying`` class.
:param do_xcom_push: Whether we should push run_id and run_page_url to xcom.
+ :param wait_for_termination: if we should wait for termination of the job run. ``True`` by default.
"""
# Used in airflow.models.BaseOperator
@@ -498,6 +578,7 @@ class DatabricksRunNowOperator(BaseOperator):
# Databricks brand color (blue) under white text
ui_color = '#1CB1C2'
ui_fgcolor = '#fff'
+ operator_extra_links = (DatabricksJobRunLink(),)
def __init__(
self,
@@ -509,12 +590,14 @@ def __init__(
python_params: Optional[List[str]] = None,
jar_params: Optional[List[str]] = None,
spark_submit_params: Optional[List[str]] = None,
+ python_named_parameters: Optional[Dict[str, str]] = None,
+ idempotency_token: Optional[str] = None,
databricks_conn_id: str = 'databricks_default',
polling_period_seconds: int = 30,
databricks_retry_limit: int = 3,
databricks_retry_delay: int = 1,
databricks_retry_args: Optional[Dict[Any, Any]] = None,
- do_xcom_push: bool = False,
+ do_xcom_push: bool = True,
wait_for_termination: bool = True,
**kwargs,
) -> None:
@@ -538,12 +621,16 @@ def __init__(
self.json['notebook_params'] = notebook_params
if python_params is not None:
self.json['python_params'] = python_params
+ if python_named_parameters is not None:
+ self.json['python_named_parameters'] = python_named_parameters
if jar_params is not None:
self.json['jar_params'] = jar_params
if spark_submit_params is not None:
self.json['spark_submit_params'] = spark_submit_params
+ if idempotency_token is not None:
+ self.json['idempotency_token'] = idempotency_token
- self.json = _deep_string_coerce(self.json)
+ self.json = deep_string_coerce(self.json)
# This variable will be used in case our task gets killed.
self.run_id: Optional[int] = None
self.do_xcom_push = do_xcom_push
@@ -576,3 +663,15 @@ def on_kill(self):
)
else:
self.log.error('Error: Task: %s with invalid run_id was requested to be cancelled.', self.task_id)
+
+
+class DatabricksRunNowDeferrableOperator(DatabricksRunNowOperator):
+ """Deferrable version of ``DatabricksRunNowOperator``"""
+
+ def execute(self, context):
+ hook = self._get_hook()
+ self.run_id = hook.run_now(self.json)
+ _handle_deferrable_databricks_operator_execution(self, hook, self.log, context)
+
+ def execute_complete(self, context: Optional[dict], event: dict):
+ _handle_deferrable_databricks_operator_completion(event, self.log)
diff --git a/airflow/providers/databricks/operators/databricks_repos.py b/airflow/providers/databricks/operators/databricks_repos.py
index fc50730d03d06..97b46b8e244d6 100644
--- a/airflow/providers/databricks/operators/databricks_repos.py
+++ b/airflow/providers/databricks/operators/databricks_repos.py
@@ -17,8 +17,9 @@
# under the License.
#
"""This module contains Databricks operators."""
-
+import re
from typing import TYPE_CHECKING, Optional, Sequence
+from urllib.parse import urlparse
from airflow.exceptions import AirflowException
from airflow.models import BaseOperator
@@ -28,12 +29,142 @@
from airflow.utils.context import Context
+class DatabricksReposCreateOperator(BaseOperator):
+ """
+ Creates a Databricks Repo
+ using
+ `POST api/2.0/repos `_
+ API endpoint and optionally checking it out to a specific branch or tag.
+
+ :param git_url: Required HTTPS URL of a Git repository
+ :param git_provider: Optional name of Git provider. Must be provided if we can't guess its name from URL.
+ :param repo_path: optional path for a repository. Must be in the format ``/Repos/{folder}/{repo-name}``.
+ If not specified, it will be created in the user's directory.
+ :param branch: optional name of branch to check out.
+ :param tag: optional name of tag to checkout.
+ :param ignore_existing_repo: don't throw exception if repository with given path already exists.
+ :param databricks_conn_id: Reference to the :ref:`Databricks connection `.
+ By default and in the common case this will be ``databricks_default``. To use
+ token based authentication, provide the key ``token`` in the extra field for the
+ connection and create the key ``host`` and leave the ``host`` field empty.
+ :param databricks_retry_limit: Amount of times retry if the Databricks backend is
+ unreachable. Its value must be greater than or equal to 1.
+ :param databricks_retry_delay: Number of seconds to wait between retries (it
+ might be a floating point number).
+ """
+
+ # Used in airflow.models.BaseOperator
+ template_fields: Sequence[str] = ('repo_path', 'tag', 'branch')
+
+ __git_providers__ = {
+ "github.com": "gitHub",
+ "dev.azure.com": "azureDevOpsServices",
+ "gitlab.com": "gitLab",
+ "bitbucket.org": "bitbucketCloud",
+ }
+ __aws_code_commit_regexp__ = re.compile(r"^git-codecommit\.[^.]+\.amazonaws.com$")
+ __repos_path_regexp__ = re.compile(r"/Repos/[^/]+/[^/]+/?$")
+
+ def __init__(
+ self,
+ *,
+ git_url: str,
+ git_provider: Optional[str] = None,
+ branch: Optional[str] = None,
+ tag: Optional[str] = None,
+ repo_path: Optional[str] = None,
+ ignore_existing_repo: bool = False,
+ databricks_conn_id: str = 'databricks_default',
+ databricks_retry_limit: int = 3,
+ databricks_retry_delay: int = 1,
+ **kwargs,
+ ) -> None:
+ """Creates a new ``DatabricksReposCreateOperator``."""
+ super().__init__(**kwargs)
+ self.databricks_conn_id = databricks_conn_id
+ self.databricks_retry_limit = databricks_retry_limit
+ self.databricks_retry_delay = databricks_retry_delay
+ self.git_url = git_url
+ self.ignore_existing_repo = ignore_existing_repo
+ if git_provider is None:
+ self.git_provider = self.__detect_repo_provider__(git_url)
+ if self.git_provider is None:
+ raise AirflowException(
+ f"git_provider isn't specified and couldn't be guessed for URL {git_url}"
+ )
+ else:
+ self.git_provider = git_provider
+ self.repo_path = repo_path
+ if branch is not None and tag is not None:
+ raise AirflowException("Only one of branch or tag should be provided, but not both")
+ self.branch = branch
+ self.tag = tag
+
+ @staticmethod
+ def __detect_repo_provider__(url):
+ provider = None
+ try:
+ netloc = urlparse(url).netloc
+ idx = netloc.rfind("@")
+ if idx != -1:
+ netloc = netloc[(idx + 1) :]
+ netloc = netloc.lower()
+ provider = DatabricksReposCreateOperator.__git_providers__.get(netloc)
+ if provider is None and DatabricksReposCreateOperator.__aws_code_commit_regexp__.match(netloc):
+ provider = "awsCodeCommit"
+ except ValueError:
+ pass
+ return provider
+
+ def _get_hook(self) -> DatabricksHook:
+ return DatabricksHook(
+ self.databricks_conn_id,
+ retry_limit=self.databricks_retry_limit,
+ retry_delay=self.databricks_retry_delay,
+ )
+
+ def execute(self, context: 'Context'):
+ """
+ Creates a Databricks Repo
+
+ :param context: context
+ :return: Repo ID
+ """
+ payload = {
+ "url": self.git_url,
+ "provider": self.git_provider,
+ }
+ if self.repo_path is not None:
+ if not self.__repos_path_regexp__.match(self.repo_path):
+ raise AirflowException(
+ f"repo_path should have form of /Repos/{{folder}}/{{repo-name}}, got '{self.repo_path}'"
+ )
+ payload["path"] = self.repo_path
+ hook = self._get_hook()
+ existing_repo_id = None
+ if self.repo_path is not None:
+ existing_repo_id = hook.get_repo_by_path(self.repo_path)
+ if existing_repo_id is not None and not self.ignore_existing_repo:
+ raise AirflowException(f"Repo with path '{self.repo_path}' already exists")
+ if existing_repo_id is None:
+ result = hook.create_repo(payload)
+ repo_id = result["id"]
+ else:
+ repo_id = existing_repo_id
+ # update repo if necessary
+ if self.branch is not None:
+ hook.update_repo(str(repo_id), {'branch': str(self.branch)})
+ elif self.tag is not None:
+ hook.update_repo(str(repo_id), {'tag': str(self.tag)})
+
+ return repo_id
+
+
class DatabricksReposUpdateOperator(BaseOperator):
"""
- Updates specified repository to a given branch or tag using
- `api/2.0/repos/
- `_
- API endpoint.
+ Updates specified repository to a given branch or tag
+ using `PATCH api/2.0/repos
+ `_ API endpoint.
:param branch: optional name of branch to update to. Should be specified if ``tag`` is omitted
:param tag: optional name of tag to update to. Should be specified if ``branch`` is omitted
@@ -64,7 +195,7 @@ def __init__(
databricks_retry_delay: int = 1,
**kwargs,
) -> None:
- """Creates a new ``DatabricksSubmitRunOperator``."""
+ """Creates a new ``DatabricksReposUpdateOperator``."""
super().__init__(**kwargs)
self.databricks_conn_id = databricks_conn_id
self.databricks_retry_limit = databricks_retry_limit
@@ -76,7 +207,7 @@ def __init__(
if repo_id is not None and repo_path is not None:
raise AirflowException("Only one of repo_id or repo_path should be provided, but not both")
if repo_id is None and repo_path is None:
- raise AirflowException("One of repo_id repo_path tag should be provided")
+ raise AirflowException("One of repo_id or repo_path should be provided")
self.repo_path = repo_path
self.repo_id = repo_id
self.branch = branch
@@ -102,3 +233,63 @@ def execute(self, context: 'Context'):
result = hook.update_repo(str(self.repo_id), payload)
return result['head_commit_id']
+
+
+class DatabricksReposDeleteOperator(BaseOperator):
+ """
+ Deletes specified repository
+ using `DELETE api/2.0/repos
+ `_ API endpoint.
+
+ :param repo_id: optional ID of existing repository. Should be specified if ``repo_path`` is omitted
+ :param repo_path: optional path of existing repository. Should be specified if ``repo_id`` is omitted
+ :param databricks_conn_id: Reference to the :ref:`Databricks connection `.
+ By default and in the common case this will be ``databricks_default``. To use
+ token based authentication, provide the key ``token`` in the extra field for the
+ connection and create the key ``host`` and leave the ``host`` field empty.
+ :param databricks_retry_limit: Amount of times retry if the Databricks backend is
+ unreachable. Its value must be greater than or equal to 1.
+ :param databricks_retry_delay: Number of seconds to wait between retries (it
+ might be a floating point number).
+ """
+
+ # Used in airflow.models.BaseOperator
+ template_fields: Sequence[str] = ('repo_path',)
+
+ def __init__(
+ self,
+ *,
+ repo_id: Optional[str] = None,
+ repo_path: Optional[str] = None,
+ databricks_conn_id: str = 'databricks_default',
+ databricks_retry_limit: int = 3,
+ databricks_retry_delay: int = 1,
+ **kwargs,
+ ) -> None:
+ """Creates a new ``DatabricksReposDeleteOperator``."""
+ super().__init__(**kwargs)
+ self.databricks_conn_id = databricks_conn_id
+ self.databricks_retry_limit = databricks_retry_limit
+ self.databricks_retry_delay = databricks_retry_delay
+ if repo_id is not None and repo_path is not None:
+ raise AirflowException("Only one of repo_id or repo_path should be provided, but not both")
+ if repo_id is None and repo_path is None:
+ raise AirflowException("One of repo_id repo_path tag should be provided")
+ self.repo_path = repo_path
+ self.repo_id = repo_id
+
+ def _get_hook(self) -> DatabricksHook:
+ return DatabricksHook(
+ self.databricks_conn_id,
+ retry_limit=self.databricks_retry_limit,
+ retry_delay=self.databricks_retry_delay,
+ )
+
+ def execute(self, context: 'Context'):
+ hook = self._get_hook()
+ if self.repo_path is not None:
+ self.repo_id = hook.get_repo_by_path(self.repo_path)
+ if self.repo_id is None:
+ raise AirflowException(f"Can't find Repo ID for path '{self.repo_path}'")
+
+ hook.delete_repo(str(self.repo_id))
diff --git a/airflow/providers/databricks/operators/databricks_sql.py b/airflow/providers/databricks/operators/databricks_sql.py
index 48a6713517b46..9e6298bc21263 100644
--- a/airflow/providers/databricks/operators/databricks_sql.py
+++ b/airflow/providers/databricks/operators/databricks_sql.py
@@ -20,9 +20,9 @@
import csv
import json
-from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Mapping, Optional, Sequence, Union
+from typing import TYPE_CHECKING, Any, Dict, Iterable, List, Mapping, Optional, Sequence, Tuple, Union
-from databricks.sql.common import ParamEscaper
+from databricks.sql.utils import ParamEscaper
from airflow.exceptions import AirflowException
from airflow.models import BaseOperator
@@ -53,13 +53,18 @@ class DatabricksSqlOperator(BaseOperator):
:param parameters: (optional) the parameters to render the SQL query with.
:param session_configuration: An optional dictionary of Spark session parameters. Defaults to None.
If not specified, it could be specified in the Databricks connection's extra parameters.
+ :param client_parameters: Additional parameters internal to Databricks SQL Connector parameters
+ :param http_headers: An optional list of (k, v) pairs that will be set as HTTP headers on every request.
+ (templated)
+ :param catalog: An optional initial catalog to use. Requires DBR version 9.0+ (templated)
+ :param schema: An optional initial schema to use. Requires DBR version 9.0+ (templated)
:param output_path: optional string specifying the file to which write selected data. (templated)
:param output_format: format of output data if ``output_path` is specified.
Possible values are ``csv``, ``json``, ``jsonl``. Default is ``csv``.
:param csv_params: parameters that will be passed to the ``csv.DictWriter`` class used to write CSV data.
"""
- template_fields: Sequence[str] = ('sql', '_output_path')
+ template_fields: Sequence[str] = ('sql', '_output_path', 'schema', 'catalog', 'http_headers')
template_ext: Sequence[str] = ('.sql',)
template_fields_renderers = {'sql': 'sql'}
@@ -72,10 +77,14 @@ def __init__(
sql_endpoint_name: Optional[str] = None,
parameters: Optional[Union[Mapping, Iterable]] = None,
session_configuration=None,
+ http_headers: Optional[List[Tuple[str, str]]] = None,
+ catalog: Optional[str] = None,
+ schema: Optional[str] = None,
do_xcom_push: bool = False,
output_path: Optional[str] = None,
output_format: str = 'csv',
csv_params: Optional[Dict[str, Any]] = None,
+ client_parameters: Optional[Dict[str, Any]] = None,
**kwargs,
) -> None:
"""Creates a new ``DatabricksSqlOperator``."""
@@ -90,6 +99,10 @@ def __init__(
self.parameters = parameters
self.do_xcom_push = do_xcom_push
self.session_config = session_configuration
+ self.http_headers = http_headers
+ self.catalog = catalog
+ self.schema = schema
+ self.client_parameters = client_parameters or {}
def _get_hook(self) -> DatabricksSqlHook:
return DatabricksSqlHook(
@@ -97,6 +110,10 @@ def _get_hook(self) -> DatabricksSqlHook:
http_path=self._http_path,
session_configuration=self.session_config,
sql_endpoint_name=self._sql_endpoint_name,
+ http_headers=self.http_headers,
+ catalog=self.catalog,
+ schema=self.schema,
+ **self.client_parameters,
)
def _format_output(self, schema, results):
@@ -165,11 +182,18 @@ class DatabricksCopyIntoOperator(BaseOperator):
or ``sql_endpoint_name`` must be specified.
:param sql_endpoint_name: Optional name of Databricks SQL Endpoint.
If not specified, ``http_path`` must be provided as described above.
+ :param session_configuration: An optional dictionary of Spark session parameters. Defaults to None.
+ If not specified, it could be specified in the Databricks connection's extra parameters.
+ :param http_headers: An optional list of (k, v) pairs that will be set as HTTP headers on every request
+ :param catalog: An optional initial catalog to use. Requires DBR version 9.0+
+ :param schema: An optional initial schema to use. Requires DBR version 9.0+
+ :param client_parameters: Additional parameters internal to Databricks SQL Connector parameters
:param files: optional list of files to import. Can't be specified together with ``pattern``. (templated)
:param pattern: optional regex string to match file names to import.
Can't be specified together with ``files``.
:param expression_list: optional string that will be used in the ``SELECT`` expression.
- :param credential: optional credential configuration for authentication against a specified location.
+ :param credential: optional credential configuration for authentication against a source location.
+ :param storage_credential: optional Unity Catalog storage credential for destination.
:param encryption: optional encryption configuration for a specified location.
:param format_options: optional dictionary with options specific for a given file format.
:param force_copy: optional bool to control forcing of data import
@@ -195,10 +219,15 @@ def __init__(
http_path: Optional[str] = None,
sql_endpoint_name: Optional[str] = None,
session_configuration=None,
+ http_headers: Optional[List[Tuple[str, str]]] = None,
+ client_parameters: Optional[Dict[str, Any]] = None,
+ catalog: Optional[str] = None,
+ schema: Optional[str] = None,
files: Optional[List[str]] = None,
pattern: Optional[str] = None,
expression_list: Optional[str] = None,
credential: Optional[Dict[str, str]] = None,
+ storage_credential: Optional[str] = None,
encryption: Optional[Dict[str, str]] = None,
format_options: Optional[Dict[str, str]] = None,
force_copy: Optional[bool] = None,
@@ -224,13 +253,18 @@ def __init__(
self._sql_endpoint_name = sql_endpoint_name
self.session_config = session_configuration
self._table_name = table_name
+ self._catalog = catalog
+ self._schema = schema
self._file_location = file_location
self._expression_list = expression_list
self._credential = credential
+ self._storage_credential = storage_credential
self._encryption = encryption
self._format_options = format_options
self._copy_options = copy_options or {}
self._validate = validate
+ self._http_headers = http_headers
+ self._client_parameters = client_parameters or {}
if force_copy is not None:
self._copy_options["force"] = 'true' if force_copy else 'false'
@@ -240,6 +274,10 @@ def _get_hook(self) -> DatabricksSqlHook:
http_path=self._http_path,
session_configuration=self.session_config,
sql_endpoint_name=self._sql_endpoint_name,
+ http_headers=self._http_headers,
+ catalog=self._catalog,
+ schema=self._schema,
+ **self._client_parameters,
)
@staticmethod
@@ -280,6 +318,9 @@ def _create_sql_query(self) -> str:
files_or_pattern = f"FILES = {escaper.escape_item(self._files)}\n"
format_options = self._generate_options("FORMAT_OPTIONS", escaper, self._format_options) + "\n"
copy_options = self._generate_options("COPY_OPTIONS", escaper, self._copy_options) + "\n"
+ storage_cred = ""
+ if self._storage_credential:
+ storage_cred = f" WITH (CREDENTIAL {self._storage_credential})"
validation = ""
if self._validate is not None:
if isinstance(self._validate, bool):
@@ -292,9 +333,11 @@ def _create_sql_query(self) -> str:
)
validation = f"VALIDATE {self._validate} ROWS\n"
else:
- raise AirflowException("Incorrect data type for validate parameter: " + type(self._validate))
+ raise AirflowException(
+ "Incorrect data type for validate parameter: " + str(type(self._validate))
+ )
# TODO: think on how to make sure that table_name and expression_list aren't used for SQL injection
- sql = f"""COPY INTO {self._table_name}
+ sql = f"""COPY INTO {self._table_name}{storage_cred}
FROM {location}
FILEFORMAT = {self._file_format}
{validation}{files_or_pattern}{format_options}{copy_options}
diff --git a/airflow/providers/databricks/provider.yaml b/airflow/providers/databricks/provider.yaml
index ae307c7b79cff..470209604572d 100644
--- a/airflow/providers/databricks/provider.yaml
+++ b/airflow/providers/databricks/provider.yaml
@@ -22,6 +22,9 @@ description: |
`Databricks `.
+ :param polling_period_seconds: Controls the rate of the poll for the result of this run.
+ By default, the trigger will poll every 30 seconds.
+ """
+
+ def __init__(self, run_id: int, databricks_conn_id: str, polling_period_seconds: int = 30) -> None:
+ super().__init__()
+ self.run_id = run_id
+ self.databricks_conn_id = databricks_conn_id
+ self.polling_period_seconds = polling_period_seconds
+ self.hook = DatabricksHook(databricks_conn_id)
+
+ def serialize(self) -> Tuple[str, Dict[str, Any]]:
+ return (
+ 'airflow.providers.databricks.triggers.databricks.DatabricksExecutionTrigger',
+ {
+ 'run_id': self.run_id,
+ 'databricks_conn_id': self.databricks_conn_id,
+ 'polling_period_seconds': self.polling_period_seconds,
+ },
+ )
+
+ async def run(self):
+ async with self.hook:
+ run_page_url = await self.hook.a_get_run_page_url(self.run_id)
+ while True:
+ run_state = await self.hook.a_get_run_state(self.run_id)
+ if run_state.is_terminal:
+ yield TriggerEvent(
+ {
+ 'run_id': self.run_id,
+ 'run_state': run_state.to_json(),
+ 'run_page_url': run_page_url,
+ }
+ )
+ break
+ else:
+ await asyncio.sleep(self.polling_period_seconds)
diff --git a/airflow/providers/databricks/utils/__init__.py b/airflow/providers/databricks/utils/__init__.py
new file mode 100644
index 0000000000000..13a83393a9124
--- /dev/null
+++ b/airflow/providers/databricks/utils/__init__.py
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
diff --git a/airflow/providers/databricks/utils/databricks.py b/airflow/providers/databricks/utils/databricks.py
new file mode 100644
index 0000000000000..96935d806344e
--- /dev/null
+++ b/airflow/providers/databricks/utils/databricks.py
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from typing import Union
+
+from airflow.exceptions import AirflowException
+from airflow.providers.databricks.hooks.databricks import RunState
+
+
+def deep_string_coerce(content, json_path: str = 'json') -> Union[str, list, dict]:
+ """
+ Coerces content or all values of content if it is a dict to a string. The
+ function will throw if content contains non-string or non-numeric types.
+ The reason why we have this function is because the ``self.json`` field must be a
+ dict with only string values. This is because ``render_template`` will fail
+ for numerical values.
+ """
+ coerce = deep_string_coerce
+ if isinstance(content, str):
+ return content
+ elif isinstance(
+ content,
+ (
+ int,
+ float,
+ ),
+ ):
+ # Databricks can tolerate either numeric or string types in the API backend.
+ return str(content)
+ elif isinstance(content, (list, tuple)):
+ return [coerce(e, f'{json_path}[{i}]') for i, e in enumerate(content)]
+ elif isinstance(content, dict):
+ return {k: coerce(v, f'{json_path}[{k}]') for k, v in list(content.items())}
+ else:
+ param_type = type(content)
+ msg = f'Type {param_type} used for parameter {json_path} is not a number or a string'
+ raise AirflowException(msg)
+
+
+def validate_trigger_event(event: dict):
+ """
+ Validates correctness of the event
+ received from :class:`~airflow.providers.databricks.triggers.databricks.DatabricksExecutionTrigger`
+ """
+ keys_to_check = ['run_id', 'run_page_url', 'run_state']
+ for key in keys_to_check:
+ if key not in event:
+ raise AirflowException(f'Could not find `{key}` in the event: {event}')
+
+ try:
+ RunState.from_json(event['run_state'])
+ except Exception:
+ raise AirflowException(f'Run state returned by the Trigger is incorrect: {event["run_state"]}')
diff --git a/airflow/providers/datadog/.latest-doc-only-change.txt b/airflow/providers/datadog/.latest-doc-only-change.txt
index 28124098645cf..ab24993f57139 100644
--- a/airflow/providers/datadog/.latest-doc-only-change.txt
+++ b/airflow/providers/datadog/.latest-doc-only-change.txt
@@ -1 +1 @@
-6c3a67d4fccafe4ab6cd9ec8c7bacf2677f17038
+8b6b0848a3cacf9999477d6af4d2a87463f03026
diff --git a/airflow/providers/datadog/CHANGELOG.rst b/airflow/providers/datadog/CHANGELOG.rst
index 5cc4b3b066711..c6d15452d1209 100644
--- a/airflow/providers/datadog/CHANGELOG.rst
+++ b/airflow/providers/datadog/CHANGELOG.rst
@@ -19,6 +19,14 @@
Changelog
---------
+2.0.4
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix mistakenly added install_requires for all providers (#22382)``
+
2.0.3
.....
diff --git a/airflow/providers/datadog/hooks/datadog.py b/airflow/providers/datadog/hooks/datadog.py
index dabe9111f9611..574b85efc0bb0 100644
--- a/airflow/providers/datadog/hooks/datadog.py
+++ b/airflow/providers/datadog/hooks/datadog.py
@@ -19,7 +19,7 @@
import time
from typing import Any, Dict, List, Optional, Union
-from datadog import api, initialize
+from datadog import api, initialize # type: ignore[attr-defined]
from airflow.exceptions import AirflowException
from airflow.hooks.base import BaseHook
@@ -43,6 +43,7 @@ def __init__(self, datadog_conn_id: str = 'datadog_default') -> None:
conn = self.get_connection(datadog_conn_id)
self.api_key = conn.extra_dejson.get('api_key', None)
self.app_key = conn.extra_dejson.get('app_key', None)
+ self.api_host = conn.extra_dejson.get('api_host', None)
self.source_type_name = conn.extra_dejson.get('source_type_name', None)
# If the host is populated, it will use that hostname instead.
@@ -53,7 +54,7 @@ def __init__(self, datadog_conn_id: str = 'datadog_default') -> None:
raise AirflowException("api_key must be specified in the Datadog connection details")
self.log.info("Setting up api keys for Datadog")
- initialize(api_key=self.api_key, app_key=self.app_key)
+ initialize(api_key=self.api_key, app_key=self.app_key, api_host=self.api_host)
def validate_response(self, response: Dict[str, Any]) -> None:
"""Validate Datadog response"""
diff --git a/airflow/providers/datadog/provider.yaml b/airflow/providers/datadog/provider.yaml
index b2b2dea7cd0e0..9276349e242ad 100644
--- a/airflow/providers/datadog/provider.yaml
+++ b/airflow/providers/datadog/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Datadog `__
versions:
+ - 3.0.3
+ - 3.0.2
- 3.0.1
- 3.0.0
- 2.2.0
diff --git a/airflow/providers/exasol/CHANGELOG.rst b/airflow/providers/exasol/CHANGELOG.rst
index f493309b2bd49..5cd54d10a1def 100644
--- a/airflow/providers/exasol/CHANGELOG.rst
+++ b/airflow/providers/exasol/CHANGELOG.rst
@@ -19,6 +19,14 @@
Changelog
---------
+2.1.3
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix mistakenly added install_requires for all providers (#22382)``
+
2.1.2
.....
diff --git a/airflow/providers/exasol/hooks/exasol.py b/airflow/providers/exasol/hooks/exasol.py
index c2f8e13d99810..2233ce1e2c347 100644
--- a/airflow/providers/exasol/hooks/exasol.py
+++ b/airflow/providers/exasol/hooks/exasol.py
@@ -134,7 +134,7 @@ def export_to_file(
def run(
self, sql: Union[str, list], autocommit: bool = False, parameters: Optional[dict] = None, handler=None
- ) -> None:
+ ) -> Optional[list]:
"""
Runs a command or a list of commands. Pass a list of sql
statements to the sql parameter to get them to execute
@@ -150,6 +150,11 @@ def run(
if isinstance(sql, str):
sql = [sql]
+ if sql:
+ self.log.debug("Executing %d statements against Exasol DB", len(sql))
+ else:
+ raise ValueError("List of SQL statements is empty")
+
with closing(self.get_conn()) as conn:
if self.supports_autocommit:
self.set_autocommit(conn, autocommit)
@@ -157,12 +162,23 @@ def run(
for query in sql:
self.log.info(query)
with closing(conn.execute(query, parameters)) as cur:
+ results = []
+
+ if handler is not None:
+ cur = handler(cur)
+
+ for row in cur:
+ self.log.info("Statement execution info - %s", row)
+ results.append(row)
+
self.log.info(cur.row_count)
# If autocommit was set to False for db that supports autocommit,
- # or if db does not supports autocommit, we do a manual commit.
+ # or if db does not support autocommit, we do a manual commit.
if not self.get_autocommit(conn):
conn.commit()
+ return results
+
def set_autocommit(self, conn, autocommit: bool) -> None:
"""
Sets the autocommit flag on the connection
diff --git a/airflow/providers/exasol/provider.yaml b/airflow/providers/exasol/provider.yaml
index 7c50479bd4813..6e6d8cbd7ca92 100644
--- a/airflow/providers/exasol/provider.yaml
+++ b/airflow/providers/exasol/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Exasol `__
versions:
+ - 2.1.3
- 2.1.2
- 2.1.1
- 2.1.0
diff --git a/airflow/providers/facebook/CHANGELOG.rst b/airflow/providers/facebook/CHANGELOG.rst
index 28e82d7d3136f..e0a8dbe15bc2f 100644
--- a/airflow/providers/facebook/CHANGELOG.rst
+++ b/airflow/providers/facebook/CHANGELOG.rst
@@ -19,6 +19,14 @@
Changelog
---------
+2.2.3
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix mistakenly added install_requires for all providers (#22382)``
+
2.2.2
.....
diff --git a/airflow/providers/facebook/provider.yaml b/airflow/providers/facebook/provider.yaml
index 419346f3988fe..8b2471e64b0ee 100644
--- a/airflow/providers/facebook/provider.yaml
+++ b/airflow/providers/facebook/provider.yaml
@@ -22,6 +22,7 @@ description: |
`Facebook Ads `__
versions:
+ - 2.1.2
- 2.1.1
- 2.1.0
- 2.0.1
diff --git a/airflow/providers/github/.latest-doc-only-change.txt b/airflow/providers/github/.latest-doc-only-change.txt
new file mode 100644
index 0000000000000..ab24993f57139
--- /dev/null
+++ b/airflow/providers/github/.latest-doc-only-change.txt
@@ -0,0 +1 @@
+8b6b0848a3cacf9999477d6af4d2a87463f03026
diff --git a/airflow/providers/github/CHANGELOG.rst b/airflow/providers/github/CHANGELOG.rst
index a9066632d5cce..c800f7904856d 100644
--- a/airflow/providers/github/CHANGELOG.rst
+++ b/airflow/providers/github/CHANGELOG.rst
@@ -19,6 +19,14 @@
Changelog
+1.0.3
+.....
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix mistakenly added install_requires for all providers (#22382)``
+
1.0.2
.....
diff --git a/airflow/providers/github/example_dags/example_github.py b/airflow/providers/github/example_dags/example_github.py
index 7f3fd8578b839..642bdd43e26dc 100644
--- a/airflow/providers/github/example_dags/example_github.py
+++ b/airflow/providers/github/example_dags/example_github.py
@@ -56,10 +56,10 @@ def tag_checker(repo: Any, tag_name: str) -> Optional[bool]:
all_tags = [x.name for x in repo.get_tags()]
result = tag_name in all_tags
- except GithubException as github_error:
+ except GithubException as github_error: # type: ignore[misc]
raise AirflowException(f"Failed to execute GithubSensor, error: {str(github_error)}")
except Exception as e:
- raise AirflowException(f"Github operator error: {str(e)}")
+ raise AirflowException(f"GitHub operator error: {str(e)}")
return result
diff --git a/airflow/providers/github/hooks/github.py b/airflow/providers/github/hooks/github.py
index 3514b77d35b7e..07a8566a7f575 100644
--- a/airflow/providers/github/hooks/github.py
+++ b/airflow/providers/github/hooks/github.py
@@ -69,11 +69,11 @@ def get_ui_field_behaviour() -> Dict:
return {
"hidden_fields": ['schema', 'port', 'login', 'extra'],
"relabeling": {
- 'host': 'Github Enterprise Url (Optional)',
- 'password': 'Github Access Token',
+ 'host': 'GitHub Enterprise Url (Optional)',
+ 'password': 'GitHub Access Token',
},
"placeholders": {
- 'host': 'https://{hostname}/api/v3 (for Github Enterprise Connection)',
+ 'host': 'https://{hostname}/api/v3 (for GitHub Enterprise Connection)',
'password': 'token credentials auth',
},
}
diff --git a/airflow/providers/github/provider.yaml b/airflow/providers/github/provider.yaml
index 7e91fe9166395..090176fcfdfc1 100644
--- a/airflow/providers/github/provider.yaml
+++ b/airflow/providers/github/provider.yaml
@@ -19,8 +19,9 @@
package-name: apache-airflow-providers-github
name: Github
description: |
- `Github -``, where timestamp is in
+ YYYY-MM-DDThh:mm:ss.sssZ ISO-8601 format. Inside of it files ``predictions_0001.``,
+ ``predictions_0002.``, ..., ``predictions_N.`` are created where
+ ```` depends on chosen ``predictions_format``, and N may equal 0001 and depends on the
+ total number of successfully predicted instances. If the Model has both ``instance`` and
+ ``prediction`` schemata defined then each such file contains predictions as per the
+ ``predictions_format``. If prediction for any instance failed (partially or completely), then an
+ additional ``errors_0001.``, ``errors_0002.``,..., ``errors_N.``
+ files are created (N depends on total number of failed predictions). These files contain the
+ failed instances, as per their schema, followed by an additional ``error`` field which as value
+ has ```google.rpc.Status`` `__ containing only ``code`` and ``message`` fields.
+ :param bigquery_destination_prefix: The BigQuery project location where the output is to be written
+ to. In the given project a new dataset is created with name
+ ``prediction__`` where is made BigQuery-dataset-name
+ compatible (for example, most special characters become underscores), and timestamp is in
+ YYYY_MM_DDThh_mm_ss_sssZ "based on ISO-8601" format. In the dataset two tables will be created,
+ ``predictions``, and ``errors``. If the Model has both ``instance`` and ``prediction`` schemata
+ defined then the tables have columns as follows: The ``predictions`` table contains instances for
+ which the prediction succeeded, it has columns as per a concatenation of the Model's instance and
+ prediction schemata. The ``errors`` table contains rows for which the prediction has failed, it
+ has instance columns, as per the instance schema, followed by a single "errors" column, which as
+ values has ```google.rpc.Status`` `__ represented as a STRUCT, and containing only
+ ``code`` and ``message``.
+ :param model_parameters: The parameters that govern the predictions. The schema of the parameters may
+ be specified via the Model's `parameters_schema_uri`.
+ :param machine_type: The type of machine for running batch prediction on dedicated resources. Not
+ specifying machine type will result in batch prediction job being run with automatic resources.
+ :param accelerator_type: The type of accelerator(s) that may be attached to the machine as per
+ `accelerator_count`. Only used if `machine_type` is set.
+ :param accelerator_count: The number of accelerators to attach to the `machine_type`. Only used if
+ `machine_type` is set.
+ :param starting_replica_count: The number of machine replicas used at the start of the batch
+ operation. If not set, Vertex AI decides starting number, not greater than `max_replica_count`.
+ Only used if `machine_type` is set.
+ :param max_replica_count: The maximum number of machine replicas the batch operation may be scaled
+ to. Only used if `machine_type` is set. Default is 10.
+ :param generate_explanation: Optional. Generate explanation along with the batch prediction results.
+ This will cause the batch prediction output to include explanations based on the
+ `prediction_format`:
+ - `bigquery`: output includes a column named `explanation`. The value is a struct that conforms
+ to the [aiplatform.gapic.Explanation] object.
+ - `jsonl`: The JSON objects on each line include an additional entry keyed `explanation`. The
+ value of the entry is a JSON object that conforms to the [aiplatform.gapic.Explanation] object.
+ - `csv`: Generating explanations for CSV format is not supported.
+ :param explanation_metadata: Optional. Explanation metadata configuration for this
+ BatchPredictionJob. Can be specified only if `generate_explanation` is set to `True`.
+ This value overrides the value of `Model.explanation_metadata`. All fields of
+ `explanation_metadata` are optional in the request. If a field of the `explanation_metadata`
+ object is not populated, the corresponding field of the `Model.explanation_metadata` object is
+ inherited. For more details, see `Ref docs `
+ :param explanation_parameters: Optional. Parameters to configure explaining for Model's predictions.
+ Can be specified only if `generate_explanation` is set to `True`.
+ This value overrides the value of `Model.explanation_parameters`. All fields of
+ `explanation_parameters` are optional in the request. If a field of the `explanation_parameters`
+ object is not populated, the corresponding field of the `Model.explanation_parameters` object is
+ inherited. For more details, see `Ref docs `
+ :param labels: Optional. The labels with user-defined metadata to organize your BatchPredictionJobs.
+ Label keys and values can be no longer than 64 characters (Unicode codepoints), can only contain
+ lowercase letters, numeric characters, underscores and dashes. International characters are
+ allowed. See https://goo.gl/xmQnxf for more information and examples of labels.
+ :param encryption_spec_key_name: Optional. The Cloud KMS resource identifier of the customer managed
+ encryption key used to protect the job. Has the form:
+ ``projects/my-project/locations/my-region/keyRings/my-kr/cryptoKeys/my-key``. The key needs to be
+ in the same region as where the compute resource is created.
+ If this is set, then all resources created by the BatchPredictionJob will be encrypted with the
+ provided encryption key.
+ Overrides encryption_spec_key_name set in aiplatform.init.
+ :param sync: Whether to execute this method synchronously. If False, this method will be executed in
+ concurrent Future and any downstream object will be immediately returned and synced when the
+ Future has completed.
+ """
+ self._batch_prediction_job = BatchPredictionJob.create(
+ job_display_name=job_display_name,
+ model_name=model_name,
+ instances_format=instances_format,
+ predictions_format=predictions_format,
+ gcs_source=gcs_source,
+ bigquery_source=bigquery_source,
+ gcs_destination_prefix=gcs_destination_prefix,
+ bigquery_destination_prefix=bigquery_destination_prefix,
+ model_parameters=model_parameters,
+ machine_type=machine_type,
+ accelerator_type=accelerator_type,
+ accelerator_count=accelerator_count,
+ starting_replica_count=starting_replica_count,
+ max_replica_count=max_replica_count,
+ generate_explanation=generate_explanation,
+ explanation_metadata=explanation_metadata,
+ explanation_parameters=explanation_parameters,
+ labels=labels,
+ project=project_id,
+ location=region,
+ credentials=self._get_credentials(),
+ encryption_spec_key_name=encryption_spec_key_name,
+ sync=sync,
+ )
+ return self._batch_prediction_job
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def delete_batch_prediction_job(
+ self,
+ project_id: str,
+ region: str,
+ batch_prediction_job: str,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> Operation:
+ """
+ Deletes a BatchPredictionJob. Can only be called on jobs that already finished.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param batch_prediction_job: The name of the BatchPredictionJob resource to be deleted.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_job_service_client(region)
+ name = client.batch_prediction_job_path(project_id, region, batch_prediction_job)
+
+ result = client.delete_batch_prediction_job(
+ request={
+ 'name': name,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def get_batch_prediction_job(
+ self,
+ project_id: str,
+ region: str,
+ batch_prediction_job: str,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> BatchPredictionJob:
+ """
+ Gets a BatchPredictionJob
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param batch_prediction_job: Required. The name of the BatchPredictionJob resource.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_job_service_client(region)
+ name = client.batch_prediction_job_path(project_id, region, batch_prediction_job)
+
+ result = client.get_batch_prediction_job(
+ request={
+ 'name': name,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def list_batch_prediction_jobs(
+ self,
+ project_id: str,
+ region: str,
+ filter: Optional[str] = None,
+ page_size: Optional[int] = None,
+ page_token: Optional[str] = None,
+ read_mask: Optional[str] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> ListBatchPredictionJobsPager:
+ """
+ Lists BatchPredictionJobs in a Location.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param filter: The standard list filter.
+ Supported fields:
+ - ``display_name`` supports = and !=.
+ - ``state`` supports = and !=.
+ - ``model_display_name`` supports = and !=
+ Some examples of using the filter are:
+ - ``state="JOB_STATE_SUCCEEDED" AND display_name="my_job"``
+ - ``state="JOB_STATE_RUNNING" OR display_name="my_job"``
+ - ``NOT display_name="my_job"``
+ - ``state="JOB_STATE_FAILED"``
+ :param page_size: The standard list page size.
+ :param page_token: The standard list page token.
+ :param read_mask: Mask specifying which fields to read.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_job_service_client(region)
+ parent = client.common_location_path(project_id, region)
+
+ result = client.list_batch_prediction_jobs(
+ request={
+ 'parent': parent,
+ 'filter': filter,
+ 'page_size': page_size,
+ 'page_token': page_token,
+ 'read_mask': read_mask,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
diff --git a/airflow/providers/google/cloud/hooks/vertex_ai/custom_job.py b/airflow/providers/google/cloud/hooks/vertex_ai/custom_job.py
index cd73da149aa0e..bd6987868512c 100644
--- a/airflow/providers/google/cloud/hooks/vertex_ai/custom_job.py
+++ b/airflow/providers/google/cloud/hooks/vertex_ai/custom_job.py
@@ -20,6 +20,8 @@
from typing import Dict, List, Optional, Sequence, Tuple, Union
+from google.api_core.client_options import ClientOptions
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
from google.api_core.operation import Operation
from google.api_core.retry import Retry
from google.cloud.aiplatform import (
@@ -69,10 +71,10 @@ def get_pipeline_service_client(
region: Optional[str] = None,
) -> PipelineServiceClient:
"""Returns PipelineServiceClient."""
- client_options = None
if region and region != 'global':
- client_options = {'api_endpoint': f'{region}-aiplatform.googleapis.com:443'}
-
+ client_options = ClientOptions(api_endpoint=f'{region}-aiplatform.googleapis.com:443')
+ else:
+ client_options = ClientOptions()
return PipelineServiceClient(
credentials=self._get_credentials(), client_info=CLIENT_INFO, client_options=client_options
)
@@ -82,9 +84,10 @@ def get_job_service_client(
region: Optional[str] = None,
) -> JobServiceClient:
"""Returns JobServiceClient"""
- client_options = None
if region and region != 'global':
- client_options = {'api_endpoint': f'{region}-aiplatform.googleapis.com:443'}
+ client_options = ClientOptions(api_endpoint=f'{region}-aiplatform.googleapis.com:443')
+ else:
+ client_options = ClientOptions()
return JobServiceClient(
credentials=self._get_credentials(), client_info=CLIENT_INFO, client_options=client_options
@@ -338,7 +341,7 @@ def cancel_pipeline_job(
project_id: str,
region: str,
pipeline_job: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> None:
@@ -377,7 +380,7 @@ def cancel_training_pipeline(
project_id: str,
region: str,
training_pipeline: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> None:
@@ -416,7 +419,7 @@ def cancel_custom_job(
project_id: str,
region: str,
custom_job: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> None:
@@ -456,7 +459,7 @@ def create_pipeline_job(
region: str,
pipeline_job: PipelineJob,
pipeline_job_id: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> PipelineJob:
@@ -495,7 +498,7 @@ def create_training_pipeline(
project_id: str,
region: str,
training_pipeline: TrainingPipeline,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> TrainingPipeline:
@@ -529,7 +532,7 @@ def create_custom_job(
project_id: str,
region: str,
custom_job: CustomJob,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> CustomJob:
@@ -1649,7 +1652,7 @@ def delete_pipeline_job(
project_id: str,
region: str,
pipeline_job: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Operation:
@@ -1682,7 +1685,7 @@ def delete_training_pipeline(
project_id: str,
region: str,
training_pipeline: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Operation:
@@ -1715,7 +1718,7 @@ def delete_custom_job(
project_id: str,
region: str,
custom_job: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Operation:
@@ -1748,7 +1751,7 @@ def get_pipeline_job(
project_id: str,
region: str,
pipeline_job: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> PipelineJob:
@@ -1781,7 +1784,7 @@ def get_training_pipeline(
project_id: str,
region: str,
training_pipeline: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> TrainingPipeline:
@@ -1814,7 +1817,7 @@ def get_custom_job(
project_id: str,
region: str,
custom_job: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> CustomJob:
@@ -1850,7 +1853,7 @@ def list_pipeline_jobs(
page_token: Optional[str] = None,
filter: Optional[str] = None,
order_by: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> ListPipelineJobsPager:
@@ -1938,7 +1941,7 @@ def list_training_pipelines(
page_token: Optional[str] = None,
filter: Optional[str] = None,
read_mask: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> ListTrainingPipelinesPager:
@@ -1999,7 +2002,7 @@ def list_custom_jobs(
page_token: Optional[str],
filter: Optional[str],
read_mask: Optional[str],
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> ListCustomJobsPager:
diff --git a/airflow/providers/google/cloud/hooks/vertex_ai/dataset.py b/airflow/providers/google/cloud/hooks/vertex_ai/dataset.py
index 485c86cb99001..e693424c08b29 100644
--- a/airflow/providers/google/cloud/hooks/vertex_ai/dataset.py
+++ b/airflow/providers/google/cloud/hooks/vertex_ai/dataset.py
@@ -20,6 +20,8 @@
from typing import Dict, Optional, Sequence, Tuple, Union
+from google.api_core.client_options import ClientOptions
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
from google.api_core.operation import Operation
from google.api_core.retry import Retry
from google.cloud.aiplatform_v1 import DatasetServiceClient
@@ -41,9 +43,10 @@ class DatasetHook(GoogleBaseHook):
def get_dataset_service_client(self, region: Optional[str] = None) -> DatasetServiceClient:
"""Returns DatasetServiceClient."""
- client_options = None
if region and region != 'global':
- client_options = {'api_endpoint': f'{region}-aiplatform.googleapis.com:443'}
+ client_options = ClientOptions(api_endpoint=f'{region}-aiplatform.googleapis.com:443')
+ else:
+ client_options = ClientOptions()
return DatasetServiceClient(
credentials=self._get_credentials(), client_info=CLIENT_INFO, client_options=client_options
@@ -68,7 +71,7 @@ def create_dataset(
project_id: str,
region: str,
dataset: Union[Dataset, Dict],
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Operation:
@@ -102,7 +105,7 @@ def delete_dataset(
project_id: str,
region: str,
dataset: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Operation:
@@ -136,7 +139,7 @@ def export_data(
region: str,
dataset: str,
export_config: Union[ExportDataConfig, Dict],
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Operation:
@@ -173,7 +176,7 @@ def get_annotation_spec(
dataset: str,
annotation_spec: str,
read_mask: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> AnnotationSpec:
@@ -210,7 +213,7 @@ def get_dataset(
region: str,
dataset: str,
read_mask: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Dataset:
@@ -246,7 +249,7 @@ def import_data(
region: str,
dataset: str,
import_configs: Sequence[ImportDataConfig],
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Operation:
@@ -288,7 +291,7 @@ def list_annotations(
page_token: Optional[str] = None,
read_mask: Optional[str] = None,
order_by: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> ListAnnotationsPager:
@@ -338,7 +341,7 @@ def list_data_items(
page_token: Optional[str] = None,
read_mask: Optional[str] = None,
order_by: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> ListDataItemsPager:
@@ -386,7 +389,7 @@ def list_datasets(
page_token: Optional[str] = None,
read_mask: Optional[str] = None,
order_by: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> ListDatasetsPager:
@@ -430,7 +433,7 @@ def update_dataset(
dataset_id: str,
dataset: Union[Dataset, Dict],
update_mask: Union[FieldMask, Dict],
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Dataset:
diff --git a/airflow/providers/google/cloud/hooks/vertex_ai/endpoint_service.py b/airflow/providers/google/cloud/hooks/vertex_ai/endpoint_service.py
new file mode 100644
index 0000000000000..63307fed15715
--- /dev/null
+++ b/airflow/providers/google/cloud/hooks/vertex_ai/endpoint_service.py
@@ -0,0 +1,379 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""This module contains a Google Cloud Vertex AI hook.
+
+.. spelling::
+
+ undeployed
+ undeploy
+ Undeploys
+ aiplatform
+ FieldMask
+ unassigns
+"""
+
+from typing import Dict, Optional, Sequence, Tuple, Union
+
+from google.api_core.client_options import ClientOptions
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
+from google.api_core.operation import Operation
+from google.api_core.retry import Retry
+from google.cloud.aiplatform_v1 import EndpointServiceClient
+from google.cloud.aiplatform_v1.services.endpoint_service.pagers import ListEndpointsPager
+from google.cloud.aiplatform_v1.types import DeployedModel, Endpoint
+from google.protobuf.field_mask_pb2 import FieldMask
+
+from airflow import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class EndpointServiceHook(GoogleBaseHook):
+ """Hook for Google Cloud Vertex AI Endpoint Service APIs."""
+
+ def get_endpoint_service_client(self, region: Optional[str] = None) -> EndpointServiceClient:
+ """Returns EndpointServiceClient."""
+ if region and region != 'global':
+ client_options = ClientOptions(api_endpoint=f'{region}-aiplatform.googleapis.com:443')
+ else:
+ client_options = ClientOptions()
+
+ return EndpointServiceClient(
+ credentials=self._get_credentials(), client_info=self.client_info, client_options=client_options
+ )
+
+ def wait_for_operation(self, operation: Operation, timeout: Optional[float] = None):
+ """Waits for long-lasting operation to complete."""
+ try:
+ return operation.result(timeout=timeout)
+ except Exception:
+ error = operation.exception(timeout=timeout)
+ raise AirflowException(error)
+
+ @staticmethod
+ def extract_endpoint_id(obj: Dict) -> str:
+ """Returns unique id of the endpoint."""
+ return obj["name"].rpartition("/")[-1]
+
+ @staticmethod
+ def extract_deployed_model_id(obj: Dict) -> str:
+ """Returns unique id of the deploy model."""
+ return obj["deployed_model"]["id"]
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def create_endpoint(
+ self,
+ project_id: str,
+ region: str,
+ endpoint: Union[Endpoint, Dict],
+ endpoint_id: Optional[str] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> Operation:
+ """
+ Creates an Endpoint.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param endpoint: Required. The Endpoint to create.
+ :param endpoint_id: The ID of Endpoint. This value should be 1-10 characters, and valid characters
+ are /[0-9]/. If not provided, Vertex AI will generate a value for this ID.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_endpoint_service_client(region)
+ parent = client.common_location_path(project_id, region)
+
+ result = client.create_endpoint(
+ request={
+ 'parent': parent,
+ 'endpoint': endpoint,
+ 'endpoint_id': endpoint_id,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def delete_endpoint(
+ self,
+ project_id: str,
+ region: str,
+ endpoint: str,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> Operation:
+ """
+ Deletes an Endpoint.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param endpoint: Required. The Endpoint to delete.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_endpoint_service_client(region)
+ name = client.endpoint_path(project_id, region, endpoint)
+
+ result = client.delete_endpoint(
+ request={
+ 'name': name,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def deploy_model(
+ self,
+ project_id: str,
+ region: str,
+ endpoint: str,
+ deployed_model: Union[DeployedModel, Dict],
+ traffic_split: Optional[Union[Sequence, Dict]] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> Operation:
+ """
+ Deploys a Model into this Endpoint, creating a DeployedModel within it.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param endpoint: Required. The name of the Endpoint resource into which to deploy a Model. Format:
+ ``projects/{project}/locations/{location}/endpoints/{endpoint}``
+ :param deployed_model: Required. The DeployedModel to be created within the Endpoint. Note that
+ [Endpoint.traffic_split][google.cloud.aiplatform.v1.Endpoint.traffic_split] must be updated for
+ the DeployedModel to start receiving traffic, either as part of this call, or via
+ [EndpointService.UpdateEndpoint][google.cloud.aiplatform.v1.EndpointService.UpdateEndpoint].
+ :param traffic_split: A map from a DeployedModel's ID to the percentage of this Endpoint's traffic
+ that should be forwarded to that DeployedModel.
+
+ If this field is non-empty, then the Endpoint's
+ [traffic_split][google.cloud.aiplatform.v1.Endpoint.traffic_split] will be overwritten with it. To
+ refer to the ID of the just being deployed Model, a "0" should be used, and the actual ID of the
+ new DeployedModel will be filled in its place by this method. The traffic percentage values must
+ add up to 100.
+
+ If this field is empty, then the Endpoint's
+ [traffic_split][google.cloud.aiplatform.v1.Endpoint.traffic_split] is not updated.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_endpoint_service_client(region)
+ endpoint_path = client.endpoint_path(project_id, region, endpoint)
+
+ result = client.deploy_model(
+ request={
+ 'endpoint': endpoint_path,
+ 'deployed_model': deployed_model,
+ 'traffic_split': traffic_split,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def get_endpoint(
+ self,
+ project_id: str,
+ region: str,
+ endpoint: str,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> Endpoint:
+ """
+ Gets an Endpoint.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param endpoint: Required. The Endpoint to get.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_endpoint_service_client(region)
+ name = client.endpoint_path(project_id, region, endpoint)
+
+ result = client.get_endpoint(
+ request={
+ 'name': name,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def list_endpoints(
+ self,
+ project_id: str,
+ region: str,
+ filter: Optional[str] = None,
+ page_size: Optional[int] = None,
+ page_token: Optional[str] = None,
+ read_mask: Optional[str] = None,
+ order_by: Optional[str] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> ListEndpointsPager:
+ """
+ Lists Endpoints in a Location.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param filter: The standard list filter.
+ Supported fields:
+ - ``display_name`` supports = and !=.
+ - ``state`` supports = and !=.
+ - ``model_display_name`` supports = and !=
+ Some examples of using the filter are:
+ - ``state="JOB_STATE_SUCCEEDED" AND display_name="my_job"``
+ - ``state="JOB_STATE_RUNNING" OR display_name="my_job"``
+ - ``NOT display_name="my_job"``
+ - ``state="JOB_STATE_FAILED"``
+ :param page_size: The standard list page size.
+ :param page_token: The standard list page token.
+ :param read_mask: Mask specifying which fields to read.
+ :param order_by: A comma-separated list of fields to order by, sorted in
+ ascending order. Use "desc" after a field name for
+ descending. Supported fields:
+
+ - ``display_name``
+ - ``create_time``
+ - ``update_time``
+
+ Example: ``display_name, create_time desc``.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_endpoint_service_client(region)
+ parent = client.common_location_path(project_id, region)
+
+ result = client.list_endpoints(
+ request={
+ 'parent': parent,
+ 'filter': filter,
+ 'page_size': page_size,
+ 'page_token': page_token,
+ 'read_mask': read_mask,
+ 'order_by': order_by,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def undeploy_model(
+ self,
+ project_id: str,
+ region: str,
+ endpoint: str,
+ deployed_model_id: str,
+ traffic_split: Optional[Union[Sequence, Dict]] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> Operation:
+ """
+ Undeploys a Model from an Endpoint, removing a DeployedModel from it, and freeing all resources it's
+ using.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param endpoint: Required. The name of the Endpoint resource from which to undeploy a Model.
+ :param deployed_model_id: Required. The ID of the DeployedModel to be undeployed from the Endpoint.
+ :param traffic_split: If this field is provided, then the Endpoint's
+ [traffic_split][google.cloud.aiplatform.v1.Endpoint.traffic_split] will be overwritten with it. If
+ last DeployedModel is being undeployed from the Endpoint, the [Endpoint.traffic_split] will always
+ end up empty when this call returns. A DeployedModel will be successfully undeployed only if it
+ doesn't have any traffic assigned to it when this method executes, or if this field unassigns any
+ traffic to it.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_endpoint_service_client(region)
+ endpoint_path = client.endpoint_path(project_id, region, endpoint)
+
+ result = client.undeploy_model(
+ request={
+ 'endpoint': endpoint_path,
+ 'deployed_model_id': deployed_model_id,
+ 'traffic_split': traffic_split,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def update_endpoint(
+ self,
+ project_id: str,
+ region: str,
+ endpoint_id: str,
+ endpoint: Union[Endpoint, Dict],
+ update_mask: Union[FieldMask, Dict],
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> Endpoint:
+ """
+ Updates an Endpoint.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param endpoint: Required. The Endpoint which replaces the resource on the server.
+ :param update_mask: Required. The update mask applies to the resource. See
+ [google.protobuf.FieldMask][google.protobuf.FieldMask].
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_endpoint_service_client(region)
+ endpoint["name"] = client.endpoint_path(project_id, region, endpoint_id)
+
+ result = client.update_endpoint(
+ request={
+ 'endpoint': endpoint,
+ 'update_mask': update_mask,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
diff --git a/airflow/providers/google/cloud/hooks/vertex_ai/hyperparameter_tuning_job.py b/airflow/providers/google/cloud/hooks/vertex_ai/hyperparameter_tuning_job.py
new file mode 100644
index 0000000000000..2659636c76bdf
--- /dev/null
+++ b/airflow/providers/google/cloud/hooks/vertex_ai/hyperparameter_tuning_job.py
@@ -0,0 +1,418 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""This module contains a Google Cloud Vertex AI hook.
+
+.. spelling::
+
+ irreproducible
+ codepoints
+ Tensorboard
+ aiplatform
+ myVPC
+"""
+
+from typing import Dict, List, Optional, Sequence, Tuple, Union
+
+from google.api_core.client_options import ClientOptions
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
+from google.api_core.operation import Operation
+from google.api_core.retry import Retry
+from google.cloud.aiplatform import CustomJob, HyperparameterTuningJob, gapic, hyperparameter_tuning
+from google.cloud.aiplatform_v1 import JobServiceClient, types
+from google.cloud.aiplatform_v1.services.job_service.pagers import ListHyperparameterTuningJobsPager
+
+from airflow import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class HyperparameterTuningJobHook(GoogleBaseHook):
+ """Hook for Google Cloud Vertex AI Hyperparameter Tuning Job APIs."""
+
+ def __init__(
+ self,
+ gcp_conn_id: str = "google_cloud_default",
+ delegate_to: Optional[str] = None,
+ impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
+ ) -> None:
+ super().__init__(
+ gcp_conn_id=gcp_conn_id,
+ delegate_to=delegate_to,
+ impersonation_chain=impersonation_chain,
+ )
+ self._hyperparameter_tuning_job: Optional[HyperparameterTuningJob] = None
+
+ def get_job_service_client(self, region: Optional[str] = None) -> JobServiceClient:
+ """Returns JobServiceClient."""
+ if region and region != 'global':
+ client_options = ClientOptions(api_endpoint=f'{region}-aiplatform.googleapis.com:443')
+ else:
+ client_options = ClientOptions()
+
+ return JobServiceClient(
+ credentials=self._get_credentials(), client_info=self.client_info, client_options=client_options
+ )
+
+ def get_hyperparameter_tuning_job_object(
+ self,
+ display_name: str,
+ custom_job: CustomJob,
+ metric_spec: Dict[str, str],
+ parameter_spec: Dict[str, hyperparameter_tuning._ParameterSpec],
+ max_trial_count: int,
+ parallel_trial_count: int,
+ max_failed_trial_count: int = 0,
+ search_algorithm: Optional[str] = None,
+ measurement_selection: Optional[str] = "best",
+ project: Optional[str] = None,
+ location: Optional[str] = None,
+ labels: Optional[Dict[str, str]] = None,
+ encryption_spec_key_name: Optional[str] = None,
+ ) -> HyperparameterTuningJob:
+ """Returns HyperparameterTuningJob object"""
+ return HyperparameterTuningJob(
+ display_name=display_name,
+ custom_job=custom_job,
+ metric_spec=metric_spec,
+ parameter_spec=parameter_spec,
+ max_trial_count=max_trial_count,
+ parallel_trial_count=parallel_trial_count,
+ max_failed_trial_count=max_failed_trial_count,
+ search_algorithm=search_algorithm,
+ measurement_selection=measurement_selection,
+ project=project,
+ location=location,
+ credentials=self._get_credentials(),
+ labels=labels,
+ encryption_spec_key_name=encryption_spec_key_name,
+ )
+
+ def get_custom_job_object(
+ self,
+ display_name: str,
+ worker_pool_specs: Union[List[Dict], List[gapic.WorkerPoolSpec]],
+ base_output_dir: Optional[str] = None,
+ project: Optional[str] = None,
+ location: Optional[str] = None,
+ labels: Optional[Dict[str, str]] = None,
+ encryption_spec_key_name: Optional[str] = None,
+ staging_bucket: Optional[str] = None,
+ ) -> CustomJob:
+ """Returns CustomJob object"""
+ return CustomJob(
+ display_name=display_name,
+ worker_pool_specs=worker_pool_specs,
+ base_output_dir=base_output_dir,
+ project=project,
+ location=location,
+ credentials=self._get_credentials,
+ labels=labels,
+ encryption_spec_key_name=encryption_spec_key_name,
+ staging_bucket=staging_bucket,
+ )
+
+ @staticmethod
+ def extract_hyperparameter_tuning_job_id(obj: Dict) -> str:
+ """Returns unique id of the hyperparameter_tuning_job."""
+ return obj["name"].rpartition("/")[-1]
+
+ def wait_for_operation(self, operation: Operation, timeout: Optional[float] = None):
+ """Waits for long-lasting operation to complete."""
+ try:
+ return operation.result(timeout=timeout)
+ except Exception:
+ error = operation.exception(timeout=timeout)
+ raise AirflowException(error)
+
+ def cancel_hyperparameter_tuning_job(self) -> None:
+ """Cancel HyperparameterTuningJob"""
+ if self._hyperparameter_tuning_job:
+ self._hyperparameter_tuning_job.cancel()
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def create_hyperparameter_tuning_job(
+ self,
+ project_id: str,
+ region: str,
+ display_name: str,
+ metric_spec: Dict[str, str],
+ parameter_spec: Dict[str, hyperparameter_tuning._ParameterSpec],
+ max_trial_count: int,
+ parallel_trial_count: int,
+ # START: CustomJob param
+ worker_pool_specs: Union[List[Dict], List[gapic.WorkerPoolSpec]],
+ base_output_dir: Optional[str] = None,
+ custom_job_labels: Optional[Dict[str, str]] = None,
+ custom_job_encryption_spec_key_name: Optional[str] = None,
+ staging_bucket: Optional[str] = None,
+ # END: CustomJob param
+ max_failed_trial_count: int = 0,
+ search_algorithm: Optional[str] = None,
+ measurement_selection: Optional[str] = "best",
+ hyperparameter_tuning_job_labels: Optional[Dict[str, str]] = None,
+ hyperparameter_tuning_job_encryption_spec_key_name: Optional[str] = None,
+ # START: run param
+ service_account: Optional[str] = None,
+ network: Optional[str] = None,
+ timeout: Optional[int] = None, # seconds
+ restart_job_on_worker_restart: bool = False,
+ enable_web_access: bool = False,
+ tensorboard: Optional[str] = None,
+ sync: bool = True,
+ # END: run param
+ ) -> HyperparameterTuningJob:
+ """
+ Create a HyperparameterTuningJob.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param display_name: Required. The user-defined name of the HyperparameterTuningJob. The name can be
+ up to 128 characters long and can be consist of any UTF-8 characters.
+ :param metric_spec: Required. Dictionary representing metrics to optimize. The dictionary key is the
+ metric_id, which is reported by your training job, and the dictionary value is the optimization
+ goal of the metric('minimize' or 'maximize').
+ example: metric_spec = {'loss': 'minimize', 'accuracy': 'maximize'}
+ :param parameter_spec: Required. Dictionary representing parameters to optimize. The dictionary key
+ is the metric_id, which is passed into your training job as a command line key word argument, and
+ the dictionary value is the parameter specification of the metric.
+ :param max_trial_count: Required. The desired total number of Trials.
+ :param parallel_trial_count: Required. The desired number of Trials to run in parallel.
+ :param worker_pool_specs: Required. The spec of the worker pools including machine type and Docker
+ image. Can provided as a list of dictionaries or list of WorkerPoolSpec proto messages.
+ :param base_output_dir: Optional. GCS output directory of job. If not provided a timestamped
+ directory in the staging directory will be used.
+ :param custom_job_labels: Optional. The labels with user-defined metadata to organize CustomJobs.
+ Label keys and values can be no longer than 64 characters (Unicode codepoints), can only contain
+ lowercase letters, numeric characters, underscores and dashes. International characters are
+ allowed. See https://goo.gl/xmQnxf for more information and examples of labels.
+ :param custom_job_encryption_spec_key_name: Optional.Customer-managed encryption key name for a
+ CustomJob. If this is set, then all resources created by the CustomJob will be encrypted with the
+ provided encryption key.
+ :param staging_bucket: Optional. Bucket for produced custom job artifacts. Overrides staging_bucket
+ set in aiplatform.init.
+ :param max_failed_trial_count: Optional. The number of failed Trials that need to be seen before
+ failing the HyperparameterTuningJob. If set to 0, Vertex AI decides how many Trials must fail
+ before the whole job fails.
+ :param search_algorithm: The search algorithm specified for the Study. Accepts one of the following:
+ `None` - If you do not specify an algorithm, your job uses the default Vertex AI algorithm. The
+ default algorithm applies Bayesian optimization to arrive at the optimal solution with a more
+ effective search over the parameter space.
+
+ 'grid' - A simple grid search within the feasible space. This option is particularly useful if
+ you want to specify a quantity of trials that is greater than the number of points in the
+ feasible space. In such cases, if you do not specify a grid search, the Vertex AI default
+ algorithm may generate duplicate suggestions. To use grid search, all parameter specs must be of
+ type `IntegerParameterSpec`, `CategoricalParameterSpace`, or `DiscreteParameterSpec`.
+
+ 'random' - A simple random search within the feasible space.
+ :param measurement_selection: This indicates which measurement to use if/when the service
+ automatically selects the final measurement from previously reported intermediate measurements.
+ Accepts: 'best', 'last'
+ Choose this based on two considerations:
+ A) Do you expect your measurements to monotonically improve? If so, choose 'last'. On the other
+ hand, if you're in a situation where your system can "over-train" and you expect the performance
+ to get better for a while but then start declining, choose 'best'.
+ B) Are your measurements significantly noisy and/or irreproducible? If so, 'best' will tend to be
+ over-optimistic, and it may be better to choose 'last'.
+ If both or neither of (A) and (B) apply, it doesn't matter which selection type is chosen.
+ :param hyperparameter_tuning_job_labels: Optional. The labels with user-defined metadata to organize
+ HyperparameterTuningJobs. Label keys and values can be no longer than 64 characters (Unicode
+ codepoints), can only contain lowercase letters, numeric characters, underscores and dashes.
+ International characters are allowed. See https://goo.gl/xmQnxf for more information and examples
+ of labels.
+ :param hyperparameter_tuning_job_encryption_spec_key_name: Optional. Customer-managed encryption key
+ options for a HyperparameterTuningJob. If this is set, then all resources created by the
+ HyperparameterTuningJob will be encrypted with the provided encryption key.
+ :param service_account: Optional. Specifies the service account for workload run-as account. Users
+ submitting jobs must have act-as permission on this run-as account.
+ :param network: Optional. The full name of the Compute Engine network to which the job should be
+ peered. For example, projects/12345/global/networks/myVPC. Private services access must already
+ be configured for the network. If left unspecified, the job is not peered with any network.
+ :param timeout: The maximum job running time in seconds. The default is 7 days.
+ :param restart_job_on_worker_restart: Restarts the entire CustomJob if a worker gets restarted. This
+ feature can be used by distributed training jobs that are not resilient to workers leaving and
+ joining a job.
+ :param enable_web_access: Whether you want Vertex AI to enable interactive shell access to training
+ containers. https://cloud.google.com/vertex-ai/docs/training/monitor-debug-interactive-shell
+ :param tensorboard: Optional. The name of a Vertex AI
+ [Tensorboard][google.cloud.aiplatform.v1beta1.Tensorboard] resource to which this CustomJob will
+ upload Tensorboard logs. Format:
+ ``projects/{project}/locations/{location}/tensorboards/{tensorboard}`` The training script should
+ write Tensorboard to following Vertex AI environment variable: AIP_TENSORBOARD_LOG_DIR
+ `service_account` is required with provided `tensorboard`. For more information on configuring
+ your service account please visit:
+ https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-training
+ :param sync: Whether to execute this method synchronously. If False, this method will unblock and it
+ will be executed in a concurrent Future.
+ """
+ custom_job = self.get_custom_job_object(
+ project=project_id,
+ location=region,
+ display_name=display_name,
+ worker_pool_specs=worker_pool_specs,
+ base_output_dir=base_output_dir,
+ labels=custom_job_labels,
+ encryption_spec_key_name=custom_job_encryption_spec_key_name,
+ staging_bucket=staging_bucket,
+ )
+
+ self._hyperparameter_tuning_job = self.get_hyperparameter_tuning_job_object(
+ project=project_id,
+ location=region,
+ display_name=display_name,
+ custom_job=custom_job,
+ metric_spec=metric_spec,
+ parameter_spec=parameter_spec,
+ max_trial_count=max_trial_count,
+ parallel_trial_count=parallel_trial_count,
+ max_failed_trial_count=max_failed_trial_count,
+ search_algorithm=search_algorithm,
+ measurement_selection=measurement_selection,
+ labels=hyperparameter_tuning_job_labels,
+ encryption_spec_key_name=hyperparameter_tuning_job_encryption_spec_key_name,
+ )
+ self._hyperparameter_tuning_job.run(
+ service_account=service_account,
+ network=network,
+ timeout=timeout, # seconds
+ restart_job_on_worker_restart=restart_job_on_worker_restart,
+ enable_web_access=enable_web_access,
+ tensorboard=tensorboard,
+ sync=sync,
+ )
+ self._hyperparameter_tuning_job.wait()
+ return self._hyperparameter_tuning_job
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def get_hyperparameter_tuning_job(
+ self,
+ project_id: str,
+ region: str,
+ hyperparameter_tuning_job: str,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> types.HyperparameterTuningJob:
+ """
+ Gets a HyperparameterTuningJob
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param hyperparameter_tuning_job: Required. The name of the HyperparameterTuningJob resource.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_job_service_client(region)
+ name = client.hyperparameter_tuning_job_path(project_id, region, hyperparameter_tuning_job)
+
+ result = client.get_hyperparameter_tuning_job(
+ request={
+ 'name': name,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def list_hyperparameter_tuning_jobs(
+ self,
+ project_id: str,
+ region: str,
+ filter: Optional[str] = None,
+ page_size: Optional[int] = None,
+ page_token: Optional[str] = None,
+ read_mask: Optional[str] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> ListHyperparameterTuningJobsPager:
+ """
+ Lists HyperparameterTuningJobs in a Location.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param filter: The standard list filter.
+ Supported fields:
+ - ``display_name`` supports = and !=.
+ - ``state`` supports = and !=.
+ - ``model_display_name`` supports = and !=
+ Some examples of using the filter are:
+ - ``state="JOB_STATE_SUCCEEDED" AND display_name="my_job"``
+ - ``state="JOB_STATE_RUNNING" OR display_name="my_job"``
+ - ``NOT display_name="my_job"``
+ - ``state="JOB_STATE_FAILED"``
+ :param page_size: The standard list page size.
+ :param page_token: The standard list page token.
+ :param read_mask: Mask specifying which fields to read.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_job_service_client(region)
+ parent = client.common_location_path(project_id, region)
+
+ result = client.list_hyperparameter_tuning_jobs(
+ request={
+ 'parent': parent,
+ 'filter': filter,
+ 'page_size': page_size,
+ 'page_token': page_token,
+ 'read_mask': read_mask,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def delete_hyperparameter_tuning_job(
+ self,
+ project_id: str,
+ region: str,
+ hyperparameter_tuning_job: str,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> Operation:
+ """
+ Deletes a HyperparameterTuningJob.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param hyperparameter_tuning_job: Required. The name of the HyperparameterTuningJob resource to be
+ deleted.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_job_service_client(region)
+ name = client.hyperparameter_tuning_job_path(project_id, region, hyperparameter_tuning_job)
+
+ result = client.delete_hyperparameter_tuning_job(
+ request={
+ 'name': name,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
diff --git a/airflow/providers/google/cloud/hooks/vertex_ai/model_service.py b/airflow/providers/google/cloud/hooks/vertex_ai/model_service.py
new file mode 100644
index 0000000000000..c8f58eb394ea9
--- /dev/null
+++ b/airflow/providers/google/cloud/hooks/vertex_ai/model_service.py
@@ -0,0 +1,228 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+"""This module contains a Google Cloud Vertex AI hook.
+
+.. spelling::
+
+ aiplatform
+ camelCase
+"""
+
+from typing import Dict, Optional, Sequence, Tuple, Union
+
+from google.api_core.client_options import ClientOptions
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
+from google.api_core.operation import Operation
+from google.api_core.retry import Retry
+from google.cloud.aiplatform_v1 import ModelServiceClient
+from google.cloud.aiplatform_v1.services.model_service.pagers import ListModelsPager
+from google.cloud.aiplatform_v1.types import Model, model_service
+
+from airflow import AirflowException
+from airflow.providers.google.common.hooks.base_google import GoogleBaseHook
+
+
+class ModelServiceHook(GoogleBaseHook):
+ """Hook for Google Cloud Vertex AI Endpoint Service APIs."""
+
+ def get_model_service_client(self, region: Optional[str] = None) -> ModelServiceClient:
+ """Returns ModelServiceClient."""
+ if region and region != 'global':
+ client_options = ClientOptions(api_endpoint=f'{region}-aiplatform.googleapis.com:443')
+ else:
+ client_options = ClientOptions()
+
+ return ModelServiceClient(
+ credentials=self._get_credentials(), client_info=self.client_info, client_options=client_options
+ )
+
+ @staticmethod
+ def extract_model_id(obj: Dict) -> str:
+ """Returns unique id of the model."""
+ return obj["model"].rpartition("/")[-1]
+
+ def wait_for_operation(self, operation: Operation, timeout: Optional[float] = None):
+ """Waits for long-lasting operation to complete."""
+ try:
+ return operation.result(timeout=timeout)
+ except Exception:
+ error = operation.exception(timeout=timeout)
+ raise AirflowException(error)
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def delete_model(
+ self,
+ project_id: str,
+ region: str,
+ model: str,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> Operation:
+ """
+ Deletes a Model.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param model: Required. The name of the Model resource to be deleted.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_model_service_client(region)
+ name = client.model_path(project_id, region, model)
+
+ result = client.delete_model(
+ request={
+ 'name': name,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def export_model(
+ self,
+ project_id: str,
+ region: str,
+ model: str,
+ output_config: Union[model_service.ExportModelRequest.OutputConfig, Dict],
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> Operation:
+ """
+ Exports a trained, exportable Model to a location specified by the user.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param model: Required. The resource name of the Model to export.
+ :param output_config: Required. The desired output location and configuration.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_model_service_client(region)
+ name = client.model_path(project_id, region, model)
+
+ result = client.export_model(
+ request={
+ "name": name,
+ "output_config": output_config,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def list_models(
+ self,
+ project_id: str,
+ region: str,
+ filter: Optional[str] = None,
+ page_size: Optional[int] = None,
+ page_token: Optional[str] = None,
+ read_mask: Optional[str] = None,
+ order_by: Optional[str] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> ListModelsPager:
+ r"""
+ Lists Models in a Location.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param filter: An expression for filtering the results of the request. For field names both
+ snake_case and camelCase are supported.
+ - ``model`` supports = and !=. ``model`` represents the Model ID, i.e. the last segment of the
+ Model's [resource name][google.cloud.aiplatform.v1.Model.name].
+ - ``display_name`` supports = and !=
+ - ``labels`` supports general map functions that is:
+ -- ``labels.key=value`` - key:value equality
+ -- \`labels.key:\* or labels:key - key existence
+ -- A key including a space must be quoted. ``labels."a key"``.
+ :param page_size: The standard list page size.
+ :param page_token: The standard list page token. Typically obtained via
+ [ListModelsResponse.next_page_token][google.cloud.aiplatform.v1.ListModelsResponse.next_page_token]
+ of the previous
+ [ModelService.ListModels][google.cloud.aiplatform.v1.ModelService.ListModels]
+ call.
+ :param read_mask: Mask specifying which fields to read.
+ :param order_by: A comma-separated list of fields to order by, sorted in ascending order. Use "desc"
+ after a field name for descending.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_model_service_client(region)
+ parent = client.common_location_path(project_id, region)
+
+ result = client.list_models(
+ request={
+ 'parent': parent,
+ 'filter': filter,
+ 'page_size': page_size,
+ 'page_token': page_token,
+ 'read_mask': read_mask,
+ 'order_by': order_by,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def upload_model(
+ self,
+ project_id: str,
+ region: str,
+ model: Union[Model, Dict],
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
+ timeout: Optional[float] = None,
+ metadata: Sequence[Tuple[str, str]] = (),
+ ) -> Operation:
+ """
+ Uploads a Model artifact into Vertex AI.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param region: Required. The ID of the Google Cloud region that the service belongs to.
+ :param model: Required. The Model to create.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = self.get_model_service_client(region)
+ parent = client.common_location_path(project_id, region)
+
+ result = client.upload_model(
+ request={
+ "parent": parent,
+ "model": model,
+ },
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return result
diff --git a/airflow/providers/google/cloud/hooks/video_intelligence.py b/airflow/providers/google/cloud/hooks/video_intelligence.py
index ff4addce15a88..cc3f00e94eed8 100644
--- a/airflow/providers/google/cloud/hooks/video_intelligence.py
+++ b/airflow/providers/google/cloud/hooks/video_intelligence.py
@@ -18,6 +18,7 @@
"""This module contains a Google Cloud Video Intelligence Hook."""
from typing import Dict, List, Optional, Sequence, Tuple, Union
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
from google.api_core.operation import Operation
from google.api_core.retry import Retry
from google.cloud.videointelligence_v1 import VideoIntelligenceServiceClient
@@ -82,7 +83,7 @@ def annotate_video(
video_context: Union[Dict, VideoContext] = None,
output_uri: Optional[str] = None,
location: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Operation:
diff --git a/airflow/providers/google/cloud/hooks/vision.py b/airflow/providers/google/cloud/hooks/vision.py
index 379a9a0ef86bb..1d4d232a1eb4b 100644
--- a/airflow/providers/google/cloud/hooks/vision.py
+++ b/airflow/providers/google/cloud/hooks/vision.py
@@ -27,6 +27,7 @@
else:
from cached_property import cached_property
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
from google.api_core.retry import Retry
from google.cloud.vision_v1 import ImageAnnotatorClient, ProductSearchClient
from google.cloud.vision_v1.types import (
@@ -173,7 +174,7 @@ def create_product_set(
product_set: Union[dict, ProductSet],
project_id: str = PROVIDE_PROJECT_ID,
product_set_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> str:
@@ -208,7 +209,7 @@ def get_product_set(
location: str,
product_set_id: str,
project_id: str = PROVIDE_PROJECT_ID,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> dict:
@@ -232,7 +233,7 @@ def update_product_set(
location: Optional[str] = None,
product_set_id: Optional[str] = None,
update_mask: Union[dict, FieldMask] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> dict:
@@ -258,7 +259,7 @@ def delete_product_set(
location: str,
product_set_id: str,
project_id: str = PROVIDE_PROJECT_ID,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> None:
@@ -279,7 +280,7 @@ def create_product(
product: Union[dict, Product],
project_id: str = PROVIDE_PROJECT_ID,
product_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
):
@@ -314,7 +315,7 @@ def get_product(
location: str,
product_id: str,
project_id: str = PROVIDE_PROJECT_ID,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
):
@@ -338,7 +339,7 @@ def update_product(
location: Optional[str] = None,
product_id: Optional[str] = None,
update_mask: Optional[Dict[str, FieldMask]] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
):
@@ -362,7 +363,7 @@ def delete_product(
location: str,
product_id: str,
project_id: str = PROVIDE_PROJECT_ID,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> None:
@@ -384,7 +385,7 @@ def create_reference_image(
reference_image: Union[dict, ReferenceImage],
project_id: str,
reference_image_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> str:
@@ -424,7 +425,7 @@ def delete_reference_image(
product_id: str,
reference_image_id: str,
project_id: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> dict:
@@ -455,7 +456,7 @@ def add_product_to_product_set(
product_id: str,
project_id: str,
location: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> None:
@@ -483,7 +484,7 @@ def remove_product_from_product_set(
product_id: str,
project_id: str,
location: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> None:
@@ -507,7 +508,7 @@ def remove_product_from_product_set(
def annotate_image(
self,
request: Union[dict, AnnotateImageRequest],
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
) -> Dict:
"""
@@ -528,7 +529,7 @@ def annotate_image(
def batch_annotate_images(
self,
requests: Union[List[dict], List[AnnotateImageRequest]],
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
) -> dict:
"""
@@ -550,7 +551,7 @@ def text_detection(
self,
image: Union[dict, Image],
max_results: Optional[int] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
additional_properties: Optional[Dict] = None,
) -> dict:
@@ -580,7 +581,7 @@ def document_text_detection(
self,
image: Union[dict, Image],
max_results: Optional[int] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
additional_properties: Optional[dict] = None,
) -> dict:
@@ -610,7 +611,7 @@ def label_detection(
self,
image: Union[dict, Image],
max_results: Optional[int] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
additional_properties: Optional[dict] = None,
) -> dict:
@@ -640,7 +641,7 @@ def safe_search_detection(
self,
image: Union[dict, Image],
max_results: Optional[int] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
additional_properties: Optional[dict] = None,
) -> dict:
diff --git a/airflow/providers/google/cloud/hooks/workflows.py b/airflow/providers/google/cloud/hooks/workflows.py
index 03fd21c0053c1..fb9963160f074 100644
--- a/airflow/providers/google/cloud/hooks/workflows.py
+++ b/airflow/providers/google/cloud/hooks/workflows.py
@@ -17,6 +17,7 @@
from typing import Dict, Optional, Sequence, Tuple, Union
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
from google.api_core.operation import Operation
from google.api_core.retry import Retry
from google.cloud.workflows.executions_v1beta import Execution, ExecutionsClient
@@ -52,7 +53,7 @@ def create_workflow(
workflow_id: str,
location: str,
project_id: str = PROVIDE_PROJECT_ID,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Operation:
@@ -88,7 +89,7 @@ def get_workflow(
workflow_id: str,
location: str,
project_id: str = PROVIDE_PROJECT_ID,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Workflow:
@@ -113,7 +114,7 @@ def update_workflow(
self,
workflow: Union[Dict, Workflow],
update_mask: Optional[FieldMask] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Operation:
@@ -149,7 +150,7 @@ def delete_workflow(
workflow_id: str,
location: str,
project_id: str = PROVIDE_PROJECT_ID,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Operation:
@@ -179,7 +180,7 @@ def list_workflows(
project_id: str = PROVIDE_PROJECT_ID,
filter_: Optional[str] = None,
order_by: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> ListWorkflowsPager:
@@ -218,7 +219,7 @@ def create_execution(
location: str,
execution: Dict,
project_id: str = PROVIDE_PROJECT_ID,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Execution:
@@ -253,7 +254,7 @@ def get_execution(
execution_id: str,
location: str,
project_id: str = PROVIDE_PROJECT_ID,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Execution:
@@ -282,7 +283,7 @@ def cancel_execution(
execution_id: str,
location: str,
project_id: str = PROVIDE_PROJECT_ID,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> Execution:
@@ -312,7 +313,7 @@ def list_executions(
workflow_id: str,
location: str,
project_id: str = PROVIDE_PROJECT_ID,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
) -> ListExecutionsPager:
diff --git a/airflow/providers/google/cloud/links/bigquery.py b/airflow/providers/google/cloud/links/bigquery.py
new file mode 100644
index 0000000000000..a80818e2034ed
--- /dev/null
+++ b/airflow/providers/google/cloud/links/bigquery.py
@@ -0,0 +1,77 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains Google BigQuery links."""
+from typing import TYPE_CHECKING
+
+from airflow.models import BaseOperator
+from airflow.providers.google.cloud.links.base import BaseGoogleLink
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+BIGQUERY_BASE_LINK = "https://console.cloud.google.com/bigquery"
+BIGQUERY_DATASET_LINK = (
+ BIGQUERY_BASE_LINK + "?referrer=search&project={project_id}&d={dataset_id}&p={project_id}&page=dataset"
+)
+BIGQUERY_TABLE_LINK = (
+ BIGQUERY_BASE_LINK
+ + "?referrer=search&project={project_id}&d={dataset_id}&p={project_id}&page=table&t={table_id}"
+)
+
+
+class BigQueryDatasetLink(BaseGoogleLink):
+ """Helper class for constructing BigQuery Dataset Link"""
+
+ name = "BigQuery Dataset"
+ key = "bigquery_dataset"
+ format_str = BIGQUERY_DATASET_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance: BaseOperator,
+ dataset_id: str,
+ project_id: str,
+ ):
+ task_instance.xcom_push(
+ context,
+ key=BigQueryDatasetLink.key,
+ value={"dataset_id": dataset_id, "project_id": project_id},
+ )
+
+
+class BigQueryTableLink(BaseGoogleLink):
+ """Helper class for constructing BigQuery Table Link"""
+
+ name = "BigQuery Table"
+ key = "bigquery_table"
+ format_str = BIGQUERY_TABLE_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance: BaseOperator,
+ dataset_id: str,
+ project_id: str,
+ table_id: str,
+ ):
+ task_instance.xcom_push(
+ context,
+ key=BigQueryTableLink.key,
+ value={"dataset_id": dataset_id, "project_id": project_id, "table_id": table_id},
+ )
diff --git a/airflow/providers/google/cloud/links/bigquery_dts.py b/airflow/providers/google/cloud/links/bigquery_dts.py
new file mode 100644
index 0000000000000..4a73be51d896a
--- /dev/null
+++ b/airflow/providers/google/cloud/links/bigquery_dts.py
@@ -0,0 +1,50 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains Google BigQuery Data Transfer links."""
+from typing import TYPE_CHECKING
+
+from airflow.models import BaseOperator
+from airflow.providers.google.cloud.links.base import BaseGoogleLink
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+BIGQUERY_BASE_LINK = "https://console.cloud.google.com/bigquery/transfers"
+BIGQUERY_DTS_LINK = BIGQUERY_BASE_LINK + "/locations/{region}/configs/{config_id}/runs?project={project_id}"
+
+
+class BigQueryDataTransferConfigLink(BaseGoogleLink):
+ """Helper class for constructing BigQuery Data Transfer Config Link"""
+
+ name = "BigQuery Data Transfer Config"
+ key = "bigquery_dts_config"
+ format_str = BIGQUERY_DTS_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance: BaseOperator,
+ region: str,
+ config_id: str,
+ project_id: str,
+ ):
+ task_instance.xcom_push(
+ context,
+ key=BigQueryDataTransferConfigLink.key,
+ value={"project_id": project_id, "region": region, "config_id": config_id},
+ )
diff --git a/airflow/providers/google/cloud/links/bigtable.py b/airflow/providers/google/cloud/links/bigtable.py
new file mode 100644
index 0000000000000..cc06129021798
--- /dev/null
+++ b/airflow/providers/google/cloud/links/bigtable.py
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING
+
+from airflow.providers.google.cloud.links.base import BaseGoogleLink
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+BASE_LINK = "https://console.cloud.google.com"
+BIGTABLE_BASE_LINK = BASE_LINK + "/bigtable"
+BIGTABLE_INSTANCE_LINK = BIGTABLE_BASE_LINK + "/instances/{instance_id}/overview?project={project_id}"
+BIGTABLE_CLUSTER_LINK = (
+ BIGTABLE_BASE_LINK + "/instances/{instance_id}/clusters/{cluster_id}?project={project_id}"
+)
+BIGTABLE_TABLES_LINK = BIGTABLE_BASE_LINK + "/instances/{instance_id}/tables?project={project_id}"
+
+
+class BigtableInstanceLink(BaseGoogleLink):
+ """Helper class for constructing Bigtable Instance link"""
+
+ name = "Bigtable Instance"
+ key = "instance_key"
+ format_str = BIGTABLE_INSTANCE_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance,
+ ):
+ task_instance.xcom_push(
+ context=context,
+ key=BigtableInstanceLink.key,
+ value={
+ "instance_id": task_instance.instance_id,
+ "project_id": task_instance.project_id,
+ },
+ )
+
+
+class BigtableClusterLink(BaseGoogleLink):
+ """Helper class for constructing Bigtable Cluster link"""
+
+ name = "Bigtable Cluster"
+ key = "cluster_key"
+ format_str = BIGTABLE_CLUSTER_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance,
+ ):
+ task_instance.xcom_push(
+ context=context,
+ key=BigtableClusterLink.key,
+ value={
+ "instance_id": task_instance.instance_id,
+ "cluster_id": task_instance.cluster_id,
+ "project_id": task_instance.project_id,
+ },
+ )
+
+
+class BigtableTablesLink(BaseGoogleLink):
+ """Helper class for constructing Bigtable Tables link"""
+
+ name = "Bigtable Tables"
+ key = "tables_key"
+ format_str = BIGTABLE_TABLES_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance,
+ ):
+ task_instance.xcom_push(
+ context=context,
+ key=BigtableTablesLink.key,
+ value={
+ "instance_id": task_instance.instance_id,
+ "project_id": task_instance.project_id,
+ },
+ )
diff --git a/airflow/providers/google/cloud/links/cloud_sql.py b/airflow/providers/google/cloud/links/cloud_sql.py
new file mode 100644
index 0000000000000..58a0b3c3a5c6d
--- /dev/null
+++ b/airflow/providers/google/cloud/links/cloud_sql.py
@@ -0,0 +1,74 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains Google Cloud SQL links."""
+from typing import TYPE_CHECKING, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.google.cloud.links.base import BaseGoogleLink
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+
+CLOUD_SQL_BASE_LINK = "https://console.cloud.google.com/sql"
+CLOUD_SQL_INSTANCE_LINK = CLOUD_SQL_BASE_LINK + "/instances/{instance}/overview?project={project_id}"
+CLOUD_SQL_INSTANCE_DATABASE_LINK = (
+ CLOUD_SQL_BASE_LINK + "/instances/{instance}/databases?project={project_id}"
+)
+
+
+class CloudSQLInstanceLink(BaseGoogleLink):
+ """Helper class for constructing Cloud SQL Instance Link"""
+
+ name = "Cloud SQL Instance"
+ key = "cloud_sql_instance"
+ format_str = CLOUD_SQL_INSTANCE_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance: BaseOperator,
+ cloud_sql_instance: str,
+ project_id: Optional[str],
+ ):
+ task_instance.xcom_push(
+ context,
+ key=CloudSQLInstanceLink.key,
+ value={"instance": cloud_sql_instance, "project_id": project_id},
+ )
+
+
+class CloudSQLInstanceDatabaseLink(BaseGoogleLink):
+ """Helper class for constructing Cloud SQL Instance Database Link"""
+
+ name = "Cloud SQL Instance Database"
+ key = "cloud_sql_instance_database"
+ format_str = CLOUD_SQL_INSTANCE_DATABASE_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance: BaseOperator,
+ cloud_sql_instance: str,
+ project_id: Optional[str],
+ ):
+ task_instance.xcom_push(
+ context,
+ key=CloudSQLInstanceDatabaseLink.key,
+ value={"instance": cloud_sql_instance, "project_id": project_id},
+ )
diff --git a/airflow/providers/google/cloud/links/cloud_tasks.py b/airflow/providers/google/cloud/links/cloud_tasks.py
new file mode 100644
index 0000000000000..16a28ee4b62eb
--- /dev/null
+++ b/airflow/providers/google/cloud/links/cloud_tasks.py
@@ -0,0 +1,81 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains Google Cloud Tasks links."""
+from typing import TYPE_CHECKING, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.google.cloud.links.base import BaseGoogleLink
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+CLOUD_TASKS_BASE_LINK = "https://pantheon.corp.google.com/cloudtasks"
+CLOUD_TASKS_QUEUE_LINK = CLOUD_TASKS_BASE_LINK + "/queue/{location}/{queue_id}/tasks?project={project_id}"
+CLOUD_TASKS_LINK = CLOUD_TASKS_BASE_LINK + "?project={project_id}"
+
+
+class CloudTasksQueueLink(BaseGoogleLink):
+ """Helper class for constructing Cloud Task Queue Link"""
+
+ name = "Cloud Tasks Queue"
+ key = "cloud_task_queue"
+ format_str = CLOUD_TASKS_QUEUE_LINK
+
+ @staticmethod
+ def extract_parts(queue_name: Optional[str]):
+ """
+ Extract project_id, location and queue id from queue name:
+ projects/PROJECT_ID/locations/LOCATION_ID/queues/QUEUE_ID
+ """
+ if not queue_name:
+ return "", "", ""
+ parts = queue_name.split("/")
+ return parts[1], parts[3], parts[5]
+
+ @staticmethod
+ def persist(
+ operator_instance: BaseOperator,
+ context: "Context",
+ queue_name: Optional[str],
+ ):
+ project_id, location, queue_id = CloudTasksQueueLink.extract_parts(queue_name)
+ operator_instance.xcom_push(
+ context,
+ key=CloudTasksQueueLink.key,
+ value={"project_id": project_id, "location": location, "queue_id": queue_id},
+ )
+
+
+class CloudTasksLink(BaseGoogleLink):
+ """Helper class for constructing Cloud Task Link"""
+
+ name = "Cloud Tasks"
+ key = "cloud_task"
+ format_str = CLOUD_TASKS_LINK
+
+ @staticmethod
+ def persist(
+ operator_instance: BaseOperator,
+ context: "Context",
+ project_id: Optional[str],
+ ):
+ operator_instance.xcom_push(
+ context,
+ key=CloudTasksLink.key,
+ value={"project_id": project_id},
+ )
diff --git a/airflow/providers/google/cloud/links/datastore.py b/airflow/providers/google/cloud/links/datastore.py
new file mode 100644
index 0000000000000..d17a6a8ef0ef2
--- /dev/null
+++ b/airflow/providers/google/cloud/links/datastore.py
@@ -0,0 +1,73 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING
+
+from airflow.providers.google.cloud.links.base import BaseGoogleLink
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+BASE_LINK = "https://console.cloud.google.com"
+DATASTORE_BASE_LINK = BASE_LINK + "/datastore"
+DATASTORE_IMPORT_EXPORT_LINK = DATASTORE_BASE_LINK + "/import-export?project={project_id}"
+DATASTORE_EXPORT_ENTITIES_LINK = (
+ BASE_LINK + "/storage/browser/{bucket_name}/{export_name}?project={project_id}"
+)
+DATASTORE_ENTITIES_LINK = DATASTORE_BASE_LINK + "/entities/query/kind?project={project_id}"
+
+
+class CloudDatastoreImportExportLink(BaseGoogleLink):
+ """Helper class for constructing Cloud Datastore Import/Export Link"""
+
+ name = "Import/Export Page"
+ key = "import_export_conf"
+ format_str = DATASTORE_IMPORT_EXPORT_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance,
+ ):
+ task_instance.xcom_push(
+ context=context,
+ key=CloudDatastoreImportExportLink.key,
+ value={
+ "project_id": task_instance.project_id,
+ },
+ )
+
+
+class CloudDatastoreEntitiesLink(BaseGoogleLink):
+ """Helper class for constructing Cloud Datastore Entities Link"""
+
+ name = "Entities"
+ key = "entities_conf"
+ format_str = DATASTORE_ENTITIES_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance,
+ ):
+ task_instance.xcom_push(
+ context=context,
+ key=CloudDatastoreEntitiesLink.key,
+ value={
+ "project_id": task_instance.project_id,
+ },
+ )
diff --git a/airflow/providers/google/cloud/links/spanner.py b/airflow/providers/google/cloud/links/spanner.py
new file mode 100644
index 0000000000000..0834944dc177f
--- /dev/null
+++ b/airflow/providers/google/cloud/links/spanner.py
@@ -0,0 +1,74 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains Google Spanner links."""
+from typing import TYPE_CHECKING, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.google.cloud.links.base import BaseGoogleLink
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+SPANNER_BASE_LINK = "https://console.cloud.google.com/spanner/instances"
+SPANNER_INSTANCE_LINK = SPANNER_BASE_LINK + "/{instance_id}/details/databases?project={project_id}"
+SPANNER_DATABASE_LINK = (
+ SPANNER_BASE_LINK + "/{instance_id}/databases/{database_id}/details/tables?project={project_id}"
+)
+
+
+class SpannerInstanceLink(BaseGoogleLink):
+ """Helper class for constructing Spanner Instance Link"""
+
+ name = "Spanner Instance"
+ key = "spanner_instance"
+ format_str = SPANNER_INSTANCE_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance: BaseOperator,
+ instance_id: str,
+ project_id: Optional[str],
+ ):
+ task_instance.xcom_push(
+ context,
+ key=SpannerInstanceLink.key,
+ value={"instance_id": instance_id, "project_id": project_id},
+ )
+
+
+class SpannerDatabaseLink(BaseGoogleLink):
+ """Helper class for constructing Spanner Database Link"""
+
+ name = "Spanner Database"
+ key = "spanner_database"
+ format_str = SPANNER_DATABASE_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance: BaseOperator,
+ instance_id: str,
+ database_id: str,
+ project_id: Optional[str],
+ ):
+ task_instance.xcom_push(
+ context,
+ key=SpannerDatabaseLink.key,
+ value={"instance_id": instance_id, "database_id": database_id, "project_id": project_id},
+ )
diff --git a/airflow/providers/google/cloud/links/stackdriver.py b/airflow/providers/google/cloud/links/stackdriver.py
new file mode 100644
index 0000000000000..5870266422bc2
--- /dev/null
+++ b/airflow/providers/google/cloud/links/stackdriver.py
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""This module contains Google Stackdriver links."""
+from typing import TYPE_CHECKING, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.google.cloud.links.base import BaseGoogleLink
+
+if TYPE_CHECKING:
+ from airflow.utils.context import Context
+
+STACKDRIVER_BASE_LINK = "https://pantheon.corp.google.com/monitoring/alerting"
+STACKDRIVER_NOTIFICATIONS_LINK = STACKDRIVER_BASE_LINK + "/notifications?project={project_id}"
+STACKDRIVER_POLICIES_LINK = STACKDRIVER_BASE_LINK + "/policies?project={project_id}"
+
+
+class StackdriverNotificationsLink(BaseGoogleLink):
+ """Helper class for constructing Stackdriver Notifications Link"""
+
+ name = "Cloud Monitoring Notifications"
+ key = "stackdriver_notifications"
+ format_str = STACKDRIVER_NOTIFICATIONS_LINK
+
+ @staticmethod
+ def persist(
+ operator_instance: BaseOperator,
+ context: "Context",
+ project_id: Optional[str],
+ ):
+ operator_instance.xcom_push(
+ context,
+ key=StackdriverNotificationsLink.key,
+ value={"project_id": project_id},
+ )
+
+
+class StackdriverPoliciesLink(BaseGoogleLink):
+ """Helper class for constructing Stackdriver Policies Link"""
+
+ name = "Cloud Monitoring Policies"
+ key = "stackdriver_policies"
+ format_str = STACKDRIVER_POLICIES_LINK
+
+ @staticmethod
+ def persist(
+ operator_instance: BaseOperator,
+ context: "Context",
+ project_id: Optional[str],
+ ):
+ operator_instance.xcom_push(
+ context,
+ key=StackdriverPoliciesLink.key,
+ value={"project_id": project_id},
+ )
diff --git a/airflow/providers/google/cloud/links/vertex_ai.py b/airflow/providers/google/cloud/links/vertex_ai.py
index 6642793ac7688..910049c81a5a0 100644
--- a/airflow/providers/google/cloud/links/vertex_ai.py
+++ b/airflow/providers/google/cloud/links/vertex_ai.py
@@ -22,15 +22,35 @@
if TYPE_CHECKING:
from airflow.utils.context import Context
-VERTEX_AI_BASE_LINK = "https://console.cloud.google.com/vertex-ai"
+BASE_LINK = "https://console.cloud.google.com"
+VERTEX_AI_BASE_LINK = BASE_LINK + "/vertex-ai"
VERTEX_AI_MODEL_LINK = (
VERTEX_AI_BASE_LINK + "/locations/{region}/models/{model_id}/deploy?project={project_id}"
)
+VERTEX_AI_MODEL_LIST_LINK = VERTEX_AI_BASE_LINK + "/models?project={project_id}"
+VERTEX_AI_MODEL_EXPORT_LINK = (
+ BASE_LINK + "/storage/browser/{bucket_name}/model-{model_id}?project={project_id}"
+)
+VERTEX_AI_TRAINING_LINK = (
+ VERTEX_AI_BASE_LINK + "/locations/{region}/training/{training_id}/cpu?project={project_id}"
+)
VERTEX_AI_TRAINING_PIPELINES_LINK = VERTEX_AI_BASE_LINK + "/training/training-pipelines?project={project_id}"
VERTEX_AI_DATASET_LINK = (
VERTEX_AI_BASE_LINK + "/locations/{region}/datasets/{dataset_id}/analyze?project={project_id}"
)
VERTEX_AI_DATASET_LIST_LINK = VERTEX_AI_BASE_LINK + "/datasets?project={project_id}"
+VERTEX_AI_HYPERPARAMETER_TUNING_JOB_LIST_LINK = (
+ VERTEX_AI_BASE_LINK + "/training/hyperparameter-tuning-jobs?project={project_id}"
+)
+VERTEX_AI_BATCH_PREDICTION_JOB_LINK = (
+ VERTEX_AI_BASE_LINK
+ + "/locations/{region}/batch-predictions/{batch_prediction_job_id}?project={project_id}"
+)
+VERTEX_AI_BATCH_PREDICTION_JOB_LIST_LINK = VERTEX_AI_BASE_LINK + "/batch-predictions?project={project_id}"
+VERTEX_AI_ENDPOINT_LINK = (
+ VERTEX_AI_BASE_LINK + "/locations/{region}/endpoints/{endpoint_id}?project={project_id}"
+)
+VERTEX_AI_ENDPOINT_LIST_LINK = VERTEX_AI_BASE_LINK + "/endpoints?project={project_id}"
class VertexAIModelLink(BaseGoogleLink):
@@ -57,6 +77,79 @@ def persist(
)
+class VertexAIModelListLink(BaseGoogleLink):
+ """Helper class for constructing Vertex AI Models Link"""
+
+ name = "Model List"
+ key = "models_conf"
+ format_str = VERTEX_AI_MODEL_LIST_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance,
+ ):
+ task_instance.xcom_push(
+ context=context,
+ key=VertexAIModelListLink.key,
+ value={
+ "project_id": task_instance.project_id,
+ },
+ )
+
+
+class VertexAIModelExportLink(BaseGoogleLink):
+ """Helper class for constructing Vertex AI Model Export Link"""
+
+ name = "Export Model"
+ key = "export_conf"
+ format_str = VERTEX_AI_MODEL_EXPORT_LINK
+
+ @staticmethod
+ def extract_bucket_name(config):
+ """Returns bucket name from output configuration."""
+ return config["artifact_destination"]["output_uri_prefix"].rpartition("gs://")[-1]
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance,
+ ):
+ task_instance.xcom_push(
+ context=context,
+ key=VertexAIModelExportLink.key,
+ value={
+ "project_id": task_instance.project_id,
+ "model_id": task_instance.model_id,
+ "bucket_name": VertexAIModelExportLink.extract_bucket_name(task_instance.output_config),
+ },
+ )
+
+
+class VertexAITrainingLink(BaseGoogleLink):
+ """Helper class for constructing Vertex AI Training link"""
+
+ name = "Vertex AI Training"
+ key = "training_conf"
+ format_str = VERTEX_AI_TRAINING_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance,
+ training_id: str,
+ ):
+ task_instance.xcom_push(
+ context=context,
+ key=VertexAITrainingLink.key,
+ value={
+ "training_id": training_id,
+ "region": task_instance.region,
+ "project_id": task_instance.project_id,
+ },
+ )
+
+
class VertexAITrainingPipelinesLink(BaseGoogleLink):
"""Helper class for constructing Vertex AI Training Pipelines link"""
@@ -117,3 +210,114 @@ def persist(
"project_id": task_instance.project_id,
},
)
+
+
+class VertexAIHyperparameterTuningJobListLink(BaseGoogleLink):
+ """Helper class for constructing Vertex AI HyperparameterTuningJobs Link"""
+
+ name = "Hyperparameter Tuning Job List"
+ key = "hyperparameter_tuning_jobs_conf"
+ format_str = VERTEX_AI_HYPERPARAMETER_TUNING_JOB_LIST_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance,
+ ):
+ task_instance.xcom_push(
+ context=context,
+ key=VertexAIHyperparameterTuningJobListLink.key,
+ value={
+ "project_id": task_instance.project_id,
+ },
+ )
+
+
+class VertexAIBatchPredictionJobLink(BaseGoogleLink):
+ """Helper class for constructing Vertex AI BatchPredictionJob link"""
+
+ name = "Batch Prediction Job"
+ key = "batch_prediction_job_conf"
+ format_str = VERTEX_AI_BATCH_PREDICTION_JOB_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance,
+ batch_prediction_job_id: str,
+ ):
+ task_instance.xcom_push(
+ context=context,
+ key=VertexAIBatchPredictionJobLink.key,
+ value={
+ "batch_prediction_job_id": batch_prediction_job_id,
+ "region": task_instance.region,
+ "project_id": task_instance.project_id,
+ },
+ )
+
+
+class VertexAIBatchPredictionJobListLink(BaseGoogleLink):
+ """Helper class for constructing Vertex AI BatchPredictionJobList link"""
+
+ name = "Batch Prediction Job List"
+ key = "batch_prediction_jobs_conf"
+ format_str = VERTEX_AI_BATCH_PREDICTION_JOB_LIST_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance,
+ ):
+ task_instance.xcom_push(
+ context=context,
+ key=VertexAIBatchPredictionJobListLink.key,
+ value={
+ "project_id": task_instance.project_id,
+ },
+ )
+
+
+class VertexAIEndpointLink(BaseGoogleLink):
+ """Helper class for constructing Vertex AI Endpoint link"""
+
+ name = "Endpoint"
+ key = "endpoint_conf"
+ format_str = VERTEX_AI_ENDPOINT_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance,
+ endpoint_id: str,
+ ):
+ task_instance.xcom_push(
+ context=context,
+ key=VertexAIEndpointLink.key,
+ value={
+ "endpoint_id": endpoint_id,
+ "region": task_instance.region,
+ "project_id": task_instance.project_id,
+ },
+ )
+
+
+class VertexAIEndpointListLink(BaseGoogleLink):
+ """Helper class for constructing Vertex AI EndpointList link"""
+
+ name = "Endpoint List"
+ key = "endpoints_conf"
+ format_str = VERTEX_AI_ENDPOINT_LIST_LINK
+
+ @staticmethod
+ def persist(
+ context: "Context",
+ task_instance,
+ ):
+ task_instance.xcom_push(
+ context=context,
+ key=VertexAIEndpointListLink.key,
+ value={
+ "project_id": task_instance.project_id,
+ },
+ )
diff --git a/airflow/providers/google/cloud/operators/automl.py b/airflow/providers/google/cloud/operators/automl.py
index 11a4d5ae0dc15..d42b4f1a248dc 100644
--- a/airflow/providers/google/cloud/operators/automl.py
+++ b/airflow/providers/google/cloud/operators/automl.py
@@ -21,6 +21,7 @@
import ast
from typing import TYPE_CHECKING, Dict, List, Optional, Sequence, Tuple, Union
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
from google.api_core.retry import Retry
from google.cloud.automl_v1beta1 import (
BatchPredictResult,
@@ -83,7 +84,7 @@ def __init__(
project_id: Optional[str] = None,
metadata: MetaData = (),
timeout: Optional[float] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
@@ -168,7 +169,7 @@ def __init__(
project_id: Optional[str] = None,
metadata: MetaData = (),
timeout: Optional[float] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
@@ -263,7 +264,7 @@ def __init__(
prediction_params: Optional[Dict[str, str]] = None,
metadata: MetaData = (),
timeout: Optional[float] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
@@ -349,7 +350,7 @@ def __init__(
project_id: Optional[str] = None,
metadata: MetaData = (),
timeout: Optional[float] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
@@ -435,7 +436,7 @@ def __init__(
project_id: Optional[str] = None,
metadata: MetaData = (),
timeout: Optional[float] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
@@ -530,7 +531,7 @@ def __init__(
project_id: Optional[str] = None,
metadata: MetaData = (),
timeout: Optional[float] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
@@ -618,7 +619,7 @@ def __init__(
update_mask: Optional[dict] = None,
metadata: MetaData = (),
timeout: Optional[float] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
@@ -695,7 +696,7 @@ def __init__(
project_id: Optional[str] = None,
metadata: MetaData = (),
timeout: Optional[float] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
@@ -771,7 +772,7 @@ def __init__(
project_id: Optional[str] = None,
metadata: MetaData = (),
timeout: Optional[float] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
@@ -856,7 +857,7 @@ def __init__(
image_detection_metadata: Optional[dict] = None,
metadata: Sequence[Tuple[str, str]] = (),
timeout: Optional[float] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
@@ -945,7 +946,7 @@ def __init__(
project_id: Optional[str] = None,
metadata: MetaData = (),
timeout: Optional[float] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
@@ -1024,7 +1025,7 @@ def __init__(
project_id: Optional[str] = None,
metadata: MetaData = (),
timeout: Optional[float] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
@@ -1106,7 +1107,7 @@ def __init__(
project_id: Optional[str] = None,
metadata: MetaData = (),
timeout: Optional[float] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
diff --git a/airflow/providers/google/cloud/operators/bigquery.py b/airflow/providers/google/cloud/operators/bigquery.py
index 9dc9eebba1719..268bbc1cad8c2 100644
--- a/airflow/providers/google/cloud/operators/bigquery.py
+++ b/airflow/providers/google/cloud/operators/bigquery.py
@@ -29,6 +29,8 @@
import attr
from google.api_core.exceptions import Conflict
+from google.api_core.retry import Retry
+from google.cloud.bigquery import DEFAULT_RETRY
from airflow.exceptions import AirflowException
from airflow.models import BaseOperator, BaseOperatorLink
@@ -36,6 +38,7 @@
from airflow.operators.sql import SQLCheckOperator, SQLIntervalCheckOperator, SQLValueCheckOperator
from airflow.providers.google.cloud.hooks.bigquery import BigQueryHook, BigQueryJob
from airflow.providers.google.cloud.hooks.gcs import GCSHook, _parse_gcs_url
+from airflow.providers.google.cloud.links.bigquery import BigQueryDatasetLink, BigQueryTableLink
if TYPE_CHECKING:
from airflow.models.taskinstance import TaskInstanceKey
@@ -44,10 +47,6 @@
BIGQUERY_JOB_DETAILS_LINK_FMT = "https://console.cloud.google.com/bigquery?j={job_id}"
-_DEPRECATION_MSG = (
- "The bigquery_conn_id parameter has been deprecated. You should pass the gcp_conn_id parameter."
-)
-
class BigQueryUIColors(enum.Enum):
"""Hex colors for BigQuery operators"""
@@ -159,8 +158,6 @@ class BigQueryCheckOperator(_BigQueryDbHookMixin, SQLCheckOperator):
:param sql: the sql to be executed
:param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
- :param bigquery_conn_id: (Deprecated) The connection ID used to connect to Google Cloud.
- This parameter has been deprecated. You should pass the gcp_conn_id parameter instead.
:param use_legacy_sql: Whether to use legacy SQL (true)
or standard SQL (false).
:param location: The geographic location of the job. See details at:
@@ -190,7 +187,6 @@ def __init__(
*,
sql: str,
gcp_conn_id: str = 'google_cloud_default',
- bigquery_conn_id: Optional[str] = None,
use_legacy_sql: bool = True,
location: Optional[str] = None,
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
@@ -198,10 +194,6 @@ def __init__(
**kwargs,
) -> None:
super().__init__(sql=sql, **kwargs)
- if bigquery_conn_id:
- warnings.warn(_DEPRECATION_MSG, DeprecationWarning, stacklevel=3)
- gcp_conn_id = bigquery_conn_id
-
self.gcp_conn_id = gcp_conn_id
self.sql = sql
self.use_legacy_sql = use_legacy_sql
@@ -222,8 +214,6 @@ class BigQueryValueCheckOperator(_BigQueryDbHookMixin, SQLValueCheckOperator):
:param use_legacy_sql: Whether to use legacy SQL (true)
or standard SQL (false).
:param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
- :param bigquery_conn_id: (Deprecated) The connection ID used to connect to Google Cloud.
- This parameter has been deprecated. You should pass the gcp_conn_id parameter instead.
:param location: The geographic location of the job. See details at:
https://cloud.google.com/bigquery/docs/locations#specifying_your_location
:param impersonation_chain: Optional service account to impersonate using short-term
@@ -254,7 +244,6 @@ def __init__(
pass_value: Any,
tolerance: Any = None,
gcp_conn_id: str = 'google_cloud_default',
- bigquery_conn_id: Optional[str] = None,
use_legacy_sql: bool = True,
location: Optional[str] = None,
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
@@ -262,11 +251,6 @@ def __init__(
**kwargs,
) -> None:
super().__init__(sql=sql, pass_value=pass_value, tolerance=tolerance, **kwargs)
-
- if bigquery_conn_id:
- warnings.warn(_DEPRECATION_MSG, DeprecationWarning, stacklevel=3)
- gcp_conn_id = bigquery_conn_id
-
self.location = location
self.gcp_conn_id = gcp_conn_id
self.use_legacy_sql = use_legacy_sql
@@ -297,8 +281,6 @@ class BigQueryIntervalCheckOperator(_BigQueryDbHookMixin, SQLIntervalCheckOperat
:param use_legacy_sql: Whether to use legacy SQL (true)
or standard SQL (false).
:param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
- :param bigquery_conn_id: (Deprecated) The connection ID used to connect to Google Cloud.
- This parameter has been deprecated. You should pass the gcp_conn_id parameter instead.
:param location: The geographic location of the job. See details at:
https://cloud.google.com/bigquery/docs/locations#specifying_your_location
:param impersonation_chain: Optional service account to impersonate using short-term
@@ -330,7 +312,6 @@ def __init__(
date_filter_column: str = 'ds',
days_back: SupportsAbs[int] = -7,
gcp_conn_id: str = 'google_cloud_default',
- bigquery_conn_id: Optional[str] = None,
use_legacy_sql: bool = True,
location: Optional[str] = None,
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
@@ -345,10 +326,6 @@ def __init__(
**kwargs,
)
- if bigquery_conn_id:
- warnings.warn(_DEPRECATION_MSG, DeprecationWarning, stacklevel=3)
- gcp_conn_id = bigquery_conn_id
-
self.gcp_conn_id = gcp_conn_id
self.use_legacy_sql = use_legacy_sql
self.location = location
@@ -395,8 +372,6 @@ class BigQueryGetDataOperator(BaseOperator):
:param selected_fields: List of fields to return (comma-separated). If
unspecified, all fields are returned.
:param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
- :param bigquery_conn_id: (Deprecated) The connection ID used to connect to Google Cloud.
- This parameter has been deprecated. You should pass the gcp_conn_id parameter instead.
:param delegate_to: The account to impersonate using domain-wide delegation of authority,
if any. For this to work, the service account making the request must have
domain-wide delegation enabled.
@@ -428,7 +403,6 @@ def __init__(
max_results: int = 100,
selected_fields: Optional[str] = None,
gcp_conn_id: str = 'google_cloud_default',
- bigquery_conn_id: Optional[str] = None,
delegate_to: Optional[str] = None,
location: Optional[str] = None,
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
@@ -436,15 +410,6 @@ def __init__(
) -> None:
super().__init__(**kwargs)
- if bigquery_conn_id:
- warnings.warn(
- "The bigquery_conn_id parameter has been deprecated. You should pass "
- "the gcp_conn_id parameter.",
- DeprecationWarning,
- stacklevel=3,
- )
- gcp_conn_id = bigquery_conn_id
-
self.dataset_id = dataset_id
self.table_id = table_id
self.max_results = int(max_results)
@@ -460,11 +425,19 @@ def execute(self, context: 'Context') -> list:
)
hook = BigQueryHook(
- bigquery_conn_id=self.gcp_conn_id,
+ gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
impersonation_chain=self.impersonation_chain,
)
+ if not self.selected_fields:
+ schema: Dict[str, list] = hook.get_schema(
+ dataset_id=self.dataset_id,
+ table_id=self.table_id,
+ )
+ if "fields" in schema:
+ self.selected_fields = ','.join([field["name"] for field in schema["fields"]])
+
rows = hook.list_rows(
dataset_id=self.dataset_id,
table_id=self.table_id,
@@ -503,8 +476,6 @@ class BigQueryExecuteQueryOperator(BaseOperator):
must be ``true`` if this is set to ``false``. For standard SQL queries, this
flag is ignored and results are never flattened.
:param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
- :param bigquery_conn_id: (Deprecated) The connection ID used to connect to Google Cloud.
- This parameter has been deprecated. You should pass the gcp_conn_id parameter instead.
:param delegate_to: The account to impersonate using domain-wide delegation of authority,
if any. For this to work, the service account making the request must have
domain-wide delegation enabled.
@@ -588,7 +559,6 @@ def __init__(
allow_large_results: bool = False,
flatten_results: Optional[bool] = None,
gcp_conn_id: str = 'google_cloud_default',
- bigquery_conn_id: Optional[str] = None,
delegate_to: Optional[str] = None,
udf_config: Optional[list] = None,
use_legacy_sql: bool = True,
@@ -608,16 +578,6 @@ def __init__(
**kwargs,
) -> None:
super().__init__(**kwargs)
-
- if bigquery_conn_id:
- warnings.warn(
- "The bigquery_conn_id parameter has been deprecated. You should pass "
- "the gcp_conn_id parameter.",
- DeprecationWarning,
- stacklevel=2,
- )
- gcp_conn_id = bigquery_conn_id
-
warnings.warn(
"This operator is deprecated. Please use `BigQueryInsertJobOperator`.",
DeprecationWarning,
@@ -839,6 +799,7 @@ class BigQueryCreateEmptyTableOperator(BaseOperator):
)
template_fields_renderers = {"table_resource": "json", "materialized_view": "json"}
ui_color = BigQueryUIColors.TABLE.value
+ operator_extra_links = (BigQueryTableLink(),)
def __init__(
self,
@@ -920,6 +881,13 @@ def execute(self, context: 'Context') -> None:
table_resource=self.table_resource,
exists_ok=self.exists_ok,
)
+ BigQueryTableLink.persist(
+ context=context,
+ task_instance=self,
+ dataset_id=table.to_api_repr()["tableReference"]["datasetId"],
+ project_id=table.to_api_repr()["tableReference"]["projectId"],
+ table_id=table.to_api_repr()["tableReference"]["tableId"],
+ )
self.log.info(
'Table %s.%s.%s created successfully', table.project, table.dataset_id, table.table_id
)
@@ -962,6 +930,9 @@ class BigQueryCreateExternalTableOperator(BaseOperator):
:param schema_object: If set, a GCS object path pointing to a .json file that
contains the schema for the table. (templated)
:param source_format: File format of the data.
+ :param autodetect: Try to detect schema and format options automatically.
+ The schema_fields and schema_object options will be honored when specified explicitly.
+ https://cloud.google.com/bigquery/docs/schema-detect#schema_auto-detection_for_external_data_sources
:param compression: [Optional] The compression type of the data source.
Possible values include GZIP and NONE.
The default value is NONE.
@@ -1015,6 +986,7 @@ class BigQueryCreateExternalTableOperator(BaseOperator):
)
template_fields_renderers = {"table_resource": "json"}
ui_color = BigQueryUIColors.TABLE.value
+ operator_extra_links = (BigQueryTableLink(),)
def __init__(
self,
@@ -1026,6 +998,7 @@ def __init__(
schema_fields: Optional[List] = None,
schema_object: Optional[str] = None,
source_format: Optional[str] = None,
+ autodetect: bool = False,
compression: Optional[str] = None,
skip_leading_rows: Optional[int] = None,
field_delimiter: Optional[str] = None,
@@ -1055,6 +1028,7 @@ def __init__(
skip_leading_rows,
field_delimiter,
max_bad_records,
+ autodetect,
quote_character,
allow_quoted_newlines,
allow_jagged_rows,
@@ -1114,6 +1088,7 @@ def __init__(
self.bigquery_conn_id = bigquery_conn_id
self.google_cloud_storage_conn_id = google_cloud_storage_conn_id
self.delegate_to = delegate_to
+ self.autodetect = autodetect
self.src_fmt_configs = src_fmt_configs or {}
self.labels = labels
@@ -1129,9 +1104,16 @@ def execute(self, context: 'Context') -> None:
impersonation_chain=self.impersonation_chain,
)
if self.table_resource:
- bq_hook.create_empty_table(
+ table = bq_hook.create_empty_table(
table_resource=self.table_resource,
)
+ BigQueryTableLink.persist(
+ context=context,
+ task_instance=self,
+ dataset_id=table.to_api_repr()["tableReference"]["datasetId"],
+ project_id=table.to_api_repr()["tableReference"]["projectId"],
+ table_id=table.to_api_repr()["tableReference"]["tableId"],
+ )
return
if not self.schema_fields and self.schema_object and self.source_format != 'DATASTORE_BACKUP':
@@ -1146,11 +1128,12 @@ def execute(self, context: 'Context') -> None:
source_uris = [f"gs://{self.bucket}/{source_object}" for source_object in self.source_objects]
- bq_hook.create_external_table(
+ table = bq_hook.create_external_table(
external_project_dataset_table=self.destination_project_dataset_table,
schema_fields=schema_fields,
source_uris=source_uris,
source_format=self.source_format,
+ autodetect=self.autodetect,
compression=self.compression,
skip_leading_rows=self.skip_leading_rows,
field_delimiter=self.field_delimiter,
@@ -1162,6 +1145,13 @@ def execute(self, context: 'Context') -> None:
labels=self.labels,
encryption_configuration=self.encryption_configuration,
)
+ BigQueryTableLink.persist(
+ context=context,
+ task_instance=self,
+ dataset_id=table.to_api_repr()["tableReference"]["datasetId"],
+ project_id=table.to_api_repr()["tableReference"]["projectId"],
+ table_id=table.to_api_repr()["tableReference"]["tableId"],
+ )
class BigQueryDeleteDatasetOperator(BaseOperator):
@@ -1180,8 +1170,6 @@ class BigQueryDeleteDatasetOperator(BaseOperator):
Will raise HttpError 400: "{dataset_id} is still in use" if set to False and dataset is not empty.
The default value is False.
:param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
- :param bigquery_conn_id: (Deprecated) The connection ID used to connect to Google Cloud.
- This parameter has been deprecated. You should pass the gcp_conn_id parameter instead.
:param delegate_to: The account to impersonate using domain-wide delegation of authority,
if any. For this to work, the service account making the request must have
domain-wide delegation enabled.
@@ -1219,20 +1207,10 @@ def __init__(
project_id: Optional[str] = None,
delete_contents: bool = False,
gcp_conn_id: str = 'google_cloud_default',
- bigquery_conn_id: Optional[str] = None,
delegate_to: Optional[str] = None,
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
**kwargs,
) -> None:
- if bigquery_conn_id:
- warnings.warn(
- "The bigquery_conn_id parameter has been deprecated. You should pass "
- "the gcp_conn_id parameter.",
- DeprecationWarning,
- stacklevel=3,
- )
- gcp_conn_id = bigquery_conn_id
-
self.dataset_id = dataset_id
self.project_id = project_id
self.delete_contents = delete_contents
@@ -1272,8 +1250,6 @@ class BigQueryCreateEmptyDatasetOperator(BaseOperator):
More info:
https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets#resource
:param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
- :param bigquery_conn_id: (Deprecated) The connection ID used to connect to Google Cloud.
- This parameter has been deprecated. You should pass the gcp_conn_id parameter instead.
:param delegate_to: The account to impersonate using domain-wide delegation of authority,
if any. For this to work, the service account making the request must have
domain-wide delegation enabled.
@@ -1305,6 +1281,7 @@ class BigQueryCreateEmptyDatasetOperator(BaseOperator):
)
template_fields_renderers = {"dataset_reference": "json"}
ui_color = BigQueryUIColors.DATASET.value
+ operator_extra_links = (BigQueryDatasetLink(),)
def __init__(
self,
@@ -1314,22 +1291,12 @@ def __init__(
dataset_reference: Optional[Dict] = None,
location: Optional[str] = None,
gcp_conn_id: str = 'google_cloud_default',
- bigquery_conn_id: Optional[str] = None,
delegate_to: Optional[str] = None,
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
exists_ok: bool = False,
**kwargs,
) -> None:
- if bigquery_conn_id:
- warnings.warn(
- "The bigquery_conn_id parameter has been deprecated. You should pass "
- "the gcp_conn_id parameter.",
- DeprecationWarning,
- stacklevel=3,
- )
- gcp_conn_id = bigquery_conn_id
-
self.dataset_id = dataset_id
self.project_id = project_id
self.location = location
@@ -1350,13 +1317,19 @@ def execute(self, context: 'Context') -> None:
)
try:
- bq_hook.create_empty_dataset(
+ dataset = bq_hook.create_empty_dataset(
project_id=self.project_id,
dataset_id=self.dataset_id,
dataset_reference=self.dataset_reference,
location=self.location,
exists_ok=self.exists_ok,
)
+ BigQueryDatasetLink.persist(
+ context=context,
+ task_instance=self,
+ dataset_id=dataset["datasetReference"]["datasetId"],
+ project_id=dataset["datasetReference"]["projectId"],
+ )
except Conflict:
dataset_id = self.dataset_reference.get("datasetReference", {}).get("datasetId", self.dataset_id)
self.log.info('Dataset %s already exists.', dataset_id)
@@ -1397,6 +1370,7 @@ class BigQueryGetDatasetOperator(BaseOperator):
'impersonation_chain',
)
ui_color = BigQueryUIColors.DATASET.value
+ operator_extra_links = (BigQueryDatasetLink(),)
def __init__(
self,
@@ -1425,7 +1399,14 @@ def execute(self, context: 'Context'):
self.log.info('Start getting dataset: %s:%s', self.project_id, self.dataset_id)
dataset = bq_hook.get_dataset(dataset_id=self.dataset_id, project_id=self.project_id)
- return dataset.to_api_repr()
+ dataset = dataset.to_api_repr()
+ BigQueryDatasetLink.persist(
+ context=context,
+ task_instance=self,
+ dataset_id=dataset["datasetReference"]["datasetId"],
+ project_id=dataset["datasetReference"]["projectId"],
+ )
+ return dataset
class BigQueryGetDatasetTablesOperator(BaseOperator):
@@ -1616,6 +1597,7 @@ class BigQueryUpdateTableOperator(BaseOperator):
)
template_fields_renderers = {"table_resource": "json"}
ui_color = BigQueryUIColors.TABLE.value
+ operator_extra_links = (BigQueryTableLink(),)
def __init__(
self,
@@ -1647,7 +1629,7 @@ def execute(self, context: 'Context'):
impersonation_chain=self.impersonation_chain,
)
- return bq_hook.update_table(
+ table = bq_hook.update_table(
table_resource=self.table_resource,
fields=self.fields,
dataset_id=self.dataset_id,
@@ -1655,6 +1637,16 @@ def execute(self, context: 'Context'):
project_id=self.project_id,
)
+ BigQueryTableLink.persist(
+ context=context,
+ task_instance=self,
+ dataset_id=table["tableReference"]["datasetId"],
+ project_id=table["tableReference"]["projectId"],
+ table_id=table["tableReference"]["tableId"],
+ )
+
+ return table
+
class BigQueryUpdateDatasetOperator(BaseOperator):
"""
@@ -1699,6 +1691,7 @@ class BigQueryUpdateDatasetOperator(BaseOperator):
)
template_fields_renderers = {"dataset_resource": "json"}
ui_color = BigQueryUIColors.DATASET.value
+ operator_extra_links = (BigQueryDatasetLink(),)
def __init__(
self,
@@ -1735,7 +1728,15 @@ def execute(self, context: 'Context'):
dataset_id=self.dataset_id,
fields=fields,
)
- return dataset.to_api_repr()
+
+ dataset = dataset.to_api_repr()
+ BigQueryDatasetLink.persist(
+ context=context,
+ task_instance=self,
+ dataset_id=dataset["datasetReference"]["datasetId"],
+ project_id=dataset["datasetReference"]["projectId"],
+ )
+ return dataset
class BigQueryDeleteTableOperator(BaseOperator):
@@ -1750,8 +1751,6 @@ class BigQueryDeleteTableOperator(BaseOperator):
``(.|:).`` that indicates which table
will be deleted. (templated)
:param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
- :param bigquery_conn_id: (Deprecated) The connection ID used to connect to Google Cloud.
- This parameter has been deprecated. You should pass the gcp_conn_id parameter instead.
:param delegate_to: The account to impersonate using domain-wide delegation of authority,
if any. For this to work, the service account making the request must have
domain-wide delegation enabled.
@@ -1779,7 +1778,6 @@ def __init__(
*,
deletion_dataset_table: str,
gcp_conn_id: str = 'google_cloud_default',
- bigquery_conn_id: Optional[str] = None,
delegate_to: Optional[str] = None,
ignore_if_missing: bool = False,
location: Optional[str] = None,
@@ -1788,15 +1786,6 @@ def __init__(
) -> None:
super().__init__(**kwargs)
- if bigquery_conn_id:
- warnings.warn(
- "The bigquery_conn_id parameter has been deprecated. You should pass "
- "the gcp_conn_id parameter.",
- DeprecationWarning,
- stacklevel=3,
- )
- gcp_conn_id = bigquery_conn_id
-
self.deletion_dataset_table = deletion_dataset_table
self.gcp_conn_id = gcp_conn_id
self.delegate_to = delegate_to
@@ -1831,8 +1820,6 @@ class BigQueryUpsertTableOperator(BaseOperator):
:param project_id: The name of the project where we want to update the dataset.
Don't need to provide, if projectId in dataset_reference.
:param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
- :param bigquery_conn_id: (Deprecated) The connection ID used to connect to Google Cloud.
- This parameter has been deprecated. You should pass the gcp_conn_id parameter instead.
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
@@ -1854,6 +1841,7 @@ class BigQueryUpsertTableOperator(BaseOperator):
)
template_fields_renderers = {"table_resource": "json"}
ui_color = BigQueryUIColors.TABLE.value
+ operator_extra_links = (BigQueryTableLink(),)
def __init__(
self,
@@ -1862,7 +1850,6 @@ def __init__(
table_resource: dict,
project_id: Optional[str] = None,
gcp_conn_id: str = 'google_cloud_default',
- bigquery_conn_id: Optional[str] = None,
delegate_to: Optional[str] = None,
location: Optional[str] = None,
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
@@ -1870,15 +1857,6 @@ def __init__(
) -> None:
super().__init__(**kwargs)
- if bigquery_conn_id:
- warnings.warn(
- "The bigquery_conn_id parameter has been deprecated. You should pass "
- "the gcp_conn_id parameter.",
- DeprecationWarning,
- stacklevel=3,
- )
- gcp_conn_id = bigquery_conn_id
-
self.dataset_id = dataset_id
self.table_resource = table_resource
self.project_id = project_id
@@ -1890,16 +1868,23 @@ def __init__(
def execute(self, context: 'Context') -> None:
self.log.info('Upserting Dataset: %s with table_resource: %s', self.dataset_id, self.table_resource)
hook = BigQueryHook(
- bigquery_conn_id=self.gcp_conn_id,
+ gcp_conn_id=self.gcp_conn_id,
delegate_to=self.delegate_to,
location=self.location,
impersonation_chain=self.impersonation_chain,
)
- hook.run_table_upsert(
+ table = hook.run_table_upsert(
dataset_id=self.dataset_id,
table_resource=self.table_resource,
project_id=self.project_id,
)
+ BigQueryTableLink.persist(
+ context=context,
+ task_instance=self,
+ dataset_id=table["tableReference"]["datasetId"],
+ project_id=table["tableReference"]["projectId"],
+ table_id=table["tableReference"]["tableId"],
+ )
class BigQueryUpdateTableSchemaOperator(BaseOperator):
@@ -1938,8 +1923,6 @@ class BigQueryUpdateTableSchemaOperator(BaseOperator):
:param project_id: The name of the project where we want to update the dataset.
Don't need to provide, if projectId in dataset_reference.
:param gcp_conn_id: (Optional) The connection ID used to connect to Google Cloud.
- :param bigquery_conn_id: (Deprecated) The connection ID used to connect to Google Cloud.
- This parameter has been deprecated. You should pass the gcp_conn_id parameter instead.
:param delegate_to: The account to impersonate, if any.
For this to work, the service account making the request must have domain-wide
delegation enabled.
@@ -1963,6 +1946,7 @@ class BigQueryUpdateTableSchemaOperator(BaseOperator):
)
template_fields_renderers = {"schema_fields_updates": "json"}
ui_color = BigQueryUIColors.TABLE.value
+ operator_extra_links = (BigQueryTableLink(),)
def __init__(
self,
@@ -1994,7 +1978,7 @@ def execute(self, context: 'Context'):
impersonation_chain=self.impersonation_chain,
)
- return bq_hook.update_table_schema(
+ table = bq_hook.update_table_schema(
schema_fields_updates=self.schema_fields_updates,
include_policy_tags=self.include_policy_tags,
dataset_id=self.dataset_id,
@@ -2002,6 +1986,15 @@ def execute(self, context: 'Context'):
project_id=self.project_id,
)
+ BigQueryTableLink.persist(
+ context=context,
+ task_instance=self,
+ dataset_id=table["tableReference"]["datasetId"],
+ project_id=table["tableReference"]["projectId"],
+ table_id=table["tableReference"]["tableId"],
+ )
+ return table
+
class BigQueryInsertJobOperator(BaseOperator):
"""
@@ -2052,6 +2045,8 @@ class BigQueryInsertJobOperator(BaseOperator):
Service Account Token Creator IAM role to the directly preceding identity, with first
account from the list granting this role to the originating account (templated).
:param cancel_on_kill: Flag which indicates whether cancel the hook's job or not, when on_kill is called
+ :param result_retry: How to retry the `result` call that retrieves rows
+ :param result_timeout: The number of seconds to wait for `result` method before using `result_retry`
"""
template_fields: Sequence[str] = (
@@ -2059,9 +2054,13 @@ class BigQueryInsertJobOperator(BaseOperator):
"job_id",
"impersonation_chain",
)
- template_ext: Sequence[str] = (".json",)
+ template_ext: Sequence[str] = (
+ ".json",
+ ".sql",
+ )
template_fields_renderers = {"configuration": "json", "configuration.query.query": "sql"}
ui_color = BigQueryUIColors.QUERY.value
+ operator_extra_links = (BigQueryTableLink(),)
def __init__(
self,
@@ -2075,6 +2074,8 @@ def __init__(
delegate_to: Optional[str] = None,
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
cancel_on_kill: bool = True,
+ result_retry: Retry = DEFAULT_RETRY,
+ result_timeout: Optional[float] = None,
**kwargs,
) -> None:
super().__init__(**kwargs)
@@ -2088,6 +2089,8 @@ def __init__(
self.reattach_states: Set[str] = reattach_states or set()
self.impersonation_chain = impersonation_chain
self.cancel_on_kill = cancel_on_kill
+ self.result_retry = result_retry
+ self.result_timeout = result_timeout
self.hook: Optional[BigQueryHook] = None
def prepare_template(self) -> None:
@@ -2107,6 +2110,8 @@ def _submit_job(
project_id=self.project_id,
location=self.location,
job_id=job_id,
+ timeout=self.result_timeout,
+ retry=self.result_retry,
)
@staticmethod
@@ -2140,6 +2145,7 @@ def execute(self, context: Any):
job_id = self._job_id(context)
try:
+ self.log.info(f"Executing: {self.configuration}")
job = self._submit_job(hook, job_id)
self._handle_job_error(job)
except Conflict:
@@ -2151,7 +2157,7 @@ def execute(self, context: Any):
)
if job.state in self.reattach_states:
# We are reattaching to a job
- job.result()
+ job.result(timeout=self.result_timeout, retry=self.result_retry)
self._handle_job_error(job)
else:
# Same job configuration so we need force_rerun
@@ -2161,6 +2167,14 @@ def execute(self, context: Any):
f"Or, if you want to reattach in this scenario add {job.state} to `reattach_states`"
)
+ table = job.to_api_repr()["configuration"]["query"]["destinationTable"]
+ BigQueryTableLink.persist(
+ context=context,
+ task_instance=self,
+ dataset_id=table["datasetId"],
+ project_id=table["projectId"],
+ table_id=table["tableId"],
+ )
self.job_id = job.job_id
return job.job_id
diff --git a/airflow/providers/google/cloud/operators/bigquery_dts.py b/airflow/providers/google/cloud/operators/bigquery_dts.py
index c3bb734ba5e6d..54c87505897ca 100644
--- a/airflow/providers/google/cloud/operators/bigquery_dts.py
+++ b/airflow/providers/google/cloud/operators/bigquery_dts.py
@@ -18,16 +18,23 @@
"""This module contains Google BigQuery Data Transfer Service operators."""
from typing import TYPE_CHECKING, Optional, Sequence, Tuple, Union
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
from google.api_core.retry import Retry
from google.cloud.bigquery_datatransfer_v1 import StartManualTransferRunsResponse, TransferConfig
from airflow.models import BaseOperator
from airflow.providers.google.cloud.hooks.bigquery_dts import BiqQueryDataTransferServiceHook, get_object_id
+from airflow.providers.google.cloud.links.bigquery_dts import BigQueryDataTransferConfigLink
if TYPE_CHECKING:
from airflow.utils.context import Context
+def _get_transfer_config_details(config_transfer_name: str):
+ config_details = config_transfer_name.split("/")
+ return {"project_id": config_details[1], "region": config_details[3], "config_id": config_details[5]}
+
+
class BigQueryCreateDataTransferOperator(BaseOperator):
"""
Creates a new data transfer configuration.
@@ -67,6 +74,7 @@ class BigQueryCreateDataTransferOperator(BaseOperator):
"gcp_conn_id",
"impersonation_chain",
)
+ operator_extra_links = (BigQueryDataTransferConfigLink(),)
def __init__(
self,
@@ -75,7 +83,7 @@ def __init__(
project_id: Optional[str] = None,
location: Optional[str] = None,
authorization_code: Optional[str] = None,
- retry: Retry = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id="google_cloud_default",
@@ -106,6 +114,16 @@ def execute(self, context: 'Context'):
timeout=self.timeout,
metadata=self.metadata,
)
+
+ transfer_config = _get_transfer_config_details(response.name)
+ BigQueryDataTransferConfigLink.persist(
+ context=context,
+ task_instance=self,
+ region=transfer_config["region"],
+ config_id=transfer_config["config_id"],
+ project_id=transfer_config["project_id"],
+ )
+
result = TransferConfig.to_dict(response)
self.log.info("Created DTS transfer config %s", get_object_id(result))
self.xcom_push(context, key="transfer_config_id", value=get_object_id(result))
@@ -154,7 +172,7 @@ def __init__(
transfer_config_id: str,
project_id: Optional[str] = None,
location: Optional[str] = None,
- retry: Retry = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id="google_cloud_default",
@@ -231,6 +249,7 @@ class BigQueryDataTransferServiceStartTransferRunsOperator(BaseOperator):
"gcp_conn_id",
"impersonation_chain",
)
+ operator_extra_links = (BigQueryDataTransferConfigLink(),)
def __init__(
self,
@@ -240,7 +259,7 @@ def __init__(
location: Optional[str] = None,
requested_time_range: Optional[dict] = None,
requested_run_time: Optional[dict] = None,
- retry: Retry = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id="google_cloud_default",
@@ -273,6 +292,16 @@ def execute(self, context: 'Context'):
timeout=self.timeout,
metadata=self.metadata,
)
+
+ transfer_config = _get_transfer_config_details(response.runs[0].name)
+ BigQueryDataTransferConfigLink.persist(
+ context=context,
+ task_instance=self,
+ region=transfer_config["region"],
+ config_id=transfer_config["config_id"],
+ project_id=transfer_config["project_id"],
+ )
+
result = StartManualTransferRunsResponse.to_dict(response)
run_id = get_object_id(result['runs'][0])
self.xcom_push(context, key="run_id", value=run_id)
diff --git a/airflow/providers/google/cloud/operators/bigtable.py b/airflow/providers/google/cloud/operators/bigtable.py
index 7520a655ca285..fdecd22089a27 100644
--- a/airflow/providers/google/cloud/operators/bigtable.py
+++ b/airflow/providers/google/cloud/operators/bigtable.py
@@ -26,6 +26,11 @@
from airflow.exceptions import AirflowException
from airflow.models import BaseOperator
from airflow.providers.google.cloud.hooks.bigtable import BigtableHook
+from airflow.providers.google.cloud.links.bigtable import (
+ BigtableClusterLink,
+ BigtableInstanceLink,
+ BigtableTablesLink,
+)
if TYPE_CHECKING:
from airflow.utils.context import Context
@@ -65,9 +70,6 @@ class BigtableCreateInstanceOperator(BaseOperator, BigtableValidationMixin):
:param replica_clusters: (optional) A list of replica clusters for the new
instance. Each cluster dictionary contains an id and a zone.
Example: [{"id": "replica-1", "zone": "us-west1-a"}]
- :param replica_cluster_id: (deprecated) The ID for replica cluster for the new
- instance.
- :param replica_cluster_zone: (deprecated) The zone for replica cluster.
:param instance_type: (optional) The type of the instance.
:param instance_display_name: (optional) Human-readable name of the instance. Defaults
to ``instance_id``.
@@ -96,6 +98,7 @@ class BigtableCreateInstanceOperator(BaseOperator, BigtableValidationMixin):
'main_cluster_zone',
'impersonation_chain',
)
+ operator_extra_links = (BigtableInstanceLink(),)
def __init__(
self,
@@ -105,8 +108,6 @@ def __init__(
main_cluster_zone: str,
project_id: Optional[str] = None,
replica_clusters: Optional[List[Dict[str, str]]] = None,
- replica_cluster_id: Optional[str] = None,
- replica_cluster_zone: Optional[str] = None,
instance_display_name: Optional[str] = None,
instance_type: Optional[enums.Instance.Type] = None,
instance_labels: Optional[Dict] = None,
@@ -122,8 +123,6 @@ def __init__(
self.main_cluster_id = main_cluster_id
self.main_cluster_zone = main_cluster_zone
self.replica_clusters = replica_clusters
- self.replica_cluster_id = replica_cluster_id
- self.replica_cluster_zone = replica_cluster_zone
self.instance_display_name = instance_display_name
self.instance_type = instance_type
self.instance_labels = instance_labels
@@ -148,6 +147,7 @@ def execute(self, context: 'Context') -> None:
"The instance '%s' already exists in this project. Consider it as created",
self.instance_id,
)
+ BigtableInstanceLink.persist(context=context, task_instance=self)
return
try:
hook.create_instance(
@@ -156,8 +156,6 @@ def execute(self, context: 'Context') -> None:
main_cluster_id=self.main_cluster_id,
main_cluster_zone=self.main_cluster_zone,
replica_clusters=self.replica_clusters,
- replica_cluster_id=self.replica_cluster_id,
- replica_cluster_zone=self.replica_cluster_zone,
instance_display_name=self.instance_display_name,
instance_type=self.instance_type,
instance_labels=self.instance_labels,
@@ -165,6 +163,7 @@ def execute(self, context: 'Context') -> None:
cluster_storage_type=self.cluster_storage_type,
timeout=self.timeout,
)
+ BigtableInstanceLink.persist(context=context, task_instance=self)
except google.api_core.exceptions.GoogleAPICallError as e:
self.log.error('An error occurred. Exiting.')
raise e
@@ -207,6 +206,7 @@ class BigtableUpdateInstanceOperator(BaseOperator, BigtableValidationMixin):
'instance_id',
'impersonation_chain',
)
+ operator_extra_links = (BigtableInstanceLink(),)
def __init__(
self,
@@ -250,6 +250,7 @@ def execute(self, context: 'Context') -> None:
instance_labels=self.instance_labels,
timeout=self.timeout,
)
+ BigtableInstanceLink.persist(context=context, task_instance=self)
except google.api_core.exceptions.GoogleAPICallError as e:
self.log.error('An error occurred. Exiting.')
raise e
@@ -360,6 +361,7 @@ class BigtableCreateTableOperator(BaseOperator, BigtableValidationMixin):
'table_id',
'impersonation_chain',
)
+ operator_extra_links = (BigtableTablesLink(),)
def __init__(
self,
@@ -421,6 +423,7 @@ def execute(self, context: 'Context') -> None:
initial_split_keys=self.initial_split_keys,
column_families=self.column_families,
)
+ BigtableTablesLink.persist(context=context, task_instance=self)
except google.api_core.exceptions.AlreadyExists:
if not self._compare_column_families(hook, instance):
raise AirflowException(
@@ -542,6 +545,7 @@ class BigtableUpdateClusterOperator(BaseOperator, BigtableValidationMixin):
'nodes',
'impersonation_chain',
)
+ operator_extra_links = (BigtableClusterLink(),)
def __init__(
self,
@@ -574,6 +578,7 @@ def execute(self, context: 'Context') -> None:
try:
hook.update_cluster(instance=instance, cluster_id=self.cluster_id, nodes=self.nodes)
+ BigtableClusterLink.persist(context=context, task_instance=self)
except google.api_core.exceptions.NotFound:
raise AirflowException(
f"Dependency: cluster '{self.cluster_id}' does not exist for instance '{self.instance_id}'."
diff --git a/airflow/providers/google/cloud/operators/cloud_build.py b/airflow/providers/google/cloud/operators/cloud_build.py
index fdb6c21efcdb7..c377af6732d7f 100644
--- a/airflow/providers/google/cloud/operators/cloud_build.py
+++ b/airflow/providers/google/cloud/operators/cloud_build.py
@@ -20,12 +20,12 @@
import json
import re
-import warnings
from copy import deepcopy
from typing import TYPE_CHECKING, Any, Dict, Optional, Sequence, Tuple, Union
from urllib.parse import unquote, urlparse
import yaml
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
from google.api_core.retry import Retry
from google.cloud.devtools.cloudbuild_v1.types import Build, BuildTrigger, RepoSource
@@ -76,7 +76,7 @@ def __init__(
*,
id_: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -112,11 +112,8 @@ class CloudBuildCreateBuildOperator(BaseOperator):
For more information on how to use this operator, take a look at the guide:
:ref:`howto/operator:CloudBuildCreateBuildOperator`
- :param build: Optional, the build resource to create. If a dict is provided, it must be of
+ :param build: The build resource to create. If a dict is provided, it must be of
the same form as the protobuf message `google.cloud.devtools.cloudbuild_v1.types.Build`.
- Only either build or body should be passed.
- :param body: (Deprecated) The build resource to create.
- This parameter has been deprecated. You should pass the build parameter instead.
:param project_id: Optional, Google Cloud Project project_id where the function belongs.
If set to None or missing, the default project_id from the GCP connection is used.
:param wait: Optional, wait for operation to finish.
@@ -138,16 +135,15 @@ class CloudBuildCreateBuildOperator(BaseOperator):
:rtype: dict
"""
- template_fields: Sequence[str] = ("project_id", "build", "body", "gcp_conn_id", "impersonation_chain")
+ template_fields: Sequence[str] = ("project_id", "build", "gcp_conn_id", "impersonation_chain")
def __init__(
self,
*,
- build: Optional[Union[Dict, Build]] = None,
- body: Optional[Dict] = None,
+ build: Union[Dict, Build],
project_id: Optional[str] = None,
wait: bool = True,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -162,25 +158,9 @@ def __init__(
self.metadata = metadata
self.gcp_conn_id = gcp_conn_id
self.impersonation_chain = impersonation_chain
- self.body = body
-
- if body and build:
- raise AirflowException("You should not pass both build or body parameters. Both are set.")
- if body is not None:
- warnings.warn(
- "The body parameter has been deprecated. You should pass body using the build parameter.",
- DeprecationWarning,
- stacklevel=4,
- )
- actual_build = body
- else:
- if build is None:
- raise AirflowException("You should pass one of the build or body parameters. Both are None")
- actual_build = build
-
- self.build = actual_build
+ self.build = build
# Not template fields to keep original value
- self.build_raw = actual_build
+ self.build_raw = build
def prepare_template(self) -> None:
# if no file is specified, skip
@@ -245,7 +225,7 @@ def __init__(
*,
trigger: Union[dict, BuildTrigger],
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -307,7 +287,7 @@ def __init__(
*,
trigger_id: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -370,7 +350,7 @@ def __init__(
*,
id_: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -434,7 +414,7 @@ def __init__(
*,
trigger_id: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -502,7 +482,7 @@ def __init__(
project_id: Optional[str] = None,
page_size: Optional[int] = None,
page_token: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -574,7 +554,7 @@ def __init__(
project_id: Optional[str] = None,
page_size: Optional[int] = None,
filter_: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -645,7 +625,7 @@ def __init__(
id_: str,
project_id: Optional[str] = None,
wait: bool = True,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -716,7 +696,7 @@ def __init__(
source: Union[dict, RepoSource],
project_id: Optional[str] = None,
wait: bool = True,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -787,7 +767,7 @@ def __init__(
trigger_id: str,
trigger: Union[dict, BuildTrigger],
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
diff --git a/airflow/providers/google/cloud/operators/cloud_composer.py b/airflow/providers/google/cloud/operators/cloud_composer.py
index 589914133d72b..8c4a8a533480a 100644
--- a/airflow/providers/google/cloud/operators/cloud_composer.py
+++ b/airflow/providers/google/cloud/operators/cloud_composer.py
@@ -18,6 +18,7 @@
from typing import TYPE_CHECKING, Dict, Optional, Sequence, Tuple, Union
from google.api_core.exceptions import AlreadyExists
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
from google.api_core.retry import Retry
from google.cloud.orchestration.airflow.service_v1 import ImageVersion
from google.cloud.orchestration.airflow.service_v1.types import Environment
@@ -133,7 +134,7 @@ def __init__(
gcp_conn_id: str = "google_cloud_default",
impersonation_chain: Optional[Union[str, Sequence[str]]] = None,
delegate_to: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
deferrable: bool = False,
@@ -265,7 +266,7 @@ def __init__(
project_id: str,
region: str,
environment_id: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -361,7 +362,7 @@ def __init__(
project_id: str,
region: str,
environment_id: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -441,7 +442,7 @@ def __init__(
region: str,
page_size: Optional[int] = None,
page_token: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -529,7 +530,7 @@ def __init__(
environment_id: str,
environment: Union[Dict, Environment],
update_mask: Union[Dict, FieldMask],
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -647,7 +648,7 @@ def __init__(
page_size: Optional[int] = None,
page_token: Optional[str] = None,
include_past_releases: bool = False,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
diff --git a/airflow/providers/google/cloud/operators/cloud_memorystore.py b/airflow/providers/google/cloud/operators/cloud_memorystore.py
index d02bdb09b3495..cbb4ccd025fb3 100644
--- a/airflow/providers/google/cloud/operators/cloud_memorystore.py
+++ b/airflow/providers/google/cloud/operators/cloud_memorystore.py
@@ -25,6 +25,7 @@
"""
from typing import TYPE_CHECKING, Dict, Optional, Sequence, Tuple, Union
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
from google.api_core.retry import Retry
from google.cloud.memcache_v1beta2.types import cloud_memcache
from google.cloud.redis_v1 import FailoverInstanceRequest, InputConfig, Instance, OutputConfig
@@ -101,7 +102,7 @@ def __init__(
instance_id: str,
instance: Union[Dict, Instance],
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -180,7 +181,7 @@ def __init__(
location: str,
instance: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -264,7 +265,7 @@ def __init__(
instance: str,
output_config: Union[Dict, OutputConfig],
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -348,7 +349,7 @@ def __init__(
instance: str,
data_protection_mode: FailoverInstanceRequest.DataProtectionMode,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -426,7 +427,7 @@ def __init__(
location: str,
instance: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -512,7 +513,7 @@ def __init__(
instance: str,
input_config: Union[Dict, InputConfig],
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -594,7 +595,7 @@ def __init__(
location: str,
page_size: int,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -691,7 +692,7 @@ def __init__(
location: Optional[str] = None,
instance_id: Optional[str] = None,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -774,7 +775,7 @@ def __init__(
location: Optional[str] = None,
instance_id: Optional[str] = None,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -877,7 +878,7 @@ def __init__(
instance: Union[Dict, Instance],
input_config: Union[Dict, InputConfig],
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -976,7 +977,7 @@ def __init__(
instance: str,
output_config: Union[Dict, OutputConfig],
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1063,7 +1064,7 @@ def __init__(
location: str,
instance_id: str,
project_id: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1149,7 +1150,7 @@ def __init__(
instance_id: str,
instance: Union[Dict, cloud_memcache.Instance],
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1215,7 +1216,7 @@ def __init__(
location: str,
instance: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1288,7 +1289,7 @@ def __init__(
location: str,
instance: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1365,7 +1366,7 @@ def __init__(
*,
location: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1457,7 +1458,7 @@ def __init__(
location: Optional[str] = None,
instance_id: Optional[str] = None,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1540,7 +1541,7 @@ def __init__(
location: str,
instance_id: str,
project_id: str,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
diff --git a/airflow/providers/google/cloud/operators/cloud_sql.py b/airflow/providers/google/cloud/operators/cloud_sql.py
index 1441f518b46b4..d6c9fc7c10c51 100644
--- a/airflow/providers/google/cloud/operators/cloud_sql.py
+++ b/airflow/providers/google/cloud/operators/cloud_sql.py
@@ -24,7 +24,9 @@
from airflow.hooks.base import BaseHook
from airflow.models import BaseOperator, Connection
from airflow.providers.google.cloud.hooks.cloud_sql import CloudSQLDatabaseHook, CloudSQLHook
+from airflow.providers.google.cloud.links.cloud_sql import CloudSQLInstanceDatabaseLink, CloudSQLInstanceLink
from airflow.providers.google.cloud.utils.field_validator import GcpBodyFieldValidator
+from airflow.providers.google.common.links.storage import FileDetailsLink
from airflow.providers.mysql.hooks.mysql import MySqlHook
from airflow.providers.postgres.hooks.postgres import PostgresHook
@@ -151,9 +153,27 @@
fields=[
dict(name="tables", optional=True, type="list"),
dict(name="schemaOnly", optional=True),
+ dict(
+ name="mysqlExportOptions",
+ type="dict",
+ optional=True,
+ fields=[dict(name="masterData")],
+ ),
+ ],
+ ),
+ dict(
+ name="csvExportOptions",
+ type="dict",
+ optional=True,
+ fields=[
+ dict(name="selectQuery"),
+ dict(name="escapeCharacter", optional=True),
+ dict(name="quoteCharacter", optional=True),
+ dict(name="fieldsTerminatedBy", optional=True),
+ dict(name="linesTerminatedBy", optional=True),
],
),
- dict(name="csvExportOptions", type="dict", optional=True, fields=[dict(name="selectQuery")]),
+ dict(name="offload", optional=True),
],
)
]
@@ -298,6 +318,8 @@ class CloudSQLCreateInstanceOperator(CloudSQLBaseOperator):
'impersonation_chain',
)
# [END gcp_sql_create_template_fields]
+ ui_color = '#FADBDA'
+ operator_extra_links = (CloudSQLInstanceLink(),)
def __init__(
self,
@@ -345,6 +367,13 @@ def execute(self, context: 'Context') -> None:
else:
self.log.info("Cloud SQL instance with ID %s already exists. Aborting create.", self.instance)
+ CloudSQLInstanceLink.persist(
+ context=context,
+ task_instance=self,
+ cloud_sql_instance=self.instance,
+ project_id=self.project_id or hook.project_id,
+ )
+
instance_resource = hook.get_instance(project_id=self.project_id, instance=self.instance)
service_account_email = instance_resource["serviceAccountEmailAddress"]
task_instance = context['task_instance']
@@ -393,6 +422,8 @@ class CloudSQLInstancePatchOperator(CloudSQLBaseOperator):
'impersonation_chain',
)
# [END gcp_sql_patch_template_fields]
+ ui_color = '#FBDAC8'
+ operator_extra_links = (CloudSQLInstanceLink(),)
def __init__(
self,
@@ -432,6 +463,13 @@ def execute(self, context: 'Context'):
'Please specify another instance to patch.'
)
else:
+ CloudSQLInstanceLink.persist(
+ context=context,
+ task_instance=self,
+ cloud_sql_instance=self.instance,
+ project_id=self.project_id or hook.project_id,
+ )
+
return hook.patch_instance(project_id=self.project_id, body=self.body, instance=self.instance)
@@ -467,6 +505,7 @@ class CloudSQLDeleteInstanceOperator(CloudSQLBaseOperator):
'impersonation_chain',
)
# [END gcp_sql_delete_template_fields]
+ ui_color = '#FEECD2'
def execute(self, context: 'Context') -> Optional[bool]:
hook = CloudSQLHook(
@@ -517,6 +556,8 @@ class CloudSQLCreateInstanceDatabaseOperator(CloudSQLBaseOperator):
'impersonation_chain',
)
# [END gcp_sql_db_create_template_fields]
+ ui_color = '#FFFCDB'
+ operator_extra_links = (CloudSQLInstanceDatabaseLink(),)
def __init__(
self,
@@ -567,6 +608,12 @@ def execute(self, context: 'Context') -> Optional[bool]:
api_version=self.api_version,
impersonation_chain=self.impersonation_chain,
)
+ CloudSQLInstanceDatabaseLink.persist(
+ context=context,
+ task_instance=self,
+ cloud_sql_instance=self.instance,
+ project_id=self.project_id or hook.project_id,
+ )
if self._check_if_db_exists(database, hook):
self.log.info(
"Cloud SQL instance with ID %s already contains database '%s'. Aborting database insert.",
@@ -617,6 +664,8 @@ class CloudSQLPatchInstanceDatabaseOperator(CloudSQLBaseOperator):
'impersonation_chain',
)
# [END gcp_sql_db_patch_template_fields]
+ ui_color = '#ECF4D9'
+ operator_extra_links = (CloudSQLInstanceDatabaseLink(),)
def __init__(
self,
@@ -669,6 +718,12 @@ def execute(self, context: 'Context') -> None:
"Please specify another database to patch."
)
else:
+ CloudSQLInstanceDatabaseLink.persist(
+ context=context,
+ task_instance=self,
+ cloud_sql_instance=self.instance,
+ project_id=self.project_id or hook.project_id,
+ )
return hook.patch_database(
project_id=self.project_id, instance=self.instance, database=self.database, body=self.body
)
@@ -708,6 +763,7 @@ class CloudSQLDeleteInstanceDatabaseOperator(CloudSQLBaseOperator):
'impersonation_chain',
)
# [END gcp_sql_db_delete_template_fields]
+ ui_color = '#D5EAD8'
def __init__(
self,
@@ -793,6 +849,8 @@ class CloudSQLExportInstanceOperator(CloudSQLBaseOperator):
'impersonation_chain',
)
# [END gcp_sql_export_template_fields]
+ ui_color = '#D4ECEA'
+ operator_extra_links = (CloudSQLInstanceLink(), FileDetailsLink())
def __init__(
self,
@@ -835,6 +893,18 @@ def execute(self, context: 'Context') -> None:
api_version=self.api_version,
impersonation_chain=self.impersonation_chain,
)
+ CloudSQLInstanceLink.persist(
+ context=context,
+ task_instance=self,
+ cloud_sql_instance=self.instance,
+ project_id=self.project_id or hook.project_id,
+ )
+ FileDetailsLink.persist(
+ context=context,
+ task_instance=self,
+ uri=self.body["exportContext"]["uri"][5:],
+ project_id=self.project_id or hook.project_id,
+ )
return hook.export_instance(project_id=self.project_id, instance=self.instance, body=self.body)
@@ -890,6 +960,8 @@ class CloudSQLImportInstanceOperator(CloudSQLBaseOperator):
'impersonation_chain',
)
# [END gcp_sql_import_template_fields]
+ ui_color = '#D3EDFB'
+ operator_extra_links = (CloudSQLInstanceLink(), FileDetailsLink())
def __init__(
self,
@@ -932,6 +1004,18 @@ def execute(self, context: 'Context') -> None:
api_version=self.api_version,
impersonation_chain=self.impersonation_chain,
)
+ CloudSQLInstanceLink.persist(
+ context=context,
+ task_instance=self,
+ cloud_sql_instance=self.instance,
+ project_id=self.project_id or hook.project_id,
+ )
+ FileDetailsLink.persist(
+ context=context,
+ task_instance=self,
+ uri=self.body["importContext"]["uri"][5:],
+ project_id=self.project_id or hook.project_id,
+ )
return hook.import_instance(project_id=self.project_id, instance=self.instance, body=self.body)
@@ -965,6 +1049,7 @@ class CloudSQLExecuteQueryOperator(BaseOperator):
template_ext: Sequence[str] = ('.sql',)
template_fields_renderers = {'sql': 'sql'}
# [END gcp_sql_query_template_fields]
+ ui_color = '#D3DEF1'
def __init__(
self,
diff --git a/airflow/providers/google/cloud/operators/datacatalog.py b/airflow/providers/google/cloud/operators/datacatalog.py
index 79b448accbb25..145aeaa4c6233 100644
--- a/airflow/providers/google/cloud/operators/datacatalog.py
+++ b/airflow/providers/google/cloud/operators/datacatalog.py
@@ -18,6 +18,7 @@
from typing import TYPE_CHECKING, Dict, Optional, Sequence, Tuple, Union
from google.api_core.exceptions import AlreadyExists, NotFound
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
from google.api_core.retry import Retry
from google.cloud.datacatalog_v1beta1 import DataCatalogClient, SearchCatalogResult
from google.cloud.datacatalog_v1beta1.types import (
@@ -95,7 +96,7 @@ def __init__(
entry_id: str,
entry: Union[Dict, Entry],
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -202,7 +203,7 @@ def __init__(
entry_group_id: str,
entry_group: Union[Dict, EntryGroup],
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -310,7 +311,7 @@ def __init__(
tag: Union[Dict, Tag],
template_id: Optional[str] = None,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -432,7 +433,7 @@ def __init__(
tag_template_id: str,
tag_template: Union[Dict, TagTemplate],
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -540,7 +541,7 @@ def __init__(
tag_template_field_id: str,
tag_template_field: Union[Dict, TagTemplateField],
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -640,7 +641,7 @@ def __init__(
entry_group: str,
entry: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -724,7 +725,7 @@ def __init__(
location: str,
entry_group: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -810,7 +811,7 @@ def __init__(
entry: str,
tag: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -899,7 +900,7 @@ def __init__(
tag_template: str,
force: bool,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -987,7 +988,7 @@ def __init__(
field: str,
force: bool,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1074,7 +1075,7 @@ def __init__(
entry_group: str,
entry: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1160,7 +1161,7 @@ def __init__(
entry_group: str,
read_mask: FieldMask,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1240,7 +1241,7 @@ def __init__(
location: str,
tag_template: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1327,7 +1328,7 @@ def __init__(
entry: str,
page_size: int = 100,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1412,7 +1413,7 @@ def __init__(
linked_resource: Optional[str] = None,
sql_resource: Optional[str] = None,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1497,7 +1498,7 @@ def __init__(
field: str,
new_tag_template_field_id: str,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1608,7 +1609,7 @@ def __init__(
query: str,
page_size: int = 100,
order_by: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1704,7 +1705,7 @@ def __init__(
entry_group: Optional[str] = None,
entry_id: Optional[str] = None,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1805,7 +1806,7 @@ def __init__(
entry: Optional[str] = None,
tag_id: Optional[str] = None,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -1908,7 +1909,7 @@ def __init__(
location: Optional[str] = None,
tag_template_id: Optional[str] = None,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -2015,7 +2016,7 @@ def __init__(
tag_template: Optional[str] = None,
tag_template_field_id: Optional[str] = None,
project_id: Optional[str] = None,
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
diff --git a/airflow/providers/google/cloud/operators/dataflow.py b/airflow/providers/google/cloud/operators/dataflow.py
index 5e50521dee82b..98a17d800cbe5 100644
--- a/airflow/providers/google/cloud/operators/dataflow.py
+++ b/airflow/providers/google/cloud/operators/dataflow.py
@@ -124,6 +124,7 @@ class DataflowConfiguration:
WaitForRun = wait until job finished and the run job.
Supported only by:
:py:class:`~airflow.providers.apache.beam.operators.beam.BeamRunJavaPipelineOperator`
+ :param service_account: Run the job as a specific service account, instead of the default GCE robot.
"""
template_fields: Sequence[str] = ("job_name", "location")
@@ -144,6 +145,7 @@ def __init__(
wait_until_finished: Optional[bool] = None,
multiple_jobs: Optional[bool] = None,
check_if_running: CheckJobRunning = CheckJobRunning.WaitForRun,
+ service_account: Optional[str] = None,
) -> None:
self.job_name = job_name
self.append_job_name = append_job_name
@@ -158,6 +160,7 @@ def __init__(
self.wait_until_finished = wait_until_finished
self.multiple_jobs = multiple_jobs
self.check_if_running = check_if_running
+ self.service_account = service_account
class DataflowCreateJavaJobOperator(BaseOperator):
@@ -316,7 +319,7 @@ class DataflowCreateJavaJobOperator(BaseOperator):
"labels": {"foo": "bar"},
},
gcp_conn_id="airflow-conn-id",
- dag=my - dag,
+ dag=my_dag,
)
"""
@@ -410,33 +413,33 @@ def set_current_job_id(job_id):
tmp_gcs_file = exit_stack.enter_context(gcs_hook.provide_file(object_url=self.jar))
self.jar = tmp_gcs_file.name
- is_running = False
- if self.check_if_running != CheckJobRunning.IgnoreJob:
+ is_running = False
+ if self.check_if_running != CheckJobRunning.IgnoreJob:
+ is_running = self.dataflow_hook.is_job_dataflow_running(
+ name=self.job_name,
+ variables=pipeline_options,
+ )
+ while is_running and self.check_if_running == CheckJobRunning.WaitForRun:
+
is_running = self.dataflow_hook.is_job_dataflow_running(
name=self.job_name,
variables=pipeline_options,
)
- while is_running and self.check_if_running == CheckJobRunning.WaitForRun:
-
- is_running = self.dataflow_hook.is_job_dataflow_running(
- name=self.job_name,
- variables=pipeline_options,
- )
- if not is_running:
- pipeline_options["jobName"] = job_name
- with self.dataflow_hook.provide_authorized_gcloud():
- self.beam_hook.start_java_pipeline(
- variables=pipeline_options,
- jar=self.jar,
- job_class=self.job_class,
- process_line_callback=process_line_callback,
- )
- self.dataflow_hook.wait_for_done(
- job_name=job_name,
- location=self.location,
- job_id=self.job_id,
- multiple_jobs=self.multiple_jobs,
+ if not is_running:
+ pipeline_options["jobName"] = job_name
+ with self.dataflow_hook.provide_authorized_gcloud():
+ self.beam_hook.start_java_pipeline(
+ variables=pipeline_options,
+ jar=self.jar,
+ job_class=self.job_class,
+ process_line_callback=process_line_callback,
)
+ self.dataflow_hook.wait_for_done(
+ job_name=job_name,
+ location=self.location,
+ job_id=self.job_id,
+ multiple_jobs=self.multiple_jobs,
+ )
return {"job_id": self.job_id}
@@ -560,7 +563,7 @@ class DataflowTemplatedJobStartOperator(BaseOperator):
"outputFile": "gs://bucket/output/my_output.txt",
},
gcp_conn_id="airflow-conn-id",
- dag=my - dag,
+ dag=my_dag,
)
``template``, ``dataflow_default_options``, ``parameters``, and ``job_name`` are
@@ -587,6 +590,7 @@ class DataflowTemplatedJobStartOperator(BaseOperator):
"gcp_conn_id",
"impersonation_chain",
"environment",
+ "dataflow_default_options",
)
ui_color = "#0273d4"
operator_extra_links = (DataflowJobLink(),)
diff --git a/airflow/providers/google/cloud/operators/dataplex.py b/airflow/providers/google/cloud/operators/dataplex.py
index 21a89ea7d060e..7b225a0c010ad 100644
--- a/airflow/providers/google/cloud/operators/dataplex.py
+++ b/airflow/providers/google/cloud/operators/dataplex.py
@@ -22,6 +22,7 @@
if TYPE_CHECKING:
from airflow.utils.context import Context
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
from google.api_core.retry import Retry, exponential_sleep_generator
from google.cloud.dataplex_v1.types import Task
from googleapiclient.errors import HttpError
@@ -84,7 +85,7 @@ def __init__(
dataplex_task_id: str,
validate_only: Optional[bool] = None,
api_version: str = "v1",
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -198,7 +199,7 @@ def __init__(
lake_id: str,
dataplex_task_id: str,
api_version: str = "v1",
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -297,7 +298,7 @@ def __init__(
filter: Optional[str] = None,
order_by: Optional[str] = None,
api_version: str = "v1",
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
@@ -384,7 +385,7 @@ def __init__(
lake_id: str,
dataplex_task_id: str,
api_version: str = "v1",
- retry: Optional[Retry] = None,
+ retry: Union[Retry, _MethodDefault] = DEFAULT,
timeout: Optional[float] = None,
metadata: Sequence[Tuple[str, str]] = (),
gcp_conn_id: str = "google_cloud_default",
diff --git a/airflow/providers/google/cloud/operators/dataproc.py b/airflow/providers/google/cloud/operators/dataproc.py
index 7cf833357838e..7a160d1ba97dd 100644
--- a/airflow/providers/google/cloud/operators/dataproc.py
+++ b/airflow/providers/google/cloud/operators/dataproc.py
@@ -30,6 +30,7 @@
from google.api_core import operation # type: ignore
from google.api_core.exceptions import AlreadyExists, NotFound
+from google.api_core.gapic_v1.method import DEFAULT, _MethodDefault
from google.api_core.retry import Retry, exponential_sleep_generator
from google.cloud.dataproc_v1 import Batch, Cluster
from google.protobuf.duration_pb2 import Duration
@@ -411,6 +412,10 @@ class DataprocCreateClusterOperator(BaseOperator):
:param cluster_config: Required. The cluster config to create.
If a dict is provided, it must be of the same form as the protobuf message
:class:`~google.cloud.dataproc_v1.types.ClusterConfig`
+ :param virtual_cluster_config: Optional. The virtual cluster config, used when creating a Dataproc
+ cluster that does not directly control the underlying compute resources, for example, when creating a
+ `Dataproc-on-GKE cluster
+
Setup and develop using Gitpod online workspaces
- - - -Setup Airflow with Breeze -------------------------- - - -Forking and cloning Project -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1. Goto |airflow_github| and fork the project. - - .. |airflow_github| raw:: html - - https://github.com/apache/airflow/ - - .. raw:: html - -
-
-
-
-2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
-
- .. raw:: html
-
-
-
-
-
-
-3. Add goto https://gitpod.io/#
-
- 
-
-
-Setting up Breeze
-~~~~~~~~~~~~~~~~~
-
-1. Breeze is already initialized in one of the terminals in Gitpod
-
-2. Once the breeze environment is initialized, create airflow tables and users from the breeze CLI. ``airflow db reset``
- is required to execute at least once for Airflow Breeze to get the database/tables created.
-
-.. note::
-
- This step is needed when you would like to run/use webserver.
-
-.. code-block:: bash
-
- root@b76fcb399bb6:/opt/airflow# airflow db reset
- root@b76fcb399bb6:/opt/airflow# airflow users create --role Admin --username admin --password admin \
- --email admin@example.com --firstname foo --lastname bar
-
-
-3. Closing Breeze environment. After successfully finishing above command will leave you in container,
- type ``exit`` to exit the container
+Using your IDE
+##############
-.. code-block:: bash
-
- root@b76fcb399bb6:/opt/airflow#
- root@b76fcb399bb6:/opt/airflow# exit
-
-.. code-block:: bash
-
- $ ./breeze stop
-
-
-Installing Airflow with Breeze.
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Gitpod default image have all the required packages installed.
-
-1. Add following line to ~/.bashrc in order to call breeze command from anywhere.
-
-.. code-block:: bash
-
- export PATH=${PATH}:"/workspace/airflow"
- source ~/.bashrc
-
-
-Using Breeze
-~~~~~~~~~~~~
-
-1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run.
- It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left
- and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate.
-
-.. code-block:: bash
-
- $ breeze start-airflow
-
- Use CI image.
-
- Branch name: main
- Docker image: ghcr.io/apache/airflow/main/ci/python3.8:latest
- Airflow source version: 2.3.0.dev0
- Python version: 3.8
- Backend: mysql 5.7
-
-
- Port forwarding:
-
- Ports are forwarded to the running docker containers for webserver and database
- * 12322 -> forwarded to Airflow ssh server -> airflow:22
- * 28080 -> forwarded to Airflow webserver -> airflow:8080
- * 25555 -> forwarded to Flower dashboard -> airflow:5555
- * 25433 -> forwarded to Postgres database -> postgres:5432
- * 23306 -> forwarded to MySQL database -> mysql:3306
- * 21433 -> forwarded to MSSQL database -> mssql:1443
- * 26379 -> forwarded to Redis broker -> redis:6379
-
- Here are links to those services that you can use on host:
- * ssh connection for remote debugging: ssh -p 12322 airflow@127.0.0.1 pw: airflow
- * Webserver: http://127.0.0.1:28080
- * Flower: http://127.0.0.1:25555
- * Postgres: jdbc:postgresql://127.0.0.1:25433/airflow?user=postgres&password=airflow
- * Mysql: jdbc:mysql://127.0.0.1:23306/airflow?user=root
- * Redis: redis://127.0.0.1:26379/0
-
-.. raw:: html
-
-
-
- 
-
-
-2. You can access the ports as shown
-
-.. raw:: html
-
-
-
- 
-
+If you are familiar with Python development and use your favourite editors, Airflow can be setup
+similarly to other projects of yours. However, if you need specific instructions for your IDE you
+will find more detailed instructions here:
+* `Pycharm/IntelliJ
-
-
-
-
-2. Give a name to a branch and checkout
-
- .. raw:: html
-
-
-
-
-
-
-
-
-Testing
-~~~~~~~
-
-All Tests are inside ``./tests`` directory.
-
-- Running Unit tests inside Breeze environment.
-
- Just run ``pytest filepath+filename`` to run the tests.
-
-.. code-block:: bash
-
- root@4a2143c17426:/opt/airflow# pytest tests/utils/test_session.py
- ======================================= test session starts =======================================
- platform linux -- Python 3.7.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/bin/python
- cachedir: .pytest_cache
- rootdir: /opt/airflow, configfile: pytest.ini
- plugins: anyio-3.3.4, flaky-3.7.0, asyncio-0.16.0, cov-3.0.0, forked-1.3.0, httpx-0.15.0, instafail-0.4.2, rerunfailures-9.1.1, timeouts-1.2.1, xdist-2.4.0, requests-mock-1.9.3
- setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
- collected 4 items
-
- tests/utils/test_session.py::TestSession::test_raised_provide_session PASSED [ 25%]
- tests/utils/test_session.py::TestSession::test_provide_session_without_args_and_kwargs PASSED [ 50%]
- tests/utils/test_session.py::TestSession::test_provide_session_with_args PASSED [ 75%]
- tests/utils/test_session.py::TestSession::test_provide_session_with_kwargs PASSED [100%]
-
- ====================================== 4 passed, 11 warnings in 33.14s ======================================
+Using Remote development environments
+#####################################
-- Running All the tests with Breeze by specifying required Python version, backend, backend version
+In order to use remote development environment, you usually need a paid account, but you do not have to
+setup local machine for development.
-.. code-block:: bash
-
- $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
-
-
-- Running specific test in container using shell scripts. Testing in container scripts are located in
- ``./scripts/in_container`` directory.
-
-.. code-block:: bash
-
- root@4a2143c17426:/opt/airflow# ls ./scripts/in_container/
- _in_container_script_init.sh quarantine_issue_header.md run_mypy.sh
- _in_container_utils.sh run_anything.sh run_prepare_airflow_packages.sh
- airflow_ci.cfg run_ci_tests.sh run_prepare_provider_documentation.sh
- bin run_docs_build.sh run_prepare_provider_packages.sh
- check_environment.sh run_extract_tests.sh run_resource_check.sh
- check_junitxml_result.py run_fix_ownership.sh run_system_tests.sh
- configure_environment.sh run_flake8.sh run_tmux_welcome.sh
- entrypoint_ci.sh run_generate_constraints.sh stop_tmux_airflow.sh
- entrypoint_exec.sh run_init_script.sh update_quarantined_test_status.py
- prod run_install_and_test_provider_packages.sh
-
- root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
-
-- Running specific type of test
-
- - Types of tests
-
- - Running specific type of test
-
- .. note::
-
- Before starting a new instance, let's clear the volume and databases "fresh like a daisy". You
- can do this by:
-
- .. code-block::bash
-
- $ breeze stop
-
- .. code-block:: bash
-
- $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type Core
-
-
-- Running Integration test for specific test type
-
- - Running an Integration Test
-
- .. code-block:: bash
-
- $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All --integration mongo
+* `GitPod
-
+
+
+
+2. Follow `Codespaces Quickstart
+
+
+
+
+2. Goto your github account's fork of airflow click on ``Code`` and copy the clone link.
+
+ .. raw:: html
+
+
+
+
+
+
+3. Add goto https://gitpod.io/#
+
+
+
+
+Set up Breeze in Gitpod
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Gitpod default image have all the required packages installed.
+
+1. Run ``pipx install -e ./dev/breeze`` to install Breeze
+
+2. Run ``breeze`` to enter breeze in Gitpod.
+
+Setting up database in Breeze
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once you enter breeze environment is initialized, create airflow tables and users from the breeze CLI.
+The ``airflow db reset`` command is required to execute at least once for Airflow Breeze to
+get the database/tables created. When you run the tests, your database will be initialized automatically
+the first time you run tests.
+
+.. note::
+
+ This step is needed when you would like to run/use webserver.
+
+.. code-block:: bash
+
+ root@b76fcb399bb6:/opt/airflow# airflow db reset
+ root@b76fcb399bb6:/opt/airflow# airflow users create --role Admin --username admin --password admin \
+ --email admin@example.com --firstname foo --lastname bar
+
+Follow the `Quick start
+
+
+
+
+
+2. Paste the repository link in the URL field and submit.
+
+ .. raw:: html
+
+
+
+
+
+
+Setting up debugging
+####################
+
+It requires "airflow-env" virtual environment configured locally.
+
+1. Configuring Airflow database connection
+
+- Airflow is by default configured to use SQLite database. Configuration can be seen on local machine
+ ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
+
+- Installing required dependency for MySQL connection in ``airflow-env`` on local machine.
+
+ .. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install PyMySQL
+
+- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
+ ``~/airflow/airflow.cfg`` on local machine.
+
+2. Debugging an example DAG
+
+- Add Interpreter to PyCharm pointing interpreter path to ``~/.pyenv/versions/airflow-env/bin/python``, which is virtual
+ environment ``airflow-env`` created with pyenv earlier. For adding an Interpreter go to ``File -> Setting -> Project:
+ airflow -> Python Interpreter``.
+
+ .. raw:: html
+
+
+
+
+
+
+- In PyCharm IDE open airflow project, directory ``/files/dags`` of local machine is by default mounted to docker
+ machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
+ scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
+
+- Copy any example DAG present in the ``/airflow/example_dags`` directory to ``/files/dags/``.
+
+- Add a ``__main__`` block at the end of your DAG file to make it runnable. It will run a ``back_fill`` job:
+
+ .. code-block:: python
+
+ if __name__ == "__main__":
+ dag.clear()
+ dag.run()
+
+- Add ``AIRFLOW__CORE__EXECUTOR=DebugExecutor`` to Environment variable of Run Configuration.
+
+ - Click on Add configuration
+
+ .. raw:: html
+
+
+
+
+
+
+ - Add Script Path and Environment Variable to new Python configuration
+
+ .. raw:: html
+
+
+
+
+
+
+- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in MySQL Workbench.
+
+Creating a branch
+#################
+
+1. Click on the branch symbol in the status bar
+
+ .. raw:: html
+
+
+
+
+
+
+2. Give a name to a branch and checkout
+
+ .. raw:: html
+
+
+
+
+
+
+Follow the `Quick start
+
+
+
+
+
+2. Paste the copied clone link in the URL field and submit.
+
+ .. raw:: html
+
+
+
+
+
+
+
+Setting up debugging
+####################
+
+1. Configuring Airflow database connection
+
+- Airflow is by default configured to use SQLite database. Configuration can be seen on local machine
+ ``~/airflow/airflow.cfg`` under ``sql_alchemy_conn``.
+
+- Installing required dependency for MySQL connection in ``airflow-env`` on local machine.
+
+ .. code-block:: bash
+
+ $ pyenv activate airflow-env
+ $ pip install PyMySQL
+
+- Now set ``sql_alchemy_conn = mysql+pymysql://root:@127.0.0.1:23306/airflow?charset=utf8mb4`` in file
+ ``~/airflow/airflow.cfg`` on local machine.
+
+1. Debugging an example DAG
+
+- In Visual Studio Code open airflow project, directory ``/files/dags`` of local machine is by default mounted to docker
+ machine when breeze airflow is started. So any DAG file present in this directory will be picked automatically by
+ scheduler running in docker machine and same can be seen on ``http://127.0.0.1:28080``.
+
+- Copy any example DAG present in the ``/airflow/example_dags`` directory to ``/files/dags/``.
+
+- Add a ``__main__`` block at the end of your DAG file to make it runnable. It will run a ``back_fill`` job:
+
+ .. code-block:: python
+
+
+ if __name__ == "__main__":
+ dag.clear()
+ dag.run()
+
+- Add ``"AIRFLOW__CORE__EXECUTOR": "DebugExecutor"`` to the ``"env"`` field of Debug configuration.
+
+ - Using the ``Run`` view click on ``Create a launch.json file``
+
+ .. raw:: html
+
+
+
+
+
+
+
+
+ - Change ``"program"`` to point to an example dag and add ``"env"`` and ``"python"`` fields to the new Python configuration
+
+ .. raw:: html
+
+
+
+
+
+
+
+
+- Now Debug an example dag and view the entries in tables such as ``dag_run, xcom`` etc in mysql workbench.
+
+Creating a branch
+#################
+
+1. Click on the branch symbol in the status bar
+
+ .. raw:: html
+
+
+
+
+
+
+2. Give a name to a branch and checkout
+
+ .. raw:: html
+
+
+
+
+
+
+Follow the `Quick start
+ ') default_subject = 'Airflow alert: {{ti}}' @@ -2167,13 +2226,20 @@ def get_email_subject_content(self, exception: BaseException) -> Tuple[str, str, html_content_err = jinja_env.from_string(default_html_content_err).render(**default_context) else: - jinja_env = self.task.get_template_env() + # Use the DAG's get_template_env() to set force_sandboxed. Don't add + # the flag to the function on task object -- that function can be + # overridden, and adding a flag breaks backward compatibility. + dag = self.task.get_dag() + if dag: + jinja_env = dag.get_template_env(force_sandboxed=True) + else: + jinja_env = SandboxedEnvironment(cache_size=0) jinja_context = self.get_template_context() context_merge(jinja_context, additional_context) def render(key: str, content: str) -> str: if conf.has_option('email', key): - path = conf.get('email', key) + path = conf.get_mandatory_value('email', key) with open(path) as f: content = f.read() return render_template_to_string(jinja_env.from_string(content), jinja_context) @@ -2184,13 +2250,14 @@ def render(key: str, content: str) -> str: return subject, html_content, html_content_err - def email_alert(self, exception): + def email_alert(self, exception, task: "BaseOperator"): """Send alert email with exception information.""" - subject, html_content, html_content_err = self.get_email_subject_content(exception) + subject, html_content, html_content_err = self.get_email_subject_content(exception, task=task) + assert task.email try: - send_email(self.task.email, subject, html_content) + send_email(task.email, subject, html_content) except Exception: - send_email(self.task.email, subject, html_content_err) + send_email(task.email, subject, html_content_err) def set_duration(self) -> None: """Set TI duration""" @@ -2200,15 +2267,24 @@ def set_duration(self) -> None: self.duration = None self.log.debug("Task Duration set to %s", self.duration) - def _record_task_map_for_downstreams(self, value: Any, *, session: Session) -> None: - if not self.task.has_mapped_dependants(): + def _record_task_map_for_downstreams(self, task: "Operator", value: Any, *, session: Session) -> None: + # TODO: We don't push TaskMap for mapped task instances because it's not + # currently possible for a downstream to depend on one individual mapped + # task instance, only a task as a whole. This will change in AIP-42 + # Phase 2, and we'll need to further analyze the mapped task case. + if next(task.iter_mapped_dependants(), None) is None: + return + if value is None: + raise XComForMappingNotPushed() + if task.is_mapped: return if not isinstance(value, collections.abc.Collection) or isinstance(value, (bytes, str)): raise UnmappableXComTypePushed(value) + task_map = TaskMap.from_task_instance_xcom(self, value) max_map_length = conf.getint("core", "max_map_length", fallback=1024) - if len(value) > max_map_length: + if task_map.length > max_map_length: raise UnmappableXComLengthPushed(value, max_map_length) - session.merge(TaskMap.from_task_instance_xcom(self, value)) + session.merge(task_map) @provide_session def xcom_push( @@ -2244,6 +2320,7 @@ def xcom_push( task_id=self.task_id, dag_id=self.dag_id, run_id=self.run_id, + map_index=self.map_index, session=session, ) @@ -2255,32 +2332,39 @@ def xcom_pull( key: str = XCOM_RETURN_KEY, include_prior_dates: bool = False, session: Session = NEW_SESSION, + *, + map_indexes: Optional[Union[int, Iterable[int]]] = None, + default: Any = None, ) -> Any: - """ - Pull XComs that optionally meet certain criteria. - - The default value for `key` limits the search to XComs - that were returned by other tasks (as opposed to those that were pushed - manually). To remove this filter, pass key=None (or any desired value). - - If a single task_id string is provided, the result is the value of the - most recent matching XCom from that task_id. If multiple task_ids are - provided, a tuple of matching values is returned. None is returned - whenever no matches are found. + """Pull XComs that optionally meet certain criteria. :param key: A key for the XCom. If provided, only XComs with matching - keys will be returned. The default key is 'return_value', also - available as a constant XCOM_RETURN_KEY. This key is automatically + keys will be returned. The default key is ``'return_value'``, also + available as constant ``XCOM_RETURN_KEY``. This key is automatically given to XComs returned by tasks (as opposed to being pushed - manually). To remove the filter, pass key=None. + manually). To remove the filter, pass *None*. :param task_ids: Only XComs from tasks with matching ids will be - pulled. Can pass None to remove the filter. - :param dag_id: If provided, only pulls XComs from this DAG. - If None (default), the DAG of the calling task is used. + pulled. Pass *None* to remove the filter. + :param dag_id: If provided, only pulls XComs from this DAG. If *None* + (default), the DAG of the calling task is used. + :param map_indexes: If provided, only pull XComs with matching indexes. + If *None* (default), this is inferred from the task(s) being pulled + (see below for details). :param include_prior_dates: If False, only XComs from the current - execution_date are returned. If True, XComs from previous dates + execution_date are returned. If *True*, XComs from previous dates are returned as well. - :param session: Sqlalchemy ORM Session + + When pulling one single task (``task_id`` is *None* or a str) without + specifying ``map_indexes``, the return value is inferred from whether + the specified task is mapped. If not, value from the one single task + instance is returned. If the task to pull is mapped, an iterator (not a + list) yielding XComs from mapped task instances is returned. In either + case, ``default`` (*None* if not specified) is returned if no matching + XComs are found. + + When pulling multiple tasks (i.e. either ``task_id`` or ``map_index`` is + a non-str iterable), a list of matching XComs is returned. Elements in + the list is ordered by item ordering in ``task_id`` and ``map_index``. """ if dag_id is None: dag_id = self.dag_id @@ -2290,25 +2374,53 @@ def xcom_pull( run_id=self.run_id, dag_ids=dag_id, task_ids=task_ids, + map_indexes=map_indexes, include_prior_dates=include_prior_dates, session=session, ) - # Since we're only fetching the values field, and not the - # whole class, the @recreate annotation does not kick in. - # Therefore we need to deserialize the fields by ourselves. - if task_ids is None or isinstance(task_ids, str): - xcom = query.with_entities(XCom.value).first() - if xcom: - return XCom.deserialize_value(xcom) + # NOTE: Since we're only fetching the value field and not the whole + # class, the @recreate annotation does not kick in. Therefore we need to + # call XCom.deserialize_value() manually. + + # We are only pulling one single task. + if (task_ids is None or isinstance(task_ids, str)) and not isinstance(map_indexes, Iterable): + first = query.with_entities( + XCom.run_id, XCom.task_id, XCom.dag_id, XCom.map_index, XCom.value + ).first() + if first is None: # No matching XCom at all. + return default + if map_indexes is not None or first.map_index < 0: + return XCom.deserialize_value(first) + + return _LazyXComAccess.build_from_single_xcom(first, query) + + # At this point either task_ids or map_indexes is explicitly multi-value. + + results = ( + (r.task_id, r.map_index, XCom.deserialize_value(r)) + for r in query.with_entities(XCom.task_id, XCom.map_index, XCom.value) + ) + + if task_ids is None: + task_id_pos: Dict[str, int] = defaultdict(int) + elif isinstance(task_ids, str): + task_id_pos = {task_ids: 0} + else: + task_id_pos = {task_id: i for i, task_id in enumerate(task_ids)} + if map_indexes is None: + map_index_pos: Dict[int, int] = defaultdict(int) + elif isinstance(map_indexes, int): + map_index_pos = {map_indexes: 0} else: - vals_kv = { - result.task_id: XCom.deserialize_value(result) - for result in query.with_entities(XCom.task_id, XCom.value) - } + map_index_pos = {map_index: i for i, map_index in enumerate(map_indexes)} - values_ordered_by_id = [vals_kv.get(task_id) for task_id in task_ids] - return values_ordered_by_id + def _arg_pos(item: Tuple[str, int, Any]) -> Tuple[int, int]: + task_id, map_index, _ = item + return task_id_pos[task_id], map_index_pos[map_index] + + results_sorted_by_arg_pos = sorted(results, key=_arg_pos) + return [value for _, _, value in results_sorted_by_arg_pos] @provide_session def get_num_running_task_instances(self, session): @@ -2369,20 +2481,35 @@ def filter_for_tis(tis: Iterable[Union["TaskInstance", TaskInstanceKey]]) -> Opt TaskInstance.task_id == first_task_id, ) - if settings.engine.dialect.name == 'mssql': - return or_( - and_( - TaskInstance.dag_id == ti.dag_id, - TaskInstance.task_id == ti.task_id, - TaskInstance.run_id == ti.run_id, - TaskInstance.map_index == ti.map_index, - ) - for ti in tis - ) - else: - return tuple_( - TaskInstance.dag_id, TaskInstance.task_id, TaskInstance.run_id, TaskInstance.map_index - ).in_([ti.key.primary for ti in tis]) + return tuple_in_condition( + (TaskInstance.dag_id, TaskInstance.task_id, TaskInstance.run_id, TaskInstance.map_index), + (ti.key.primary for ti in tis), + ) + + @classmethod + def ti_selector_condition(cls, vals: Collection[Union[str, Tuple[str, int]]]) -> ColumnOperators: + """ + Build an SQLAlchemy filter for a list where each element can contain + whether a task_id, or a tuple of (task_id,map_index) + + :meta private: + """ + # Compute a filter for TI.task_id and TI.map_index based on input values + # For each item, it will either be a task_id, or (task_id, map_index) + task_id_only = [v for v in vals if isinstance(v, str)] + with_map_index = [v for v in vals if not isinstance(v, str)] + + filters: List[ColumnOperators] = [] + if task_id_only: + filters.append(cls.task_id.in_(task_id_only)) + if with_map_index: + filters.append(tuple_in_condition((cls.task_id, cls.map_index), with_map_index)) + + if not filters: + return false() + if len(filters) == 1: + return filters[0] + return or_(*filters) # State of the task instance. @@ -2402,9 +2529,10 @@ def __init__( dag_id: str, task_id: str, run_id: str, - start_date: datetime, - end_date: datetime, + start_date: Optional[datetime], + end_date: Optional[datetime], try_number: int, + map_index: int, state: str, executor_config: Any, pool: str, @@ -2413,21 +2541,20 @@ def __init__( run_as_user: Optional[str] = None, priority_weight: Optional[int] = None, ): - self._dag_id: str = dag_id - self._task_id: str = task_id - self._run_id: str = run_id - self._start_date: datetime = start_date - self._end_date: datetime = end_date - self._try_number: int = try_number - self._state: str = state - self._executor_config: Any = executor_config - self._run_as_user: Optional[str] = None - self._run_as_user = run_as_user - self._pool: str = pool - self._priority_weight: Optional[int] = None - self._priority_weight = priority_weight - self._queue: str = queue - self._key = key + self.dag_id = dag_id + self.task_id = task_id + self.run_id = run_id + self.map_index = map_index + self.start_date = start_date + self.end_date = end_date + self.try_number = try_number + self.state = state + self.executor_config = executor_config + self.run_as_user = run_as_user + self.pool = pool + self.priority_weight = priority_weight + self.queue = queue + self.key = key def __eq__(self, other): if isinstance(other, self.__class__): @@ -2440,6 +2567,7 @@ def from_ti(cls, ti: TaskInstance): dag_id=ti.dag_id, task_id=ti.task_id, run_id=ti.run_id, + map_index=ti.map_index, start_date=ti.start_date, end_date=ti.end_date, try_number=ti.try_number, @@ -2454,80 +2582,16 @@ def from_ti(cls, ti: TaskInstance): @classmethod def from_dict(cls, obj_dict: dict): - ti_key = obj_dict.get('_key', []) - start_date: Union[Any, datetime] = ( - datetime.fromisoformat(str(obj_dict.get('_start_date'))) - if obj_dict.get('_start_date') is not None - else None - ) - end_date: Union[Any, datetime] = ( - datetime.fromisoformat(str(obj_dict.get('_end_date'))) - if obj_dict.get('_end_date') is not None - else None - ) - return cls( - dag_id=str(obj_dict['_dag_id']), - task_id=str(obj_dict.get('_task_id')), - run_id=str(obj_dict.get('_run_id')), - start_date=start_date, - end_date=end_date, - try_number=obj_dict.get('_try_number', 1), - state=str(obj_dict.get('_state')), - executor_config=obj_dict.get('_executor_config'), - run_as_user=obj_dict.get('_run_as_user', None), - pool=str(obj_dict.get('_pool')), - priority_weight=obj_dict.get('_priority_weight', None), - queue=str(obj_dict.get('_queue')), - key=TaskInstanceKey(ti_key[0], ti_key[1], ti_key[2], ti_key[3], ti_key[4]), - ) - - @property - def dag_id(self) -> str: - return self._dag_id - - @property - def task_id(self) -> str: - return self._task_id - - @property - def run_id(self) -> str: - return self._run_id - - @property - def start_date(self) -> datetime: - return self._start_date - - @property - def end_date(self) -> datetime: - return self._end_date - - @property - def try_number(self) -> int: - return self._try_number - - @property - def state(self) -> str: - return self._state - - @property - def pool(self) -> str: - return self._pool - - @property - def priority_weight(self) -> Optional[int]: - return self._priority_weight - - @property - def queue(self) -> str: - return self._queue - - @property - def key(self) -> TaskInstanceKey: - return self._key - - @property - def executor_config(self): - return self._executor_config + ti_key = TaskInstanceKey(*obj_dict.pop('key')) + start_date = None + end_date = None + start_date_str: Optional[str] = obj_dict.pop('start_date') + end_date_str: Optional[str] = obj_dict.pop('end_date') + if start_date_str: + start_date = timezone.parse(start_date_str) + if end_date_str: + end_date = timezone.parse(end_date_str) + return cls(**obj_dict, start_date=start_date, end_date=end_date, key=ti_key) STATICA_HACK = True diff --git a/airflow/models/taskmixin.py b/airflow/models/taskmixin.py index ad6404869eb36..1d66a719a6aec 100644 --- a/airflow/models/taskmixin.py +++ b/airflow/models/taskmixin.py @@ -28,6 +28,7 @@ from logging import Logger from airflow.models.dag import DAG + from airflow.models.mappedoperator import MappedOperator from airflow.utils.edgemodifier import EdgeModifier from airflow.utils.task_group import TaskGroup @@ -54,29 +55,16 @@ def leaves(self) -> Sequence["DependencyMixin"]: raise NotImplementedError() @abstractmethod - def set_upstream( - self, - other: Union["DependencyMixin", Sequence["DependencyMixin"]], - edge_modifier: Optional["EdgeModifier"] = None, - ): + def set_upstream(self, other: Union["DependencyMixin", Sequence["DependencyMixin"]]): """Set a task or a task list to be directly upstream from the current task.""" raise NotImplementedError() @abstractmethod - def set_downstream( - self, - other: Union["DependencyMixin", Sequence["DependencyMixin"]], - edge_modifier: Optional["EdgeModifier"] = None, - ): + def set_downstream(self, other: Union["DependencyMixin", Sequence["DependencyMixin"]]): """Set a task or a task list to be directly downstream from the current task.""" raise NotImplementedError() - def update_relative( - self, - other: "DependencyMixin", - upstream=True, - edge_modifier: Optional["EdgeModifier"] = None, - ) -> None: + def update_relative(self, other: "DependencyMixin", upstream=True) -> None: """ Update relationship information about another TaskMixin. Default is no-op. Override if necessary. @@ -146,6 +134,13 @@ def label(self) -> Optional[str]: def has_dag(self) -> bool: return self.dag is not None + @property + def dag_id(self) -> str: + """Returns dag id if it has one or an adhoc/meaningless ID""" + if self.dag: + return self.dag.dag_id + return "_in_memory_dag_" + @property def log(self) -> "Logger": raise NotImplementedError() @@ -176,7 +171,7 @@ def _set_relatives( task_list: List[Operator] = [] for task_object in task_or_task_list: - task_object.update_relative(self, not upstream, edge_modifier=edge_modifier) + task_object.update_relative(self, not upstream) relatives = task_object.leaves if upstream else task_object.roots for task in relatives: if not isinstance(task, (BaseOperator, MappedOperator)): @@ -283,16 +278,23 @@ def serialize_for_task_group(self) -> Tuple[DagAttributeTypes, Any]: """This is used by SerializedTaskGroup to serialize a task group's content.""" raise NotImplementedError() - def mapped_dependants(self) -> Iterator["DAGNode"]: - """Return any mapped nodes that are direct dependencies of the current task + def _iter_all_mapped_downstreams(self) -> Iterator["MappedOperator"]: + """Return mapped nodes that are direct dependencies of the current task. For now, this walks the entire DAG to find mapped nodes that has this current task as an upstream. We cannot use ``downstream_list`` since it only contains operators, not task groups. In the future, we should provide a way to record an DAG node's all downstream nodes instead. + + Note that this does not guarantee the returned tasks actually use the + current task for task mapping, but only checks those task are mapped + operators, and are downstreams of the current task. + + To get a list of tasks that uses the current task for task mapping, use + :meth:`iter_mapped_dependants` instead. """ from airflow.models.mappedoperator import MappedOperator - from airflow.utils.task_group import MappedTaskGroup, TaskGroup + from airflow.utils.task_group import TaskGroup def _walk_group(group: TaskGroup) -> Iterable[Tuple[str, DAGNode]]: """Recursively walk children in a task group. @@ -307,21 +309,25 @@ def _walk_group(group: TaskGroup) -> Iterable[Tuple[str, DAGNode]]: tg = self.task_group if not tg: - raise RuntimeError("Cannot check for mapped_dependants when not attached to a DAG") + raise RuntimeError("Cannot check for mapped dependants when not attached to a DAG") for key, child in _walk_group(tg): if key == self.node_id: continue - if not isinstance(child, (MappedOperator, MappedTaskGroup)): + if not isinstance(child, MappedOperator): continue if self.node_id in child.upstream_task_ids: yield child - def has_mapped_dependants(self) -> bool: - """Whether any downstream dependencies depend on this task for mapping. + def iter_mapped_dependants(self) -> Iterator["MappedOperator"]: + """Return mapped nodes that depend on the current task the expansion. For now, this walks the entire DAG to find mapped nodes that has this current task as an upstream. We cannot use ``downstream_list`` since it only contains operators, not task groups. In the future, we should provide a way to record an DAG node's all downstream nodes instead. """ - return any(self.mapped_dependants()) + return ( + downstream + for downstream in self._iter_all_mapped_downstreams() + if any(p.node_id == self.node_id for p in downstream.iter_mapped_dependencies()) + ) diff --git a/airflow/models/trigger.py b/airflow/models/trigger.py index c1ccdd4964678..dc91fdccdf2ba 100644 --- a/airflow/models/trigger.py +++ b/airflow/models/trigger.py @@ -24,6 +24,7 @@ from airflow.models.taskinstance import TaskInstance from airflow.triggers.base import BaseTrigger from airflow.utils import timezone +from airflow.utils.retries import run_with_db_retries from airflow.utils.session import provide_session from airflow.utils.sqlalchemy import ExtendedJSON, UtcDateTime from airflow.utils.state import State @@ -88,9 +89,11 @@ def clean_unused(cls, session=None): (triggers have a one-to-many relationship to both) """ # Update all task instances with trigger IDs that are not DEFERRED to remove them - session.query(TaskInstance).filter( - TaskInstance.state != State.DEFERRED, TaskInstance.trigger_id.isnot(None) - ).update({TaskInstance.trigger_id: None}) + for attempt in run_with_db_retries(): + with attempt: + session.query(TaskInstance).filter( + TaskInstance.state != State.DEFERRED, TaskInstance.trigger_id.isnot(None) + ).update({TaskInstance.trigger_id: None}) # Get all triggers that have no task instances depending on them... ids = [ trigger_id @@ -168,7 +171,7 @@ def assign_unassigned(cls, triggerer_id, capacity, session=None): """ from airflow.jobs.base_job import BaseJob # To avoid circular import - count = session.query(cls.id).filter(cls.triggerer_id == triggerer_id).count() + count = session.query(func.count(cls.id)).filter(cls.triggerer_id == triggerer_id).scalar() capacity -= count if capacity <= 0: diff --git a/airflow/models/xcom.py b/airflow/models/xcom.py index 7456fd293d778..d67160d1fa901 100644 --- a/airflow/models/xcom.py +++ b/airflow/models/xcom.py @@ -26,7 +26,7 @@ from typing import TYPE_CHECKING, Any, Iterable, List, Optional, Type, Union, cast, overload import pendulum -from sqlalchemy import Column, Index, Integer, LargeBinary, String +from sqlalchemy import Column, ForeignKeyConstraint, Index, Integer, LargeBinary, String from sqlalchemy.ext.associationproxy import association_proxy from sqlalchemy.orm import Query, Session, reconstructor, relationship from sqlalchemy.orm.exc import NoResultFound @@ -46,10 +46,6 @@ MAX_XCOM_SIZE = 49344 XCOM_RETURN_KEY = 'return_value' -# Stand-in value for 'airflow task test' generating a temporary in-memory DAG -# run without storing it in the database. -IN_MEMORY_RUN_ID = "__airflow_in_memory_dagrun__" - if TYPE_CHECKING: from airflow.models.taskinstance import TaskInstanceKey @@ -71,26 +67,33 @@ class BaseXCom(Base, LoggingMixin): value = Column(LargeBinary) timestamp = Column(UtcDateTime, default=timezone.utcnow, nullable=False) + __table_args__ = ( + # Ideally we should create a unique index over (key, dag_id, task_id, run_id), + # but it goes over MySQL's index length limit. So we instead index 'key' + # separately, and enforce uniqueness with DagRun.id instead. + Index("idx_xcom_key", key), + ForeignKeyConstraint( + [dag_id, task_id, run_id, map_index], + [ + "task_instance.dag_id", + "task_instance.task_id", + "task_instance.run_id", + "task_instance.map_index", + ], + name="xcom_task_instance_fkey", + ondelete="CASCADE", + ), + ) + dag_run = relationship( "DagRun", - primaryjoin="""and_( - BaseXCom.dag_id == foreign(DagRun.dag_id), - BaseXCom.run_id == foreign(DagRun.run_id), - )""", + primaryjoin="BaseXCom.dag_run_id == foreign(DagRun.id)", uselist=False, lazy="joined", passive_deletes="all", ) execution_date = association_proxy("dag_run", "execution_date") - __table_args__ = ( - # Ideally we should create a unique index over (key, dag_id, task_id, run_id), - # but it goes over MySQL's index length limit. So we instead create indexes - # separately, and enforce uniqueness with DagRun.id instead. - Index("idx_xcom_key", key), - Index("idx_xcom_ti_id", dag_id, task_id, run_id, map_index), - ) - @reconstructor def init_on_load(self): """ @@ -100,7 +103,9 @@ def init_on_load(self): self.value = self.orm_deserialize_value() def __repr__(self): - return f'
)}
+
+
+ ),
+ },
+ ...cols,
+ ]);
+ }]
+ : [];
+
+ const {
+ getTableProps,
+ getTableBodyProps,
+ allColumns,
+ prepareRow,
+ page,
+ canPreviousPage,
+ canNextPage,
+ nextPage,
+ previousPage,
+ selectedFlatRows,
+ state: { pageIndex, sortBy, selectedRowIds },
+ } = useTable(
+ {
+ columns,
+ data,
+ pageCount,
+ manualPagination: !!manualPagination,
+ manualSortBy: !!setSortBy,
+ initialState: {
+ pageIndex: offset ? offset / pageSize : 0,
+ pageSize,
+ },
+ },
+ useSortBy,
+ usePagination,
+ ...selectProps,
+ );
+
+ const handleNext = () => {
+ nextPage();
+ if (setOffset) setOffset((pageIndex + 1) * pageSize);
+ };
+
+ const handlePrevious = () => {
+ previousPage();
+ if (setOffset) setOffset((pageIndex - 1 || 0) * pageSize);
+ };
+
+ useEffect(() => {
+ if (setSortBy) setSortBy(sortBy);
+ }, [sortBy, setSortBy]);
+
+ useEffect(() => {
+ if (selectRows) selectRows(selectedFlatRows.map((row) => row.original.mapIndex));
+ // eslint-disable-next-line react-hooks/exhaustive-deps
+ }, [selectedRowIds, selectRows]);
+
+ return (
+ <>
+
{ti}))}
+ | |
|
+ |
+ + {key} + | +
| + | |
| Total Runs Displayed | ++ {durations.length} + | +
| First Run Start | ++ + | +
| Last Run Start | ++ + | +
| Max Run Duration | ++ {formatDuration(max)} + | +
| Mean Run Duration | ++ {formatDuration(avg)} + | +
| Min Run Duration | ++ {formatDuration(min)} + | +
|
+ |
+ + |
| Total Tasks | +{totalEntries} | +
| + {key} + {value > 1 && 's'} + | +{value} | +
+
+ {startDate && ( +
+
+ + )} + {mappedStates && mappedStates.length > 0 && ( +
+
+
+
+