Skip to content
Permalink
Browse files
Seperate provider verification as standalone breeze command (#23454)
This is another step in simplifying and converting to Python all of
the CI/local development tooling.

This PR separates out verification of providers as a separate
breeze command `verify-provider-packages`. It was previously part of
"prepare_provider_packages.py" but it has been now
extracted to a separate in-container python file and it was
wrapped with breeze's `verify-provider-packages` command.

No longer provider verification is run with "preparing provider docs"
nor "preparing provider packages" - it's a standaline command.

This command is also used in CI now to run the tests:

* all provider packages are built and created on CI together with
  airflow version
* the packages are installed inside the CI image and providers are
  verified
* the 2.1 version of Airflow is installed together with all 2.1
  - compatible providers and provider verification is run there too.

This all is much simpler now - we got rediof some 500 lines of bash
code again in favour of breeze python code.

Fixes: #23430
  • Loading branch information
potiuk committed May 6, 2022
1 parent ccb5ce9 commit 3ed07474649b1e202f9b106105fef21f7b2cfddc
Showing 62 changed files with 2,282 additions and 2,938 deletions.
@@ -41,7 +41,6 @@ env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
LOGIN_TO_GITHUB_REGISTRY: "true"
IMAGE_TAG_FOR_THE_BUILD: "${{ github.event.pull_request.head.sha || github.sha }}"
WITH_CI_GROUPS: "true"

concurrency:
group: build-${{ github.event.pull_request.number || github.ref }}
@@ -47,7 +47,6 @@ env:
LOGIN_TO_GITHUB_REGISTRY: "true"
ENABLE_TEST_COVERAGE: "${{ github.event_name == 'push' }}"
IMAGE_TAG_FOR_THE_BUILD: "${{ github.event.pull_request.head.sha || github.sha }}"
WITH_CI_GROUPS: "true"

concurrency:
group: ci-${{ github.event.pull_request.number || github.ref }}
@@ -792,17 +791,35 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- name: "Prepare airflow package: wheel"
run: breeze prepare-airflow-package --package-format wheel --version-suffix-for-pypi dev0
- name: "Install and test provider packages and airflow via wheel files"
run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
env:
USE_AIRFLOW_VERSION: "wheel"
PACKAGE_FORMAT: "wheel"
- name: "Replace non-compliant providers with their 2.1-compliant versions"
run: ./scripts/ci/provider_packages/ci_make_providers_2_1_compliant.sh
run: >
breeze verify-provider-packages --use-airflow-version wheel --use-packages-from-dist
--package-format wheel
- name: "Remove airflow package and replace providers with 2.1-compliant versions"
run: |
rm -vf dist/apache_airflow-*.whl \
dist/apache_airflow_providers_cncf_kubernetes*.whl \
dist/apache_airflow_providers_celery*.whl
pip download --no-deps --dest dist \
apache-airflow-providers-cncf-kubernetes==3.0.0 \
apache-airflow-providers-celery==2.1.3
- name: "Install and test provider packages and airflow on Airflow 2.1 files"
run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
run: >
breeze verify-provider-packages --use-airflow-version 2.1.0
--use-packages-from-dist --package-format wheel --airflow-constraints-reference constraints-2.1.0
env:
USE_AIRFLOW_VERSION: "2.1.0"
PACKAGE_FORMAT: "wheel"
# The extras below are all extras that should be installed with Airflow 2.1.0
AIRFLOW_EXTRAS: "airbyte,alibaba,amazon,apache.atlas.apache.beam,apache.cassandra,apache.drill,\
apache.druid,apache.hdfs,apache.hive,apache.kylin,apache.livy,apache.pig,apache.pinot,\
apache.spark,apache.sqoop,apache.webhdfs,arangodb,asana,async,\
celery,cgroups,cloudant,cncf.kubernetes,dask,databricks,datadog,dbt.cloud,\
deprecated_api,dingding,discord,docker,\
elasticsearch,exasol,facebook,ftp,github,github_enterprise,google,google_auth,\
grpc,hashicorp,http,imap,influxdb,jdbc,jenkins,jira,kerberos,ldap,\
leveldb,microsoft.azure,microsoft.mssql,microsoft.psrp,microsoft.winrm,mongo,mysql,\
neo4j,odbc,openfaas,opsgenie,oracle,pagerduty,pandas,papermill,password,plexus,\
postgres,presto,qubole,rabbitmq,redis,salesforce,samba,segment,sendgrid,sentry,\
sftp,singularity,slack,snowflake,sqlite,ssh,statsd,tableau,telegram,trino,vertica,\
virtualenv,yandex,zendesk"
- name: "Fix ownership"
run: breeze fix-ownership
if: always()
@@ -851,11 +868,9 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
path: "./dist/apache-airflow-providers-*.tar.gz"
retention-days: 1
- name: "Install and test provider packages and airflow via sdist files"
run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
env:
USE_AIRFLOW_VERSION: "sdist"
PACKAGE_FORMAT: "sdist"
AIRFLOW_EXTRAS: "all"
run: >
breeze verify-provider-packages --use-airflow-version sdist --use-packages-from-dist
--package-format sdist
- name: "Fix ownership"
run: breeze fix-ownership
if: always()
@@ -314,18 +314,22 @@ Customize your environment

When you enter the Breeze environment, automatically an environment file is sourced from
``files/airflow-breeze-config/variables.env``.
a

You can also add ``files/airflow-breeze-config/init.sh`` and the script will be sourced always
when you enter Breeze. For example you can add ``pip install`` commands if you want to install
custom dependencies - but there are no limits to add your own customizations.

The ``files`` folder from your local sources is automatically mounted to the container under
``/files`` path and you can put there any files you want to make available for the Breeze container.

You can also copy any .whl or .sdist packages to dist and when you pass ``--use-packages-from-dist`` flag
as ``wheel`` or ``sdist`` line parameter, breeze will automatically install the packages found there
when you enter Breeze.

You can also add your local tmux configuration in ``files/airflow-breeze-config/.tmux.conf`` and
these configurations will be available for your tmux environment.

there is a symlink between ``files/airflow-breeze-config/.tmux.conf`` and ``~/.tmux.conf`` in the container,
There is a symlink between ``files/airflow-breeze-config/.tmux.conf`` and ``~/.tmux.conf`` in the container,
so you can change it at any place, and run

.. code-block:: bash
@@ -546,6 +550,8 @@ do not need or have no access to run). Those are usually connected with releasin
``breeze build-prod image --prepare-build-cache``(needs buildx plugin and write access to registry ghcr.io)
* Generate constraints with ``breeze generate-constraints`` (needed when conflicting changes are merged)
* Prepare airflow packages: ``breeze prepare-airflow-package`` (when releasing Airflow)
* Verify providers: ``breeze verify-provider-packages`` (when releasing provider packages) - including importing
the providers in an earlier airflow version.
* Prepare provider documentation ``breeze prepare-provider-documentation`` and prepare provider packages
``breeze prepare-provider-packages`` (when releasing provider packages)
* Finding the updated dependencies since the last successful build when we have conflict with
@@ -750,6 +756,27 @@ Those are all available flags of ``verify-image`` command:
:width: 100%
:alt: Breeze verify-image

Verifying providers
-------------------

Breeze can also be used to verify if provider classes are importable and if they are following the
right naming conventions. This happens automatically on CI but you can also run it manually.

.. code-block:: bash
breeze verify-provider-packages
You can also run the verification with an earlier airflow version to check for compatibility.

.. code-block:: bash
breeze verify-provider-packages --use-airflow-version 2.1.0
All the command parameters are here:

.. image:: ./images/breeze/output-verify-provider-packages.svg
:width: 100%
:alt: Breeze verify-provider-packages

Preparing packages
------------------
@@ -646,27 +646,6 @@ All Tests are inside ./tests directory.
$ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
- Running specific test in container using shell scripts. Testing in container scripts are located in
``./scripts/in_container`` directory.

.. code-block:: bash
root@df8927308887:/opt/airflow# ./scripts/in_container/
bin/ run_flake8.sh*
check_environment.sh* run_generate_constraints.sh*
entrypoint_ci.sh* run_init_script.sh*
entrypoint_exec.sh* run_install_and_test_provider_packages.sh*
_in_container_script_init.sh* run_mypy.sh*
prod/ run_prepare_provider_packages.sh*
run_ci_tests.sh* run_prepare_provider_documentation.sh*
run_clear_tmp.sh* run_system_tests.sh*
run_docs_build.sh* run_tmux_welcome.sh*
run_extract_tests.sh* stop_tmux_airflow.sh*
run_fix_ownership.sh* update_quarantined_test_status.py*
root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
- Running specific type of test

- Types of tests
@@ -835,40 +814,15 @@ To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be r
7. Running specific checks in container using shell scripts. Scripts are located in ``./scripts/in_container``
directory.

.. code-block:: bash
root@df8927308887:/opt/airflow# ./scripts/in_container/
bin/ run_flake8.sh*
check_environment.sh* run_generate_constraints.sh*
entrypoint_ci.sh* run_init_script.sh*
entrypoint_exec.sh* run_install_and_test_provider_packages.sh*
_in_container_script_init.sh* run_mypy.sh*
prod/ run_prepare_provider_packages.sh*
run_ci_tests.sh* run_prepare_provider_documentation.sh*
run_clear_tmp.sh* run_system_tests.sh*
run_docs_build.sh* run_tmux_welcome.sh*
run_extract_tests.sh* stop_tmux_airflow.sh*
run_fix_ownership.sh* update_quarantined_test_status.py*
root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
8. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
7. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit

.. code-block:: bash
$ cd ~/Projects/airflow
$ pre-commit install
$ git commit -m "Added xyz"
9. To disable Pre-commit
8. To disable Pre-commit

.. code-block:: bash
@@ -1457,26 +1411,6 @@ All Tests are inside ./tests directory.
$ ./breeze-legacy --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All tests
- Running specific test in container using shell scripts. Testing in container scripts are located in
``./scripts/in_container`` directory.

.. code-block:: bash
root@df8927308887:/opt/airflow# ./scripts/in_container/
bin/ run_flake8.sh*
check_environment.sh* run_generate_constraints.sh*
entrypoint_ci.sh* run_init_script.sh*
entrypoint_exec.sh* run_install_and_test_provider_packages.sh*
_in_container_script_init.sh* run_mypy.sh*
prod/ run_prepare_provider_packages.sh*
run_ci_tests.sh* run_prepare_provider_documentation.sh*
run_clear_tmp.sh* run_system_tests.sh*
run_docs_build.sh* run_tmux_welcome.sh*
run_extract_tests.sh* stop_tmux_airflow.sh*
run_fix_ownership.sh* update_quarantined_test_status.py*
root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
- Running specific type of test

- Types of tests
@@ -1643,39 +1577,15 @@ To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be r
Run flake8..........................................................Passed
7. Running specific checks in container using shell scripts. Scripts are located in ``./scripts/in_container``
directory.

.. code-block:: bash
root@df8927308887:/opt/airflow# ./scripts/in_container/
bin/ run_flake8.sh*
check_environment.sh* run_generate_constraints.sh*
entrypoint_ci.sh* run_init_script.sh*
entrypoint_exec.sh* run_install_and_test_provider_packages.sh*
_in_container_script_init.sh* run_mypy.sh*
prod/ run_prepare_provider_packages.sh*
run_ci_tests.sh* run_prepare_provider_documentation.sh*
run_clear_tmp.sh* run_system_tests.sh*
run_docs_build.sh* run_tmux_welcome.sh*
run_extract_tests.sh* stop_tmux_airflow.sh*
run_fix_ownership.sh* update_quarantined_test_status.py*
root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
8. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
7. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit

.. code-block:: bash
$ cd ~/Projects/airflow
$ pre-commit install
$ git commit -m "Added xyz"
9. To disable Pre-commit
8. To disable Pre-commit

.. code-block:: bash
@@ -1706,10 +1616,6 @@ To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be r
<a href="https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#running-static-code-checks-via-breeze"
target="_blank">Running Static Code Checks via Breeze</a>





Contribution guide
~~~~~~~~~~~~~~~~~~

0 comments on commit 3ed0747

Please sign in to comment.