Seperate provider verification as standalone breeze command (#23454)

This is another step in simplifying and converting to Python all of
the CI/local development tooling.

This PR separates out verification of providers as a separate
breeze command `verify-provider-packages`. It was previously part of
"prepare_provider_packages.py" but it has been now
extracted to a separate in-container python file and it was
wrapped with breeze's `verify-provider-packages` command.

No longer provider verification is run with "preparing provider docs"
nor "preparing provider packages" - it's a standaline command.

This command is also used in CI now to run the tests:

* all provider packages are built and created on CI together with
  airflow version
* the packages are installed inside the CI image and providers are
  verified
* the 2.1 version of Airflow is installed together with all 2.1
  - compatible providers and provider verification is run there too.

This all is much simpler now - we got rediof some 500 lines of bash
code again in favour of breeze python code.

Fixes: #23430

.github/workflows/build-images.yml

@@ -41,7 +41,6 @@ env:
   GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
   LOGIN_TO_GITHUB_REGISTRY: "true"
   IMAGE_TAG_FOR_THE_BUILD: "${{ github.event.pull_request.head.sha || github.sha }}"
-  WITH_CI_GROUPS: "true"
 
 concurrency:
   group: build-${{ github.event.pull_request.number || github.ref }}

.github/workflows/ci.yml

@@ -47,7 +47,6 @@ env:
   LOGIN_TO_GITHUB_REGISTRY: "true"
   ENABLE_TEST_COVERAGE: "${{ github.event_name == 'push' }}"
   IMAGE_TAG_FOR_THE_BUILD: "${{ github.event.pull_request.head.sha || github.sha }}"
-  WITH_CI_GROUPS: "true"
 
 concurrency:
   group: ci-${{ github.event.pull_request.number || github.ref }}
@@ -792,17 +791,35 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
       - name: "Prepare airflow package: wheel"
         run: breeze prepare-airflow-package --package-format wheel --version-suffix-for-pypi dev0
       - name: "Install and test provider packages and airflow via wheel files"
-        run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
-        env:
-          USE_AIRFLOW_VERSION: "wheel"
-          PACKAGE_FORMAT: "wheel"
-      - name: "Replace non-compliant providers with their 2.1-compliant versions"
-        run: ./scripts/ci/provider_packages/ci_make_providers_2_1_compliant.sh
+        run: >
+          breeze verify-provider-packages --use-airflow-version wheel --use-packages-from-dist
+          --package-format wheel
+      - name: "Remove airflow package and replace providers with 2.1-compliant versions"
+        run: |
+          rm -vf dist/apache_airflow-*.whl \
+             dist/apache_airflow_providers_cncf_kubernetes*.whl \
+             dist/apache_airflow_providers_celery*.whl
+          pip download --no-deps --dest dist \
+              apache-airflow-providers-cncf-kubernetes==3.0.0 \
+              apache-airflow-providers-celery==2.1.3
       - name: "Install and test provider packages and airflow on Airflow 2.1 files"
-        run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
+        run: >
+          breeze verify-provider-packages --use-airflow-version 2.1.0
+          --use-packages-from-dist --package-format wheel --airflow-constraints-reference constraints-2.1.0
         env:
-          USE_AIRFLOW_VERSION: "2.1.0"
-          PACKAGE_FORMAT: "wheel"
+          # The extras below are all extras that should be installed with Airflow 2.1.0
+          AIRFLOW_EXTRAS: "airbyte,alibaba,amazon,apache.atlas.apache.beam,apache.cassandra,apache.drill,\
+            apache.druid,apache.hdfs,apache.hive,apache.kylin,apache.livy,apache.pig,apache.pinot,\
+            apache.spark,apache.sqoop,apache.webhdfs,arangodb,asana,async,\
+            celery,cgroups,cloudant,cncf.kubernetes,dask,databricks,datadog,dbt.cloud,\
+            deprecated_api,dingding,discord,docker,\
+            elasticsearch,exasol,facebook,ftp,github,github_enterprise,google,google_auth,\
+            grpc,hashicorp,http,imap,influxdb,jdbc,jenkins,jira,kerberos,ldap,\
+            leveldb,microsoft.azure,microsoft.mssql,microsoft.psrp,microsoft.winrm,mongo,mysql,\
+            neo4j,odbc,openfaas,opsgenie,oracle,pagerduty,pandas,papermill,password,plexus,\
+            postgres,presto,qubole,rabbitmq,redis,salesforce,samba,segment,sendgrid,sentry,\
+            sftp,singularity,slack,snowflake,sqlite,ssh,statsd,tableau,telegram,trino,vertica,\
+            virtualenv,yandex,zendesk"
       - name: "Fix ownership"
         run: breeze fix-ownership
         if: always()
@@ -851,11 +868,9 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
           path: "./dist/apache-airflow-providers-*.tar.gz"
           retention-days: 1
       - name: "Install and test provider packages and airflow via sdist files"
-        run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
-        env:
-          USE_AIRFLOW_VERSION: "sdist"
-          PACKAGE_FORMAT: "sdist"
-          AIRFLOW_EXTRAS: "all"
+        run: >
+          breeze verify-provider-packages --use-airflow-version sdist --use-packages-from-dist
+          --package-format sdist
       - name: "Fix ownership"
         run: breeze fix-ownership
         if: always()

BREEZE.rst

@@ -314,18 +314,22 @@ Customize your environment
 
 When you enter the Breeze environment, automatically an environment file is sourced from
 ``files/airflow-breeze-config/variables.env``.
-a
+
 You can also add ``files/airflow-breeze-config/init.sh`` and the script will be sourced always
 when you enter Breeze. For example you can add ``pip install`` commands if you want to install
 custom dependencies - but there are no limits to add your own customizations.
 
 The ``files`` folder from your local sources is automatically mounted to the container under
 ``/files`` path and you can put there any files you want to make available for the Breeze container.
 
+You can also copy any .whl or .sdist packages to dist and when you pass ``--use-packages-from-dist`` flag
+as ``wheel`` or ``sdist`` line parameter, breeze will automatically install the packages found there
+when you enter Breeze.
+
 You can also add your local tmux configuration in ``files/airflow-breeze-config/.tmux.conf`` and
 these configurations will be available for your tmux environment.
 
-there is a symlink between ``files/airflow-breeze-config/.tmux.conf`` and ``~/.tmux.conf`` in the container,
+There is a symlink between ``files/airflow-breeze-config/.tmux.conf`` and ``~/.tmux.conf`` in the container,
 so you can change it at any place, and run
 
 .. code-block:: bash
@@ -546,6 +550,8 @@ do not need or have no access to run). Those are usually connected with releasin
   ``breeze build-prod image --prepare-build-cache``(needs buildx plugin and write access to registry ghcr.io)
 * Generate constraints with ``breeze generate-constraints`` (needed when conflicting changes are merged)
 * Prepare airflow packages: ``breeze prepare-airflow-package`` (when releasing Airflow)
+* Verify providers: ``breeze verify-provider-packages`` (when releasing provider packages) - including importing
+  the providers in an earlier airflow version.
 * Prepare provider documentation ``breeze prepare-provider-documentation`` and prepare provider packages
   ``breeze prepare-provider-packages`` (when releasing provider packages)
 * Finding the updated dependencies since the last successful build when we have conflict with
@@ -750,6 +756,27 @@ Those are all available flags of ``verify-image`` command:
   :width: 100%
   :alt: Breeze verify-image
 
+Verifying providers
+-------------------
+
+Breeze can also be used to verify if provider classes are importable and if they are following the
+right naming conventions. This happens automatically on CI but you can also run it manually.
+
+.. code-block:: bash
+
+     breeze verify-provider-packages
+
+You can also run the verification with an earlier airflow version to check for compatibility.
+
+.. code-block:: bash
+
+    breeze verify-provider-packages --use-airflow-version 2.1.0
+
+All the command parameters are here:
+
+.. image:: ./images/breeze/output-verify-provider-packages.svg
+  :width: 100%
+  :alt: Breeze verify-provider-packages
 
 Preparing packages
 ------------------

CONTRIBUTORS_QUICK_START.rst

@@ -646,27 +646,6 @@ All Tests are inside ./tests directory.
 
    $ breeze --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All  tests
 
-
-- Running specific test in container using shell scripts. Testing in container scripts are located in
-  ``./scripts/in_container`` directory.
-
-.. code-block:: bash
-
-   root@df8927308887:/opt/airflow# ./scripts/in_container/
-      bin/                                        run_flake8.sh*
-      check_environment.sh*                       run_generate_constraints.sh*
-      entrypoint_ci.sh*                           run_init_script.sh*
-      entrypoint_exec.sh*                         run_install_and_test_provider_packages.sh*
-      _in_container_script_init.sh*               run_mypy.sh*
-      prod/                                       run_prepare_provider_packages.sh*
-      run_ci_tests.sh*                            run_prepare_provider_documentation.sh*
-      run_clear_tmp.sh*                           run_system_tests.sh*
-      run_docs_build.sh*                          run_tmux_welcome.sh*
-      run_extract_tests.sh*                       stop_tmux_airflow.sh*
-      run_fix_ownership.sh*                       update_quarantined_test_status.py*
-
-   root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
-
 - Running specific type of test
 
   - Types of tests
@@ -835,40 +814,15 @@ To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be r
 
 
 
-
-7. Running specific checks in container using shell scripts. Scripts are located in ``./scripts/in_container``
-   directory.
-
-.. code-block:: bash
-
-   root@df8927308887:/opt/airflow# ./scripts/in_container/
-      bin/                                        run_flake8.sh*
-      check_environment.sh*                       run_generate_constraints.sh*
-      entrypoint_ci.sh*                           run_init_script.sh*
-      entrypoint_exec.sh*                         run_install_and_test_provider_packages.sh*
-      _in_container_script_init.sh*               run_mypy.sh*
-      prod/                                       run_prepare_provider_packages.sh*
-      run_ci_tests.sh*                            run_prepare_provider_documentation.sh*
-      run_clear_tmp.sh*                           run_system_tests.sh*
-      run_docs_build.sh*                          run_tmux_welcome.sh*
-      run_extract_tests.sh*                       stop_tmux_airflow.sh*
-      run_fix_ownership.sh*                       update_quarantined_test_status.py*
-
-
-   root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
-
-
-
-
-8. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
+7. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
 
 .. code-block:: bash
 
   $ cd ~/Projects/airflow
   $ pre-commit install
   $ git commit -m "Added xyz"
 
-9. To disable Pre-commit
+8. To disable Pre-commit
 
 .. code-block:: bash
 
@@ -1457,26 +1411,6 @@ All Tests are inside ./tests directory.
    $ ./breeze-legacy --backend mysql --mysql-version 5.7 --python 3.8 --db-reset --test-type All  tests
 
 
-- Running specific test in container using shell scripts. Testing in container scripts are located in
-  ``./scripts/in_container`` directory.
-
-.. code-block:: bash
-
-   root@df8927308887:/opt/airflow# ./scripts/in_container/
-      bin/                                        run_flake8.sh*
-      check_environment.sh*                       run_generate_constraints.sh*
-      entrypoint_ci.sh*                           run_init_script.sh*
-      entrypoint_exec.sh*                         run_install_and_test_provider_packages.sh*
-      _in_container_script_init.sh*               run_mypy.sh*
-      prod/                                       run_prepare_provider_packages.sh*
-      run_ci_tests.sh*                            run_prepare_provider_documentation.sh*
-      run_clear_tmp.sh*                           run_system_tests.sh*
-      run_docs_build.sh*                          run_tmux_welcome.sh*
-      run_extract_tests.sh*                       stop_tmux_airflow.sh*
-      run_fix_ownership.sh*                       update_quarantined_test_status.py*
-
-   root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
-
 - Running specific type of test
 
   - Types of tests
@@ -1643,39 +1577,15 @@ To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be r
     Run flake8..........................................................Passed
 
 
-7. Running specific checks in container using shell scripts. Scripts are located in ``./scripts/in_container``
-   directory.
-
-.. code-block:: bash
-
-   root@df8927308887:/opt/airflow# ./scripts/in_container/
-      bin/                                        run_flake8.sh*
-      check_environment.sh*                       run_generate_constraints.sh*
-      entrypoint_ci.sh*                           run_init_script.sh*
-      entrypoint_exec.sh*                         run_install_and_test_provider_packages.sh*
-      _in_container_script_init.sh*               run_mypy.sh*
-      prod/                                       run_prepare_provider_packages.sh*
-      run_ci_tests.sh*                            run_prepare_provider_documentation.sh*
-      run_clear_tmp.sh*                           run_system_tests.sh*
-      run_docs_build.sh*                          run_tmux_welcome.sh*
-      run_extract_tests.sh*                       stop_tmux_airflow.sh*
-      run_fix_ownership.sh*                       update_quarantined_test_status.py*
-
-
-   root@df8927308887:/opt/airflow# ./scripts/in_container/run_docs_build.sh
-
-
-
-
-8. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
+7. Enabling Pre-commit check before push. It will run pre-commit automatically before committing and stops the commit
 
 .. code-block:: bash
 
   $ cd ~/Projects/airflow
   $ pre-commit install
   $ git commit -m "Added xyz"
 
-9. To disable Pre-commit
+8. To disable Pre-commit
 
 .. code-block:: bash
 
@@ -1706,10 +1616,6 @@ To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be r
    <a href="https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#running-static-code-checks-via-breeze"
    target="_blank">Running Static Code Checks via Breeze</a>
 
-
-
-
-
 Contribution guide
 ~~~~~~~~~~~~~~~~~~