Add verificationy that provider docs are as expected (#35424)

Validate that documentation in provider files is properly added and adds missing documentation if not there. The provider documentation for all community providers follows the same structure. This PR adds pre-commit that verifies that some common structure for all provider documentation is in place and is properly indexed. Since we are adding a lot of providers now, it's worth making a common pre-commit that will verify, and if needed update and regenerate the content to follow the same patterns. It also turned out that Kubernetes and AWS executors were missing and entry for those in provider.yaml file - which made them not discoverable via "airflow providers executors" CLI - this is also verified now. List of providers that are contributing their configuration is removed from Airflow - instead the documentation points to the common "providers" documentation where list of packages and links to their configuration are generated automatically. https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/configurations.html
apache · Nov 4, 2023 · a61da3c · a61da3c
1 parent 706878e
commit a61da3c
Show file tree

Hide file tree

Showing 11 changed files with 355 additions and 71 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -972,6 +972,13 @@ repos:
         entry: ./scripts/ci/pre_commit/pre_commit_check_deferrable_default.py
         pass_filenames: false
         files: ^airflow/.*/sensors/.*\.py$|^airflow/.*/operators/.*\.py$
+      - id: check-provider-docs-valid
+        name: Validate provider doc files
+        entry: ./scripts/ci/pre_commit/pre_commit_check_provider_docs.py
+        language: python
+        files: ^airflow/providers/.*/provider\.yaml|^docs/.*
+        additional_dependencies: ['rich>=12.4.4', 'pyyaml', 'jinja2']
+        require_serial: true
         ## ADD MOST PRE-COMMITS ABOVE THAT LINE
         # The below pre-commits are those requiring CI image to be built
       - id: mypy-dev

diff --git a/STATIC_CODE_CHECKS.rst b/STATIC_CODE_CHECKS.rst
@@ -215,6 +215,8 @@ require Breeze Docker image to be built locally.
 +-----------------------------------------------------------+--------------------------------------------------------------+---------+
 | check-provide-create-sessions-imports                     | Check provide_session and create_session imports             |         |
 +-----------------------------------------------------------+--------------------------------------------------------------+---------+
+| check-provider-docs-valid                                 | Validate provider doc files                                  |         |
++-----------------------------------------------------------+--------------------------------------------------------------+---------+
 | check-provider-yaml-valid                                 | Validate provider.yaml files                                 | *       |
 +-----------------------------------------------------------+--------------------------------------------------------------+---------+
 | check-providers-init-file-missing                         | Provider init file is missing                                |         |

diff --git a/airflow/providers/amazon/provider.yaml b/airflow/providers/amazon/provider.yaml
@@ -894,3 +894,6 @@ config:
         type: boolean
         example: "True"
         default: "True"
+
+executors:
+  - airflow.providers.amazon.aws.executors.ecs.ecs_executor.AwsEcsExecutor
diff --git a/airflow/providers/cncf/kubernetes/provider.yaml b/airflow/providers/cncf/kubernetes/provider.yaml
@@ -342,3 +342,6 @@ config:
         type: string
         example: ~
         default: ""
+
+executors:
+  - airflow.providers.cncf.kubernetes.kubernetes_executor.KubernetesExecutor
diff --git a/dev/breeze/src/airflow_breeze/pre_commit_ids.py b/dev/breeze/src/airflow_breeze/pre_commit_ids.py
@@ -61,6 +61,7 @@
     "check-persist-credentials-disabled-in-github-workflows",
     "check-pre-commit-information-consistent",
     "check-provide-create-sessions-imports",
+    "check-provider-docs-valid",
     "check-provider-yaml-valid",
     "check-providers-init-file-missing",
     "check-providers-subpackages-init-file-exist",

diff --git a/docs/apache-airflow/configurations-ref.rst b/docs/apache-airflow/configurations-ref.rst
@@ -32,20 +32,21 @@ generated using the secret key has a short expiry time though - make sure that t
 that you run airflow components on is synchronized (for example using ntpd) otherwise you might get
 "forbidden" errors when the logs are accessed.
 
-Some of the providers have their own configuration options, you will find details of their configuration
-in the provider's documentation. The pre-installed providers that you may want to configure are:
-
-* :doc:`Configuration Reference for Celery Provider <apache-airflow-providers-celery:configurations-ref>`
-* :doc:`Configuration Reference for Apache Hive Provider <apache-airflow-providers-apache-hive:configurations-ref>`
-* :doc:`Configuration Reference for CNCF Kubernetes Provider <apache-airflow-providers-cncf-kubernetes:configurations-ref>`
-* :doc:`Configuration Reference for SMTP Provider <apache-airflow-providers-smtp:configurations-ref>`
-* :doc:`Configuration Reference for IMAP Provider <apache-airflow-providers-imap:configurations-ref>`
-* :doc:`Configuration Reference for OpenLineage Provider <apache-airflow-providers-openlineage:configurations-ref>`
-* :doc:`Configuration Reference for Elasticsearch Provider <apache-airflow-providers-elasticsearch:configurations-ref>`
-* :doc:`Configuration Reference for Amazon Provider <apache-airflow-providers-amazon:configurations-ref>`
-* :doc:`Configuration Reference for Yandex Provider <apache-airflow-providers-yandex:configurations-ref>`
-
 .. note::
     For more information see :doc:`/howto/set-config`.
 
+
+Provider-specific configuration options
+---------------------------------------
+
+Some of the providers have their own configuration options, you will find details of their configuration
+in the provider's documentation.
+
+You can find all the provider configuration in
+:doc:`configurations specific for provider packages <apache-airflow-providers:core-extensions/configurations>`
+
+Airflow configuration options
+-----------------------------
+
+
 .. include:: ../exts/includes/sections-and-options.rst
diff --git a/docs/exts/operators_and_hooks_ref.py b/docs/exts/operators_and_hooks_ref.py
@@ -314,7 +314,9 @@ def render_content(
         self, *, tags: set[str] | None, header_separator: str = DEFAULT_HEADER_SEPARATOR
     ) -> str:
         tabular_data = [
-            provider["package-name"] for provider in load_package_data() if provider.get("config") is not None
+            (provider["name"], provider["package-name"])
+            for provider in load_package_data()
+            if provider.get("config") is not None
         ]
         return _render_template(
             "configuration.rst.jinja2", items=tabular_data, header_separator=header_separator

diff --git a/docs/exts/templates/configuration.rst.jinja2 b/docs/exts/templates/configuration.rst.jinja2
@@ -16,6 +16,6 @@
  specific language governing permissions and limitations
  under the License.
 #}
-{%for provider_package_name in items %}
-* :doc:`Configuration for {{ provider_package_name }} <{{ provider_package_name }}:configurations-ref>`
+{%for name, provider_package in items %}
+* :doc:`Configuration for {{ name }} ({{ provider_package }})<{{ provider_package }}:configurations-ref>`
 {% endfor %}
diff --git a/images/breeze/output-commands-hash.txt b/images/breeze/output-commands-hash.txt
@@ -65,7 +65,7 @@ setup:version:be116d90a21c2afe01087f7609774e1e
 setup:f60eda462f5bc70fee369806fedad538
 shell:aa92fe60473e4b5d0f41b5b182c02468
 start-airflow:f09871892c61bc889e6b56791115c923
-static-checks:8ed3127b97b5b56dae600aa4b15d4002
+static-checks:06290ff4d6f24bbda96421385ba29ca2
 testing:db-tests:e08e3f30ddc34d95ae56de5222392b59
 testing:docker-compose-tests:fd154a058082fcfda12eb877a9a89338
 testing:helm-tests:98a9ba6631249762b1633b76a29f4461