Add pre-commit check to check integrations table.

[timonviola]

Add pre-commit check to check if integrations and TESTING documentation are synchronized.

The new pre-commit test:

• reads integration-*.yml files
• reads the table of integrations in TESTING.rst
• compares the two lists and fails if they are not synced.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[timonviola]

@potiuk Could you give some first feedback on this?

Particularly, I am not sure about auto generating the table, as the description part is missing. Could the description be pulled from the yml files?

[potiuk]

Particularly, I am not sure about auto generating the table, as the description part is missing. Could the description be pulled from the yml files?

Generally looks cool. We do not have descriptions yet. But we can possibly also already improve the `integration-*.yml" files and add description there (and read it from there)

See: https://docs.docker.com/compose/compose-file/compose-file-v3/#reference-and-guidelines - you could add "labels" and we could standardize to use breeze.description label for it (you can also add version as well to all the integration compose files - I tihnk "3.8" seems to make most sense as it is "19.03.0+" and we are already "24.0.0+".

version: "3.8"
services:
  kafka:
    image: ....
    labels:
      breeze.description: "This label will appear on all containers for the web service"

As a follow-up we could update "version" and "labels" for other files (for airflow service we only need to set label for "base.yml".

[timonviola]

@potiuk Thank's for the feedback, pushed 2 commits, which should cover it.

As a follow-up we could update "version" and "labels" for other files (for airflow service we only need to set label for "base.yml".

I did not change the base.yml in this PR as it is not part of integrations-*.yml

Maybe this is what you meant, but would it make sense to create another PR for standardizing breeze.description?

• Add a hook, which checks whether breeze.description exists and
• add breeze.description to other files as well e.g. base.yml?

[potiuk]

❤️ it

[potiuk]

Maybe this is what you meant, but would it make sense to create another PR for standardizing breeze.description?

Add a hook, which checks whether breeze.description exists and
add breeze.description to other files as well e.g. base.yml?

Yep. Absolutely. Separate PR, And having pre-commit to check it is a good idea :)

[potiuk]

Some other pre-commits need to be run and fixed :).

I highly recomment pre-commit install - and then you do not have to even think about it

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[potiuk]

Cool !