Apply pypi suffix to provider dependencies (including apache-airflow)

So far we only applied PyPi suffix for cross-provider dependencies -
i.e. when one airflow package depended on another one released at
the same time we added "dev0" suffix to their cross-dependencies,
when building them on CI so that they can be installed together
without conflicts (this is because of decisions made with PEP-440
where version suffixes are in different namespaces than the final
versions, so it is impossible to specify "final" dependency as
minimal and have the ".dev0" satisfy it only for that single package.

With openlineage provider, we need to release it before Airflow 2.7.0
gets released, and openlineage provider depends on Airflow 2.7.0 so
we need to also handle the situation, where OpenLineage has >= 2.7.0
for Airflow, but the dependency that is used to resolve dependencies
in CI uses 2.7.0dev0. This is done by dynamically manipulating the
dependencies in setup.py based on VERSION_SUFFIX_FOR_PYPI variable.

This variable in CI is set to "dev0" thus all packages built have
"dev0" added as version, with this change if any package has
>= <CURRENT_AIRFLOW_VERSION> specified, it will also be extended
with the same "dev0" suffix.

dev/breeze/README.md

@@ -52,6 +52,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.
 
 ---------------------------------------------------------------------------------------------------------
 
-Package config hash: c2bade421865f94fb91f652317d76f3686ad017067349265d8a1d465bbfc2f0796d570bf4777acfb31a7703f1ccb6b0a63c655b97cec9eb9fdeeb1676c804c02
+Package config hash: 9d095d522c9f6fcf0c5834fcdc050bc98231d17fad07ec054c4e437580129d547b693b66b61442757f81fc1a505483da5267cc973dbf86babba7cd2c11697708
 
 ---------------------------------------------------------------------------------------------------------

setup.py

@@ -61,7 +61,35 @@
 CURRENT_PYTHON_VERSION = f"{sys.version_info.major}.{sys.version_info.minor}"
 
 
-#
+def apply_pypi_suffix_to_airflow_packages(dependencies: list[str]) -> None:
+    """
+    Looks through the list of dependencies, finds which one are airflow or airflow providers packages
+    and applies the version suffix to those of them that do not have the suffix applied yet.
+
+    :param dependencies: list of dependencies to add suffix to
+    """
+    for i in range(len(dependencies)):
+        dependency = dependencies[i]
+        if dependency.startswith("apache-airflow"):
+            # in case we want to depend on other airflow package, the chance is the package
+            # has not yet been released to PyPI and we only see it as a local package that is
+            # being installed with .dev0 suffix in CI. Unfortunately, there is no way in standard
+            # PEP-440 compliant way to specify version that would be both - releasable, and
+            # testable to install on CI with .dev0 or .rc suffixes. We could add `--pre` flag to
+            # enable it, but `--pre` flag is not selective and will work for all packages so
+            # we would automatically install all "pre-release" packages for all packages that
+            # we install from PyPI - and this is definitely not what we want. So in order to
+            # install only airflow packages that are available in sources in .dev0 or .rc version
+            # we need to dynamically modify the dependencies here.
+            if ">=" in dependency:
+                package, version = dependency.split(">=")
+                version_spec = f">={version}"
+                version_suffix = os.environ.get("VERSION_SUFFIX_FOR_PYPI")
+                if version_suffix and version_suffix not in version_spec:
+                    version_spec += version_suffix
+                dependencies[i] = f"{package}{version_spec}"
+
+
 # NOTE! IN Airflow 2.4.+ dependencies for providers are maintained in `provider.yaml` files for each
 # provider separately. They are loaded here and if you want to modify them, you need to modify
 # corresponding provider.yaml file.
@@ -74,11 +102,13 @@ def fill_provider_dependencies() -> dict[str, dict[str, list[str]]]:
     try:
         with AIRFLOW_SOURCES_ROOT.joinpath("generated", "provider_dependencies.json").open() as f:
             dependencies = json.load(f)
-        return {
-            key: value
-            for key, value in dependencies.items()
-            if CURRENT_PYTHON_VERSION not in value["excluded-python-versions"] or skip_python_version_check
-        }
+        provider_dict = {}
+        for key, value in dependencies.items():
+            if value.get(DEPS):
+                apply_pypi_suffix_to_airflow_packages(value[DEPS])
+            if CURRENT_PYTHON_VERSION not in value["excluded-python-versions"] or skip_python_version_check:
+                provider_dict[key] = value
+        return provider_dict
     except Exception as e:
         print(f"Exception while loading provider dependencies {e}")
         # we can ignore loading dependencies when they are missing - they are only used to generate